Turaga’s lab to improve computer reliability in identifying objects in images

The director of the School of Arts, Media and Engineering at Arizona State University has won a million-dollar federal grant to improve the way that computers discern images for the U.S. Department of Defense.

Pavan Turaga, who is a professor in both the School of Arts, Media and Engineering and the School of Electrical, Computer and Energy Engineering, received an 18-month AI-Explorations grant from the Defense Advanced Research Projects Agency earlier this month.

Turaga’s ASU lab, Geometric Media Lab, has been working on fusing different methods, including mathematical, physical and data-driven, to better process and understand imaging data.

“This is a big defense agency that is traditionally known to invest in high-risk, high-reward projects,” said Turaga, who uses they/them pronouns.

“This specific call was a response to an ongoing issue in the machine-learning and artificial intelligence world.”

Computers use neural networks to process images. The networks use deep-learning techniques to elicit data to identify objects in images.

“The past decade of work in neural networks has shown that they’re very brittle and not very reliable,” Turaga said.

“We don’t know why they do what they do. There is a picture of a cat and they will say it’s a dog, and there is generally no satisfactory explanation for that.”

The images that the neural networks are processing are affected by numerous factors, such as lighting and camera angle.

In the DARPA project, Turaga’s team will embed knowledge of physics into the neural networks to lead to more reliable and robust models of machine learning.

They’ll use Riemannian geometry, which originally was invented in the early 20th century mathematics community to study the curvature of surfaces. With careful reworking and fusion with deep networks, we will be able to assess how the shape and geometry of an object can make things easier to identify.

The Geometric Media Lab at ASU is well positioned to be the bridge between the neural networks world and the physics world, said Turaga, whose PhD thesis was about advancing a similar technique of using geometric methods to encode models of physics.

A neural network can be trained, through repetition, to recognize images. The networks do that by focusing on regions of the image. 

“The attention of networks is often on color, and shading – it’s very diffuse and not precise,” Turaga said. “It’s looking at a lot and coming up with a decision that may or may not be optimal.

“We’re proposing, let’s look at more detailed geometry. What is the shape? And encode that definition in a way that neural networks can train under.”

The network will process the pixels the same way but with better reliability.

“When we say, ‘Tell me what you’re paying attention to while you’re reaching this decision,’ it’s more clearly focusing on the features of the objects that make it what it is,” they said.

“We’re able to get the neural network to pay attention to the relevant part of the object, such as shape and texture. If you don’t do that, you get a neural network that pays attention to a lot of distracting things.”

Turaga’s lab is able to show how their technique improves the network’s processing through “attention map” images that look like heat maps.

“Our approach shows that we can train networks to pay attention by highlighting shapes more precisely.

“We’re trying to make the neural networks more trustworthy by creating these visualizations that help us trust what they’re doing.”

This image shows how the Geometric Media Lab’s technology is improving computers’ ability to reliably identify objects in images by teaching the computer to pay attention to the objects’ shapes. The top row is the object in the image. The second row shows what the computer neural networks pay attention to while discerning the objects. The attention is diffuse. The bottom row shows how the computer, using the lab’s method, pays closer attention to specific areas of the objects, increasing the reliability of the identification. Image courtesy the Geometric Media Lab at ASU

Trustworthiness of the systems is key because DARPA is interested in defense applications, such as drone images.

Using standard benchmark data tests, Turaga’s method shows an increase of four percentage points in accuracy, from 68% to 72%

“The interesting thing is not that we pushed (the accuracy), but to say it went up, because we’re paying attention to the underlying property of the object.

“We’re actually encoding knowledge of the object shape and geometry.”

Turaga said they couldn’t have done this kind of work without a joint appointment between the schools.

“That role allows me time and space to think about these kinds of speculative questions, which would not be easy if I were only in mainstream engineering.

“It lets me step back and say, ‘Let’s think deeper about how media technologies are encoding knowledge.’”

The Geometric Media Lab is working with the Global Security Initiative at ASU on the grant project, part of which is subcontracted out to Johns Hopkins and Florida State universities. The project will last 18 months and include teams of students and faculty members. The co-principal investigators are Visar Berisha, an associate professor, and Gautam Dasarathy, assistant professor, both in the School of Electrical, Computer and Energy Engineering at ASU, and also includes student Rajhans Singh and postdoc Ankitha Shukla as key contributors.

Top graphic by Alejandro Cabrera/ASU News