By Patrick Anderson
PBN Staff Writer

Brown University assistant professor James Hays knew computers have a good eye for art, especially detecting texture and shading in life-like representations of real-world images.

But Hays and fellow researchers wanted to find out how artificial intelligence reacted to more rudimentary art, the kind of sketches amateurs might dash off on a napkin or notebook.

Could a computer identify the abstract shapes and outlines commonly used across cultures to represent sharks, cars or zebras? Could a computer play the party-game “Pictionary”?

To their surprise, Hays and two visiting researchers from Berlin found they could teach a computer to recognize our cruder visual renderings, if not quite as well as other humans can.

Thanks to the emerging capabilities of crowdsourcing, Hays and colleagues created a program that correctly identified the subject of a simple sketch more than half the time. A random selection would have gotten the identification right less than 1 percent of the time and humans generally get it right three-quarters of the time.

“I was positively surprised,” Hays said. “Part of the reason that there are a lot of recognition problems in computer vision is that the part they are good at is texture. A computer is worse at things defined by shape, with no texture and just outline.”

And simple, abstract outlines are the most common ways people, especially those without great artistic talent, render images when trying to communicate through drawings.

So the capacity for computers to recognize simple, iconographic human drawings could have vast uses, such as pictorial Internet searching.

“Sketch-based search is an existing field, but now that we don’t just rely on matching sketches and photographs, it might be better,” Hays said, “allowing children, maybe people who are not literate or don’t speak the right language, to interact with a computer.”

Hays’ background is in computer graphics and vision and it was only in discussions with the visiting researchers from the Technical University in Berlin, Mathias Eitz and Marc Alexa, that the three began thinking about sketch recognition.

Software already existed that allowed computers to match images, such as photographs or professional police drawings, but no one had tried to teach a computer to recognize hand-drawn visual symbols, such as four legs and stripes representing a zebra, on a conceptual level.