James Hays, Manning assistant professor of computer science at Brown University, has been awarded a fellowship grant by the Alfred P. Sloan Foundation. Hays was recognized for his work in computer graphics, computer vision and computational photography. At Brown, he has facilitated program development for advanced photo-editing and image-recognition software, including one program that enables users to change the weather, season or time of day of a photograph by using text commands. He holds a B.S. in computer science from the Georgia Institute of Technology and a Ph.D. in computer science from Carnegie Mellon University.
Creating these algorithms requires a substantial amount of data, what are some ways you have sourced this information?
The raw data (webcam images) comes from the work of other research teams. But for our algorithm we need to know the “attributes” of every image so we use Amazon Mechanical Turk to crowdsource the annotation of thousands of webcam frames.
After the user types “more rain,” into the software, what is happening on the back-end of the program to change a photo’s attributes?
First, the program finds a webcam that observes a scene similar to your photo. Within the images from that webcam, it finds A) a photo as close as possible to your photo and B) a photo that has the attributes that you asked for. Since A and B depict the same scene, their difference in appearance is due to the change in attributes. The program “learns” for each part of the photo what color changes are associated with the attribute changes and then it replays those color changes in your photo.
What does this type of software mean for the future of photography?
Better image-editing tools make it easier to get the photo you want (rather than the photo “reality” was willing to give you). If you’re a photography purist, then better photo-editing tools are scary and dishonest. The cat is already out of the bag, though. People have been manipulating photos for more than a century. This just makes it easier for the average person to do. •