AI-packed camera fits in your hand; captures 3D, exact color in each shot

Researchers from Karlsruhe Institute of Technology in Germany have developed a new compact multispectral camera using inkjet printers. The new device, which fits in the palm, is a version of a light field camera.

It could be useful for many applications, including autonomous driving, classification of recycled materials, and remote sensing. The camera is based on something called 3D spectral imaging, which can aid in object and material classification.

Traditionally, capturing the spatial and spectral information necessary for this process has required the use of multiple devices or time-intensive scanning procedures. The new light field camera addresses this challenge by capturing 3D information and spectral data simultaneously in a single snapshot.

“To our knowledge, this is the most advanced and integrated version of a multispectral light field camera,” said research team leader Uli Lemmer from the Karlsruhe Institute of Technology in Germany.

“We combined it with new AI methods for reconstructing the depth and spectral properties of the scene to create an advanced sensor system for acquiring 3D information.”

Inkjet technology for better optics

Reconstructed 3D data from camera images are finding widespread use in various applications, including virtual and augmented reality, autonomous cars, robotics, smart home devices, and remote sensing, asserts Michael Heizmann, a member of the research team.

“This new technology could, for example, allow robots to better interact with humans or improve the accuracy of classifying and separating materials in recycling. It could also be potentially used to classify healthy and diseased tissues,” he added.

Light field cameras , also known as plenoptic cameras, are specialized imaging devices that can capture both the direction and intensity of light rays. After the images are captured, computational processing is used to reconstruct 3D image information from the collected data.

These cameras typically utilize microlens arrays that are aligned with the pixels of a high-resolution camera chip.

To make the new device, researchers used inkjet printing to create a multispectral light field camera. They deposited a single droplet of material to form each individual lens on one side of ultrathin microscope slides.

Then, they printed fully aligned color filter arrays on the opposite side of the microscope slides. The resulting optical component was integrated directly into a CMOS camera chip.

In the palm of your hand

The inkjet printing method allowed for precise alignment between the optical components, significantly reducing manufacturing complexity and enhancing efficiency. The researchers discovered that an approach based on deep learning was most effective in extracting the desired information directly from the acquired measurements.

“Tackling the challenge of creating a multispectral light field camera was only possible by combining recent advances from manufacturing, system design, and AI-based image reconstruction,” said Qiaoshuang Zhang, first author of the paper.

He added that this work pushed the boundaries of inkjet printing, a versatile method with high precision and industrial scalability, for manufacturing photonic components.

The researchers tested the camera by recording a scene containing multicolored 3D objects at various distances. The image reconstruction algorithm was then trained and tested using many synthetic and real multispectral images.

According to the researchers, the new camera worked better than expected. It could, for example, distinguish different objects based on their spectral composition and depth information within a single snapshot.

With this proof-of-concept in hand, the team is now exploring areas where a light field camera could be useful. The research was published in the journal Optics Express .