A team from King Abdullah University of Science & Technology (KAUST) have developed a smart in-memory digital sensor with visual perception capabilities, including image and scene recognition. The technology for such artificial visual perception is built on the charge-coupled device (CCD) image sensors found in early digital cameras, adapting and enhancing their core structure to create light-sensitive memory devices able to be programmed by light. This breakthrough could help in the development of optoelectronic devices that perform optical sensing and storage with computing capabilities.
The rapid growth of sensory nodes driven by artificial intelligence (AI) and the internet of things (IoT) means an increasing volume of raw data. The traditional von Neumann architecture, which is comprised of discrete devices, results in delays in accessing and analyzing data, as well as high power consumption, a problem for applications such as autonomous cars and robotics that have strict requirements for minimal delay and power consumption.
However, in-memory light sensors can perform the roles of multiple devices at once, including optical sensing, storage and computation. Here, the team wanted them to detect different stimuli and compute, thus overcoming the memory wall and allowing faster and more real-time data analysis based on less power consumption, a requirement for the next level of applications.
As reported in Light: Science & Applications [Kumar et al. Light Sci. Appl. (2023) DOI: 10.1038/s41377-023-01166-7], here the 2D material MoS2 was embedded in a semiconductor capacitor (MOSCAP) structure that functions as a charge-trapping in-memory sensor that is sensitive to visible light rather than electrical stimulation and can be programmed optically and erased electrically.
Their findings show how to merge different traditionally discrete components with a single multifunctional memory device. The MoS2 MOSCAP structure was also combined with a neural network to demonstrate that it was possible to carry out straightforward binary image recognition, achieving a differentiation between images of a dog and an automobile with an accuracy of 91%.
The study will further the development of optoelectronic MOS memory devices for neuromorphic visual perception, adaptive parallel processing networks for in-memory light sensing, and smart CCD cameras with artificial visual perception capabilities. As team leader Nazek El-Atab told Materials Today, “this study [is] a significant step towards the development of smart CCD cameras with artificial visual perception capabilities.”
The researchers now hope to demonstrate the ability to perform sensing, storage and computing in a single device without compromising the performance of the conventionally discrete devices, such as sensitivity, selectivity to different wavelengths of light, endurance, retention and speed. They are also keen to investigate in-memory sensors that detect different stimuli, which will require further optimization of materials and device structure.
“This study [is] a significant step towards the development of smart CCD cameras with artificial visual perception capabilities.”Nazek El-Atab