Fraunhofer Institute for Silicon TechnologyNeurOSmart: Development of a smart LiDAR sensor with integrated neuromorphic data processing picks up speed
In the Fraunhofer lead project NeurOSmart, five institutes (ISIT, IPMS, IMS, IWU, IAIS)* are jointly researching a particularly energy-efficient and intelligent sensor for the next generation of autonomously acting systems. After just under a year of project work, the Fraunhofer researchers will present the expected sensor system and the hybrid, neuromorphic data processing pipeline at Electronica.
The current trend for complex, demanding applications, such as autonomous driving, is toward mobile supercomputers with significantly increasing energy consumption. Especially for mobile systems, this leads to shorter operating times or ranges and, according to current forecasts, will even reach the limits of global energy production in the next decades. To counteract this escalation, the Fraunhofer researchers involved in NeurOSmart are relying on decentralized intelligence that is tailored to the respective sensor and new approaches to energy-efficient data processing.
The basic idea behind the ambitious project is to integrate data processing directly into the sensor as far as possible and to become as energy-efficient as possible through coordinated components
The desired result is a range sensor (LiDAR) that detects and classifies objects on the basis of individual pixels in a field of view of 90° x 90° at working distances of up to 12 meters. All this in real time with frame rates of over 20 Hz and with an energy consumption of a few 10 W. This system can thus be used as a safety system for collaboration between humans and industrial robots and, in the future, also for mobile applications.
This performance is made possible by a variety of innovations in data processing and sensor technology, which are combined in this project.
We would like to particularly highlight the raw data preparation and pre-processing, which provides an integrated solution for the energy-efficient computation of AI algorithms on embedded hardware in real-time via the connection of the RISC-V Manycore cluster AIRISC-N with the embedded AI framework AIfES. This way, the amount of data to be interpreted is reduced by a factor of 10 and allow to observe even only parts of the scene (region of interest), which again reduces energy consumption and increases response time.
Best of all - the Fraunhofer technologies of AIRISC Core and AI Framework AIfES are available for free via GitHub and can recently also be easily downloaded as a pre-configured system from the Intel Pathfinder Shop and integrated into own designs. In addition to image processing applications (LiDAR, thermal imaging, Visible), time series analysis (acoustic monitoring, medical technology) and AI-based predictions (predictive maintenance, failure prediction), among others, are also supported via preconfigured extensions.
Advanced data interpretation is implemented on a novel mixed-signal in-memory accelerator that outperforms current alternatives by at least two orders of magnitude with an expected energy efficiency of more than 1000 TOPS/W. At the heart of the accelerator are so-called ferroelectric field-effect transistors (FeFETs), which can both store and process information (in-memory compute). This way, the energy-intensive bottleneck in the communication between memory and computational core is resolved and complex AI processes can also be realized in mobile end devices.
The advantages of the new approach are underlined by the parallel development of particularly small and efficient models for object recognition and classification. They are specifically adapted to the sensor, to new possibilities of directly integrated electronics and to their applications. The result is fast response time, increased data protection and significant energy savings compared to the current trend of off-practice or cloud-based solutions that prefer to rely on ever larger, more energy-intensive models.
At Electronica 2022, you will find the project's researchers at the Fraunhofer-Gesellschaft booth in Hall B4, Booth 258.
