Applications
Mobile

Smartphone (Intelligence)

Overview

With the expansion of AI-based recognition processing using text, audio, and image data, Sony Semiconductor Solutions Corporation (SSS) has developed the Intelligent Vision Sensor, which combines the hardware engine for AI processing and a streamlined AI algorithm, realizing the world’s first*1 sensor with AI processing functionality.

The Intelligent Vision Sensor is a revolutionary image sensor that enables high-speed edge AI processing within the sensor unit, on top of normal signal processing. Leveraging SSS’s stacking technology, we have successfully outfitted a logic chip with an AI-based image processing function, making it possible to integrate AI processing functionality into the tiny sensor.

This has reduced the data transfer time, which was a challenge for conventional systems that transmit image data to the cloud for AI processing. It also achieves privacy considerations, low power consumption, and reduced communication cost.

The image sensors with AI processing functionality opens up countless new future possibilities, such as mobile appliances, where it can improve camera function and be rolled out into new app-based services.
*1) Among image sensors. According to Sony research.(as of the announcement of May 14, 2020)

Applications

Photography (reflection removal, digital zoom)

Night views are always difficult to capture from behind window glass because of the reflection. This problem can be solved by AI that integrates the data from the color sensor and polarization image sensor, enabling to remove glare from the image while retaining a photogenic nightscape.

Photography (reflection removal, digital zoom) before
Photography (reflection removal, digital zoom) after

Also, smartphones that cannot be equipped with optical zoom due to physical restrictions often use digital zoom, which uses digital processing to enlarge part of the image, but it has the drawback of leaving images rough and unclear. SSS’s Intelligent Vision Sensor utilizes AI when using digital zoom to apply high-precision super-resolution imaging suited to the characteristics of the target. In the future, it may be possible to shoot images that rival the sharpness of optical zoom.
*) This is intended for mobile usage and does not represent the specifications of SSS’s image sensors.

Object recognition

The mobile image sensor’s AI can identify the subject as a dog simply by pointing a camera at it. With more fine-tuned feature analysis, it could even identify the breed as a corgi, for example, or display other such information. It is possible to enhance the accuracy by leveraging cloud technology.

AI can learn the images of your pet dog, so it will follow and focus on your dog automatically when you point a camera to it. It would be also possible that AI cuts out the best-angle shots and save them. The images are automatically tagged with keywords such as “dog,” “park,” and “lawns,” which will possibly enable you to easily search for the images you want among a large number of saved photos.
*) This is intended for mobile usage and does not represent the specifications of SSS’s image sensors.

Object recognition

Learning

The mobile image sensor’s AI can analyze image information such as brightness or color elements obtained by an SWIR or polarization image sensor, making it possible to read the subject’s pulse or diagnose dermatological conditions. In the future, AI processing may be able to provide detailed health checks based on diverse biometric information.
*) This is intended for mobile usage and does not represent the specifications of SSS’s image sensors.

Learning

AR

It can display AR (Augmented Reality) elements by superpositioning a variety of information over images projected by a camera. The mobile image sensor’s AI is used to integrate the camera image data and spatial information obtained by ToF image sensors, and then it overlays CG images and metadata, such as the names of places and buildings. This will enable useful and enjoyable AR experiences through visually enriched images, such as arrows appearing on a captured image to indicate the route direction or anime characters popping out from behind a tree for entertainment. In the future it may be possible to play games with others in a shared AR space.
*) This is intended for mobile usage and does not represent the specifications of SSS’s image sensors.

AR

Technology

ToF (Time of Flight)

Find out more about the ToF(Time of Flight) used in this field.

Related Products & Solutions

Contact Us

For inquiries about Sony Semiconductor Solutions Group and products / solutions, specifications, quotation / purchase requests, etc., please contact us using the Inquiry form from the button below.