Overview

An Event-based Vision Sensor (EVS) realizes high-speed data output with low latency by limiting the output data to luminance changes from each pixel, combined with information on coordinates and time. With a focus on movement, they can be applied in a wide variety of fields.

Technology part

Application part

What is EVS?

EVS captures movements (luminance changes)

EVS is designed to emulate how the human eye senses light.The human eye functions in such a way that the receptors on the retina, when exposed to light, convert it to visual signals to be sent to the brain. The subsequent neuronal cells identify the light and shade, and the information is conveyed to the visual cortex in the brain via the retinal ganglion cells.

In the EVS the incident light is converted into electric signals in the imager’s light receiving circuit. The signals pass through the amp unit and reach a comparator where the differential luminance data is separated and divided into positive and negative signals which are then processed and output as events.

EVS captures movements (luminance changes)
Zoom Zoom
EVS captures movements (luminance changes)

EVS mechanism

In the Event-based Vision Sensor, the luminance changes detected by each pixel are filtered to extract only those that exceed the preset threshold value. This event data is then combined with the pixel coordinate, time, and polarity information before being output. Each pixel operates asynchronously, independently from any other.
Figures illustrate how the sensor captures the ball movement.

EVS mechanism1
Event-based Vision Sensor Data Output Zoom Zoom
EVS mechanism1

Event-based Vision Sensor Data Output


Each pixel consists of a light receiving and luminance detection unit. The incident light is converted into a voltage in the light receiving unit. The differential detection circuit in the luminance detection unit detects the changes between the reference voltage and the converted incident light voltage. If the changes are greater than the set threshold value in either a positive or negative direction then the comparator identifies it as an event and this data is output.

EVS mechanism2
Zoom Zoom
EVS mechanism2

The circuit is reset with the detected event luminance as the reference, and the threshold values are set from this new reference voltage in the positive (light) and negative (dark) directions. If the incident light changes in luminance by an amount greater than the value set as a threshold in the positive direction (i.e. the output voltage surpasses the positive threshold) a positive event is output; conversely, if the voltage is lower than the negative threshold, a negative event is output.

EVS mechanism3
ZoomZoom
EVS mechanism3

(1) The reference voltage and positive/negative thresholds are set.
(2) A negative event is output when the incident light luminance goes below the negative threshold value.
(3) The reference voltage and positive/negative threshold values are reset based on the value at the event output.
(4) Another negative event is output if the incident light luminance further lowers below the negative threshold.
(5) The reference voltage and positive/negative threshold values are reset again based on the value at the second event output.
(6) If subsequently the luminance increases and surpasses the positive threshold value, a positive event is output.


As illustrated in the diagram below, pixels convert incident light luminance into electric voltage logarithmically. This makes it possible for the sensor to detect subtle differences in the low luminance range while it responds to a wide luminance difference in the high luminance range to prevent event saturation, allowing it to realize a wide dynamic range.

EVS mechanism4

This mechanism produces EVS images as shown below (right).

EEVS mechanism5

Since changes in the luminance of pixels will occur when a target has moved, images of moving targets will appear as though their outlines have been extracted (photograph was taken by a camera equipped with EVS mounted on an automobile dashboard).

Technical Features

The industry’s smallest*1 pixel: miniaturized high-resolution sensor made possible by utilizing a stacked structure

Unlike conventional technology that has the light-receiving circuit and luminance detection circuit on the same layer, this technology incorporates them on different layers: a pixel chip (upper layer) and logic chip (lower layer) which includes integrated signal processing circuits. These chips are stacked and connected using Cu-Cu technology within each pixel. The industry’s smallest pixel (4.86 μm) is integrated with the logic chip on a 40nm process, resulting in a 1/2.5 type sensor with HD resolution of 1,280 x 720.

*1) Among stacked event-based vision sensors. According to Sony research (as of announcement on September 9, 2021).

The industry’s smallest pixel: miniaturized high-resolution sensor made possible by utilizing a stacked structure

Highspeed and lower latency

Each pixel detects luminance changes asynchronously and will output event data immediately. When multiple pixels produce events the arbitration circuit controls the output order based on the earliest-received event. In this way the sensor outputs events as they are generated, making it possible to only output necessary data at the microsecond order while keeping power consumption low.

Highspeed and lower latency

Built-in Hardware event filter

In order to cater to various applications the sensor is equipped with several filter functions specifically designed for event data.
This feature enables removal of unnecessary events such as periodical events due to LEDs flickering and other events that are highly unlikely to be the outline of a moving object. It can also restrict the data volume when necessary to ensure it falls below the event rate that can be processed in downstream systems.

Built-in Hardware event filter
Built-in Hardware event filter

(Results of software simulation. Data of the left image is reduced by approx. 92%.)

By turning on the filter (right image), the overall data volume can be reduced while retaining the information required for a given application.
The right image (with the filter turned on) shows the white lines on the road as the significant information.

Examples of Event Signal Processing

(1) Liquid monitoring

Frame-based sensor image

Frame-based sensor image

The flow of water appears linear due to the blurring effect,making it impossible to discern individual drops.

  • EVS image

    EVS image

    Each drop of water can be captured [high frame rate]

  • EVS(super slow motion)

    EVS(super slow motion)

    This data is retained chronologically and output seamlessly, making it possible to extract images in super slow motion, capturing the image at a specific time stamp, etc.

(2) Human tracking

Frame-based sensor image

Frame-based sensor image

The person in dark-color clothes is harder to detect.
In a high illuminance environment, the frame-based sensor can also capture images.

EVS image

EVS image

The outlines of the persons are extracted.
The attributes of the clothes do not affect the detection.
The non-significant data (background, etc.) are not output [minimal data output]

Application output

Application output

Detection results enhanced by machine learning. The figures are recognized as pedestrians.
The recognition is sustained continuously even if the targets run, overtake, etc.

(3) Metal process monitoring

Frame-based sensor image

Frame-based sensor image

The sparks are overexposed due to high luminance,
making linear trails in the image.

EVS image

EVS image

Each fast-moving spark is captured individually [high frame rate]
Data other than the sparks (such as the machinery) are not output [high efficiency: minimal data output]

Application output

Application output

Each spark is tagged with ID and tracked
⇒ analyzable in terms of the number, size, speed, etc

(4) 3D measurement

Frame-based sensor image

Frame-based sensor image

The laser is projected onto a box moving on a conveyer belt and the reflected light is captured by the EVS camera.

EVS image

EVS image

The difference between the laser reflected on the box surface and that on the surface of the belt (default surface) are combined with temporal information to generate a 3D image.

Application output

Application output

It is possible to configure the sensor so that the height information is taken only when the reflected laser beam enters the EVS. The high temporal resolution enables to obtain detailed height information with enhanced accuracy.

(5) Vibration monitoring

Frame-based sensor image

Frame-based sensor image

It is impossible to discern the vibration in the model car on the platform with naked eye.

EVS image

EVS image

Only the vibrating areas are processed and output so that the vibration is clearly visualized.

Application output

Application output

The frequency is analyzed per pixel and can be mapped out in two dimensions.

*) Metavision® Intelligence Suite is the Event-Based Vision software developed by Prophesee.
Metavision® is a registered trademark of PROPHESEE S.A.

Applications

Visual inspections and contaminant detection inspections

EVS can detect the luminance differences of scratches or stains by capturing relative images of changes in moving products. For example, the application of EVS is expected in inspections such as for scratches and dust, or for damage in materials such as large film wound into rolls.

Their small data sizes and compatibility with machine learning are also notable features. With RGB cameras, the way scratches are perceived will differ depending on the way colors appear, so their data volume will become larger. Comparatively, EVS detects differences in luminance for each pixel, and only use the differential data for output after combining it with coordinate and temporal information, so their data is small in size and high in output speed. In the future, they will be able to contribute to the automation of inspections in combination with machine learning.

Related sectors

Electronics Manufacturing
Food/Medicine/Cosmetic Manufacturing

Visual inspections and contaminant detection inspections

Machine abnormality detection

Since EVS can detect and track the movement of objects moving at high speeds, they can identify even minor abnormalities in behavior and output data which is small in size with no latency. They can be applied to systems for detecting abnormalities in combination with machine learning.

Related sectors

Electronics Manufacturing
Heavy Industry and Civil Engineering

Machine abnormality detection

Robotics

In the robotics industry, demand for autonomous robots is currently growing. EVS can detect movement by converting the external characteristics of objects into silhouettes. They are capable of high-speed processing for obstacle detection, so they are expected to be applied in automatically-traveling robots such as robots for automatic transport or home delivery. Their use as care-giving robots is also being investigated at sites for long-term care support.

Related sectors

Logistics

Robotics

Monitoring with consideration for privacy

The movement of humans is perceived by EVS as changes in luminance. They do not capture specific details such as human faces, color, or shapes, so they are well suited to applications for safety monitoring with consideration for privacy at medical or long-term care sites.

Also, since their wide dynamic range characteristics allow them to detect movement without being affected by lighting conditions, they can output data which is small in size with no latency. For example, they are able to monitor the movement paths of people even in dark locations where image sensors for security applications have difficulty in capturing images. By combining other types of image sensors with EVS, even more advanced monitoring systems can be created.

Related sectors

Safety Monitoring

Monitoring with consideration for privacy

Scientific measurement and investigation

EVS is applicable to the scientific measurement, as it can detect high-speed moving objects. In the field of customer behavior investigation, EVS can achieve privacy-conscious investigations since they will only acquire data in response to movement.

Related sectors

Reserch and Investigation

Scientific measurement and investigation

UI/UX development

Event-based Vision Sensors perceive the movement of individual pixels as changes in luminance and can output only data which has changed with low latency, so they are ideal for gesture tracking such as of hand movements. Their application in combination with ToF image sensors is also greatly expected.

Related sectors

Reserch and Investigation

UI/UX development

Download

A List of Cameras with Event-based Vision Sensors (EVS)

Download a list of cameras that incorporate Sony's Event-based Vision Sensor

Related Products & Solutions

Related Links

Product & Service Inquiries

Click here to request for a datasheet/quotation (RFQ)!

* This button will redirect you to the salesforce.com Co., Ltd. website,
which we have entrusted.

E-mail Newsletters

Find the latest information on our newsletter for industrial and security applications.