INDUSTRIAL

Image sensor

Introducing a new standard of security cameras, our security image sensor provides a wider dynamic range and captures images of moving objects without blur or color tints

January 28, 2022

As the importance of image sensors for security cameras which monitor spaces is growing day by day, the characteristics required of them are also becoming more advanced. Fulfilling all of these requirements, which include low-illuminance performance to allow targets to be discerned even in dark locations, a high dynamic range for face recognition even against a light source such as in entranceways, and performance capable of recording images of moving targets accurately without causing blur or color tints, was a task at a level impossible to accomplish just with improvements to conventional sensors. That is why STARVIS 2 was developed. By redesigning the structure of its image sensor from the ground up, it has successfully implemented characteristics unmatched by any other. Its development team explains that achieving these outstanding characteristics was an extremely difficult project in which the two procedures of developing completely new technology and commercializing it into a product proceeded simultaneously, despite the highly unfavorable conditions of development during the COVID-19 crisis. There were two major keys to the success of this demanding project: the expertise and the teamwork only possible for Sony Semiconductor Solutions Group.

Y.Shimizu

Sony Semiconductor Solutions Corporation
Imaging System Business Div.

S.Iwabuchi

Sony Semiconductor Solutions Corporation
Research Div.1

T.Kawamura

Sony Semiconductor Solutions Corporation
Research Div.1

A.Shirahama

Sony Semiconductor Solutions Corporation
Sensor Product Design Div.

* Information at the time of the interview in October 2021.

Praised as “Impressive!” by customers, STARVIS combined various characteristics for excellent low-illuminance performance

―― What kinds of issues were previously being faced by conventional security cameras?

Shimizu:Conventional security cameras had issues with image quality in low-illuminance environments. To address such matters, it is necessary to increase the sensitivity when the surroundings are dark and light sources are insufficient, but raising the sensitivity requires a larger pixel size. However, if the pixel size is increased, the image sensor itself will become larger and lead to higher unit prices. Moreover, the lens will also become larger, so the size of the camera unit itself will also inevitably increase. Larger cameras will be limited in the places where they can be installed, so we needed to develop a sensor with high sensitivity while keeping it small in size. By changing the front-illuminated structure of the image sensor to a newly-developed back-illuminated structure, we were able to develop STARVIS with higher sensitivity.

―― Did the development of STARVIS proceed smoothly?

Shimizu:There were issues with reflections and flare ghosts*¹ appearing when we tried to use a back-illuminated CMOS image sensor with conventional packages. While a back-illuminated structure makes it easy to bring in light, too much light entering can cause problems like those we saw. Therefore, after repeated trial and error we developed a new device structure that would prevent too much light from entering the image sensor, and completed a package which prevented reflections and flare ghosts from occurring.

*1:A phenomenon in which, when recording images with the lens pointed toward a strong light source, photographs become tinted white and rings or spheres of light appear.

―― What was the response of the market toward the world’s first*2 product equipped with a back-illuminated CMOS image sensor for security cameras?

*2:Launch of STARVIS in August 2014, according to data from Sony.

Shimizu:When STARVIS was introduced, we had both front-illuminated CMOS image sensors and back-illuminated CMOS image sensors with the same resolution of full HD and the same number of pixels. Comparisons of these two products could be made under the same conditions, so we held business discussions with customers while having them examine the actual differences between them. There were considerable differences in the performance of these two image sensors, and anyone could clearly see the difference in the resulting videos themselves. Our customers were astonished as they made comments such as “Impressive!”, and we received positive evaluations from many sources.

By proceeding simultaneously with the establishment of new technology and product development,
STARVIS 2 achieved unmatched characteristics which were the first to be seen in the world*3

*3: Launch of STARVIS 2 in June 2021, according to data from Sony.

―― What was the trigger for the development of STARVIS 2?

Shimizu:It was triggered by requests from our customers. Although STARVIS had achieved greater low-illuminance sensitivity and improved image quality performance, there were also separate requests to expand its dynamic range so that faces could be accurately recognized even in situations such as entranceways, where the exterior is bright and shooting images against that light source causes faces to become dark and shadowed. Development on STARVIS 2 began to answer these requests.

―― How did you proceed with the development of STARVIS 2?

Shirahama:In order to widen the dynamic range further without sacrificing the low-illuminance performance of STARVIS, we thought that we could apply process technology and circuit technology whose development was proceeding in our company. We therefore focused on establishing those two types of technology.
Although their development was still under investigation and had therefore not yet been technologically established, our commercialization schedule had already been decided, so we had to proceed concurrently with both technological establishment and commercialization.

Image sensor incorporated in STARVIS 2

IMX678
F=1.6, Exp=33.3ms, Gain=48dB

Image sensor incorporated in STARVIS

IMX334
F=1.6, 33.3ms, Gain=57dB

A high dynamic range was achieved without sacrificing the low-illuminance performance of STARVIS

―― That means its progress was an unusual case different from standard product development, so did it proceed smoothly?

Shirahama:Normally, we move on to commercialization only after a product’s technology is in place, but since both proceeded simultaneously in this case, we were required to execute development and testing in extremely short cycles. We had no choice but to move forward while resolving each issue one at a time. Because of these conditions, the first prototype was full of problems and was completed in a state that was entirely unacceptable for commercialization.
However, since there was no leeway in the commercialization schedule, delays were not something that could be allowed easily, so the question of how to move the overall project forward while staying aware of its issues was very important. Even when we thought “It’s no use anymore,” we kept an eye on the schedule while resolving each problem, and took one step ahead at a time. And as we did so, our thoughts of “It’s no use anymore” gradually changed more and more into feelings of “Maybe it will work!” Even though the team for this project consisted of members working together for the first time, they possessed the capabilities to solve issues and the flexibility to skillfully move the project forward even under its strict schedule. I began to think “This just may be an incredible team” and working with the team became more and more enjoyable. One of the unique experiences that arose during this project was that as it went forward, the trust we felt toward our team grew and grew.

―― Since this project was developed during the COVID-19 crisis, you probably had only a few opportunities to work on it while actually seeing each other face to face, so did you feel any difficulties with the work?

Shirahama:I believe that not being able to meet face to face is a significant bottleneck for communication. Since I am very conscious of personal impressions when working on projects, I paid extremely close attention to the way I communicated when speaking about problems or issues under these conditions where team members could not see each other’s faces. Instead of blaming someone for problems or issues, I was careful to speak in a way that focused on how we should respond to them. If I was critical of someone and got an impression among the members of being a scary person, I would not have been able to receive the opinions I wanted to hear.

―― What kinds of things did you struggle with in the development of STARVIS 2?

Kawamura:As our development policy, we are constantly making product improvements while maintaining the strengths of the products. This time, in order to address the issue of expanding the dynamic range, there was a need to increase the saturation of the photodiode. Therefore, instead of attaching the photodiode to a conventional silicon surface, we developed a structure which accumulated electric charge in the vertical direction within the silicon.

At the same time, we also introduced new technology for a high dynamic range feature*4.

*4: A feature that can capture images with a large variance in brightness (dynamic range). It enables a wider range of luminosity to be captured, and automatic brightness adjustment reduces blackout or blowout.

We had to proceed with the development of these two new structures that had not yet been established while staying within the schedule for commercialization, so it was a challenge under an extremely tight development period. As Shirahama-san mentioned earlier, there were problems with charge transfer in the initial prototype and it was full of issues, so we were not able to even sufficiently evaluate them.

Iwabuchi:This vertical-structure photodiode had not yet been applied to image sensors, so it was a technology with high risks regarding mass production. Still, we desperately needed a form of breakthrough technology to achieve a wider dynamic range, so since there was a section for prototypes on the development line at Atsugi TEC, we started by trying it out there. We proceeded step-by-step starting from verifying its principles as we marked each development milestone. The commercialization schedule this time had been formulated under the assumption that all parts of technological development would proceed smoothly. As we were faced with successive issues from the very start that did not go as planned, we were continually walking a tightrope to figure out how we could somehow maintain the development schedule.

Shirahama:Iwabuchi-san ability to arrange and link the schedules was truly amazing. He was a lifesaver for all of us.

―― How did you overcome and resolve these issues?

Kawamura:We had no choice but to tackle each one individually, since there was no way we could solve all of them at once. We explained and showed these situations to Shirahama-san and Iwabuchi-san without hiding anything, shared our issues, and went around to consult with anyone who had related knowledge outside of the internal project team.
Iwabuchi:To expand the dynamic range, we had to increase the saturation signal while also suppressing noise.
The new vertical structure adopted this time was an innovative structure that could increase the saturation signal amount, but we faced tremendously difficult struggles to optimize the process conditions so that those increased signals could be transmitted reliably.
Regarding the other matter of noise suppression, since both the structure and process were new, we were thoroughly aware of the possibility that the dark characteristics, which were the cause of noise, might become worse.
In order to simultaneously resolve the two issues of increasing the saturation capacity while suppressing noise, we implemented various measures such as reviewing the pixel layout and process conditions, and finally succeeded in developing characteristics to obtain image quality that would fulfill the requests of our customers.

STARVIS 2 has infinite hidden potential that breaks the boundaries for the category of past security camera image sensors

―― What kinds of solutions did you come up with to make STARVIS 2 a reality?

Shimizu:The past version of STARVIS had strong low-illuminance performance characteristics. I believe that one result of adding the characteristic of wider dynamic range is that it allows its performance to be demonstrated in security cameras installed at locations such as entranceways. At entranceways, light from the sun shines from beyond the entrance door which is the direction that people enter from, so the faces of people are set against that light source. With a conventional camera this would cause people’s faces to be completely dark, while the outside is blown out in bright white. STARVIS 2 has high dynamic range even in conditions like this with large differences in brightness, so it can clearly capture faces and can also record images of bright areas near doors without blowout.
Also, with previous high dynamic range functions, multiple photographs taken with a short exposure time and long exposure time were superimposed on each other, so if targets such as people were in motion, their bodies or faces would be blurred or would have unnatural color tints.
The newly-designed STARVIS 2 has increased saturation capacity and uses a function known as ClearHDR that records images at the same time with differences only in gain, or sensitivity to brightness, rather than time differences, which can eliminate blur and color tints.
This has made it possible to dramatically improve the rate of face recognition at locations such as entranceways.

Kawamura:It can record images accurately even in situations where there are large changes in brightness such as nighttime scenes or tunnel entrances/exits, so it also has promise for application as a drive recorder. Enhanced recognition rates for faces and objects can facilitate connection with AI, so we believe it can be easily applied in a greater variety of situations than ever before.

―― You mentioned AI just now, so did you start development of STARVIS 2 with consideration for connecting with AI?

Shimizu:Nearly all security cameras today are connected to AI. There are many different applications, such as incorporating AI into cameras themselves, or sending videos to clouds for analysis by AI, but it is image sensors alone that actually capture video images. No matter how much AI might attempt to analyze videos, it will not even begin unless accurate images are recorded by these image sensors. Therefore, from the initial stages of development, we were very conscious of connection with AI. We also believe that as connection with AI advances, the demand for security camera image sensors will become even larger.

―― Tell us about the future potential of STARVIS 2.

Shimizu:We believe that we can expand the base concept of STARVIS 2 which has achieved an expanded dynamic range, to extend its capabilities and aim for “Auto Exposure Free” functions as a future goal. Currently, a flash or other lighting is used when shooting images in dark places, but we believe that there will be a time to come when flashes and even adjustment of shutter time will not be necessary. We believe it will be possible to automatically record images under the brightest to darkest conditions.

Kawamura:Drones are even now being used to deliver packages, and I believe that there are many methods that can use modern technology to deliver items to specified locations. By combining image sensors like the one we have just developed, as well as devices such as AI and ToF image sensors, deliveries can be made safely even at night, and we believe it will become possible to confirm that deliveries are made to the proper person by recognizing a person’s face at a specified location. In addition to contributing to security, it can be used regardless of day or night in lifesaving situations such as searching for accident victims, so we also believe its possible applications will be expanded.

Iwabuchi:Regarding technical potential, together with the development of processes in this project to accumulate saturated charge in silicon in the vertical direction, we also developed a transistor to make it easier to read out the accumulated electric charge to the silicone surface. This will allow the flexibility of pixel design to grow dramatically, so the potential for achieving innovative, never-before-seen pixel structures has broadened. This product is only the beginning, so I believe it will be a trigger for us to proceed with revolutionary device development from here on.

―― Tell us about the challenges you want to undertake in the future.

Shimizu:We often show our prototypes to customers during development, and in this project our customers were shocked that we were able to create the product under the current conditions. We also received extremely positive evaluations since we were able to commit to the requested characteristics. I am confident that with this product we were able to successfully achieve performance characteristics at an overwhelmingly high level.
In addition to regions such as America, Europe, and China in which our products have been introduced in the past, we wish to widen our sales channels to areas such as Southeast Asia and Africa in the near future, and to have our image sensors be in true worldwide use.

Shirahama:Through development and commercialization of new technology which is the first of its kind in the world, I hope we will be able to bring satisfaction to people across the globe. Also, the experiences we obtained from this development project, in which we attempted a variety of tasks and overcame many challenges, were a source of true joy of our engineers. Even though it may only be a small part, we wish to make contributions to create a free-spirited company where it is easy to attempt new things and which provides the motivation for everyone to want to take on new challenges.

Kawamura:In this project, we made a change to a new pixel structure, but I believe it is still just the beginning. Changing the photodiode to a vertical configuration to make it possible to increase the saturation capacity for greater depth has an enormous amount of hidden potential. I believe it will become essential in the security field to improve image recognition rates, so we intend to further enhance our pixel characteristics such as infrared sensitivity and high dynamic range while also miniaturizing pixels, to make our range of image sensor applications wider than ever before.

Iwabuchi:I believe that Sony’s strengths lie in its ability to carry out advanced development of image sensors with unequaled characteristics that use processes which cannot be copied by competitors, like STARVIS 2. Sony has an atmosphere which encourages new internal challenges, and its engineers in each field are full of motivation to try new things. I hope that we will be able to fully apply the advantages of this environment, to continue to develop revolutionary image sensors distinctive to Sony which are the first of their kind in the world.

Related links