Pregius S - to be a bridgehead for pioneering new markets such as robot arms
through the pursuit of high speed, multi-pixel as well as size reduction
February 2, 2021
Sony Semiconductor Solutions Group (SSS Group) launched the world’s first commercialized CCD color camera. It continues to pioneer products essential to people in their day-to-day lives, including back-illuminated CMOS image sensors and stacked CMOS image sensors. We are a company that operates at the closest proximity to the future world. In this feature article, we present interviews with key product development members to inspire you with the glimpse of SSS Group product development methods as well as how they identify and resolve issues and challenges, illustrating a picture of our future world.
The first interview is about “Pregius S,” which caused a stir in the global shutter engineering for industrial equipment. Through many failed challenges, they finally succeeded in realizing a multi-pixel and high-speed shutter. We explore the future evolution of imaging technology through this story of Pregius S development, which revolutionized the imaging for industrial devices.
* Information at the time of the interview in October 2021.
From CCD to CMOS The evolution of global shutters demanded by the market despite many challenges
―― What is a global shutter?
Tanaka:CMOS image sensors (CIS) come in two broad modes. One is a rolling shutter (RS) in which the sensor processes photon-converted electrons for each line of an array and reads them out one by one, and the other is a global shutter (GS) that stores up the electrons and reads out as a complete image. Taking a photo of a fast-moving object, the RS has difficulties in producing a still image without distortion or reduction. Whereas, the GS is able to capture a still image of a fast-moving object, as the entire image of the target is captured before reading out the data.
GS is mainly installed in industrial machine vision cameras. It is used for accurate verification or inspection of fast-moving targets, and its application is wide, from factory production lines to semiconductor production systems, chip mounters, image inspection systems, food and drug tests, storage logistic lines, barcode readers, and other product verification systems, as well as the Intelligent Transport System (ITS) which requires the reading of registration numbers of motor vehicles driving on highways.
―― GS used to be equipped with CCD image sensors. Why has it been replaced with CIS?
Yamane:Many manufacturers, including Sony, have adopted CCD image sensors previously as they are compatible with GS theoretically, but these sensors are highly power-consuming. The power they consume becomes greater as more pixels are mounted on the sensor. Customers were increasingly demanding more energy-efficient products. Against this background, a power-light CIS product coupled with a GS function started to appear in the market, and it soon became a popular solution.
Tanaka:The frame rate was another disadvantage of CCD image sensors. In the industrial sector, the takt time (production rate per day) is very important. Being in a customer-facing position, I often heard customer requests for faster image sensors. It was a time when other manufactures started producing CIS with high frame-rate GS, and our clients were eagerly wanting us to develop high-speed GS.
Regarding CCD image sensors, they are structurally prone to smears (white band that results in an image if the target object is significantly brighter than the surrounding areas) when taking images against a bright light. Used in the ITS, for example, the images of registration number plates are smeared due to the vehicles’ head lights, but CIS overcomes this issue.
Yamane:Then, there is a question why CIS-based GS did not exist before, and the answer is partly because a new structure was needed to add the GS function to the previous CIS. To realize this, a diversity of technology was required for microfabrication, such as memory storage potential structure design, design to embed it in the photodiode, technology to transfer stored information without a loss, pixel designing technology, micro-processing technology, etc., to a greater extent than that required for conventional CIS.
The most decisive difference from ordinary semiconductors is that it processes light. Spectral overlaps directly result in noises in the image. It was one of the important challenges to completely eliminate spectral overlaps.
Tanaka:Through the development processes, we also worked very hard to reduce the overall noises (fixed pattern noise, or FPN). Our competitors produced CISs with superior energy efficiency or speed (higher frame rates), but we heard that many users complained of the noises compromising the picture quality. Noise in the image, and resulting compromised accuracy, was the factor that made manufacturers of inspection systems to hesitate to choose those CIS for their products, no matter how high the frame rates.
Against this backdrop, we decided to develop a CIS for GS, Pregius, leveraging our expertise in pixel technology which we cultivated through SSS Group’s pioneering RS mode CIS and CCD image sensors.
SSS Group’s ambitious specifications and capability to drive the development that overcame the delayed start and gained the market
―― While the development of the CIS for GS started somehow later than competitors, did Pregius have development concepts that could not be negotiated?
Nishide:As Yamane mentioned it earlier, it was necessary to add a special structure to CIS in order to throw in the GS function. To be more specific, it was necessary to insert memories in each single pixel, but this would result in minimizing the saturation capacity of the pixels. In order to ensure sufficient saturation capacity, pixels would have to be enlarged. Whereas, the industrial machine market, which represents our major clients, demanded size reduction. So, the development of Pregius needed to overcome these conflicting challenges of ensuring saturation capacity and reducing the pixel size.
Knowing these major challenges to be conquered, we wanted specifications for Pregius to offer high resolution and diverse aspect ratios, including 16:9, 4:3, and 1:1, in order to cover various needs in the industrial machine market. We ordered the development team to both achieve the high resolution and miniaturization, and realize the performance that superseded CCD image sensors.
Kumagai:The Business Division requested that the new product must be smaller than existing products while retaining the saturation property. This was a highly demanding task that twisted our brains. How to maximize the saturation capacity in a tiny bucket (pixel) is a fundamental problem. It was impossible to solve without innovation.
We tried every single technology that SSS Group had on the conventional memory-retaining potential design to find a break-through. This process may be described as drawing on past cases and “explore all possible ideas” to find solutions. It may sound like an unremarkable statement, but this process of exploring ideas was the most exciting yet most laborious part in the development.
―― To come up with one new idea after another, you must be perseverant and creative. Where do you find the resources for this?
Kumagai:The issue of saturation was truly shocking to me. When I first encountered this problem, I nearly quitted my job. I tried to solve it, but I could not find a breakthrough. The experience and the feeling of defeat stayed with me for a long time, and this time I was determined “to go all the way.”
So finally, we delivered increased multi-pixels and size reduction while at the same time achieved to eliminate noises, which competitors could not solve. The positive market response gives me a great sense of satisfaction.
Predecessor product being highly successful, a further development of Pregius S started immediately
There was the SSS Group insight as a leading company in semiconductors for the market projection and strategies
―― While Pregius was very successful, why did you start the development of Pregius S right away?
Tanaka:Pregius by SSS Group had impressive responses for its low-noise picture quality that was far superior to its competitions, but the market for industrial machines was moving fast, and there was an increasing demand for smaller (miniaturized) and faster GSs with more pixels.
We then decided to try integrating the back-illuminated and stacked CMOS image sensors, which was marketed in RS mode, into the GS mode. If the existing front-illuminated structure can be replaced with the back-illuminated one, more area of the photo diode will be available to take in the light, and the light condensing efficiency will be enhanced. If the light condensing efficiency is improved, the sensitivity can be maintained in a smaller pixel. In fact, Pregius S achieved a micro pixel that was smaller than ever by introducing the back-illuminated structure, realizing the resolution 1.7 times finer than an image sensor of the same size.
Meanwhile, the stacked structure allowed flexibility for freely arranging circuits underneath it, so that we could realize a higher speed by adding various functions onto the sensor. We set out with Pregius S to realize a smaller size, higher speed, and more pixels, as well as a design that met various customer needs, and we succeeded in realizing them.
Image captured by image sensor with conventional structure*1 (conceptual image)
(12 megapixels × 3 frames)
Image captured by image sensor with new structure (conceptual image)
(20 megapixels × about 4 frames)
Nishide:My top priority in determining the specifications was the size that fit in a 29 mm x 29 mm C-mount camera, which was the camera most used by clients. This being achieved, multi-pixel and miniaturization were also realized. Then, another task was to make it pin-compatible with Pregius to make it easy for its users to switch to Pregius S.
The market climate was also different from that when Pregius was first launched in that the usage of the product was far more diversified. For this reason, we decided to make a wider range of offerings in terms of the number of pixels, aspect ratio, speed, and interface types, to give clients choices to suit their needs.
Of course, it was nonnegotiable that the picture quality of Pregius was maintained or improved while the pixels became smaller.
―― Between the front- and back-illuminated structures, are there significant differences in terms of design approaches?
Schematic cross section of chip
Yamane:The back-illuminated structure enabled enhanced light saturation and increased pixels, but the most challenging task for development of Pregius S was to shield the pixels so that the light did not seep into unintended parts. This contradictory issue presented a mammoth challenge. With the front-illuminated structure, a shield was inevitably formed by the metal wiring that rested on top, which was not the case with the back-illuminated structure. This was the technology that significantly enhanced the light condensing efficiency, but ironically, it was necessary to create a light-blocking structure on the otherwise all-clear pixel surface. I knew this was going to be our challenge as soon as the back-illuminated structure was adopted for the development. So, I worked on the process design very carefully with this point in mind.
Kumagai:This shielding structure was a big challenge for me, as well, in pixel design. It was already a difficult task to install a shielding structure, but it also presented a new problem, that it necessarily disabled some areas which were previously available for pixel design.
Furthermore, it turned out that this structure interfered with the saturation and data transfer. There should not be any problem according to the design, but prototype tests always failed to produce anticipated figures. The introduction of this unanticipated structure caused a discrepancy between theoretical figures of the design and actual measures of the prototypes. It was a monumental task to both achieve the light control by the shielding structure and sound functioning of saturation/transfer.
Yamane:The requirement of maximizing the exposure and introducing another structure for shielding, this posed a difficult problem for the pixel design. It also presented us, the process design, many issues to overcome.
Introducing a structure that was not there before caused problems of noise, insufficient exposure, and so on, that were not anticipated from the design. We pursued optimization through many trials and errors.
The process design and pixel design for the shielding structure cannot be developed without collaborations. Between the technology center in Kumamoto and the development team in Atsugi, it was physically impossible to meet up so frequently. We had to communicate by means of online or meeting calls. On this note, I am grateful for Kumagai, who always responded to our queries very quickly, so we could make steady progress in the development and pursue various tests efficiently.
―― Pregius S is packed with all these technologies. Did the development take into account wide application potentials in the future?
Tanaka:Pregius S has realized size reduction, more pixels, and higher speed. We expect that it can be applied to diverse industrial contexts, such as high definition installation test systems, 3D visual inspection systems, and imaging test systems, as well as the automated guided vehicles (AGVs) for logistics and the robot vision.
The offerings cover a wide range from 24 to 5 M, the latter in particular can be applied in many industries. Conventionally, machine visions typically used the 29 x 29 mm camera type with a lens attachment, but in the future a built-in type, that contains a sensor and micro lens within a device, will become a popular choice. The built-in type sensors will be installed, not in cameras, but directly in machines such as robot arms and AGVs (automated guided vehicles) as a small sensor for embedded vision. This will enable us to approach not only camera manufacturers, but also the end-user segment in, for example, the logistics industry to promote the product.
The diverse options for interfaces include MIPI interface which can be developed easily using multi-purpose ISP. It offers an attractive option for those who start developing machine vision devices for the first time.
IMX530-AAMJ(White & Black)and IMX530-AAQJ(Color) that use Pregius S technology
Nishide:It also has high-speed recognition capability, so it will possibly be adopted by the ITS system in the future. There is a growing demand for GS in this area for the vehicle registration number recognition on highways. For this purpose, Pregius S offers an ideal sensor for its properties of multi-pixel, high definition, high speed, and small size, with significantly reduced noise.
Tanaka:In terms of the future, the areas where it can be applied are wide and diverse, such as recycling, SDGs (in the environment, education, health, etc.), science, VR, and sport spectating to use multi-view cameras.
In sports, especially, GS is suitable to capture and render clear, undistorted images of fast-moving balls in baseball or football, for example. In this sense, Pregius S is expected to be in great demand for producing higher-quality images.
Kumagai:As GS is the only technology that can secure the simultaneity, it can also be applied in commercial devices such as head-mount displays and mobile devices.
Nishide:In addition to the property of high resolution recognition of fast-moving objects, Pregius S has new features: an artifact-free HDR, which enables clear capturing of images from bright to dark places and fast-moving objects without distortion. There will be more opportunities for the product to be used in the area of recognition businesses.
Yamane:I have been working on image sensors for a long time, ever since we developed CCD image sensors. This time, we succeeded to develop CIS coupled with a GS mode by ensuring high speed and energy efficiency, and with the back-illuminated structure, we pursued to increase pixels and speed further. So far, it is mainly used in the industrial machine market, but if high precision imaging becomes in demand more widely, it has the potential to cater to various commercial and consumer uses.
SSS Group’s endeavor is never-ending
Possibilities are endless, and needs are as diverse as customers
―― Before we close, tell us what you would like to pursue in the future.
Tanaka:The world is progressing. Compared to what it was like 20 years ago, we live in a really convenient world today. Smartphones will continue evolving, and cars will be automated. People will be enjoying more prosperous lives. In such a world, I would like to be offering better image sensors by identifying people’s needs and working together with the design development team members.
Kumagai:We have succeeded in realizing the miniaturization of pixels through the development of Pregius S. I think this miniaturization is an ongoing endeavor. The pixel miniaturization leads to enhanced resolution and recognition capability. So, I would like to continue working on the miniaturization for GS and engineering new technologies.
Nishide:I would like to pursue the improvement of frame rates and interfaces in order to realize enhanced recognition capability, through-put in factories, and temperature ranges for camera operability. While Pregius S has achieved significant high speed capability, many clients request even faster speed to be realized. Increased speed leads to heat generation, which will limit the temperature range in which the device may operate. So, it will be necessary to control the heat generation, and making the device more energy-efficient is the key solution. I would therefore like to work on new technologies with the development team colleagues to realize this.
Yamane:Image sensors are used in a variety of contexts today, and I think more features will be required in the future to cater to the needs unique to specific uses. These features may vary by the purposes for which the sensor is used, such as HDR, distance measuring sensors, and automotive sensors that operate in darkness on the wavelengths invisible to human eyes. To meet those varied needs, it is necessary to develop purpose-specific technologies. I would like to pursue such new technologies that enhance performance of those features.