“Smart Eye”, an All-round Eye with Various Functions beyond Vision

“Smart Eye”, a kind of artificial eye, is expected to be able to perform a variety of tasks including simple photo taking. Improving vision by allowing to see objects far away, or recognizing very small text or objects that were not previously visible to human eyes, are two of the most basic steps. For Smart Eye, it might also be possible to capture the moment you are looking by simply blinking your eyes or sharing your vision with your friend far away from you in real-time through a network connection.

Since it is a kind of electronic device, it can be equipped with various additional functions like smartphones by utilizing the embedded semiconductor chip. In particular, when combined with AR technology Smart Eye has the potential to enrich our daily lives. For example, you might be able to display translated texts right next to foreign text in your vision at anytime, anywhere. Additionally, when you see something unfamiliar, you might be able to utilize an encyclopedia function to automatically display information about that object, right next to it.

Useful information for daily life such as time and weather can be displayed in a virtual space in your field of vision, while a stored schedule or documents can be loaded and checked with just a blink of your eyes. After you leave your home, a virtual arrow pointing to your destination will show you the way to go. Navigating stores, along with detailed information won’t be a scene of sci-fi movies anymore. All of these are pleasant changes that “Smart Eye” might bring to our daily lives.

Figure. 1 Bionic Vision Technologies’ Bionic Eye System (Source: Official website of BIONIC VISION TECHNOLOGIES)

Image Download

In the industry, “Smart Glasses”, considered to be just one step before Smart Eye, is already performing many functions as mentioned above. Furthermore, an Australian company named Bionic Vision Technologies had attempted to deliver camera images to the brain through an implanted chip behind the eye; the chip is connected to the eyeglass-shaped camera, and in the end the company was able to achieve some visible business results with this innovative attempt. In the near future, when these technologies become more sophisticated, this imaginary “Smart Eye” might be realized sooner than we expect now.

Will Numerous Ongoing Research Projects of “Artificial Eyeballs” Bring Complete Replacement for the Human Eyes?

When would Smart Eye be realized? To find this out, let’s first look at how far the research on artificial eyes has come in this journey so far.

In the current medical field, research on artificial organs has been actively carried out, contributing to the development of various artificial organs. Some meaningful results have been seen, as some of these artificial organs have been successfully transplanted to patients. Artificial eyes which might completely replace the human eyes one day, however, are still in an incomplete stage. This is because human eyes are one of the most sophisticated organs of human bodies.

Image Download

An eye is composed of cornea and lens that refract light rays from the outside, iris that controls the amount of light passing through, pupil where light rays enter, retina where light images are formed, and optic nerve that transmits visual information to the brain. Currently, the researches on artificial eyes that uses heterologous tissues or patients’ cells have reached the stage of artificially fabricating each part of human eyes. This is the stage before making the entire eye artificially.

The latest technology in terms of cornea is held by Newcastle University in the UK which succeeded in making the artificial cornea by utilizing 3D bio-printing technology in 2018. In more detail, the research team produced Mixed Stem Cells composed of cornea, alginate, and collagen, and used a 3D bio-printer to print them into a form of a human cornea, which is the first artificial cornea in the world.

Among the attempts to build artificial eyes using electronic devices, “Argus II” developed in 2013 by an American company Second Sight is one of the most well-knowns. After obtaining approval from the US Food and Drug Administration (FDA) in 2013, Argus II was proven through clinical trials that it can partly improve human eyesight. However, there is a clear limitation as it can be applied only to some patients with certain eye diseases.

Figure. 2 Artificial eye developed by a research team at University of Minnesota (Source: Official website of University of Minnesota)

Image Download

The most recent and noteworthy achievement made in this field was made by the research team at University of Minnesota. In 2018, the team succeeded in printing a receptor (a type of retina) that can receive light upon a transparent hemisphere shaped like an eyeball. With this technology, the receptor can convert light signals into electrical signals via photodiode and semiconductor. Although this “eye” has succeeded in receiving and recognizing light signals with help of semiconductor technologies, further improvement is still expected to produce commercially available artificial eyes.

The Role of Semiconductors in Realization of Smart Eye

Smart eye requires semiconductors, just like any other electronic devices. Specifically, artificial eyes require logic semiconductors to control and operate the system, as well as memory semiconductors to store data necessary for the system implementation. In the case of artificial eyes, the key mechanism is to convert light signals into electrical signals and transmit them to the optic nerve. This is the reason why the role of CMOS Image Sensor (CIS), which is frequently mounted on cameras of electronic devices to perform similar tasks, is crucial.

Figure. 3 The Working Mechanisms of CIS and Human Eyes

Image Download

Like the retina of human eyes, CIS is a semiconductor that converts color and brightness of light into electrical signals to deliver them to the central processing unit. Inserted into various types of cameras, it serves as an eye for electronic devices.

Nevertheless, the current CIS technology has not reached the level of human eyes in terms of major features such as resolution, three-dimensionality, and sensitivity. While the resolution of human eyes is 576 megapixels (MP), the highest resolution CIS can currently realize is only 108Mp. When brightness of surrounding environment changes in a sudden, CIS is also likely to suffer from latency accepting visual information.

There are some areas where CIS is superior to human eyes, however. The human eyes’ field of view is approximately 110 to 120 degrees left and right, and 150 degrees up and down, whereas CIS can cover a full 360 degree view. It also has the potential to go beyond the limits of human eyes with certain functions, such as the telephoto function which allows seeing objects far away. Once CIS technologies reach the level of human eyes in terms of resolution and sensitivity, it can not only just replace human eyes, but also present various advanced functions that cannot be delivered through naked eyes.

Ho-young Cho, Technical Leader (TL) at CIS Marketing Strategy of SK hynix said, “While human eyes’ main purpose is to recognize rather than display the collected visual information, CIS is designed for securing visual information for output. If CIS can recognize at the same level as human eyes do in the future, it will also function as a displaying device that outputs the collected information.”

Cho continued, “Unlike human eyes, CIS is designed as individual modules for various purposes. As a result, CIS is detachable, and users can equip different CIS depending on various situations. Such flexibility will make our daily lives more convenient with no doubt.”

¹BioINwatch 19-51: Core Technology for Bio Artificial Organs, “Organoids”, Biotech Policy Research Center (2019)
²2017 Technology Impact Assessment Result Report: The Future of Bio-Artificial Organs, Korea Institute of Science & Technology Evaluation and Planning, Ministry of Science and ICT (2018)
³2017 Promising Future Technology Program Organ Bio, National Research Foundation of Korea (2017)
⁴3D Printing Technology for Biomedical Applications, So Hyeon Park, Sang Gu Yim, Seung Yun Yang, and Se Hyun Kim, No 1, Volume 18 (2015), Prospectives of Industrial Chemistry (2015)
⁵Series 19: Artificial Organs for New Life, “Scientific Technology Told by Great Scholars”, The Korean Academy of Science and Technology, Young Ha Kim, Free Academy
⁶Medical Device Item Market Report: Intraocular Lens, Kyung-hwa Ko, Korea Health Industry Development Institute (2013)
⁷3D bioprinting of a corneal stroma equivalent, Abigail Isaacson, Stephen Swioklo, Che J Connon, Experimental Eye Research(2018)
⁸Research Brief: Researchers 3D print prototype for ‘bionic eye’, University of Minesota(https://twin-cities.umn.edu/research-brief-researchers-3d-print-prototype-bionic-eye)
⁹Korea Institute of Science and Technology (KIST) website, https://www.kist.re.kr
¹⁰Korea Institute of S&T Evaluation and Planning (KISTEP) website, https://www.kistep.re.kr

The post “Smart Eye”, A Computer Inside Our Eyes first appeared on SK hynix Newsroom.

“Cameras,” the Game Changer in the Smartphone War: Evolution of Mobile CIS with Demand

In the smartphone market, manufacturers are fiercely competing to equip more cameras to their smartphones. As most smartphone specifications have been leveled up, the number of cameras equipped has become a key factor to determine the competitiveness of smartphones. In addition, as the flagship market is saturated and the sales volume slows down, the competition for securing the cost-effectiveness of the mid– to low-priced smartphones has become fiercer. Accordingly, the number of cameras installed in mid- to low-priced products as well as flagship products is rapidly increasing these days.

Image Download

Image Download

According to SK hynix’s own research, the average number of cameras per smartphone almost doubled from 2.2 in 2017 to 3.9 in 2020. For the latest flagship products, even products with five can be easily found. This means that the demand for image sensors for mobile devices is increasing rapidly as well. In particular, as telephoto cameras, which were mainly used for flagship products, are now being actively adopted in low- and medium-priced smartphones, demand for related image sensors is also expected to skyrocket.

As the total number of cameras increased, the rear cameras were subdivided into wide-angle, ultra-wide-angle, telephoto, depth, and macro cameras. Accordingly, the image sensor is also evolving in a direction appropriate for each camera’s characteristics. The image sensor for the main wide-angle camera which requires versatility is increasing the size of a pixel or securing the highest pixel density possible. For a telephoto camera that zooms in on a subject, image sensors are evolving to obtain a high magnification with small pixels. For ultra-wide-angle cameras which have a much wider field of view (FOV) than general cameras, image sensors are evolving in the direction of securing image quality by maximizing the resolution.

As the design where the area occupied by the camera hole in the front display is minimized and the entire front is covered by the display becomes common, a higher level of image quality is required for the front cameras. While the front cameras are getting smaller, consumer expectations for the shooting quality are continuously increasing. Therefore, the key competitive advantage now is how to achieve higher resolutions with smaller modules than competing products.

The four new 1.0μm Black Pearl CIS line-ups, which will be presented by SK hynix in the first quarter of this year, are the result of efforts to respond to such market changes. The company plans to actively target the rapidly changing smartphone camera market with the four new products ranging from 8 megapixels (MP) to 20 MP.

Optimized for Ultra-Wide-Angle Cameras – “Hi-1634 and Hi-2021” Realize the Best Performance at 1.0μm Level

SK hynix reduced the pixel size from 1.12μm in the previous products to 1.0μm for the newly launched four 1.0μm Black Pearl CIS products, in order to respond to the resolution competition with the smaller module size. They feature Quad Pixel function to resize the pixel areas, as well as the Quad to Bayer (Q2B) Re-mosaic algorithm that is more efficient compared to the one adopted in competing products. With these features, the new Black Pearl products not only are compatible with various types of cameras, but also boast excellent shooting quality in low-light environments.

Among the new line-ups, Hi-1634 (16 MP) and Hi-2021 (20 MP) are products optimized for ultra-wide-angle cameras among smartphone rear cameras. In January, SK hynix started mass-production of these products, and they are now being supplied to customers. Ultra-wide-angle cameras require more pixels than basic cameras, because the spatial resolution decreases when the FOV increases. With the smaller pixel size, Hi-1634 and Hi-2021 are capable of fitting more pixels in the same space. In addition, by realizing the best performance that can be achieved with a pixel size of 1.0μm, these products have secured a high image quality that can be adopted even in the flagship smartphones.

Image Download

In addition, Hi-1634 and Hi-2021 products’ Quad Pixel function allows them to respond to diverse shooting environments. They also possess various advantages including high resolution, low-light adaptability, and size competitiveness, making them highly useful not only in ultra-wide-angle cameras but also in front cameras. With these advantages, Hi-2021 is recognized for its outstanding competitiveness in the market, such as being chosen for front cameras of major smartphone manufacturers’ mid- to low-priced line-ups this year.

“Hi-847” with Powerful Zoom Function and “Hi-1337” with Excellent Versatility

Hi-847 (8 MP) and Hi-1337 (13 MP) products are optimized for telephoto cameras among smartphone rear cameras. SK hynix began mass producing Hi-847 in last February, and the mass production of Hi-1337 is expected to begin in March.

Image Download

Hi-847 is an image sensor that supports 3x zoom despite a low module height. In the general module design, only 1.0μm 8 MP products are capable of 3x zoom, and only two companies including SK hynix provide such product in the current market. In addition, Hi-847 boasts higher cost-effectiveness compared to its competitors.

Hi-1337 supports 2x zoom, and it can be used in more various camera types as it has more pixels than Hi-847. Besides being applied to the rear telephoto camera, it can also be installed in a wide-angle camera, or a front camera that requires a low module height, making the application more diverse.

Image Download

Starting with 1.0μm Black Pearl line-ups in the first quarter, SK hynix plans to introduce a 0.8μm 48 MP product in the second half of the year. The company aims to secure competitiveness in the CIS business and solidify its position in the market.

Ho-young Cho, Technical Leader (TL) at CIS Marketing Strategy of SK hynix said, “SK hynix officially names all of its CIS products as “Black Pearl” starting from this year, and plans to continuously introduce products that can deliver the best value to customers in not only the memory semiconductor market, but also the CIS market. We will do our best to establish a position in the CIS market that matches the brand’s meaning, like a black pearl considered as the rarest and the most beautiful pearl among all.”

The post Everything about the “1.0μm Black Pearl,” SK hynix’s Image Sensor first appeared on SK hynix Newsroom.