Image Download

It’s very challenging to further miniaturize components that are already only a few nanometers wide.

The history of electronics is a study in the progression and evolution of miniaturization, as electrical systems matured and migrated to become what we now consider electronics. The reason older tech is huge is that it was full of individual components that were literally wired together. All of the electronics processes that we do today, with digitally driven solid-state devices, had to be done in a brute-force analog manner.

In the case of the transistors that make up our modern microcontrollers and processors, the antique analog versions were called vacuum tubes [1], variations on light bulb technology originally discovered in 1880 by Thomas Edison. Exited metal plates between filament grids in the vacuum-sealed glass bulbs regulated the passage of current but were very power inefficient (because they were modified light bulbs) and very fragile (also because they were modified light bulbs).

Figure 1. Circuit diagram of the Edison discovery.

Image Download

These issues of power and size in electronics were such a fundamental aspect of early electronics that it shaped both popular and professional history. The first computers and their rows upon rows of vacuum-tube-filled racks engulfed a large room and gave off so much heat that the first computer bug reported was an actual insect attracted to the tubes [2]. Even in science fiction, an art designed to inspire futuristic thought, most authors predicted powerful computers—none predicted small pocket-sized smart devices.

Kilby, Noyce, and the Birth of the Chip

The discovery of the semiconductor and the creation of solid-state transistors in the 1960’s began the evolutionary process in electronics [3], laying the foundation for the world we have today. Instead of using a brute-force approach of wires and grids in a vacuum tube, a semiconductor device works by controlling the passage of electrons through a solid monocrystalline piece of semiconducting material. (There is an entirely different revolution going on in semiconductor materials, which we’ll touch on later.)

These semiconductor devices, even in their first iterations, were significantly smaller, lighter, better performing, cooler operating, and much more efficient than their glass-based forefathers. The breakthrough in semiconductors enabled the creation of the integrated circuit (IC), although there were several important technology hurdles that had to be overcome. These included the actual forming of devices in the crystal, the need to electrically isolate components, and the creation of electrical connections within the IC.

The birth of the IC is a story in parallel development. Jack Kilby, a radio engineer and World War II veteran, made the first hybrid IC in May 1958 from germanium [4], followed closely in January of 1959 by Robert Noyce, who unveiled his design of a planar integrated circuit [5]. Since then, semiconductor devices have been continuously miniaturizing.

Later on, in the early 70’s, IC feature device size (the space between the components and connection vias) decreased to 10 μm, and the number of the MOSFETs in a chip exceeded 1,000 [6]. In modern days, chips have gone down to 7nm spacing, with 5nm planned, and work underway for 2nm and less.

I/O, Logic, and Power

There are many aspects that need to be addressed when making electronics smaller in general, including the integration and management of the data input/output (I/O), the logic IC itself, and the power needed to drive it all. The demands of each create a juggling act the electronic designer must perform when creating products.

At the circuit-board level, miniaturization involves high levels of component integration. For example, a product could require an inertial sensor, a compass, temperature and presence sensors, as well as those for images and light. An integrated sensor module groups sensors based on related technologies into a single package, saving significant space.

Meanwhile, package-level miniaturization forces include the shrinking of power electronics due to the advent of new wide-bandgap semiconductors. These new materials, Gallium Nitride (GaN) and Silicon Carbide (SIC) are enabling a 60%+ reduction in the size of power supplies while also increasing performance by up to 300% [7]. They are enabling significant reduction in products simply by making the power electronics smaller.

We must consider the circuit-level miniaturization process as well, which relies on simplifying and/or optimizing the system. This approach leverages advances in device integration and the new power electronics, which enable advanced circuits with smaller passive devices and significantly smaller footprints. It can be leveraged with next-generation user interfaces that eliminate the need for a keypad or other space-hungry input method.

When it comes to logic devices, all issues at the macro-level must also be addressed at the chip level. How many devices can be integrated on a chip? How closely can the components of a chip be put together before they start interfering with one another? How do you get power in the right amount (current) and level (voltage) to the parts of the chip that need it?

Shrinking the Chip

There are several limiting factors when it comes to miniaturization. The first limit is defined by basic physics, in this case the distance of atoms in the material. It’s literally impossible to make the device structures less than the atomic distance, which does define a true lower limit to nano-electronics. This lower limit isn’t the real final barrier to miniaturization, however, as there is a fundamental limit defined by the distance the electrons can tunnel in the material.

The smaller chips get, the harder it is to control aspects like the leakage current between the source and drain of the transistors involved. This is regarded as one of the main limits to miniaturization, because it’s impossible to suppress the diffusion of electrons from the source to the channel at normal temperatures.

Still, the industry will continue to shrink electronic devices of all types because they are critical to so many aspects of our future smart Cloud-enabled Internet of Things (IoT)-oriented society. Both at the macro-level and the micro-level, expect to see higher levels of component integration, both planar and 3D, as well as further advances in materials for both the chips and their interconnects.

In the sense of vertical integration, SK hynix’s 176-layer 512 Gigabit (Gb) Triple-Level Cell (TLC) 4D NAND flash is a good example of the current state of the art [8]. The third generation 4D product that allows the bit productivity to be improved by 35% and the read speed increased by 20% over legacy devices. The data transfer speed also has been improved by 33% to 1.6Gbps without increasing the number of processes.

Figure 2. SK hynix’s 176-layer 512 Gb TLC 4D NAND Flash

Image Download

In regard to material advancements, just as piezoelectric semiconductors like Gallium Nitride have revolutionized the power electronics industry, they offer the promise to do the same for logic chips. One of the major benefits of a piezoelectric semiconductor is its ability to switch at extremely high speeds. Currently advanced GaN power devices can switch up to 40 MHz [9], and if these materials can be adapted to logic circuits, could enable a jump of several orders of magnitude in processor performance at the same feature size.

Another area of approach to miniaturize chips is to improve the way we flash the mask pattern onto the wafer itself. The more we can improve pattern resolution, the closer we can get to the theoretical limits in smaller sizes. The latest lithography methods [10] use short wavelengths in the blue or ultraviolet range.

Figure 3. A diagram of EUV equipment

Image Download

For instance, in the semiconductor memory industry SK hynix recently announced the mass production of its cutting-edge 1anm 8Gb LPDDR4 DRAM by using EUV equipment [11]. As one of the first to successfully implement the technology in mass production, several other major manufacturers are expected to follow in SK hynix’s footsteps as confidence grows in the application of EUV.

Figure 4. SK hynix’s 1anm 8Gb LPDDR4 DRAM Using EUV Equipment

Image Download

Integration is key

Another aspect of design where the micro mimics the macro is in the integration of functionality. Just as a product can be made smaller by integrating various devices on the circuit board, the circuit itself can be made smaller by integrating function blocks into the chip.

By providing functions that used to require their own device, highly integrated chips enable a significant reduction in the board space used [12]. Where once even a “simple” smart product needed a microcontroller (MCU), a power management IC (PMIC), an RFIC, and a chip to control the user interface at the least, can now be served by a single-chip solution, requiring only a display, antenna, and a battery (packaging too) to create a basic product.

All of the developments in component integration, next-generation functionalities, and even packaging advances come together to create the micro-product revolution going on as you read this. Another often-overlooked aspect in the effort to shrink a product involves flexible electronics. Making flat, flexible, and efficient electronics opens the door to countless smart products that were impractical due to size or form-factor issues.

The ability to bend, fold, roll, and even stretch will enable electronics to not only expand more deeply into market sectors like medical and sports wearables, advanced portable and small robotics applications, and the Internet of Things, it will also enable those products to be rolled up and/or folded and easily stored between uses. Optimizing a product design by highly integrating its functionality at the chip, board, and product level pays major dividends.

Bulky keyboards and large screens will eventually cease to be a size issue when flexible electronics and non-contact means of data entry are further commercialized. Reducing the impact of the user interface on product design cannot be underestimated, as every external opening is removed not only reduces the form factor demands, but also increases water resistance and product durability. Optimizing a product design by highly integrating its functionality at both the chip and board level pays dividends at each.

Looking Forward

The advanced electronics and the semiconductors behind them are the sophisticated result of decades of research and development, both in functionality and scaling. The demand for smaller and smaller technology will never diminish, as the new functionalities delivered today create new expectations for tomorrow. The semiconductor industry must continue in its efforts to innovate and keep shrinking electronic components to keep pace with the demands of society in the areas of technological advancement, in even the tiniest of ways.

[Reference]

[1] ScienCentral. (n.d.). The Vacuum Tube. ScienCentral and The American Institute of Physics, 1999. https://www.pbs.org/transistor/science/events/vacuumt.html
[2] National Geographic Society. (2014, July 18). World’s First Computer Bug. https://www.nationalgeographic.org/thisday/sep9/worlds-first-computer-bug/
[3] The semiconductor revolution. (n.d.). Encyclopedia Britannica. https://www.britannica.com/technology/electronics/The-semiconductor-revolution
[4] Kilby, J. (n.d.). Jack Kilby Biography. NobelPrize.Org. https://www.nobelprize.org/prizes/physics/2000/kilby/biographical/
[5] Lovos, M. (2018, June 7). Robert N. Noyce Biography. IEEE Computer Society. https://www.computer.org/profiles/robert-noyce
[6] The Electrochemical Society. (2021, April 13). Impact of Micro-/Nano-Electronics, Miniaturization Limit, and Technology Development for the Next 10 Years and after. Newswise. https://www.newswise.com/articles/impact-of-micro-nano-electronics-miniaturization-limit-and-technology-development-for-the-next-10-years-and-after
[7] Navitas. (2021, August 27). SHARGE Upgrade 100W Fast Charging: 60% Smaller than Legacy Silicon. https://navitassemi.com/navitas-and-sharge-upgrade-100w-fast-charging-60-smaller-than-legacy-silicon/
[8] SK hynix Unveils the Industry’s Most Multilayered 176-Layer 4D NAND Flash. (2020, December 24). SK Hynix Newsroom. https://news.skhynix.com/sk-hynix-unveils-the-industrys-highest-layer-176-layer-4d-nand-flash/
[9] Jha, J., Ganguly, S., & Saha, D. (2021, May 12). GaN-based complementary inverter logic gate using InGaN/GaN superlattice capped enhancement-mode field-effect-transistors. IOP Science. https://www.scribbr.com/apa-citation-generator/new/webpage/
[10] Markoff, J. (2015, September 26). Smaller, Faster, Cheaper, Over: The Future of Computer Chips. The New York Times. https://www.nytimes.com/2015/09/27/technology/smaller-faster-cheaper-over-the-future-of-computer-chips.html
[11] SK hynix. (2021, July 12). SK hynix Starts Mass Production of 1anm DRAM Using EUV Equipment. SK Hynix Newsroom. https://news.skhynix.com/sk-hynix-starts-mass-production-of-1anm-dram-using-euv-equipment/
[12] A Trend Towards Miniaturized Electronics A Trend Towards Miniaturized Electronics. (2021, May 10). ROHM Semiconductor. https://www.rohm.com/blog/miniaturized-electronics

The post Expert Corner: A Macro Look at Micro Technology first appeared on SK hynix Newsroom.

Image Download

We are living in a highly advanced digital era from a technology perspective. Digital devices and technologies are so abundant that it is actually difficult to avoid running into one anywhere we go. Our cellphone is more capable in digital signal processing than a spacecraft from not too long ago; Watching a high-resolution video (or multiple videos for that matter!) is no longer an issue over digital wireless communication; Digital smart devices are providing automation capabilities for residences and offices that must have been unbelievable ten years ago. However, from an energy perspective, more than half of the entire electrical energy consumption in the world is still spent on electromechanical systems – things that move (e.g., motors, actuators, generators, etc.). Surprisingly – or unsurprisingly – the percentage of these electromechanical systems is projected to rapidly grow as electric vehicles are gaining traction in market share. Due to the mechanical nature, these systems unavoidably incur friction and vibration while working, eventually leading to lubricant dry-out, part wear-out, axle misalignment, mount stiffening and fracturing, etc. Therefore, frequent maintenance is an absolute necessity for these machines that move to guarantee adequate service and performance levels.

A Smarter Way to Do the Maintenance: CBM

When do we decide to do the maintenance? Traditionally, too late – when we learn of a failure via an obvious sign like a complete malfunction. For example, we finally realize the compressor in the air conditioner (AC) died after the AC does not blow cold air anymore. Sometimes, the obvious sign can be a utility bill that you happened to check today with twice the amount of what you usually pay on it.

It could have been the refrigerant in the AC that was slowly leaking and running low, causing the AC to work twice as hard to meet the temperature you set. These practices are conventionally called “run-to-failure” maintenance. We simply let the symptom or pathology develop further so that it essentially displays a giant, hard-to-miss sign over it that says “something is wrong.” As you can imagine, this run-to-failure approach is quite inconvenient for users. The down-time of the device or machine happened suddenly from your perspective and now you must invest time to perform a repair yourself or arrange a visit of a repair person. Money-wise, the repair also becomes a costlier fix as the root cause of the symptom must have quite progressed. For example, it could have been a simple touch-up a month ago like a retightening of bolts and nuts of a loose mount. If it went unnoticed for a while, however, so the loose mount finally came off while operating, it would have disastrous consequences like distorting the motor axle and damaging nearby electronics.

In a more regulated setting that should prevent a sudden, sporadic, and unpredictable “death” of equipment, like in military or commercial applications, “periodic” maintenance can be an alternative at the expense of resources (e.g., manpower, money, time, space, equipment, etc.). However, periodic maintenance still does not guarantee that all the potential issues are found during a maintenance event – the equipment can still break down during a mission. Furthermore, as the equipment group becomes larger in number and cost, a percentage of a designated down time for periodic maintenance to its all-time availability leads to an extremely excessive cost regardless of the actual percentage. For example, imagine a semiconductor fab with a hundred pieces of expensive equipment, each of which costing upward of tens or hundreds of millions of dollars. A 1% downtime of an individual machine allocated for periodic maintenance is equivalent to permanently losing one of such equipment at a fab level. Providing periodic maintenance, therefore, can easily become a multi-million (or even -billion) dollar upkeep, depending on the field. At this level, even a $10 million investment to get rid of the designated downtime for periodic maintenance, which sounds a lot of money, is in fact a tremendous deal. Furthermore, due to the pandemic and recent global shortage of semiconductors, halting wafer processes in semiconductor fabs for even a very short period can be extremely costly and should be avoided.

In order to resolve these issues, “condition-based maintenance (CBM)” – or sometimes referred to as “predictive maintenance” – is rapidly gaining popularity as a paradigm for maintenance. As the name suggests, CBM tries to trigger a maintenance event via monitoring the condition of equipment with the primary objective of predicting an equipment failure well before it happens. Actionable health information of the equipment, obtained by analyzing a continuous sensor data stream, will be immediately sent to the decision-maker if it is noteworthy.

Compared to run-to-failure and periodic maintenance, CBM can significantly improve the equipment reliability because an issue would be found at a very early stage even before it becomes symptomatic – like the bolt retightening example above. This advantage makes the repair cost yet another advantage because there are less things to fix with nearly no damage at that point. The repair, therefore, can be performed more easily by widely available labor, making it an even sweeter deal. The repair will also take less time with no or minimal equipment downtime if any. Naturally, CBM is of special interest to mission-critical areas, such as high-tech manufacturing, off-shore platforms, aircrafts, and spacecrafts.

Figure 1. The Concept of “Condition-based Maintenance”

Image Download

For CBM to work, however, there is a price to pay: we need to provide a “continuous” data on the condition of equipment, whether the equipment is in a good condition or not. In addition, for a detailed understanding of various conditions and states of the equipment, we would like “many” sensors to generate fine-grained data. A large amount of raw sensor data will be continuously generated and must be processed to draw out actionable information from it – a concise, impactful message for a decision-maker.

As you can imagine, these steps are extremely data-heavy and computation-intensive, requiring a considerable amount of powerful hardware (e.g., powerful CPU/GPU, RAM, and data storage) to perform real-time and complex computation. Furthermore, installing “many” sensors in and around the equipment is not a trivial task either. Integration or retrofit of extra sensors and computational resources into the existing equipment might not be always feasible. It will require considerable engineering of its own even if so. Another big hurdle for enabling CBM is the support for power and network. How do we power the newly retrofitted sensors and computational resources? How does a final message from one sensor reach the decision-maker?

Powering a CPS

Let us ponder on the first question. If there is a nearby power outlet for a retrofit cyber-physical system (CPS) – a recent trending name for sensor nodes, embedded systems, or Internet of Things (IoTs) – it would be an easy solution. However, not only multiple feet or meters of dangling wires from our CPS to a nearby power outlet are unsightly, but also pose various risks to the host environment: electrical and mechanical safety (after all these are vibrating or moving mechanical systems); noise and security concerns for the host systems’ electrical grid; and potential electromagnetic interference (EMI).

Because of these concerns, the “retrofit” CPS in many cases are expected to be power-independent with no grid access allowed. Then, how do we create a “non-intrusive” power supply for a CPS? A large enough battery pack might come as an attractive solution at first. However, battery alone is not a lasting solution as it eventually needs to be replaced or recharged – periodic maintenance! This essentially leaves a self-powering mechanism (i.e., wireless power transfer or energy harvesting) as the only option.

Wireless power transfer (WPT), regardless of whether it is inductive or capacitive, can send a significant amount of power through medium like air easily up to a kW level. However, it requires a dedicated transmitter on the grid/host side, bringing up the “intrusiveness” issue again at a much grander scale than dangling wires. Unless the WPT is already designed in in the host system, the high intrusiveness makes it an unattractive solution, especially when a small CPS for monitoring purposes consumes mWs or Ws at most.

Figure 2. Wireless power transfer vs. Energy harvesting

Image Download

Energy harvesting, on the other hand, does not intend to receive significant power from a dedicated, active transmitter on the other side. It extracts energy from an ambient energy source (e.g., light, temperature, electromagnetic fields, vibration, motion, friction, etc.), rather passively. The electrical installation/connection issues with respect to the host grid are, therefore, naturally nonexistent. However, being without a dedicated transmitter on the host side imply low power and energy densities, which mandate a large harvesting interface (i.e., area or volume) on the CPS side. Furthermore, an ambient energy source sometimes dictates the operating environment as well. For example, both photovoltaic (PV) and wind energy harvesting typically require outdoor installation and operation. Popular harvesting sources and their required interface sizes are presented in the table below. Here, a 100mW extraction is assumed, which is a reasonable target for a small CPS for monitoring.

Chart 1. Comparison among harvesting sources

Image Download

Supplying 100mW via traditional wiring is easy. Practically any pair of wires you can find easily does 100mW (e.g., your USB charging cable can easily do 5W = 5000mW). The real benefit of the energy harvesting is that your CPS becomes independent of the external power source and wiring such that the CPS can be placed virtually anywhere. A good real-life example is an outdoor security camera around your house solely powered by a PV cell. You do not need to create a long power wiring from a closest power outlet, which might be tens of feet or meters away from where you want to install your security camera. You also do not have to worry about drilling a hole through a door or a wall and weather-sealing the holes and power lines, which are big deterrents in installing outdoor electronics. As mentioned above, however, this approach would not work indoors as the PV cell would be nearly worthless.

Let us discuss a little bit more about energy harvesting methods for powering a CPS, especially for electromechanical systems that “move” and are mostly indoors, like Overhead Hoist Transfer (OHT) equipment in semiconductor fabs shown in Figure 3. Based on the table above, piezoelectric harvesting, magnet-based vibration harvesting, or AC field-based magnetic harvesting methods are most relevant.

Figure 3. OHT equipment running at the ceiling of SK hynix semiconductor fabs

Image Download

Piezoelectric harvesting is based on a special material that can generate voltage across two surfaces if there is a pressure across. As a motor vibrates and causes the pressure difference between two membranes of the piezoelectric interface, the voltage is induced. The power/energy will be extracted if we connect a load or an energy storage.

Magnet-based vibration harvesting is based on a permanent magnet suspended in a rigid structure – traditionally a metal cantilever. As an electromechanical system vibrates, the metal structure and permanent magnet – especially the tip of it – will also vibrate. According to Mother Nature’s fundamental physics, the change in the magnetic field due to the vibrating magnet is closely related to and can be converted into voltage via Faraday’s law. By connecting a load, the induced voltage will start flowing a current, indicating positive power generation. This is the same principle of battery-less bike wheel lights that turn on when the wheels are spinning.

Figure 4. How battery-less bike wheel lights power themselves

Image Download

AC field-based magnetic harvesting is based on the host system’s AC current while operating. Again, according to physics, the AC current of the electromechanical system must generate time-changing magnetic fields around its current carrying wires. By forming an electromagnetic coupling through magnetic material and winding – similar to a typical transformer – magnetic energy can be harnessed. Connecting a load or a charge storage, like a battery or a capacitor, will result in positive energy extraction. This approach can be very efficient because the electromechanical system should be monitored when it is operating and that coincides with the energy harvesting opportunity. Another benefit of this harvesting method is that it is significantly more power dense, compared to other approaches and less prone to mechanical issues in itself.

No matter which harvesting method is selected, the take-away point is that by having a sufficient space for a harvester or bringing the power load below the harvester’s capability, a CPS can be majorly worry-free from the power and energy perspectives, promoting the global trend of environmental, social, and governance (ESG) criteria.

Issues in a CPS: Networking

The easiest way for these CPSs to communicate with a final decision-maker is to be connected to the existing network infrastructure (e.g., WiFi (or similar) of a smart factory). However, based on the host environment, such an access is not always guaranteed. For example, external networking devices would not be allowed onto the military/utility networks for obvious security reasons. Then, a surefire way to construct a messaging channel for our CPSs is to have our own, independent network without relying on the host system’s network resources – just like our self-powering harvesters. One of the viable ways is to use the CPSs themselves to build a mesh (or partial mesh) network as they will be likely scattered over a wide area. Combined with energy harvesting, this mesh topology brings an interesting challenge at a higher level: propagation of a message to the final decision-maker.

Picture a real-life case of hundreds of electromechanical systems and CPSs scattered throughout a semiconductor fab (e.g., various pumps, actuators, and generators). The motors will operate at different times for different durations. The “monitoring/sleeping” frequencies of individual energy harvesting CPSs will be naturally different. Therefore, in the overall picture, hundreds of self-powered CPSs will come on- and off-line irregularly at their own paces and energy reserves. Based on which CPSs are alive at the moment, an important message from one CPS might or might not have a complete path to reach the final decision-maker, in which case the message must be stored somewhere in the network with a shorter expected time to reach the decision-maker than where it originated.

Figure 5. Two approaches for a CPS architecture

Image Download

“What kind of a message?” is the next question we should be asking. This concerns the architecture and data structure of a CPS. There are typically two approaches for a CPS architecture: 1) powerful onboard computation to locally produce a ‘short or no’ message (health information) and low-bandwidth communication to send it; and 2) minimal onboard computation and high-bandwidth communication to send raw sensor data. Obviously, the first option will spend minimal or no power for low-bandwidth communication (e.g., Bluetooth Low Energy (BLE)) at the expense of generally large power dissipation in the onboard computation hardware. Since the equipment health will be locally assessed, “no message” can be a response in case of a healthy motor. The second option will spend low power in computation (e.g., no processing on the raw sensor data) at the expense of large power dissipation in the high-bandwidth communication (e.g., WiFi). This option is not capable of locally deducing health information and must transmit the entire raw sensor data to the decision-maker for the “analysis.” Generally, each option has its own merits. However, in a case like this, where hundreds of CPSs can simultaneously generate raw sensor data, even a Gbps WiFi network can be easily overrun. In addition, each CPS must be able to store gigabytes or terabytes of “simultaneous and raw” sensor data temporarily, in case of no complete path to the final decision-maker at that moment – our CPS will no longer be a simple, small monitoring device. Therefore, in this case, the option 1 – the ‘short or no’ message approach with a powerful onboard computation capability – is much more sensible, feasible, and manageable.

The Challenges and Solutions

Inferring health information from vibrational and electrical sensor data conventionally requires complex mathematical operations (e.g., time-series manipulation, domain transformation, filtering, windowing, etc.). Furthermore, calculating strategies on arranging an optimal message transfer path, based on the previous history of on- and off-line timings of hundreds of CPSs, and on selecting the optimal locations of the temporary storages for messages, if there is no complete path at the moment, is also a computationally intensive task. A powerful CPU and/or GPU and large amounts of RAM and high-speed data storage are typically employed for tens of seconds to perform such real-time computation. Such a computation system can cost thousands of dollars or more with the instantaneous power consumption over hundreds of Ws or even kWs. These constraints are well beyond reasonable operating levels of an energy harvester and a small/medium CPS.

With the aid of emerging artificial intelligence (AI) and neural network (NN) technologies, recent research publications [1, 2] showed groundbreaking advancements in developing such a computation capability in a small-scale CPS. Instead of burning hundreds of Ws, only hundreds of mW are required during complex mathematical operations – a thousand times lower power consumption. This is because the AI and NN algorithms do not need to perform the original complex mathematical operations to deduce the final answer. The AI & NN algorithms reach the same answer with an extremely high probability without performing the real math in the original implementation. On top of that, instead of thousands of dollars worth of powerful computational hardware, only tens of dollars (or even less) worth of widely available hardware is required to complete the computation – a hundred times lower cost. The physical volume and space for the computational hardware is also relatively small as there is no need for big, bulky power supplies and cooling systems. This is enabled by a highly target-oriented edge computation device, implemented by a field-programmable gate array (FPGA), with tightly hardware-optimized algorithms. In simple terms, it is extremely fast and power efficient in doing a limited set of highly optimized computation – in this case, AI and NN algorithms to deduce the “health” information of an electromechanical system. However, it is not built as an all-round player like our desktop or laptop CPUs are. Dedicating toward a highly concentrated task using AI and NN and using a highly optimized set of hardware resulted in such an incredible boost in performance, power reduction, and cost reduction.

The impact and applicability of these technologies are immense. The onboard computation capability in a tiny CPS – which was the biggest hurdle in closing the gap between the large amount of continuously collected data and lower bandwidth communication constraint – is finally becoming a reality. Data filtering, signal processing, compression, and intelligent mesh network routing can be quickly done “locally” at a negligible power consumption and a cost addition. A real “smart” device that can compute like a desktop for its task and can be sustained by a tiny energy harvester without expensive Li-ion batteries will be the fundamental block for condition-based maintenance (a.k.a predictive maintenance) in the near future.

The AI and NN software technologies are currently conquering some of the most challenging engineering problems in unexpected ways. Recent advancements in semiconductor technologies have essentially enabled such research, designs, and innovations by providing explosively increasing computational power and memory capacities at lower costs. Semiconductor manufacturers, including SK hynix, will stay extremely busy to keep up with the never-ending appetites of the software technologies on critical hardware equipment, including large data storages for massive amounts of training data for deep NNs (multilayered 4D NAND flash and storage solutions – SSD/SD card/etc.), high-speed and high-capacity memories (DRAM – HBM/GDDR6+/DDR5/LPDDR5/etc.), and fast processors.

Figure 6. SK hynix’s 1anm DRAM Using EUV equipment

Image Download

Figure 7. SK hynix’s 176-layer 4D NAND flash

Image Download

[Reference]

[1] S. Kang, J. Moon and S. Jun, “FPGA-Accelerated Time Series Mining on Low-Power IoT Devices,” 2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP), 2020, pp. 33-36, doi: 10.1109/ASAP49362.2020.00015.
[2] J. Chen, S. Hong, W. He, J. Moon, S. Jun, “Eciton: Very Low-Power LSTM Neural Network Accelerator for Predictive Maintenance at the Edge,” The International Conference on Field-Programmable Logic and Applications (FPL) 2021.

The post A New Approach to Energy Efficient Maintenance: Condition-based Maintenance first appeared on SK hynix Newsroom.

Packaging, though, is now getting a fresh look as the semiconductor industry seeks to pack even more performance and capabilities into less space. In this EE Times column, SK hynix Project Leader Ho-Young Son provides new insights into how packaging can push the boundaries of memory chips and system performance overall.
He explains how advanced technologies, such as TSV (through-silicon via) and Fan-out wafer-level packaging, are helping create even smaller, thinner memory chips with higher capacities. These packaging techniques also have added benefits, such as more efficiently dissipating heat.

To find out more, please visit EE Times to read the full column titled “Advanced Packaging Technologies Overcoming the Memory System Performance and Capacity Limitation”.

Image Download

The post How Advanced Packaging Technologies Are Helping Increase System Performance first appeared on SK hynix Newsroom.

Image Download

The semiconductor industry enables and supports all aspects of modern life. Semiconductor products provide the processing for data centers, the network edge and in embedded industrial and consumer devices. It also provides fast network communication required for data centers and connected devices. Semiconductor memory/storage products such as DRAM and NAND flash provide the memory and long-term storage of information required to support data processing and to keep the results of that processing available for future access.

Vehicles are becoming mobile computers with semiconductors ensuring efficient operation, ADAS and autonomous driving services as well as infotainment. Embedded connected electronics control factory operations and enable smart cities. They are an essential element in Internet of Things (IoT) applications, including wearable devices that help us stay connected and provide ways to monitor our health and motivate us to stay healthy.

Let’s take a look at how the semiconductor industry works, the impact of the COVID-19 pandemic on the industry and how the increasing capabilities of semiconductor devices will enable an amazing array of products and services that will allow us to work more effectively, manufacture products more efficiently, keep us healthier and entertain us in more engaging and immersive ways.

Big Breakthroughs for Smaller Tech

Image Download

The semiconductor industry creates small electronic devices (chips) that can be incorporated into various products. These chips use electric energy to process data, control operations, encrypt and decrypt data and store data temporarily or long term.

Semiconductors are made on mostly silicon wafers in large factories and using sophisticated and expensive equipment. For instance, a modern 3D NAND flash fab costs more than $10B to equip and build, and if it is a brand-new plant, may take a year and a half to come online with high production volumes. The equipment required for a semiconductor fab depends upon what sort of wafers are being processed.

Some semiconductor companies, such as SK hynix, make all or most of their own wafers and chips. Many other semiconductor companies own no semiconductor fabs and instead have their chips manufactured in large semiconductor foundries. These companies are called fabless semiconductor companies. There are several large semiconductor foundry companies in various parts of the world that service these fabless companies.

Semiconductor development is following what has been known as Moore’s Law, named after Gordon Moore, one of the founders of Intel. This “law”, which is a projection of a historical trend first observed in the 1960’s, said that the number of transistors on semiconductor integrated circuits (ICs or chips) doubles about every two years. This doubling of circuit density was accompanied by various other scaling trends, increasing the performance and lowering the power required by the ICs as well.

Moore’s Law has become more difficult to sustain in recent years as the size of the features that make up a transistor have shrunk. The equipment to manufacture these small features has also increased significantly in price. Currently semiconductor devices are available in volume at down to 7nm minimum feature sizes with 5nm planned and work underway for 2nm and less.

But, because of the cost to make these small features, new ways to design chips have emerged. For instance, some companies are breaking up the capabilities that might have been built into a single chip using one lithographic node into multiple chips with different minimum lithographic feature requirements, called chiplets. These chiplets are located next to each other in a chip module. Another approach is to stack and connect wafers on top of each other that may differ in their lithographic processes and even their particular operations (e.g. CPUs, memory and specialized processing for particular tasks).

With these new approaches for semiconductor devices, chips with larger lithographic features can be used for many functions, focusing the more expensive small features processes where they do the most good and avoiding the expense of converting all the semiconductor operations to the smaller lithographic node.

The Pandemic-Induced Shortage Felt Around the World

With the COVID-19 pandemic in 2020, many semiconductor fabs went idle for a time and the semiconductor supply chain was temporarily disrupted. At the same time global demand for mobile devices, PCs and data center equipment surged in response to online learning, remote work and other activities.

Also, during the pandemic, some industries, such as the automotive industry, cut back on their orders for chips, anticipating weaker demand. Semiconductor companies responded by allocating more output to semiconductors for other applications. As the pandemic has eased with the introduction of vaccines, demand for products such as automobiles has increased, but many semiconductor companies don’t have additional capacity to meet this demand. Because of the time it takes to bring new semiconductor capacity online, several semiconductor companies have said that it could take a couple of years for capacity to meet demand.

Leveraging New Technologies, Broadening Markets

Image Download

In addition to semiconductor shortages for products like automobiles, the overall demand for semiconductors is increasing to provide advanced cloud services, wireless networks (such as 5G and WiFi6) and to support the application of artificial intelligence (AI) at the network edge and in industrial, civic and consumer endpoint devices.

5G smartphones may use $25 worth of chips compared to $18 in 4G and $8 in 3G phones. Nearly three-quarters of all cars will likely ship with cellular connectivity by 2024 and the cost of electronic components in cars was about 45% of the total car cost in 2020, up from 20% in 2000¹.

Semiconductors and wireless connectivity are enabling new wearable and embedded devices that will help us stay healthy. They will also power a new generation of robots used in human/robot manufacturing teams and for home health care and companionship.

In addition to computing, communication, healthcare and transportation, entertainment systems use lots of chips. With the introduction of high dynamic range 8K video, virtual and augmented reality and even more advanced immersive audio-video technologies, demand for semiconductors to produce, process and deliver this content will soar.

Semiconductors will help the unconnected have access to the Internet and a better life, create safer transportation and cities, make our factories more efficient, keep us healthier and entertain and educate us in new ways.

¹https://www.fool.com/investing/2021/04/27/6-causes-of-the-global-semiconductor-shortage/

The post Global Trends, Semiconductors, and Evolving Market Demands first appeared on SK hynix Newsroom.

The Cultural Agenda for SK hynix: Creating Social Values in the Era of Culture Fusion

Peter Drucker, the founder of modern management, is attributed to have said “Culture eats strategy for breakfast.” While he may not have been the first to say it, the statement is more true today than ever before.
To understand how true that statement remains today, look no further than the semiconductor industry. Mergers and acquisitions in the sector set records in 2020. And while corporate strategies may have driven those deals, their successes would actually depend on the cultures that would form from the combined companies. Our research has consistently shown that paying attention to culture during a merger increases the probability of success from 30-40 percent to 60-70 percent.

SK hynix already has a history of integrated cultures from three of South Korea’s top four business groups – but only through inbound efforts. Now, those efforts need to go global, making it critical for SK hynix to meet its own Social Values 2030 campaign (green, advance together, social safety net, corporate culture) without failing to sustain its global market leadership in semiconductor memory. Its competitors are also spending a lot of resources to meet the growing social demand by appointing diversity executives and creating alliances for inclusion.

Just as semiconductor technology has advanced, so has understanding of social values – corporate culture, in particular.

Traditional social values focus on beliefs that shape behaviors inside a company. Often described as the “roots of the tree,” they reflect the values of the company that are upheld by all employees.

Increasingly, they need to reflect the “right” culture, or the identity of the firm in the mind of its best customers today and tomorrow. Identity is about brand and its values in the minds of customers. SK hynix brand identity “Technology Innovator for a Better World” and brand slogan “We DO Technology” provide a purpose that’s real to both customers AND employees. The “right” culture means making this customer-centric brand identity real to every employee every day.

For SK hynix, this means that the four social values resonate in the marketplace with customers, investors, and communities.

• Being “Green” helps protect the planet and makes SK hynix a contributing social citizen
• “Advancing together” implies co-existence with business partners
• Creating a “social safety net” means caring for physical, emotional, and social needs of not only employees but also the community and the vulnerable social groups
• Establishing “corporate culture” focuses on diversity, inclusion and equity means that SK hynix cares for employees’ development

These four social values do not just define how SK hynix operates but are also key drivers of the technology brand identity in the marketplace. When customers see these social values in practice, they prefer to do business with SK hynix. When investors recognize the commitment that SK hynix has to these social values, they increase confidence in future earnings and increase stock price. When communities recognize SK hynix citizenship actions, they build the reputation.

With this in mind, the “right” culture starts with the aspiration of the Purpose (making a better world with all members of the society by leading the tech-based IT ecosystem) and promise of the brand. The right culture becomes less about the roots of the tree and more about the leaves that grow and change over time.

Diversity, inclusion and equity have been ingrained in SK hynix’s social values and culture from creation and have become a driving force for what matters to customers, investors, and communities today and tomorrow.

Diversity requires divergence and respecting differences. Divergence means that the company is constantly open to new ideas and ways of working. Innovation occurs not only in products and services, but in encouraging different backgrounds. Divergence comes from open dialogue, seeking others’ opinions, and looking for new answers to old problems. Divergence means that every employee has something to offer the company and that opinions will be respected. Diverse organizations create a pipeline of innovation because employees share their best ideas and disagree without being disagreeable.

Inclusion requires convergence and unity. True diversity begins with unity. Unity, or convergence, means that every employee believes in the SK hynix purpose and works to make the marketplace brand into workplace actions. Unity about the social values means that employees are all moving in the same direction, with different skills. Diversity without this unity is random action; unity without divergence leads to groupthink and a lack of innovation. Inclusion also instills a sense of belonging where each employee feels a sense of personal connection to the firm. This sense of belonging comes when employees not only share the values but enjoy working together with each other and learn and grow from that work. Equity means that everyone has an opportunity to live up to their potential.

The DI&E culture is not just for employees inside the company, but also customers, investors, and partners. These external stakeholder groups are also comprised of people who want to be respected for their diversity and feel a sense of belonging through inclusion.

A positive DI&E culture begins with senior leaders who model how to manage the divergence/convergence ongoing process. These leaders use their power to empower others by seeking input and fresh ideas (divergence) and by focusing attention on the key priorities (convergence). Employees will often do what their leaders do, and when leaders empower others, employees will not only respect their leaders, but also follow them.

Ultimately, every employee at SK hynix is responsible to help create and embed the right (DI&E) culture so that it is not an abstract ideal, but an action-oriented daily practice. Here are some questions each employee can ask to help make the right culture real:

Image Download

Peter Drucker’s insight about culture being equal to or more important than strategy applies even more today. In a world where strategies, products, and technologies are changing faster than ever, the right culture must also adapt to deliver future success.

The post The Cultural Agenda for SK hynix: Creating Social Values in the Era of Culture Fusion first appeared on SK hynix Newsroom.

Navigation has become a quintessential part in our daily lives. Smartphones double as car navigation devices, smartwatches can be a hiking trail guide and so on. But how does a device really know where we are? The most common technology is the GPS, the Global Positioning System.

Hundreds of kilometers above our heads, GPS satellites are orbiting the Earth and radiating electromagnetic (EM) signals. By detecting the minute differences in arrival times of those EM signals, a GPS device can pinpoint where someone is standing on the planet. And while the technology is essentially free and requires no subscription, it does require a device that can read GPS signals.

The core of this technology is the satellites themselves, something external that we cannot control. Without the satellites, GPS technologies are of little value. Line of sight to the satellites (even though we cannot see them with our eyes) are critical to the technology’s functionality, which is why GPS navigation frequently fails in tunnels, parking garages, mountainous regions with lush forestry and tall trees, or in crowded cities with skyscrapers. GPS signals can also be attacked and jammed by a 3rd party. When it works, however, GPS provides a relatively accurate result. Overlaying the current ‘position’ estimated by the GPS on top of a map creates a base for a navigator. The remaining job is to perform the ‘positioning’ frequently and update the display.

But does ‘positioning’ generally need a continuous, external aid such as GPS satellites? For humans, the answer is no because we don’t rely on external EM waves to have a morning jog around the neighborhood. Even for a previously unvisited area, if we are armed with a static map – whether in our heads based on previous experiences or a physical paper map – we can position ourselves correctly in that map and navigate to a friend’s new home, for example. Our eyes can perceive how fast we are moving, how far we are from reference points, or how close we are to a decision point such as a turn, landmark or destination. Our body’s ‘positioning’ system is completely integrated and self-sufficient and doesn’t require a continuous aid from an external resource.

Newer electronic devices, such as cars and robot vacuum cleaners, have taken the mimicry of this vision-based approach and even utilize a light spectrum invisible to human eyes, such as infrared, laser and RF waves, for a better ‘visualization’ of the environment. The downside of the vision system, however, is that we need to collect and interpret the data from vision sensors. Inferring direction and speed of a motion from the vision data is not a trivial task. It is a huge computational load that requires powerful processors, as well as large data storage and memory. It also demands a high power and energy consumption. Together, that all adds up to a more expensive system.

Image Download

Simple approach to location tracking

Is there a simpler approach for positioning without an extreme amount of computation? In theory, we can use one of the most ubiquitous sensors we have around us – an accelerometer. As a motion-based sensor, an accelerometer requires an almost negligible computational load to determine a position in principle, compared to a vision-based approach. At the same time, an accelerometer is extremely inexpensive. The theoretical operating principle behind it is also intuitive and straightforward.
By definition, acceleration is the change in velocity over a time duration (a=Δv/Δt) and velocity is the change in position over a time duration (v=Δs/Δt). Merging these two relationships and generalizing it for a nonlinear movement, acceleration can be related to the position:

This simple relationship tells us that the double-time differentiation of the position must be the acceleration. By holding the positional data over time, we can take the double differentiation and accurately determine the acceleration during that trip, obtaining ‘a’ from ‘s.’
Since this is a mathematical formula, we also can determine ‘s’ from ‘a’ by reversing the calculation. In this instance, we would need a double integration:

In theory, this indicates that we can perform the double integration to obtain the position if we hold the acceleration data over time, such as the data points reported from the accelerometer. This does, however, create the immense challenge of the ‘self-sufficient’ inertial navigation. To better understand, let’s go back to a lesson from the early years of college. Differentiation shrinks the expression inside and eliminates constants while integration grows the expression inside and generates a constant. ‘Double’ differentiation will eliminate up to linear terms, whereas double integration will grow the contents at a tremendously faster rate.

The challenge in this technology is simply due to this double integration and the unavoidable tiny errors in acceleration data samples. For a hypothetical, slight error in position measurement, differentiation would diminish the effect of the error over time, and even more so for the double differentiation (i.e., from s to a). On the other hand, a slight error in acceleration measurement would grow with an integration, and even bigger and quicker with the double integration (i.e., from a to )s. For example, a quantization error, a mechanical bias in an accelerometer, a miscalibration, and even undetectable defects that are under manufacturing tolerances, always exist in the captured acceleration data.

If this acceleration goes through double integration for positioning purposes, these tiny errors are all double-integrated, without bound. If we take this approach, a static object on your desk will have a moving trajectory as soon as the double integration starts. If we watch longer – that is, the integration time gets longer – then the object will continuously accelerate away from you, three-dimensionally. Within a few seconds, the double integration will report that the object has arrived at the Moon. This ‘drift’ due to the integrated error over time is a nightmarish problem for self-sufficient inertial navigation known as ‘Dead-Reckoning.’

How to reduce errors with inertial navigation

There have been efforts to limit how much error can be produced in each sampling, such as an object-dependent physical limitation (e.g., humans cannot move faster than a certain distance per step), and determination of possible motion ranges (e.g., multiple inertial measurement units (IMU) that include accelerometers, gyroscopes, and magnetometers can be placed at multiple locations of the moving object to detect and limit impossible motions.) They are effective to a certain degree as the error is at least bound by the set ‘rules’ would keep it from accelerating away from you at an astronomical speed. Yet, the problem on the snowballing error in integration remains and is fundamentally impeding ‘accurate’ positioning (e.g., a still object).

In order to completely suppress the error, the traditional approach has been the investment of more hardware, especially non-motion-based sensors, such as vision and laser. However, with the involvement of other error-bounding sensors, the benefits of inertial navigation solely with motion-based sensors – low computation complexity, cheaper construction, low power consumption and so on – now dissipates. This has limited the usage of an inertial navigation system mostly to spacecraft and aircraft applications that can afford such requirements for a short period of time, keep the inaccuracy of the positional estimate under certain levels. An inertial navigation system was used, for example, in the Apollo space shuttle and has also been used to supplement flight automation and navigation systems in Boeing 747s and US military aircraft.

Image Download

Under practical applications, the integration of the acceleration data is routinely carried out in a Kalman filter, where extra sensor outputs such as a gyroscope or magnetometer can further enhance the performance of the positional estimation. When the ‘prediction,’ or ‘estimation,’ is highly nonlinear to the input, such as our double integration, an Extended Kalman filter (EKF) is used. The “error” or “noise” characteristics will be included in the EKF system and considered a natural input to the system. The noise characteristics will be modeled with utmost precision to eliminate (or accurately account for) its effects during the double integration – again, in principle. However, aforementioned ‘tiny’ noises in measurements – such as a quantization error, a mechanical bias in an accelerometer, miscalibration, and even undetectable defects that are under manufacturing tolerances – can be dynamically changing during and between the sensor operations, rendering the precise modeling of such error sources a nearly impossible task.

Image Download

Image Download

AI-assisted Dead-Reckoning

With a recent, strong emergence of Artificial Intelligence (AI) technology and deep neural networks, a great opportunity has surfaced for enabling automatic learning of the ‘noise parameters’ and relevant customizations that are beneficial for the IMU-based self-sufficient inertial navigation. Figure 1 shows the traditional approach with the IMU measurements, noise modelings, and the EKF, whereas Figure 2 illustrates the trending approach without the noise modeling, which fully utilizes the machine learning-based engine for the automatic noise characterization. Figure 3, excerpted from the “AI-IMU Dead-Reckoning” report¹, shows a promising result for automotive applications.

Image Download

For verification purposes, the solid black curve, denoted as “GPS,” is provided as the ground truth. The blue curve, denoted as “IMU,” is the result of the direct double integration of the acceleration. As expected, it suffers from the diverging integration error and veers off from the ground truth in the initial stages. The dashed green curve, denoted as “AI Engine,” is the result with the EKF system, aided by the AI Engine producing the adaptive noise parameters. The AI approach is surprisingly effective and accurate, compared to the ground truth using GPS. An interesting aspect of this plot is that the GPS actually malfunctioned embarrassingly during this trip – denoted in the figure as “GPS outage.” The “ground truth,” in fact, was not the real truth as it could not report accurate position during the outage section. Meanwhile, the AI-enhanced IMU-only dead reckoning presented the precise location during the GPS outage. In fact, this AI-based dead reckoning is even comparable in performance to the LiDAR and powerful vision-based approaches. The physical size, power consumption, and the cost of these powerful positioning systems are unbearably high, comparable to the AI-fueled dead reckoning method.

Please note that Figure 3 is only 2-D implementation of a potentially full 3-axis travel and made at a vehicular level with the relevant units of km, km/h, and meter (resolution). It would pose different requirements in the AI-engine design and sensor capabilities for a human-level and -scale navigation, especially regarding the minimal resolution in the positional estimation. These varieties are actively being investigated by academics.

Image Download

Impact of positioning technology

The impact and applicability of this technology are immense. Compared to existing navigation technologies, it will enable navigation that is self-sufficient, extremely low-power, tremendously economical and environmentally independent – for example, weather, electromagnetic interference, trees, buildings, line-of-sight and so on. Autonomous moving objects, such as vehicles, robots or bikes, can feature an accurate, self-standing navigation technology on top of other navigation aids at a negligible cost addition. Indoor navigation for humans, pets, carts, and other objects also will have endless combinations of use cases. Likewise, it also can retrofit easily into existing devices, as the location estimation engine is purely at a software level. Already, we have accelerometers and gyroscopes all around us in abundance – via our smartphones. There might even be a cool (or fun) app soon that utilizes this incredible advancement in technology.
The AI software technologies are currently conquering some of the most challenging engineering problems in unexpected ways. Recent advancements in semiconductor technologies have essentially enabled such innovations, providing explosively increasing computational power and memory capacities at lower costs. Hardware manufacturers, including SK hynix, will stay extremely busy keeping up with the never-ending appetites of the AI software technologies on critical hardware equipment, including large data storages for massive amounts of training data for deep neural networks, such as SSD cards integrated with NAND Flash, high-speed and high-capacity memories such as DRAM (DDR4, DDR5, HBM2E, GDDR6 etc.), and fast processors.

¹M. Brossard, A. Barrau and S. Bonnabel, “AI-IMU Dead-Reckoning,” in IEEE Transactions on Intelligent Vehicles, vol. 5, no. 4, pp. 585-595, Dec. 2020, doi: 10.1109/TIV.2020.2980758.

The post Beyond GPS: Exploring Positioning Technology through Artificial Intelligence first appeared on SK hynix Newsroom.

Image Download

Visual Evolution – from the Cambrian Era to Today

Close your eyes and imagine a world without eyesight.

Millions of years ago, that was the way of life on Earth. It was nearly 540 million years ago that animals first developed the ability to see – transforming everything about the ways in which they could avoid enemies, secure food, and evolve into various species.¹ It led to a geological event known as the Cambrian explosion, where the number of animal groups exploded from three to 38.

A similar visual innovation took place much more recently with the proliferation of smartphones in the late 2000s. Suddenly, people around the world were equipped with high-performance cameras that could fit into their pocket or the palm of their hand. Photography was no longer restricted to photographers and the easy transfer of visual information became widely available.

This year, with the outbreak of COVID-19, the transition to the contactless digital era accelerated once again as video conferencing and online classes became a part of our daily lives.

¹Andrew Parker, “In the Blink of an Eye: How Vision Sparked the Big Bang of Evolution”, (2003) (URL)

“Retina of Camera” CIS Technology Development, the Heyday of Smartphone Cameras

Figure 1. Structure of CMOS Image Sensor

Image Download

A camera is designed much like the human eye.

A smartphone camera is composed of various parts such as a lens, infrared cut-off filter², auto focusing actuator,³ and CMOS image sensor (CIS).⁴ Among them, CIS is a key component that acts as a retina of the human eye. As shown in Figure 1, it is composed of a photodiode that converts light into electrons, a color filter where only the light of a specific wavelength can pass through, an analog/digital circuit that converts electrons into digital signals, and an image signal processor (ISP) responsible for correction and image processing.

Since the resolution, sensitivity, and signal-to-noise ratio (SNR)⁵ are determined by the CIS performance, it can be said that the image quality of the smartphone camera is determined by the CIS. Today, the CIS image quality of smartphone cameras has surpassed the level of compact cameras, and the gap from DSLRs is continually being narrowed.⁶

In terms of performance, CIS has developed in the direction of reinforcing the number of pixels and its functions. Since more detailed and clearer image quality can be obtained with the increase in the number of pixels, the competition for the higher number of pixels started from the early stage of smartphone cameras. In addition, with innovations in semiconductor micro-processing such as back side illumination (BSI)⁷ and deep trench isolation (DTI)⁸, more pixels are integrated in the same size of area, making it possible to easily capture images of tens of millions of pixels even in a general smartphone.

²Infrared cut-off filter: A device that only passes the visible light and blocks the infrared wavelength
³Auto focusing actuator: A lens driving device for autofocusing, implemented with a small motor
⁴CIS: A device that can detect light and convert it into an electrical signal with a structure of ‘Complementary Metal Oxide Semiconductor (CMOS)’ composed of different MOS integrated circuits.
Acts as an electronic film in fliming electronic devices such as smartphones and cameras with high speed and low power consumption.
⁵SNR (Signal-to-Noise Ration): Defined as 20 log (signal/noise)
⁶David Cardinal, “Smartphones vs Cameras: Closing the gap on image quality”, DXO Mark. (2020) (URL)
⁷BSI (Back Side Illumination): A technology that increases the amount of light received on a photodiode by accepting light from the rear of the sensor. When light enters from the front of the sensor, light loss occurs due to scattering by metal wiring.
⁸DTI (Deep Trench Isolation): Process technology to make barriers between physical pixels between adjacent photodiodes inside silicon to prevent signal interference between pixels.

The Latest Technology Trend in CIS is about Function, not Pixel

Nevertheless, this trend for high pixels in CIS is expected to face technical difficulties soon, and the innovation for a high level of functions centered on the ISP will be in full swing.

This is due to the limits of miniaturization of CIS pixels due to the diffraction limit⁹. CIS is a complex component that combines optical technologies such as micro lenses and semiconductor technologies such as devices and circuits. It is possible to reduce the critical dimension of electric circuits to several nanometers with the current semiconductor technology; however, since the light reception amount decreases as the pixel size decreases, the sensitivity and the signal level is reduced, resulting in the decline in SNR and the image quality degradation.

⁹Diffraction limit: A limit point in which the distance between two objects is too close, making it difficult to distinguish with an optical lens.

Figure 2. Airy disk diffraction image

Image Download

The optical system of the camera also has a physical limit where the performance is limited by the diffraction effect,¹⁰ and even if one point light source¹¹ is taken, the image formed on the CIS through the lens is spread out, as seen in Figure 2. This is called an airy disk¹²; given the wavelength (λ), focal length (f), and lens diameter (d), the distance (x) that can separate the two points is determined by the following formula:

¹⁰Diffraction effect: A phenomenon in which light proceeds in a curved path rather than a straight path when it encounters an obstacle.
¹¹Point light source: A light source that is small enough to be considered as a dot.
¹²Airy disk: A phenomenon that the image is spread out when one point light source is condensed on the CIS through the lens.

For example, for a 400nm blue point light source, even if a high-performance lens with an F (=f/d) number of 1.4 is used, the distance that can separate the two points is 0.68μm. In other words, to distinguish the two blue point light sources, the distance should be at least 0.68μm. Therefore, even if the size of CIS pixels is made smaller than this, it is difficult to expect substantial improvement in resolution. Since the size of the commercially available CIS pixel has already reached 0.7 to 0.8μm, it is necessary to develop a new optical technology to reduce the F number or a new application that can merge several fine pixels.

Another reason for this innovation is the emergence of stack sensor technology. Since the conventional sensor has a structure where pixels and circuits are implemented on the same substrate, it was essential to reduce the light-free area for the CIS size reduction. Therefore, only essential functions of analog/digital circuits were implemented and adding circuits for additional functions was very limited.

Figure 3. Left: Conventional sensor structure / Right: Stack sensor structure (right)

Image Download

On the contrary, the stack sensor has a structure where pixels and circuits are implemented on separate substrates as shown in Figure 3, and then the two substrates are connected electrically by Through Silicon Via (TSV)¹³ or hybrid bonding technology.¹⁴ Since pixels and circuits are stacked together, the circuits on the lower substrate can be used as much as the area occupied by the pixels on the upper substrate, leading that area free to use.

¹³TSV (Through Sillicon Via): An interconnecting technology that delivers electric signals through column-shaped paths that penetrate the entire silicon wafer thickness.
¹⁴Hybrid bonding technology: A process technology that bonds metal electrodes of two wafers together and connects them, instead of an electrode penetrating through a silicon wafer. It can reduce size while increasing performance.

For example, a 48 megapixel (8,000 x 6,000) sensor implemented with 1μm occupies an area of 48mm² or more on the upper substrate. If this size of area on the lower substrate can be used for the implementation of digital logic, it is possible to utilize a large space, enough to integrate a large number of high-performance microprocessors.¹⁵

In addition, the stack sensor has the advantage that an independent process can be applied to the pixels on the upper substrate and the circuits on the lower substrate. If an advanced logic process¹⁶ is applied to the lower substrate for circuits, even a complicated ISP algorithm can be implemented with low power, high density digital circuits. In other words, while the ISP of the conventional sensor only supported simple functions such as lens correction and defect correction due to the limitation of the circuit area, the ISP of the stack sensor can implement innovative algorithms such as image processing, computer vision, and artificial intelligence (AI) by using an advanced logic process.

¹⁵Microprocessor: A device that integrates the control functions of the processing unit and central processing unit into one chip
¹⁶Logic process: Semiconductor process that manufactures digital devices to process logical operations such as AND, OR, NOT

SK hynix’s CIS with Various Functions

Currently, SK hynix’s CIS has built-in image processing functions such as phase detection auto focus (PDAF), Quad pixel processing, and high dynamic range (HDR) processing, and new functions are constantly being added to it.

Figure 4. Left: Half Shield PDAF structure / Right: Paired PDAF structure

Image Download

PDAF is a function that applies the principle that human’s both eyes use to predict the distance to a subject. That is, for some pixels of the CIS, this method generates a phase difference¹⁷ by covering the left and right, as shown in the left side of Figure 4, or by placing the left and right pixels under one large micro lens as shown in the right side of Figure 4. Through this method, the ISP algorithm calculates the phase difference from the left and right images and converts it into a distance to the subject to focus quickly and accurately.

¹⁷Phase difference : The phase difference between two vibrations (wavelengths). The phase refers to the relative position of the vibration(wavelength) at a certain point.

Figure 5. Left: Output of a conventional sensor / Right: Output of a Quad sensor

Image Download

The Quad sensor has the function of placing four color filters of the same color adjacent to each other and processing them together. In dark places, four pixels are combined and processed to receive more light and in bright places, the individual pixels are separately processed by the ISP algorithm to improve resolution. Figure 5 shows two images taken with SK hynix’s 48 megapixel Quad sensor and a conventional sensor, respectively. This shows that it is possible to obtain a bright image without noise even in a dark place when using a Quad sensor, compared to when using a conventional sensor.

Figure 6. Left: Output of a conventional sensor / Right: Output of an HDR sensor

Image Download

The HDR sensor supports a function that makes a clear contrast between the bright and dark parts of an image by synthesizing multiple pixels with different sensitivity and exposure time. In particular, with the SK hynix’s CIS, image processing is performed by a built-in ISP, resulting in real-time processing and clear image quality even with a moving object. Figure 6 shows the output images with and without HDR application to the SK hynix’s CIS. Compared to the image where HDR is not used, the image using HDR restores the background clearly while maintaining the same brightness of the whole image.

Currently, SK hynix’s CIS, mainly the Black Pearl product line, is widely used in smartphone cameras and the application field is expected to expand to various fields such as bio, security, and autonomous vehicles.

Figure 7. Camera mounting location in self-driving car

Image Download

In particular, autonomous vehicles use at least ten cameras to detect their surroundings.¹⁸ To improve accuracy, various requirements such as high resolution support for distinguishing distance objects, HDR support for recognizing objects even in dark environments, and pre-processing of the ISP to reduce the computational amount of the processor must be satisfied.

In the security field, a function to compress and encrypt image signals in the CIS built-in ISP and to transmit them to an external processor is required. If the unencrypted image signal is transmitted to the outside as it is, the possibility of security vulnerability and information leakage increases. For this reason, the encryption function inside the CIS is essential.

¹⁸Peter Brown, “Breaking Down the Sensors Used in Self-Driving Cars”, Electronics 360. (2018) (URL)

The future of CIS: Information Sensor that supports advanced functions

In the future, CIS is expected to evolve into an information sensor that supports advanced additional functions, without being limited to image quality improvement.¹⁹ SK hynix’s stack sensor is already capable of embedding a simple AI hardware engine inside the ISP on the lower substrate, based on the advanced semiconductor process. Based on this, SK hynix is currently developing new machine learning-based technologies such as super resolution, color restoration, face recognition, and object recognition.

¹⁹Sungjoo Hong, “Smart Cloud and Information Sensor”, Smart Cloud Show. (2018) (URL)

Since it is possible to extract and classify various features from input images when using machine learning-based ISP technology, CIS will become a key component of information sensors that collect various information such as image information, location information, distance information, and biometric information.

In particular, to utilize CIS as an information sensor, it is necessary to take a new approach from a different perspective. This is because the quality goal of CIS is now “achieving the image quality optimized for machine algorithms”, while the goal was previously “achieving the image quality optimized for human eyes” so far.

Figure 7. Left: Less noise but misrecognized as a spotlight
Right: More noise but accurately recognized as an espresso

Image Download

According to the results of a research of Stanford University, even if a good-looking image to the human eye is created through complex image processing, it does not always mean that this image produces excellent results when computer vision algorithms are applied to it. For example, when an object recognition algorithm is applied to the coffee cup image in Figure 7, the image on the left with less noise is incorrectly recognized as a spotlight, but the image on the right with much more noise is accurately recognized as an espresso. This shows that the key function for the future CIS is to provide image quality optimized for the computer algorithm to be used.²⁰

²⁰Diamond, V. Sitzmann, S. Boyd, G. Wetzstein, F. Heide ‘Dirty Pixels: Optimizing Image Classification Architectures for Raw Sensor Data’, arXiv preprint arXiv:1701.06487.

As explained in this article, SK hynix is increasing the level of integration of the CIS pixels through the continuous development of device and process technologies and supporting various application fields through the ISP technology development. Especially, to pioneer new technology fields, it has established and is operating overseas research institutes in Japan and the United States, and various researches are actively being conducted with domestic and foreign universities through the academic-industrial collaboration. In the future, SK hynix’s CIS is expected to be utilized in various application fields including smartphone cameras to contribute to the creation of economic and social value and to grow as a key component of information sensors in the future.

The post The Visual Evolution & Innovation of Image Sensors first appeared on SK hynix Newsroom.