Renesas Solutions for HMI Applications―Part 2: Applying CMOS Cameras In Embedded Devices

Solutions: 12 of 20

Putting Eyes In Machines; Applying CMOS Cameras In Embedded Devices

This second installment in our series on Renesas HMI solutions examines the growing use of CMOS (Complementary Metal-Oxide Semiconductor) cameras within embedded devices. The discussion describes application requirements and solution alternatives, paying particular attention to motion-detection applications.

CMOS Cameras Are Becoming Essential Components of Embedded Devices

CMOS camera modules*1 are becoming increasingly common in embedded system applications as production increases and competitive pressures cause their cost to fall. Manufacturing volumes are growing as sales of security cameras rise, boosted further by energy-saving camera applications in TVs and other household appliances, as well as by enhanced HMIs (Human-Machine Interfaces) in many kinds of business and industrial equipment.

This digital-photography based electronics trend gives electronics manufacturers major rewards for successful efforts in creative problem solving. System engineering teams seeking new ways to enhance product features and performance find that incorporating camera capabilities into their designs can add considerable value to many embedded devices.

Especially, CMOS camera modules can implement exciting and important gains in HMI performance and capabilities for a range of rapidly expanding applications (see Figure 1). This is especially true for electronic products that apply motion-sensing technology.

Figure 1: Rapidly Growing CMOS-Camera Applications for HMIs

Figure 1: Rapidly Growing CMOS-Camera Applications for HMIs

Examples of the CMOS camera-based embedded applications now becoming commonplace put this system design trend into perspective. Camera modules installed in air-conditioning systems save money by enabling more energy-efficient operations. Modules within refrigerators allow internal conditions to be monitored from smart phones, thereby obtaining energy savings while maximizing the preservation of stored food. Cameras incorporated into vending machines gather detailed consumer usage data, allowing stocks of dispensed goods to be better managed as demand changes day and night.

In security equipment and systems—currently the biggest market for digital camera technology—camera modules are proving to be highly effective in helping to increase safety and reduce theft and fraud. Finally, many businesses are using CMOS cameras to quickly read QR codes and acquire other product-management information, thus getting the timely data essential for improving operational efficiency.

Camera Modules Are Worthwhile Upgrades from IR Sensors

Clearly, the design trend of adding camera capabilities to embedded systems is a broad, far-reaching and rapidly evolving one. It is not, however, absent of alternatives suitable for many situations.

Specifically, infrared (IR) sensors are cheaper than cameras, and they are just as effective for the basic function of detecting human presence. They have significant performance limitations, though, in many situations. IR sensors become totally inadequate as system performance requirements increase and other sensing functions come into demand.

Importantly, the cost of upgrading embedded systems from IR sensors to CMOS camera modules is often mitigated by several factors: the modules’ capacity to generate more and better data; their strengths in implementing enhanced HMI features; and their ability to eliminate the need for additional sensors, thereby helping to bring down overall material costs.

Around the world, embedded systems implementing the “Internet of Things" are producing numerous benefits—some unexpected and all welcomed. As these electronic devices become more pervasive in everyday life, Renesas expects demand for CMOS camera modules to soar.

Technical Obstacles to Camera Module Deployment Can Be Eliminated

In a typical embedded CMOS camera installation, a microcontroller (MCU) or microprocessor (MPU) receives signals from the camera module, and then displays and otherwise processes the resulting image according to the application’s specific requirements. Creating successful implementations of this process isn’t a trivial design task.

Because this HMI area is expanding, in many cases the system engineers working on such projects lack prior experience with digital cameras. Among the design issues that concern them are the following: Are high-end MCU/MPUs required? How much software must be developed? Does the system power circuitry have to be redesigned? Etc. Even engineers who have prior experience with embedded cameras often are looking for easier ways to implement the technology, yet are unsure how to proceed.

Technical assistance from application experts is invaluable in transforming HMI concepts into successful realities. Fortunately, such assistance is readily available.

The remainder of this article presents a broad introduction to development solutions for deploying CMOS camera modules in a wide span of embedded system products. This information will be helpful both to system engineers working on such projects for the first time, as well as to those seeking updated information on superior design approaches.

To supplement the technical content in this story and the data on our website, Renesas gives customers access to the massive expertise of a global staff of experienced application engineers. These professional problem solvers can review project requirements and recommend the MCU/MPUs and evaluation environments best suited for fulfilling specific technical goals and objectives.

RX and RZ/A1 Address Different Application Requirements

To meet the very diverse span of customers’ system specifications, Renesas provides embedded CMOS camera modules optimized for different frame rates and image resolutions. Our system solutions for less-intensive applications feature the easy-to-use mid-range MCUs in our RX Family: RX631 or RX64M devices. However, our system solutions for applications that must deliver higher-performance video capabilities apply advanced MPUs in our RZ Family—specifically, devices in the RZ/A1 Series, chips built around a CortexTM-A9 Arm® CPU (see Figure 2).

Figure 2 illustrates the relationship between functional requirements (horizontal axis) and clock-speed requirements (vertical axis), showing how they rise together. That is, to increase input camera resolution (the number of pixels) and frame rates, faster processors must be used.

Figure 2: Camera Resolution vs. Core Performance, and supporting MCU/MPU types

Figure 2 illustrates the relationship between functional requirements (horizontal axis) and clock-speed requirements (vertical axis), showing how they rise together. That is, to increase input camera resolution (the number of pixels) and frame rates, faster processors must be used.

RX MCUs operate at speeds up to 120 MHz. They are ideal for implementing functions such as image capture (for motion detection, security monitoring, etc.). RX MCUs are also recommended for 2D barcode scanning and for relatively simple character recognition.

The higher-end CPUs in the RZ/A1 Series run at speeds up to 400 MHz. They readily handle the vast numbers of computations needed for more complicated facial- and character-recognition applications, as well as the volumes of calculations required for gesture reading and other sophisticated motion-detection type applications.

Figure 2 also highlights the fact that as input camera resolutions (X axis) go up, greater CPU throughput (achieved via a wider bus width) becomes necessary. For example, a mid-range RX MCU would be sufficient for a 10-capture/second VGA vehicle daytime driving recorder. An RZ MPU might be necessary, though, for a nighttime driving recorder that takes higher-resolution pictures. An RZ MPU might also be needed for embedded system products that have to detect fast-moving objects, because the electronics must be able to process images at fast frame rates.

Four Techniques Are Useful for Detecting and Classifying Motion

Motion-detection applications are a prime market for CMOS camera modules. They’re also an area in which customer expectations run particularly high. Often the system design specifications mandate that the electronics detect not just the presence of motion, but also the type of the movement.

Motion-detection is typically achieved using one of four algorithmic approaches: temporal difference detection, background subtraction, template matching, and optical flow detection. These four different algorithms are illustrated in Figure 3 and described below.

Figure 3: Motion Detection Algorithmes

Figure 3: Motion Detection Algorithmes

  • Temporal Difference Detection: Motion is discovered by analyzing the differences in successive images.
  • Background Subtraction: Motion is detected by analyzing the difference between a preset background image and the incoming camera images.
  • Template Matching: To recognize motion, the electronics searches for the presence of the template image within the overall image.
  • Optical Flow: The motion-detection circuitry breaks each image up into smaller blocks, and then makes comparisons with successive images to detect motion and subsequently to calculate vectors.

The first three algorithms described above—Temporal Difference Detection, Background Subtraction and Template Matching—utilize relatively simple mathematical computations. Thus they incur relatively low processing loads. However, they don’t handle brightness fluctuations and other types of noise very well, nor are they very good at detecting fast-moving objects.

The Optical Flow detection approach delivers better performance, but it places a relatively high processing load on the CPU.

RX MCUs Are the Best Choices for Basic Motion Detection and Security Monitoring Applications

Different motion-detection approaches require different MCU or MPU characteristics. This point is illustrated by examining how our application engineers determine which processors are the best design choices for different embedded camera modules.

Some Renesas video solutions aim at applications that have relatively low computation requirements, such as security monitoring and human-presence sensing devices that use temporal difference detection or template matching to carry out motion-detection tasks. The device choice typically depends on the required frame-rate capability: If 1 to 4 fps image capture is sufficient, they apply an RX631 MCU; if 4 to 10 fps is required, they use an RX64M MCU.

To help customers’ engineering teams test and analyze different camera modules for their embedded designs, Renesas offers evaluation boards as part of our system support program. These development tools contain a CPU board and a partner-provided CMOS sensor module, as well as various peripheral-function drivers. The boards are easy to use and provide a proven, timesaving starting point for system development.

For perspective, please view the demonstration video below, which shows the capabilities of a board with an RX631 MCU. This demonstration reveals that our global technology partner community provides middleware for correcting for detection errors and imaging distortions.

Video: Demo system with RX631 MCU that detects the presence of people within images obtained from a CMOS camera module. A red frame appears on the screen whenever the electronics ‘sees’ a person or persons in the image.

System Information

RX631 Group

Evaluation Environment: Please contact us.

RX64M Group

Evaluation Environment: Coming soon.

RZ/A1 MPUs Excel for Sophisticated Facial and Gesture Recognition

MPUs in Renesas’ RZ/A1 Series have powerful processing capabilities and large-capacity internal RAM. These advanced processors are fully capable of running facial- and gesture-recognition applications based on optical-flow motion-detection algorithms.

Because RZ/A1 MPUs incorporate on-chip, hardware-driven JPEG acceleration and image correction functions, they achieve real-time processing throughputs that outperform typical middleware implementations. The devices’ internal architecture uses multiple buses to connect the CPU and RAM. This chip-design feature enables high throughput and ensures that our camera modules produce continuous frames; i.e., image streams with no dropouts.

An RZ/A1 demonstration kit is currently being prepared. A demo video will be released in the near future.

Modular Camera Solutions Cut Engineering Risk and Facilitate a Faster Time-to-Market

Electronics manufacturers in many countries and industries are showing great interest in motion-detection applications. Renesas is striving to facilitate the industry’s growth and successes in this market area.

Renesas has a long track record of providing customers around the globe with comprehensive system development support, from initial conceptualization through hardware and software design. Today we collaborate with a vast global community of technology partners to offer assistance at every step of the way as system designs move from initial specification to prototype and right through to final mass production. We can even help arrange the outsourcing of development work to application experts, if such assistance is requested.

Contact Us

The next installment of this EDGE magazine series will look at Renesas solutions for implementing touch processing in HMIs for household appliances and other embedded system products. Watch for it.

*1 A CMOS camera module consists of a lens, an image sensor, and ISP (image signal processing) circuitry. The lens focuses the image on the sensor, which converts this data into an electrical signal. The ISP circuitry then implements appropriate optical correction to derive the final electronic image.