Renesas Solutions for HMI Applications―Part 4: Low-Cost Solutions for Gesture Recognition

Solutions: 15 of 20

Renesas Solutions for HMI applications―Part 4: Low-Cost Solutions for Gesture Recognition

New low-cost system-design components can be used to overcome the limitations of traditional motion sensors, allowing system developers to create human-machine interfaces that accommodate a wide range of non-contact dynamic control gestures. This fourth article in our series on Renesas products for HMI applications highlights an advanced design solution that detects eight directions of palm motion. This solution enables exciting, intuitive and user-friendly methods for operating embedded systems.

Controlling Devices with Simple Hand and Body Movements

Some embedded systems run autonomously, but most perform functions in response to user inputs. Many machines and devices are especially convenient because their electronic circuits sense and respond to human approach. Automated entry/exit doors, for example, open by themselves when people walk up to them. Such basic non-contact human-machine interfaces (HMIs) have significant limitations, however. They detect only the presence of a body or hand, so the electronics occasionally respond inappropriately. For instance, a door might open unexpectedly when a pedestrian walking by pauses momentarily near the entrance.

Better, more predictable non-contact control is achieved when the HMI circuitry gains the ability to recognize defined sets of simple hand gestures. The easy-to-use gestures make erroneous system operations much less likely. In a typical application, for example, a person standing in front of a door could just hold out a hand and move it from left to right to open the door, or move it from right to left to close it (see Figure 1a).

Figure 1: Typical Gesture-Recognition HMI Applications

Figure 1: Typical Gesture-Recognition HMI Applications

Eliminating the Need to Make Contact with System Controls

A huge market is emerging for installing gesture-recognition HMIs in devices that currently require people to physically touch display panels or manipulate switches. These familiar controls can become contaminated from repeated use. Over time, oils and dirt accumulate, often causing operations to become unreliable. Periodic inspections and cleanings are required to correct this problem, at considerable cost.

Non-contact control approaches can be implemented that allow users to employ gestures to perform ON/OFF, Open/Close, Page Turning and other system operations. Advanced gesture-based HMIs can simplify system management and maintenance tasks. Moreover, by eliminating switches or panels on products, they facilitate simplified, more-modern exterior designs (see Figure 1b).

In business and industrial applications, gesture-control type HMIs can boost productivity. This is especially true in clean facilities such as food processing plants in which operators must wash or sterilize their hands before touching a machine's control panel. Gesture-based HMIs eliminate those touches, saving the time and effort of repeated hand washings, while also helping to minimize contamination risks (see Figure 1c).

Searching for Design Solutions for Gesture-based HMIs

For the reasons previously stated and others, device and equipment manufacturers increasingly want to incorporate gesture-recognition technology into new embedded system products. System engineers are looking for reliable and economical design solutions that deliver high sensitivity and are easy to apply.

Renesas expects that gesture-responding HMIs soon will be featured in a wide variety of products: kitchen appliances, household systems, medical and sanitary equipment and many others. With simple hand motions, users will be able make those products perform tasks without touching them.

Overcoming the Limitations of Traditional Gesture-Detector Designs

Gesture-recognition functions are not new. Traditionally, they have been built either by using multiple infrared (IR) sensors, or by combining image sensors with image-processing engines.

IR sensors are a low-cost way to detect movement along a single predetermined dimension, and they resist interference from external light. However, they're effective only over small distances, about 10 cm.

By contrast, image sensors teamed with signal processors achieve much more sophisticated recognition, but they're quite expensive. Also, they aren't suitable for locations where backlighting obscures the view and degrades motion-detection precision.

Implementing Effective Gesture-Recognition User Interfaces

Renesas has responded to this challenge by successfully developing an extremely low-cost highly sensitive implementation that detects eight directions of palm motion (vertical, horizontal, and diagonals) up to 1 meter, and that can detect body gestures at distances up to 10 meters.

The performance of our new HMI solution far exceeds that of most of the contemporary gesture-recognition systems installed on smartphones, which require a hand to be very close to the phone's screen.

Thus, the new Renesas solution is a very timely technology advance. Its functionality makes it possible to build attractive products that people can control merely by making hand motions at reasonable distances from the detection circuits. This capability is an enormous system design advantage—one that opens up enormous application opportunities for non-contact interfacing.

Applying a New Type of Very-Sensitive, Fast-Response IR Sensor

The sensing board holds four pyroelectric infrared sensors, capable of detecting infrared radiation (heat) generated by living bodies (See Figure 2)

Pyroelectric sensors are widely used for motion detection, and have proven themselves over many years. But typical implementations are generally used just to detect whether a person is present or not, since conventional pyroelectric sensors respond too slowly to detect rapid motion of small areas—and are therefore unsuitable for detection of palm movements. Because these sensors do not record pictures or otherwise invade privacy, they are widely accepted even in areas where privacy is an issue.

The Renesas solution eliminates the speed and distance limitations of conventional pyroelectric IR sensors by employing the new organic ferromagnetic film types developed by Sensors and Works, Ltd. These advanced pyroelectric devices operate at speeds over 10 times faster and work at distances up to about 1 meter. They have lower noise, too, allowing a greater dynamic range. Additionally, they are versatile and can be installed in a wide range of locations because they're inherently resistant to interference caused by external light sources.

Figure 3 compares the superlative response characteristic of the organic ferromagnetic film sensors to the performance of conventional pyroelectric IR sensors. The left end of the blue line shows that conventional types offer high sensitivity when they have a relatively prolonged exposure to a heat source such as the palm of a hand. However, the right end of that line reveals the sharp drop in sensitivity responsible for their poor dynamic performance.

The top red line in Figure 3's graph demonstrates that the new pyroelectric sensors from Sensors and Works, Ltd. maintain high sensitivity even when the exposure time becomes very short. This enables them to detect rapid hand movements.

Figure 2: Demonstration Board for Gesture-Recognition User Interfaces

Figure 2: Demonstration Board for Gesture-Recognition User Interfaces

Figure 3: Comparison of Pyroelectric IR Sensors

Figure 3: Comparison of Pyroelectric IR Sensors

Reducing the HMI Solution's Parts Count and Cost

As Figure 4 shows, the Renesas gesture-recognition circuitry feeds output signals from four organic ferromagnetic film IR sensors into a Renesas Smart Analog IC300 (RAA730300). This chip's analog front-end circuits normalize the sensor's response characteristics and then send amplified signals containing motion-detection information to an RL78/G1A microcontroller.

Using the Smart Analog IC as a bridge device to condition the sensor signals for the MCU significantly reduces the number of external passive components that would otherwise be required, lowering the cost of our gesture-recognition solution. Additionally, the Smart Analog IC eliminates the need for running digital filter software in the MCU, thereby reducing its CPU load.

The RL78/G1A-based board can drive LED and LCD displays that let system engineers watch sensor outputs in real time. This test capability expedites application development, eliminating much of the work necessary for monitoring how well the sensors track signal sources.

Figure 4: Block Diagram of Renesas Demo Gesture-Recognition Board

Figure 4: Block Diagram of Renesas Demo Gesture-Recognition Board

Running Powerful, Flexible Gesture-Recognition Software

It should be noted here that the IR sensors, by themselves, do not detect motion. Rather, each sensor has only two possible outputs: heat source detected or not detected. Consequently, the MCU has to process the incoming signals from all four sensors to determine the direction of a detected hand gesture.

Application software installed in the RL78/G1A MCU contains the algorithms essential for this processing. As previously mentioned, it distinguishes eight different palm movements that are applied in horizontal, vertical and diagonal directions (see Figure 5). Each of these gestures can be assigned to distinct system control functions, according to the embedded system's requirements.

Although the actual gesture-detection processing is quite complicated, the Renesas solution hides this complexity from system developers to help minimize their design workload. Consider, for example, that when a user executes a move-hand-right-to-left gesture, he or she will almost certainly bring their hand back to its original position after making the intended control gesture. The motion-detection software must not interpret this return motion as another gesture; otherwise, an erroneous operation might be initiated. The MCU's built-in software avoids this potential problem by recognizing and processing only the intended initial action, while effectively ignoring the return action.

Figure 5: Recognition of Eight Gesture Directions

Figure 5: Recognition of Eight Gesture Directions

The on-chip software provides fine-tuning adjustments for handling different environments and operator characteristics. Average gesture speeds might vary from one installation to another, for instance; so appropriate parameters can be modified to obtain the best HMI results. Detection sensitivity parameters can be changed, as well. User-interface performance can be optimized for an operator's hand temperature, the ambient temperature, and other environmental factors.

The parameter adjustment features in the MCU's software allow system engineers to maximize product performance without having to analyze the details of gesture recognition.

Viewing the New Renesas HMI Solution in Action

Having developed low-cost solutions for gesture-recognition HMIs that are effective and easily incorporated into end products, Renesas anticipates that customers will quickly apply them to create products offering innovative non-contact human-machine interfaces.

Important insights into our advanced HMI technology can be obtained by watching the demonstration video below. It provides a first-hand look at the superlative response characteristics of the Renesas solution, while also hinting at its application potential.

Play Video

As the previous discussion has emphasized, gesture-recognition HMIs can add compelling features and benefits to embedded system devices, systems and equipment. We urge you to contact us and take full advantage of this powerful, yet affordable, technology for creating improved user interfaces.