Moving Object Tracking Demonstration

Visual servo realized with RZ/A2 micro computer adopting DRP

Related Information

  We demonstrate Vision and Dynamics integration.
  Only two components Renesas' micro computer RZ/A2M and small embedded camera module control a robot arm, track a moving object at real time and grasp it.
  Targeting a destination by image processing eliminates a complex robot teaching*.
 *Teaching : A task of programing a robot trajectory in advance

Visual servo
  is a robot control method integrated camera image information in a robot control loop. The approach addresses changing environments robots need in the near future. On the other hand, since visual servo requires to coexist rapid image processing and real time servo control, it was difficult to realize. Renesas' dynamically reconfigurable processor DRP built into microprocessor RZ/A2M is responsible for image processing of successive frames and the CPU on the RZ/A2M controls each servos of a robot arm, which realizes visual servo control by one chip.     

Image processing by DRP
  There are two modes in image processing "moving object detection mode" and "color tracking mode". In moving object detection mode, the color information of the moving object is extracted based on difference information of successive image frames. In color tracking mode, the center position of moving object and its angle are taken based on color information extracted in moving object detection mode. Image processing by DRP computes the center position of the moving object and its angle at 2.5ms which is 4 to 10 times faster than the processing by CPU.   

 Pre processing
 ・Translation and resizing from Bayer image(640x480) taken by camera to RGB(160x120)
 ・Shading collection and white balance collection

 Moving object detection mode
 ・Generate the image of a target moving object extracted based on difference between frames.
 ・The CPU extracts the color information of the target.

 Color tracking mode
 ・DRP separates the target and others from the image based on the color information extracted in "Moving object detection mode".  
 ・The CPU calculates the center position of the target and its angle from the separated image.

Servo control by CPU
  The CPU controls robot arm based on the center position of the target and its angle calculated in the image process. In order to suppress noise of robot arm move, set the midpoint of the target position by PID. The target angles of each servos to reach the midpoint are calculated on inverse kinematics. Then issues commands to set the target angles to the servos of the robot arm through the serial connection.  
High energy efficiency
 Thanks to DRP's high energy efficiency, Successive processing of visual servo can be processed with super low power consumption at 0.6 mW -(at RZ/A2M device)

 ・RZ/A2M board ・・・RZ/A2M EVA-Lite (SHIMAFUJI)
 ・2D camera module・・・Raspberry Pi camera module
 ・Distance sensor・・・ GP2Y0A60SZ IR distance sensor (SHARP)