Visual perception system and method for a humanoid robot

• US 8,244,402 B2
• Filed: 09/22/2009
• Issued: 08/14/2012
• Est. Priority Date: 09/22/2009
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company), United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Inventors: Wells, James W., Mc Kay, Neil David, Chelian, Suhas E., Linn, Douglas Martin, Wampler, Charles W. II, Bridgwater, Lyndon
• Primary Examiner(s): Trammell, James P
• Assistant Examiner(s): Marc, Mcdieunel
• Application Number:
  US12/564,074
• Publication Number:
  US 20110071675A1
• Time in Patent Office: 1,057 Days
• Field of Search: 701/4, 701/9, 701/14, 701/200, 701/207, 701/208, 701/210, 701/211, 701/213, 701/216, 701/217, 701/214, 701/301, 340/970, 340/971, 340/979, 340/980, 340/988, 340/990, 340/995.18, 342/33, 342/352, 342/357.31, 342/457, 342/357.44, 342/357.52, 244/3.2, 244/3.15, 731/78.T, 731/78.R, 707/999.104, 707/E17.018, 600/424, 600/38, 700/245, 700/246, 700/247, 700/250, 700/252, 700/257, 700/258, 700/900, 446/95, 446/284, 446/327, 446/465, 446/473, 446/487, 318/568.11, 318/568.13, 901/1, 901/30, 706/44, 706/45, 414/402, 379/419
• US Class Current: 700/250
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A robotic system includes a humanoid robot with robotic joints each moveable using an actuator(s), and a distributed controller for controlling the movement of each of the robotic joints. The controller includes a visual perception module (VPM) for visually identifying and tracking an object in the field of view of the robot under threshold lighting conditions. The VPM includes optical devices for collecting an image of the object, a positional extraction device, and a host machine having an algorithm for processing the image and positional information. The algorithm visually identifies and tracks the object, and automatically adapts an exposure time of the optical devices to prevent feature data loss of the image under the threshold lighting conditions. A method of identifying and tracking the object includes collecting the image, extracting positional information of the object, and automatically adapting the exposure time to thereby prevent feature data loss of the image.

Claims

1. A robotic system comprising:

a humanoid robot having a plurality of robotic joints each being moveable using at least one actuator; and

a distributed controller configured to control the movement of each of the robotic joints, wherein the controller includes a visual perception module (VPM) that is configured to visually identify and track an object in the field of view of the robot under predetermined threshold ambient lighting conditions;

wherein the VPM includes;

a plurality of optical devices each configured to collect an image of the object;

at least one of a laser device and a range imaging device configured to detect positional information of the object; and

a host machine configured to;

process the image and the positional information to thereby visually identify and track the object;

automatically adapt an exposure time of the optical devices to prevent feature data loss of the image under the threshold lighting conditions; and

discriminate either of a black-on-black or a white-on-white feature of interest on the object by;

automatically evaluating a small area of interest in the image;

using a histogram to determine the extent of the range of illumination of the area of interest; and

processing the range of illumination to thereby identify the features of interest.

2. The robotic system of claim 1, wherein the humanoid robot includes a moveable head encapsulating the optical devices, the laser device, and the range imaging device, further comprising a gaze control unit configured to control the moveable head through at least four axes of movement.

3. The robotic system of claim 1, wherein the optical devices include at least a pair of identically-configured high-resolution cameras configured to capture the image with at least approximately 5 megapixels.

4. The robotic system of claim 1, wherein the range imaging device is a short-range infrared (IR) time-of-flight (TOF) device.

5. The robotic system of claim 1, wherein the VPM is configured to train the humanoid robot with respect to a new automated task by showing the robot at least one of a work sequence and a work pose within a designated work envelope.

6. The robotic system of claim 1, wherein the plurality of optical devices includes at least additional camera, wherein the VPM is configured to integrate different views of the object using information from the at least one additional camera.

7. A visual perception module (VPM) configured to visually identify and track an object in the field of view of a humanoid robot under predetermined threshold ambient lighting conditions, the VPM comprising:

a plurality of optical devices configured to collect an image of the object;

a positional extraction device including at least one of a laser device and a range imaging device, wherein the positional extraction device is configured to extract positional data of the object; and

a host machine configured to;

process the image and positional data to thereby automatically adapt an exposure time of the optical devices, and to thereby prevent feature data loss of a captured image of an object under the threshold ambient lighting conditions; and

discriminate either of a black-on-black or a white-on-white feature of interest on the object by automatically evaluating a small area of interest in the image, using a histogram to determine the extent of the range of illumination of the area of interest, and then processing the range of illumination using the host machine to thereby identify the features of interest.

8. The VPM of claim 7, wherein the humanoid robot includes a moveable head encapsulating the plurality of optical devices and the positional extraction device.

9. The VPM of claim 7, wherein the optical devices include at least a pair of identically-configured high-resolution cameras configured to capture the image with at least approximately 5 megapixels.

10. The VPM of claim 7, wherein the positional extraction device is a short-range infrared (IR) time-of-flight (TOF) device.

11. The VPM of claim 7, wherein the VPM is configured to train the humanoid robot with respect to a new automated task by showing the robot at least one of a work sequence and a work pose within a designated work envelope.

12. The VPM of claim 7, wherein the plurality of optical devices includes at least additional camera, wherein the VPM is configured to integrate different views of the object as the object is located off the robot using information from the at least one additional camera, and from a perspective of at least one of a palm, a forearm, and a chest area of the robot.

13. A method of identifying and tracking an object in the field of view of a humanoid robot under threshold lighting conditions, the method comprising:

using a plurality of optical devices to collect an image of the object;

using at least one of a laser device and a range imaging device to extract positional information of the object; and

processing the image and positional information using a host machine of a visual perception module (VPM), including discriminating either a black-on-black or a white-on-white feature of interest on the object by;

automatically evaluating a small area of interest in the image;

using a histogram to determine the extent of the range of illumination of the area of interest; and

processing the range of illumination using the host machine to thereby identify the features of interest;

wherein the host machine is configured to automatically adapt an exposure time of the optical devices to thereby prevent feature data loss of the image under the threshold lighting conditions.

14. The method of claim 13, further comprising training the humanoid robot with respect to a new automated task by showing the robot at least one of a work sequence and a work pose within a designated work envelope.

15. The method of claim 13, wherein the humanoid robot includes a plurality of compliant joints each having a measurable joint angle, the method further comprising:

positioning the object in a grasp and field of view of the humanoid robot;

moving the humanoid robot and object through a series of poses;

using the host machine to track the object as it moves through the series of poses, including measuring each of the joint angles; and

recording the joint angles using the host machine; and

using the recorded joint angles to thereby calibrate eye-hand coordination of the humanoid robot.