Mobile human interface robot

• US 9,400,503 B2
• Filed: 02/22/2011
• Issued: 07/26/2016
• Est. Priority Date: 05/20/2010
• Status: Expired due to Fees

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Kearns, Justin H., Taka, Orjeta, Pack, Robert Todd, Rosenstein, Michael T., Halloran, Michael, Farlow, Timothy S., Shamlian, Steven V., Won, Chikyung, Chiappetta, Mark, Vicenti, Jasper Fourways
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): HUANG, FRANK F
• Application Number:
  US13/032,228
• Publication Number:
  US 20120182392A1
• Time in Patent Office: 1,981 Days
• Field of Search: 348/46
• US Class Current: 1/1
• CPC Class Codes: B25J 11/009 Nursing, e.g. carrying sick persons, pushing wheelchairs, distributing drugs
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G02B 13/22 Telecentric objectives or lens systems
• CPC Class Codes: G05D 1/0227 using mechanical sensing means, e.g. for sensing treated area
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0242 using non-visible light signals, e.g. IR or UV signals
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

A method of object detection for a mobile robot includes emitting a speckle pattern of light onto a scene about the robot while maneuvering the robot across a work surface, receiving reflections of the emitted speckle pattern off surfaces of a target object in the scene, determining a distance of each reflecting surface of the target object, constructing a three-dimensional depth map of the target object, and classifying the target object.

Claims

1. A method of object detection for a mobile robot, the method comprising:

emitting a speckle pattern of light onto a scene about the robot while maneuvering the robot across a work surface of a working area;

receiving odometry measurements from a drive system of the robot while maneuvering the robot across the work surface;

determining a first motion of the robot based on the odometry measurements;

determining a current location of the robot in the working area based on the first robot motion and a previously determined location of the robot;

receiving reflections of the emitted speckle pattern off surfaces of a target object in the scene;

determining a second motion of the robot using visual odometry of the robot based on an optical flow of the received reflections;

determining a robot motion error between the first robot motion based on the drive system odometry and the second robot motion based on the visual odometry;

adjusting the current location of the robot based on the robot motion error;

determining a distance of each reflecting surface of the target object;

constructing a three-dimensional depth map of the target object using the received reflections, the determined distance of each reflecting surface of the target object, and the adjusted current location of the robot; and

classifying the target object using the three-dimensional depth map, the classifying comprising;

determining a state, a pose, or a gesture of the target object;

classifying the target object as a living object based on the state, the pose, or the gesture of the target object;

determining whether the living object needs assistance by determining one or more of whether the living object is alive, sitting, lying down, waiving, falling, or fallen based on the state, the pose, or the gesture; and

when the living object needs assistance, determining a type of assistance needed based on the state, the pose, or the gesture.

2. The method of claim 1, further comprising storing reference images of the speckle pattern as reflected off a reference object in the scene, the reference images captured at different distances from the reference object.

3. The method of claim 2, further comprising determining a primary speckle pattern on the target object and computing at least one of a respective cross-correlation and a decorrelation between the primary speckle pattern and the speckle patterns of the reference images.

4. The method of claim 1, further comprising emitting the speckle pattern of light in intermittent pulses.

5. The method of claim 4, further comprising altering a frequency of the emitted light pulses.

6. The method of claim 1, further comprising maneuvering the robot to avoid a target object classified as an obstacle.

7. The method of claim 1, further comprising:

identifying an occlusion in the scene about the robot, the occlusion indicating an un-viewable area in the scene by one or more imaging sensors of the robot from a current vantage point; and

maneuvering the robot across the work surface of the working area to obtain a new vantage point that allows the one or more image sensors to detect the target object in the occlusion.

8. The method of claim 1, wherein determining whether the living object needs assistance comprises determining whether the living object is resting or injured.

9. The method of claim 1, wherein determining whether the living object needs assistance further comprises resolving the state, the pose, and/or the gesture of the target object and determining whether the target object is alive, sitting, lying down, waiving, falling, or fallen.

10. The method of claim 1, wherein the gesture of the target object is a falling or waving gesture.

11. The method of claim 1, further comprising, when the living object needs assistance, maneuvering the robot to approach the target object based on the three-dimensional depth map.

12. The method of claim 11, further comprising:

maneuvering the robot to move toward and stop next to the target object; and

verifying a location of the target object and/or the state, the pose, and/or the gesture of the target object.