Mobile Human Interface Robot

• US 20120182392A1
• Filed: 02/22/2011
• Published: 07/19/2012
• Est. Priority Date: 05/20/2010
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Kearns, Justin H., Taka, Orjeta, Pack, Robert Todd, Rosenstein, Michael T., Halloran, Michael, Farlow, Timothy S., Shamlian, Steven V., Won, Chikyung, Chiappetta, Mark, Vicenti, Jasper Fourways
• Granted Patent: US 9,400,503 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: B25J 11/009 Nursing, e.g. carrying sick persons, pushing wheelchairs, distributing drugs
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G02B 13/22 Telecentric objectives or lens systems
• CPC Class Codes: G05D 1/0227 using mechanical sensing means, e.g. for sensing treated area
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0242 using non-visible light signals, e.g. IR or UV signals
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

A method of object detection for a mobile robot includes emitting a speckle pattern of light onto a scene about the robot while maneuvering the robot across a work surface, receiving reflections of the emitted speckle pattern off surfaces of a target object in the scene, determining a distance of each reflecting surface of the target object, constructing a three-dimensional depth map of the target object, and classifying the target object.

Claims

1. A method of operating a mobile robot, the method comprising:

receiving three-dimensional depth image data;

producing a local perceptual space corresponding to an environment around the robot;

classifying portions of the local perceptual space corresponding to sensed objects located above a ground plane and below a height of the robot as obstacles;

classifying portions of the local perceptual space corresponding to sensed objects below the ground plane as obstacles;

classifying portions of the local perceptual space corresponding to unobstructed area on the ground plane as free space;

classifying all remaining unclassified local perceptual space as unknown; and

executing a drive command to move to a location in the environment corresponding to local perceptual space classified as at least of free space and unknown.

2. The method of claim 1, wherein the classifications decay over time unless persisted with updated three-dimensional depth image data.

3. The method of claim 1, further comprising evaluating predicted robot paths corresponding to feasible robot drive commands by rejecting robot paths moving to locations having a corresponding local perceptual space classified as obstacles or unknown.

4. The method of claim 1, further comprising producing a three-dimensional voxel grid using the three-dimensional depth image data and converting the three-dimensional voxel grid into a two-dimensional grid, each cell of the two-dimensional grid corresponding to a portion of the local perceptual space.

5. The method of claim 1, further comprising producing a grid corresponding to the local perceptual space, each grid cell having the classification of the corresponding local perceptual space, for each grid cell classified as an obstacle or unknown, retrieving a grid point within that grid cell and executing a collision evaluation.

6. The method of claim 5, wherein the collision evaluation comprises rejecting grid points located within a collision circle about a location of the robot.

7. The method of claim 5, wherein the collision evaluation comprises rejecting grid points located within a collision triangle centered about a location of the robot.

8. The method of claim 1, further comprising orienting a field of view of an imaging sensor providing the three-dimensional depth image data toward an area in the environment corresponding to local perceptual space classified as unknown.

9. The method of claim 1, further comprising rejecting a drive command that moves the robot to a robot position beyond a field of view of an imaging sensor providing the three-dimensional depth image data.

10. The method of claim 1, further comprising rejecting a drive command to move holonomically perpendicular to a forward drive direction of the robot when the robot has been stationary for a threshold period of time, the imaging sensor providing the three-dimensional depth image data being aligned with the forward drive direction.

11. The method of claim 1, further comprising accepting a drive command to move holonomically perpendicular to a forward drive direction of the robot while the robot is driving forward, the imaging sensor providing the three-dimensional depth image data being aligned with the forward drive direction and having a field of view angle of at least 45 degrees.

12. The method of claim 1, further comprising accepting a drive command to move to a location in the environment having corresponding local perceptual space classified as unknown when the robot determines that a field of view of an imaging sensor providing the three-dimensional depth image data covers the location before the robot reaches the location.

13. The method of claim 1, wherein the three-dimensional depth image data is provided by a volumetric point cloud imaging device positioned on the robot to be capable of obtaining a point cloud from a volume of space adjacent the robot.

14. The method of claim 1, wherein the three-dimensional depth image data is provided by a volumetric point cloud imaging device positioned on the robot at a height of greater than 2 feet above the ground and directed to be capable of obtaining a point cloud from a volume of space that includes a floor plane in a direction of movement of the robot.

15. A method of operating a mobile robot to follow a person, the method comprising:

receiving three-dimensional image data from a volumetric point cloud imaging device positioned to be capable of obtaining a point cloud from a volume of space adjacent the robot;

segmenting the received three-dimensional image data into objects;

filtering the objects to remove objects greater than a first threshold size and smaller than a second threshold size;

identifying a person corresponding to at least a portion of the filtered objects; and

moving at least a portion of the robot with respect to the identified person.

16. The method of claim 15, wherein the three-dimensional image data comprises a two-dimensional array of pixels, each pixel containing depth information.

17. The method of claim 16, further comprising grouping the pixels into the objects based on a proximity of each pixel to a neighboring pixel.

18. The method of claim 15, further comprising driving the robot away from the identified person when the identified person is within a threshold distance of the robot.

19. The method of claim 18, further comprising maintaining a field of view of the imaging device on the identified person.

20. The method of claim 15, further comprising driving the robot to maintain a following distance between the robot and the identified person.

21. The method of claim 15, further comprising issuing waypoint drive commands to drive the robot within a following distance of the identified person.

22. The method of claim 15, further comprising maintaining a field of view of the imaging device on the identified person.

23. The method of claim 22, further comprising at least one of panning and tilting the imaging device to aim a corresponding field of view at least substantially toward the identified person.

24. The method of claim 15, further comprising driving the robot toward the identified person when the identified person is beyond a threshold distance of the robot.

25. The method of claim 15, wherein the imaging device is positioned at a height of at least about one feet above a ground surface and directed to be capable of obtaining a point cloud from a volume of space that includes a floor plane in a direction of movement of the robot.

26. The method of claim 15, wherein the first threshold size comprises a height of about 8 feet.

27. The method of claim 15, wherein the second threshold size comprises a height of about 3 feet.

28. The method of claim 15, further comprising identifying multiple people corresponding to the filtered objects.

29. The method of claim 15, further comprising using a Kalman filter to track and propagate a movement trajectory of each identified person.

30. The method of claim 29, further comprising issuing a drive command based at least in part on a movement trajectory of at least one identified person.

31. A method of object detection for a mobile robot, the method comprising:

maneuvering the robot across a work surface;

emitting light onto a scene about the robot;

capturing images of the scene along a drive direction of the robot, the images comprising at least one of (a) a three-dimensional depth image, (b) an active illumination image, and (c) an ambient illumination image;

determining a location of an object in the scene based on the images;

assigning a confidence level for the object location; and

maneuvering the robot in the scene based on the object location and corresponding confidence level.

32. The method of claim 31, further comprising constructing an object occupancy map of the scene.

33. The method of claim 31, further comprising degrading the confidence level of each object location over time until updating the respective object location with a newly determined object location.

34. The method of claim 31, further comprising maneuvering the robot to at least one of contact the object and follow along a perimeter of the object.

35. The method of claim 31, further comprising maneuvering the robot to avoid the object.

36. The method of claim 31, further comprising emitting the light onto the scene in intermittent pulses.

37. The method of claim 36, further comprising altering a frequency of the emitted light pulses.

38. The method of claim 37, further comprising emitting the light pulses at a first, power saving frequency and upon receiving a sensor event emitting the light pulses at a second, active frequency.

39. The method of claim 38, wherein the sensor event comprises a sensor signal indicative of the presence of an object in the scene.

40. The method of claim 31, further comprising constructing the three-dimensional depth image of the scene by:

emitting a speckle pattern of light onto the scene;

receiving reflections of the speckle pattern from the object in the scene;

storing reference images of the speckle pattern as reflected off a reference object in the scene, the reference images captured at different distances from the reference object;

capturing at least one target image of the speckle pattern as reflected off a target object in the scene; and

comparing the at least one target image with the reference images for determining a distance of the reflecting surfaces of the target object.

41. The method of claim 40, further comprising determining a primary speckle pattern on the target object and computing at least one of a respective cross-correlation and a decorrelation between the primary speckle pattern and the speckle patterns of the reference images.

42. A method of object detection for a mobile robot, the method comprising:

emitting a speckle pattern of light onto a scene about the robot while maneuvering the robot across a work surface;

receiving reflections of the emitted speckle pattern off surfaces of a target object in the scene;

determining a distance of each reflecting surface of the target object;

constructing a three-dimensional depth map of the target object; and

classifying the target object.

43. The method of claim 42, further comprising storing reference images of the speckle pattern as reflected off a reference object in the scene, the reference images captured at different distances from the reference object.

44. The method of claim 43, further comprising determining a primary speckle pattern on the target object and computing at least one of a respective cross-correlation and a decorrelation between the primary speckle pattern and the speckle patterns of the reference images.

45. The method of claim 42, further comprising maneuvering the robot with respect to the target object based on the classification of the target object.

46. The method of claim 42, further comprising emitting the speckle pattern of light in intermittent pulses.

47. The method of claim 46, further comprising altering a frequency of the emitted light pulses.

48. The method of claim 42, further comprising capturing frames of reflections of the emitted speckle pattern off surfaces of the target object at a frame rate.

49. The method of claim 48, wherein the frame rate is between about 10 Hz and about 90 Hz.

50. The method of claim 48, further comprising resolving differences between speckle patterns captured in successive frames for identification of the target object.