Determining a virtual representation of an environment by projecting texture patterns

• US 9,233,470 B1
• Filed: 03/14/2014
• Issued: 01/12/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Industrial Perception, Inc.
• Inventors: Rublee, Ethan, Bradski, Gary, Konolige, Kurt
• Primary Examiner(s): Nguyen, Bao Long T
• Application Number:
  US14/212,514
• Time in Patent Office: 669 Days
• Field of Search: 318/568.16, 31856822-56823
• US Class Current: 1/1
• CPC Class Codes: B25J 19/00 Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators safety-devices in general F16P; protection against radiation in general G21F
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/162 Mobile manipulator, movable base with manipulator arm mounted on it
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 41/008 frames mounted on wheels or caterpillar for movement on rail tracks B65G41/02
• CPC Class Codes: B65G 47/46 and distributing, e.g. automatically, to desired points in tube mail systems B65G51/36
• CPC Class Codes: B65G 47/50 according to destination signals stored in separate systems
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: B65H 67/065 Manipulators with gripping or holding means for transferring the packages from one station to another, e.g. from a conveyor to a creel trolley
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G05B 2219/31312 Identify pallet, bag, box code
• CPC Class Codes: G05B 2219/39391 Visual servoing, track end effector with camera image feedback
• CPC Class Codes: G05B 2219/40053 : Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40298 : Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40442 : Voxel map, 3-D grid map
• CPC Class Codes: G05B 2219/40543 : Identification and location, position of components, objects
• CPC Class Codes: G06T 17/00 : Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/529 : from texture
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: Y10S 901/01 : Mobile robot
• CPC Class Codes: Y10S 901/02 : Arm motion controller
• CPC Class Codes: Y10S 901/06 : Communication with another machine
• CPC Class Codes: Y10S 901/09 : Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Example methods and systems for determining 3D scene geometry by projecting patterns of light onto a scene are provided. In an example method, a first projector may project a first random texture pattern having a first wavelength and a second projector may project a second random texture pattern having a second wavelength. A computing device may receive sensor data that is indicative of an environment as perceived from a first viewpoint of a first optical sensor and a second viewpoint of a second optical sensor. Based on the received sensor data, the computing device may determine corresponding features between sensor data associated with the first viewpoint and sensor data associated with the second viewpoint. And based on the determined corresponding features, the computing device may determine an output including a virtual representation of the environment that includes depth measurements indicative of distances to at least one object.

Claims

1. A method comprising:

projecting a plurality of different patterns of light using a plurality of projectors, wherein projecting the plurality of different patterns of light using the plurality of projectors comprises projecting, using a first projector of the plurality of projectors, a first texture pattern having a first wavelength while projecting, using a second projector of the plurality of projectors, a second texture pattern having a second wavelength such that at least a portion of the first texture pattern overlaps at least a portion of the second texture pattern on a surface in an environment of a computing device;

receiving sensor data by the computing device and from a plurality of optical sensors, wherein the plurality of optical sensors are configured to distinguish between the plurality of different patterns of light, and wherein the sensor data comprises at least one first image of an overlapping of the first texture pattern and the second texture pattern as perceived from a first viewpoint of a first optical sensor of the plurality of optical sensors and at least one second image of the overlapping of the first texture pattern and the second texture pattern as perceived from a second viewpoint of a second optical sensor of the plurality of optical sensors;

determining, by the computing device and based on the received sensor data, corresponding features between the at least one first image and the at least one second image; and

based on the determined corresponding features, determining, by the computing device, an output including a virtual representation of the environment of the computing device, wherein the output comprises a depth measurement indicative of a distance to at least one object in the environment.

2. The method of claim 1, wherein the first projector projects the first texture pattern during a first time period, and wherein the method further comprises projecting a third texture pattern using the first projector during a second time period that is subsequent to the first time period.

3. The method of claim 2, further comprising:

determining that an expected amount of motion in the environment is less than a threshold; and

in response to determining that the expected amount of motion in the environment is less than the threshold, projecting the first texture pattern and the third texture pattern in sequence.

4. The method of claim 3, wherein the expected amount of motion in the environment comprises an expected amount of motion of a robotic manipulator configured to be controlled by the computing device, wherein the first optical sensor and the second optical sensor are coupled to the robotic manipulator.

5. The method of claim 2:

wherein the sensor data associated with the first viewpoint comprises a first sequence of infrared images corresponding to a given time period and the sensor data associated with the second viewpoint comprises a second sequence of infrared images corresponding to the given time period, and

wherein determining the corresponding features comprises;

determining a first spatio-temporal image based on a combination of the first sequence of infrared images,

determining a second spatio-temporal image based on a combination of the second sequence of infrared images, and

determining corresponding segments of pixels between the first spatio-temporal image and the second spatio-temporal image.

6. The method of claim 1:

wherein the at least one first image comprises a first infrared image and the at least one second image comprises a second infrared image, and

wherein determining the corresponding features comprises determining corresponding segments of pixels between the first infrared image and the second infrared image.

7. The method of claim 1, wherein the first optical sensor and the second optical sensor are configured to distinguish between the first wavelength and the second wavelength using pixel filters.

8. The method of claim 1, wherein the first optical sensor and the second optical sensor together form a first stereo camera, wherein the plurality of optical sensors further comprise a second stereo camera, and wherein the method further comprises:

determining, by the computing device, a first depth image using sensor data received from the first stereo camera and a second depth image using sensor data received from the second stereo camera; and

based on the first depth image, the second depth image, and a distance between a position of the first stereo camera and a position of the second stereo camera, modifying the depth measurement indicative of the distance to the at least one object in the environment.

9. The method of claim 1, wherein receiving sensor data from the plurality of optical sensors comprises receiving first sensor data corresponding to a first position of the first optical sensor and the second optical sensor and receiving second sensor data corresponding to a second position of the first optical sensor and the second optical sensor, and wherein the method further comprises:

determining, by the computing device, a first depth image using the first sensor data and a second depth image using the second sensor data; and

based on the first depth image, the second depth image, and a distance between the first position and the second position, modifying the depth measurement indicative of the distance to the at least one object in the environment.

10. The method of claim 1, wherein the plurality of projectors are located at different respective projector positions, and wherein the method further comprises determining a location of an occluding edge of the at least one object based on image segmentation of respective shadows induced by the projection of light from the different projector positions.

11. A non-transitory computer-readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

causing a plurality of projectors to project a plurality of different patterns of light, wherein causing the plurality of projectors to project the plurality of different patterns of light comprises causing a first projector of the plurality of projectors to project a first texture pattern having a first wavelength while causing a second projector of the plurality of projectors to project a second texture pattern having a second wavelength such that at least a portion of the first texture pattern overlaps at least a portion of the second texture pattern on a surface in an environment of the computing device;

receiving sensor data from a plurality of optical sensors, wherein the plurality of optical sensors are configured to distinguish between the plurality of different patterns of light, and wherein the sensor data comprises at least one first image of an overlapping of the first texture pattern and the second texture pattern as perceived from a first viewpoint of a first optical sensor of the plurality of optical sensors and at least one second image of the overlapping of the first texture pattern and the second texture pattern as perceived from a second viewpoint of a second optical sensor of the plurality of optical sensors;

based on the received sensor data, determining corresponding features between the at least one first image and the at least one second image; and

based on the determined corresponding features, determining an output including a virtual representation of the environment of the computing device, wherein the output comprises a depth measurement indicative of a distance to at least one object in the environment.

12. The non-transitory computer-readable medium of claim 11, wherein the first projector projects the first texture pattern during a first time period, and wherein the functions further comprise causing the first projector to project a third texture pattern during a second time period that is subsequent to the first time period.

13. The non-transitory computer-readable medium of claim 11:

wherein the at least one first image comprises one or more first infrared images and the at least one second image comprises one or more second infrared images, and

wherein determining the corresponding features comprises determining corresponding segments of pixels between the one or more first infrared images and the one or more second infrared images.

14. The non-transitory computer-readable medium of claim 11, wherein the first optical sensor and the second optical sensor together form a first stereo camera, wherein the plurality of optical sensors further comprise a second stereo camera, and wherein the functions further comprise:

determining a first depth image using sensor data received from the first stereo camera and a second depth image using sensor data received from the second stereo camera; and

based on the first depth image, the second depth image, and a distance between a position of the first stereo camera and a position of the second stereo camera, modifying the depth measurement indicative of the distances to the at least one object in the environment.

15. The non-transitory computer-readable medium of claim 11, wherein receiving sensor data from the plurality of optical sensors comprises receiving first sensor data corresponding to a first position of the first optical sensor and the second optical sensor and receiving second sensor data corresponding to a second position of the first optical sensor and the second optical sensor, and wherein the functions further comprise:

determining a first depth image using the first sensor data and a second depth image using the second sensor data; and

based on the first depth image, the second depth image, and a distance between the first position and the second position, modifying the depth measurement indicative of the distance to the at least one object in the environment.

16. The non-transitory computer-readable medium of claim 11, wherein the plurality of projectors are located at different respective projector positions, and wherein the functions further comprise determining a location of an occluding edge of the at least one object based on image segmentation of respective shadows induced by the projection of light from the different projector positions.

17. A system comprising:

a robotic manipulator;

at least one projector coupled to the robotic manipulator and configured to project different patterns of light;

at least one stereo camera coupled to the robotic manipulator and configured to obtain sensor data that is indicative of an environment of the system; and

a computing device configured to perform functions comprising;

determining an expected amount of motion of the robotic manipulator during a future time period,

determining that the expected amount of motion is less than a threshold, and

in response to determining that the expected amount of motion is less than the threshold, causing the at least one projector to project multiple texture patterns during the future time period.

18. The system of claim 17, wherein the functions further comprise:

receiving sensor data from the stereo camera that corresponds to the future time period, and

determining a virtual representation of the environment using the received sensor data, wherein the virtual representation comprises depth measurements indicative of distances to at least one object in the environment.

19. The system of claim 17, wherein the functions further comprise:

determining another expected amount of motion of the robotic manipulator during another future time period,

determining that the other expected amount of motion is greater than the threshold, and

in response to determining that the other expected amount of motion is greater than the threshold, causing the at least one projector to project a single texture pattern during the other future time period.