Integration of depth points into a height map

• US 9,205,562 B1
• Filed: 12/19/2014
• Issued: 12/08/2015
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Konolige, Kurt, Cagniart, Cedric
• Primary Examiner(s): Cumbess, Yolanda
• Application Number:
  US14/577,664
• Time in Patent Office: 354 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: G05B 2219/45056 Handling cases, boxes
• CPC Class Codes: G05B 2219/50378 Control height gripper as function of thickness of workpiece and height of pallet
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/46 Sensing device

Abstract

An example method includes receiving a plurality of detected depth points indicative of depths of at least one surface and determining a projection of the detected depth points onto a plane. The method may also include identifying a plurality of first detected points, where a first detected point comprises a first point at a particular location of the plane in the projection. The method may also include storing digital entries corresponding to points located within a threshold buffer from one of the first detected points relative to the plane. The method may additionally include determining values for the digital entries, where a value for a digital entry corresponding to a particular point comprises an accumulation of distances from detected depth points that cross the particular point. The method may further include determining a digital height map representative of heights of the at least one surface relative to the plane.

Claims

1. A method, comprising:

receiving, from at least one depth sensor, a plurality of detected depth points indicative of depths of at least one surface within an environment along rays from the at least one depth sensor;

determining, by a computing device, a projection of the plurality of detected depth points onto a plane within the environment;

identifying, by the computing device, a plurality of first detected points from the plurality of detected depth points, wherein a first detected point comprises a first point at a particular location of the plane in the projection;

storing, by the computing device, digital entries corresponding to points located within a threshold buffer from one of the first detected points relative to the plane;

determining, by the computing device, values for the digital entries, wherein a value for a digital entry corresponding to a particular point comprises an accumulation of distances from detected depth points along one or more of the rays that cross the particular point; and

determining, by the computing device, a digital height map representative of heights of the at least one surface in the environment relative to the plane, wherein the heights of the at least one surface at particular locations in the digital height map are determined based on points where the values for the digital entries cross zero.

2. The method of claim 1, wherein the accumulation of distances from detected depth points along one or more of the rays comprises an average of signed distance values, wherein a signed distance value for a ray from a depth sensor to a detected depth point is zero at the detected depth point, increases in a direction towards the depth sensor, and decreases in a direction away from the depth sensor.

3. The method of claim 1, wherein the plane in the environment comprises a ground floor within the environment.

4. The method of claim 1, wherein the digital entries correspond to an equal number of points above and below a first detected point relative to the plane.

5. The method of claim 1, wherein the threshold buffer comprises two pixels above and two pixels below one of the first detected points relative to the plane.

6. The method of claim 1, further comprising:

identifying a physical object within the environment based on the digital height map representative of heights of the at least one surface in the environment relative to the plane; and

providing instructions to a robotic manipulator to move the identified physical object.

7. The method of claim 6, where one or more of the at least one depth sensor is physically coupled to the robotic manipulator, and wherein the computing device comprises a control system of the robotic manipulator.

8. The method of claim 1, further comprising:

identifying a plurality of second detected points from the plurality of detected depth points, wherein a second detected point comprises a second point at a same location of the plane in the projection as a first detected point, and wherein the second detected point is further away from the first detected point than the threshold buffer; and

storing additional digital entries corresponding to points located within the threshold buffer from one of the second detected points relative to the plane.

9. The method of claim 8, further comprising using the digital entries corresponding to points located within the threshold buffer from a first detected point and points located within the threshold buffer from a second detected point at a same location of the plane in the projection to identify a first object that overlaps at least a portion of a second object relative to the plane within the environment.

10. The method of claim 8, further comprising using the digital entries corresponding to points located within the threshold buffer from a first detected point and points located within the threshold buffer from a second detected point at a same location of the plane in the projection to determine a length of an occluded edge of an object within the environment.

11. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving, from at least one depth sensor, a plurality of detected depth points indicative of depths of at least one surface within an environment along rays from the at least one depth sensor;

determining a projection of the plurality of detected depth points onto a plane within the environment;

identifying a plurality of first detected points from the plurality of detected depth points, wherein a first detected point comprises a first point at a particular location of the plane in the projection;

storing digital entries corresponding to points located within a threshold buffer from one of the first detected points relative to the plane;

determining values for the digital entries, wherein a value for a digital entry corresponding to a particular point comprises an accumulation of distances from detected depth points along one or more of the rays that cross the particular point; and

determining a digital height map representative of heights of the at least one surface in the environment relative to the plane, wherein the heights of the at least one surface at particular locations in the digital height map are determined based on points where the values for the digital entries cross zero.

12. The non-transitory computer readable medium of claim 11, wherein the plane in the environment comprises a ground floor within the environment.

13. The non-transitory computer readable medium of claim 11, wherein the threshold buffer comprises five pixels above and five pixels below one of the first detected points relative to the plane.

14. The non-transitory computer readable medium of claim 11, wherein the accumulation of distances from detected depth points along one or more of the rays comprises an average of signed distance values, wherein a signed distance value for a ray from a depth sensor to a detected depth point is zero at the detected depth point, increases in a direction towards the depth sensor, and decreases in a direction away from the depth sensor.

15. The non-transitory computer readable medium of claim 11, wherein the functions further comprise:

identifying a plurality of second detected points from the plurality of detected depth points, wherein a second detected point comprises a second point at a same location of the plane in the projection as a first detected point, and wherein the second detected point is further away from the first detected point than the threshold buffer; and

storing additional digital entries corresponding to points located within the threshold buffer from one of the second detected points relative to the plane.

19. The system of claim 15, wherein the control system is further configured to:

identify a plurality of second detected points from the plurality of detected depth points, wherein a second detected point comprises a second point at a same location of the plane in the projection as a first detected point, and wherein the second detected point is further away from the first detected point than the threshold buffer; and

store additional digital entries corresponding to points located within the threshold buffer from one of the second detected points relative to the plane.

20. The system of claim 19, wherein the control system is further configured to use the digital entries corresponding to points located within the threshold buffer from a first detected point and points located within the threshold buffer from a second detected point at a same location of the plane in the projection to identify a first object that overlaps at least a portion of a second object relative to the plane within the environment.

16. A system comprising:

at least one depth sensor; and

a control system configured to;

receive, from at least one depth sensor, a plurality of detected depth points indicative of depths of at least one surface within an environment along rays from the at least one depth sensor;

determine a projection of the plurality of detected depth points onto a plane within the environment;

identify a plurality of first detected points from the plurality of detected depth points, wherein a first detected point comprises a first point at a particular location of the plane in the projection;

store digital entries corresponding to points located within a threshold buffer from one of the first detected points relative to the plane;

determine values for the digital entries, wherein a value for a digital entry corresponding to a particular point comprises an accumulation of distances from detected depth points along one or more of the rays that cross the particular point; and

determine a digital height map representative of heights of the at least one surface in the environment relative to the plane, wherein the heights of the at least one surface at particular locations in the digital height map are determined based on points where the values for the digital entries cross zero.

17. The system of claim 16, further comprising a robotic manipulator, wherein the control system is further configured to:

identify a physical object within the environment based on the digital height map representative of heights of the at least one surface in the environment relative to the plane; and

provide instructions to the robotic manipulator to move the identified physical object.

18. The system of claim 17, where one or more of the at least one depth sensor is physically coupled to the robotic manipulator.