3D position estimation of objects from a monocular camera using a set of known 3D points on an underlying surface

• US 9,558,584 B1
• Filed: 07/29/2013
• Issued: 01/31/2017
• Est. Priority Date: 07/29/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lo, Wan-Yen, Ferguson, David I.
• Primary Examiner(s): Tryder, Gregory J
• Assistant Examiner(s): Wang, Yuehan
• Application Number:
  US13/952,853
• Time in Patent Office: 1,282 Days
• US Class Current: 1/1
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0238 using obstacle or wall sensors G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

An example method may include determining a three-dimensional (3D) location of a plurality of reference points in an environment, receiving a two-dimensional (2D) image of a portion of the environment that contains an object, selecting certain reference points from the plurality of reference points that form a polygon when projected into the 2D image that contains at least a portion of the object, determining an intersection point of a ray directed toward the object and a 3D polygon formed by the selected reference points, and based on the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points, determining a 3D location of the object in the environment.

Claims

1. A computer-implemented method, comprising:

determining three-dimensional (3D) locations of a plurality of reference points, wherein the plurality of reference points are points on one or more surfaces in an environment;

receiving a two-dimensional (2D) image of a portion of the environment from a camera on an autonomous vehicle, wherein the image contains an object;

projecting some of the plurality of reference points into the 2D image based on the 3D locations of the plurality of reference points and a location of the camera on the autonomous vehicle;

selecting certain reference points from the projected reference points such that the selected reference points form a polygon in the 2D image, wherein the polygon comprises a 2D shape that surrounds the object in the 2D image, wherein selecting the certain reference points comprises performing one or more 2D point-in-polygon tests to determine the selected reference points making up the polygon that surrounds the object in the 2D image;

determining an intersection point of a ray directed toward the object and a 3D polygon formed by the selected reference points;

based on the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points, determining a 3D location of the object in the environment; and

providing instructions to control the autonomous vehicle based at least in part on the 3D location of the object in the environment.

2. The method of claim 1, wherein the ray directed toward the object comprises a ray directed toward a center pixel of the object.

3. The method of claim 1, wherein determining the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points comprises:

determining a plane that contains the 3D polygon; and

determining an intersection point of the ray directed toward the object and the plane that contains the 3D polygon.

4. The method of claim 1, wherein:

the plurality of reference points are points on a road in the environment; and

the object is located on the road in the environment.

5. The method of claim 1, further comprising determining a 3D location of one or more continuous curves in the environment;

and

wherein determining the 3D locations of the plurality of reference points comprises selecting at least some of the reference points from one or more of the continuous curves at one or more intervals.

6. The method of claim 5, wherein:

one of the continuous curves is a first lane boundary of a road within the environment;

one of the continuous curves is a second lane boundary of the road; and

selecting the certain reference points comprises selecting at least one reference point on the first lane boundary and at least one reference point on the second lane boundary.

7. The method of claim 1, wherein:

determining the 3D locations of the plurality of reference points in the environment occurs during a first time period;

receiving the 2D image of the portion of the environment occurs during a second time period; and

the second time period begins at a time after the first time period ends.

8. A vehicle, comprising:

a camera configured to capture a 2D image of a portion of an environment, wherein the image contains an object; and

a computing system configured to;

determine 3D locations of a plurality of reference points in the environment;

project some of the plurality of reference points into the 2D image based on the 3D locations of the plurality of reference points and a location of the camera;

select certain reference points from the projected reference points such that the selected reference points form a polygon in the 2D image, wherein the polygon comprises a 2D shape that surrounds the object in the 2D image, wherein the computing system is configured to select the certain reference points by performing one or more 2D point-in-polygon tests to determine the selected reference points making up the polygon that surrounds the object in the 2D image;

determine an intersection point of a ray directed toward the object and a 3D polygon formed by the selected reference points;

based on the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points, determine a 3D location of the object in the environment; and

provide instructions to control the vehicle based at least in part on the 3D location of the object in the environment.

9. The vehicle of claim 8, wherein the computing system is configured to determine the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points by determining an intersection point of the ray directed toward the object and a plane that contains the 3D polygon.

10. The vehicle of claim 8, wherein:

the plurality of reference points are points on a road in the environment; and

the object is located on the road in the environment.

11. The vehicle of claim 8, wherein the computing system is further configured to determine a 3D location of one or more continuous curves in the environment;

and

wherein at least some of the plurality of reference points are selected from one or more of the continuous curves at one or more intervals.

12. The vehicle of claim 11, wherein:

one of the continuous curves is a first lane boundary of a road within the environment;

one of the continuous curves is a second lane boundary of the road; and

at least one reference point is located on the first lane boundary and at least one reference point is located on the second lane boundary.

13. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:

determining three-dimensional (3D) locations of a plurality of reference points, wherein the plurality of reference points are points on one or more surfaces in an environment;

receiving a two-dimensional (2D) image of a portion of the environment from a camera on an autonomous vehicle, wherein the image contains an object;

projecting some of the plurality of reference points into the 2D image based on the 3D locations of the plurality of reference points and a location of the camera on the autonomous vehicle;

selecting certain reference points from the projected reference points such that the selected reference points form a polygon in the 2D image, wherein the polygon comprises a 2D shape that surrounds the object in the 2D image, wherein selecting the certain reference points comprises performing one or more 2D point-in-polygon tests to determine the selected reference points making up the polygon that surrounds the object in the 2D image;

determining an intersection point of a ray directed toward the object and a 3D polygon formed by the selected reference points;

based on the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points, determining a 3D location of the object in the environment; and

providing instructions to control the autonomous vehicle based at least in part on the 3D location of the object in the environment.

14. The non-transitory computer readable medium of claim 13, wherein determining the intersection point of the ray directed toward the object and the 3D polygon formed by the selected reference points comprises:

determining a plane that contains the 3D polygon; and

determining an intersection point of the ray directed toward the object and the plane that contains the 3D polygon.

15. The non-transitory computer readable medium of claim 13, wherein the ray directed toward the object comprises a ray directed toward a center pixel of the object.