Camera based localization

• US 9,476,970 B1
• Filed: 03/19/2012
• Issued: 10/25/2016
• Est. Priority Date: 03/19/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Urmson, Christopher Paul, Montemerlo, Michael Steven
• Primary Examiner(s): Li, Tracy
• Assistant Examiner(s): Rahman, Mohammad J
• Application Number:
  US13/423,704
• Time in Patent Office: 1,681 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/0009 Transmission of position information to remote stations involving assistance data G01S5/0236
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0268 using internal positioning means
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Aspects of the disclosure relate generally to autonomous vehicles. Specifically, the features described may be used alone or in combination in order to improve the safety, use, driver experience, and performance of these vehicles. In particular, the disclosure includes a system and method of determining a vehicle'"'"'s location based on a comparison of images captured from the vehicle with a database of images.

Claims

1. A method for localizing a vehicle, the method comprising:

capturing a first image of a vehicle'"'"'s surroundings;

determining, by one or more processors, a roll and pitch of the vehicle at a particular moment in time;

determining, by one or more processors, an imaginary plane representing a two dimensional plane extending out in front of the vehicle using the roll and pitch of the vehicle such that the roll and pitch of the vehicle corresponds to a slope of the imaginary plane;

projecting, by the one or more processors, points of one or more objects from within the first image onto the imaginary plane using the slope of the imaginary plane;

accessing, by the one or more processors, a second image associated with a camera location of the first image and a camera orientation of the first image;

projecting, by the one or more processors, points of one or more objects from within the second image onto the imaginary plane;

determining, by the one or more processors, an offset between the first image and the second image based on at least one point projected from the first image onto the imaginary plane and at least one point projected from the second image onto the imaginary plane; and

determining, by the one or more processors, a location of the vehicle based on the offset between the first image and the second image.

2. A method of claim 1 further comprising, autonomously controlling the vehicle based on the determined location of the vehicle.

3. A method of claim 1, wherein determining the offset of the first image and the second image comprises:

identifying at least one object that appears in both the first image and second image; and

determining an offset of the at least one object within the first image and the second image.

4. A method of claim 3, wherein determining the offset of the at least one object comprises determining the position of the at least one object on the imaginary plane.

5. A method of claim 1, further comprising determining the pitch as a relative orientation of the vehicle with respect to gravity.

6. A method of claim 5, wherein the relative orientation of the vehicle is further determined by averaging a plurality of orientations over a period of time.

7. A method of claim 1, wherein determining the offset the first image and the second image comprises:

blurring the first and second images;

converting the first and second images to gradient-magnitude images; and

convolving the gradient-magnitude images.

8. A method of claim 7, wherein only a portion of the first image and a portion of the second image are used to determine the offset.

9. A method of claim 8, wherein accessing the second image comprises:

determining a likely location and orientation of the vehicle;

accessing a database containing a plurality of images, the plurality of images each being associated with a camera location and a camera orientation;

determining which of the plurality of images has a camera location and a camera orientation most similar to the likely location and orientation of the first image; and

selecting the image determined to be the most similar to the likely location and orientation of the first image as the second image.

10. A method of claim 9, wherein in the plurality of images are arranged within the database in accordance with a spherical geometry hierarchy that is based on the location and orientation of the plurality of images.

11. A method of claim 1, wherein the second image is one of a plurality of images captured by a camera that is attached to a second vehicle traveling along a predetermined path.

12. A system for controlling a vehicle, the system comprising:

a camera for capturing a first image of a vehicle'"'"'s surroundings; and

a one or more processors operable to;

determine a roll and pitch of the vehicle at a particular moment in time;

determine an imaginary plane representing a two dimensional plane extending out in front of the vehicle using the roll and pitch of the vehicle such that the roll and pitch of the vehicle corresponds to a slope of the imaginary plane;

project points of one or more objects from within the first image onto the imaginary plane using the slope of the imaginary plane;

access a second image, the second image being associated with a camera location of the first image and a camera orientation of the first image;

project points of one or more objects from within the second image onto the imaginary plane;

determine an offset between one or more objects of the first image and one or more corresponding objects of the second image based on at least one point projected from the first image onto the imaginary plan and at least one point projected from the second image onto the imaginary plane; and

determine a location of the vehicle based on the offset.

13. The system of claim 12, wherein the one or more processors are further operable to:

identify at least one object that appears in both the first image and second image;

determine a position of the at least one object within the first image and second image; and

determine the location of the vehicle based on the difference between the position of the at least one object within the first image and second image.

14. The system of claim 12, wherein the one or more processors are further configured to determine the pitch as a relative orientation of the vehicle with respect to gravity.

15. The system of claim 14, wherein the one or more processors are further operable to determine the relative orientation of the vehicle by averaging a plurality of orientations of the vehicle over a period of time.

16. The system of claim 14, wherein the one or more processors are further operable to:

blur the first and second images;

convert the first and second images to gradient-magnitude images; and

convolving the gradient-magnitude images.

17. The system of claim 16, wherein only a portion of the first image and a portion of the second image are compared.

18. The system of claim 17, wherein the one or more processors further operable to:

determine a likely location and a likely orientation of the first image;

access a database containing a plurality of images, the plurality of images each being associated with a camera location and a camera orientation;

determine which of the plurality of images has a camera location and camera orientation most similar to the likely location and the likely orientation of the first image; and

select the image determined to be the most similar to the likely location and likely orientation of the first image as the second image.

19. The system of claim 18, wherein in the plurality of images are arranged within the database in accordance with a spherical geometry hierarchy that is based on the camera location and camera orientation of the plurality of images.

20. The system of claim 12, wherein the one or more processors are further operable to autonomously control the vehicle based on the determined location.

24. The system of claim 18, further comprising the vehicle.

21. A method for navigating a vehicle, the method comprising:

capturing a plurality of images along a path;

determining a roll and pitch of a vehicle at a particular moment in time;

determining an imaginary plane representing a two dimensional plane extending out in front of the vehicle using the roll and pitch such that the roll and pitch of the vehicle corresponds to a slope of the imaginary plane;

associating each image in the plurality of images with a camera location and a camera orientation;

projecting one or more objects from within the plurality of images onto the imaginary plane using the slope of the imaginary plane;

storing the plurality of images in a database;

determining that the vehicle is traveling along the path, the vehicle having a position and orientation relative to surrounding objects;

accessing an image from the plurality of images;

identifying a first object within the accessed image, the first object having an imaginary plane position within the accessed image;

determining a location of the vehicle by comparing the imaginary plane position of the first object with the position and orientation of the vehicle relative to the object; and

using, by one or more processors, the determined location to navigate the vehicle.

22. A method of claim 21, wherein comparing the imaginary plane position of the object with the position and orientation of the vehicle relative to the object comprises:

capturing a camera image from the vehicle;

projecting the camera image onto the imaginary plane; and

comparing the camera image with the accessed image.

23. A method of claim 22, wherein comparing the camera image with the accessed image comprises:

blurring the accessed image and the camera image;

converting the accessed image and the camera image to gradient-magnitude images; and

convolving the gradient-magnitude images.