Camera calibration using structure from motion techniques

• US 9,201,424 B1
• Filed: 08/27/2013
• Issued: 12/01/2015
• Est. Priority Date: 08/27/2013
• Status: Expired due to Fees

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ogale, Abhijit
• Primary Examiner(s): Olszewski, John R
• Assistant Examiner(s): WHITTINGTON, JESS G
• Application Number:
  US14/010,937
• Time in Patent Office: 826 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

Example methods and systems for camera calibration using structure from motion techniques are described herein. Within examples, an autonomous vehicle may receive images from a vehicle camera system and may determine an image-based pose based on the images. To determine an image-bases pose, an autonomous vehicle may perform various processes related to structure from motion, such as image matching and bundle adjustment. In addition, the vehicle may determine a sensor-based pose indicative of a position and orientation of the vehicle through using information provided by vehicle sensors. The vehicle may align the image-based pose with the sensor-based pose to determine any adjustments to the position or orientation that may calibrate the cameras. In an example, a computing device of the vehicle may align the different poses using transforms, rotations, and/or scaling.

Claims

1. A method comprising:

receiving, at a computing device via a camera coupled at an orientation and position to a vehicle autonomously navigating a path in an environment, a plurality of images indicative of a plurality of objects positioned within a threshold distance from the path in the environment of the vehicle during a respective time range;

based on the plurality of images, determining, by the computing device, an image-based pose indicative of the orientation and position of the camera;

receiving from one or more laser-based sensors coupled to the vehicle as the vehicle navigates the path in the environment, sensor data indicative of an orientation and position of the vehicle relative to the plurality of objects positioned within the threshold distance from the path in the environment of the vehicle during the respective time range;

based on the sensor data, determining a sensor-based pose indicative of the orientation and position of the vehicle relative to the plurality of objects in the environment, wherein both the sensor-based pose and the image-based pose correspond to the respective time range;

aligning, by the computing device as the vehicle navigates the path in the environment, a camera-based coordinate system of the image-based pose with a vehicle-based coordinate system of the sensor-based pose using one or more of a transform, rotation, and scaling modification;

determining an adjustment for the orientation or position of the camera based on the alignment of the camera-based coordinate system of the image-based pose with the vehicle-based coordinate system of the sensor-based pose; and

causing, by the computing device as the vehicle navigates the path in the environment, calibration of the camera based on the adjustment.

2. The method of claim 1, wherein aligning the image-based pose with the sensor-based pose includes aligning a first 3D point cloud indicative of the image-based pose with a second 3D point cloud indicative of the sensor-based pose.

3. The method of claim 1, wherein the image-based pose is based on a camera-based coordinate system and the sensor-based pose is based on a vehicle-based coordinate system.

4. The method of claim 3, wherein the camera-based coordinate system differs from the vehicle-based coordinate system.

5. The method of claim 1, wherein receiving from one or more sensors coupled to the vehicle, sensor data indicative of the orientation and position of the vehicle relative to the plurality objects in the environment of the vehicle during the respective time range comprises:

receiving sensor data from a plurality of sensors comprising a combination of two or more of RADAR, LIDAR, and a global positioning system (GPS).

6. The method of claim 1, wherein determining the adjustment of the orientation or position of the camera based on the alignment of the image-based pose with the sensor-based pose comprises:

determining one or more adjustments to a rotation or translation of the camera relative to the orientation and position of the vehicle.

7. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising;

receiving, via a camera coupled at an orientation and position to a vehicle autonomously navigating a path in an environment, a plurality of images indicative of a plurality of objects positioned within a threshold distance from the path in the environment of the vehicle during a respective time range;

based on the plurality of images, determining an image-based pose indicative of the orientation and position of the camera;

receiving from one or more laser-based sensors coupled to the vehicle as the vehicle navigates the path in the environment, sensor data indicative of an orientation and position of the vehicle relative to the plurality of objects positioned within the threshold distance from the path in the environment of the vehicle during the respective time range;

based on the sensor data, determining a sensor-based pose indicative of the orientation and position of the vehicle relative to the plurality of objects in the environment, wherein both the sensor-based pose and the image-based pose correspond to the respective time range;

aligning, as the vehicle autonomously navigates the path in the environment, a camera-based coordinate system of the image-based pose with a vehicle-based coordinate system of the sensor-based pose using one or more of a transform, rotation, and scaling;

determining an adjustment for the orientation or position of the camera based on the alignment of the camera-based coordinate system of the image-based pose with the vehicle-based coordinate system of the sensor-based pose; and

causing, as the vehicle navigates the path in the environment, calibration of the camera based on the adjustment.

8. The system of claim 7, wherein determining the image-based pose indicative of the orientation and position of the camera includes determining a set of useful images for determining the image-based pose through image matching, wherein each image in the set of useful images displays a constant object.

9. The system of claim 7, wherein determining the image-based pose indicative of the orientation and position of the camera includes determining a set of three-dimensional points within the images indicative of an object in the environment;

and

adjusting the set of three-dimensional points to determine the image-based pose indicative of an orientation and position of the camera relative to the plurality of objects in the environment.

10. The system of claim 9, wherein the camera-based coordinate system differs from the vehicle-based coordinate system.

11. The system of claim 7, wherein the function of determining, via at least one sensor coupled to the vehicle, a sensor-based pose comprises:

using one or more of RADAR, LIDAR, or a global positioning system (GPS).

12. The system of claim 7, wherein determining the adjustment of the orientation or position of the camera based on the alignment of the image-based pose with the sensor-based pose comprises:

determining one or more adjustments to a rotation or translation of the camera relative to the orientation and position of the vehicle.

13. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving, via a camera coupled at an orientation and position to a vehicle autonomously navigating a path in an environment, a plurality of images indicative of a plurality of objects positioned within a threshold distance from the path in the environment of the vehicle during a respective time range;

based on the plurality of images, determining an image-based pose indicative of the orientation and position of the camera;

receiving from one or more laser-based sensors coupled to the vehicle as the vehicle navigates the path in the environment, sensor data indicative of an orientation and position of the vehicle relative to the plurality of objects positioned within the threshold distance from the path in the environment of the vehicle during the respective time range;

based on the sensor data, determining a sensor-based pose indicative of the orientation and position of the vehicle relative to the plurality of objects in the environment, wherein both the sensor-based pose and the image-based pose correspond to the respective time range;

aligning, as the vehicle autonomously navigates the path in the environment, a camera-based coordinate system of the image-based pose with a vehicle-based coordinate system of the sensor-based pose using one or more of a transform, rotation, and scaling;

determining an adjustment for the orientation or position of the camera based on the alignment of the camera-based coordinate system of the image-based pose with the vehicle-based coordinate system of the sensor-based pose; and

causing, as the vehicle navigates the path in the environment, calibration of the camera based on the adjustment.

14. The non-transitory computer readable medium of claim 13, wherein determining the image-based pose indicative of the orientation and position of the camera includes determining a set of three-dimensional points within the images indicative of an object in the environment;

and

adjusting the set of three-dimensional points to determine the image-based pose indicative of an orientation and position of the camera relative to the object in the environment.

15. The non-transitory computer readable medium of claim 13, wherein the image-based pose is based on a camera-based coordinate system and the sensor-based pose is based on a vehicle-based coordinate system.

16. The non-transitory computer readable medium of claim 15, wherein the camera-based coordinate system differs from the vehicle-based coordinate system.

17. The non-transitory computer readable medium of claim 13, wherein determining the adjustment for the orientation or position of the camera based on the alignment of the image-based pose with the sensor-based pose comprises:

determining one or more adjustments to a rotation or translation of the camera relative to the orientation and position of the vehicle.