Temporal data associations for operating autonomous vehicles

• US 10,328,934 B2
• Filed: 03/20/2017
• Issued: 06/25/2019
• Est. Priority Date: 03/20/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Minster, Gautier
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Louie, Wae L
• Application Number:
  US15/463,978
• Publication Number:
  US 20170242442A1
• Time in Patent Office: 827 Days
• Field of Search: 701 22, 701 25, 701 26
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Systems and method are provided for controlling a vehicle. In one embodiment, a method includes: selecting, by a controller onboard the vehicle, first data for a region from a first device onboard the vehicle based on a relationship between a time associated with the first data and a frequency associated with a second device, obtaining, by the controller, second data from the second device, the second data corresponding to the region, correlating, by the controller, the first data and the second data, and determining, by the controller, a command for operating one or more actuators onboard the vehicle in a manner that is influenced by the correlation between the first data and the second data.

Claims

1. A method of controlling a vehicle, the method comprising:

obtaining, by a controller onboard the vehicle from a ranging device onboard the vehicle, ranging data corresponding to a field of view of a camera onboard the vehicle, the ranging device scanning an environment around the vehicle at a first frequency;

selecting, by the controller, an image from a plurality of images captured by the camera based on a relationship between a timestamp associated with the image and the first frequency associated with the ranging device, wherein the plurality of images are captured at a second frequency different from the first frequency;

correlating, by the controller, the image and the ranging data; and

determining, by the controller, a command for operating one or more actuators onboard the vehicle in a manner that is influenced by the correlation between the image and the ranging data.

2. The method of claim 1, wherein:

correlating the image and the ranging data comprises assigning depths to the image based on the ranging data; and

determining the command comprises determining a route for the vehicle in a manner that is influenced by the depths assigned to the image.

3. The method of claim 1, further comprising:

obtaining, by the controller, first configuration data associated with the camera;

obtaining, by the controller, second configuration data associated with the ranging device; and

calculating, by the controller, a sampling time associated with the ranging device based at least in part on a relationship between the first configuration data and the second configuration data and the frequency of the ranging device, wherein selecting the image comprises selecting the image based on a difference between the timestamp associated with the image and the sampling time.

4. The method of claim 3, wherein calculating the sampling time comprises determining the sampling time when a line of sight associated with the ranging device is aligned with a line of sight associated with the camera.

5. The method of claim 1, wherein determining the command comprises determining a route for autonomously operating the vehicle based at least in part on an object in the region in a manner that is influenced by the correlation between the image and the ranging data.

6. A vehicle, comprising:

a camera onboard the vehicle capturing a plurality of images of a field of view at a first frequency;

a ranging device onboard the vehicle scanning an environment around the vehicle at a second frequency different from the first frequency to obtain ranging data;

one or more actuators onboard the vehicle; and

a controller that, by a processor, selects an image of the plurality of images based on a relationship between a timestamp associated with the image and the second frequency associated with the second device, correlates the image with a subset of the ranging data corresponding to the field of view, and autonomously operates the one or more actuators onboard the vehicle in a manner that is influenced by the correlation between the image and the subset of ranging data.

7. The vehicle of claim 6, further comprising a data storage element to maintain first calibration data associated with the camera and second calibration data associated with the ranging device, wherein the controller is coupled to the data storage element to:

determine a sampling time corresponding to alignment of the ranging device with the field of view based at least in part on the second frequency and a relationship between the first calibration data and the second calibration data; and

select the image of the plurality of images based on a difference between the timestamp of the first image and the sampling time.

8. The vehicle of claim 7, wherein the controller identifies the subset of the ranging data based on the sampling time.

9. The vehicle of claim 7, the second calibration data indicating a reference orientation for the ranging device scanning the environment, wherein the controller receives an indication from the ranging device in response to the ranging device traversing the reference orientation and calculates the sampling time based on a scan start time associated with the indication and an orientation of the field of view relative to the reference orientation.

10. The vehicle of claim 6, wherein the first frequency is greater than the second frequency.

11. A method of controlling a vehicle, the method comprising:

obtaining, by a control module onboard the vehicle, a plurality of images from a camera onboard the vehicle, the camera capturing the plurality of images of a field of view at a first frequency;

obtaining, by the control module, ranging data from a ranging device onboard the vehicle, the ranging device scanning an environment around the vehicle at a second frequency different from the first frequency;

correlating, by the control module, a first image of the plurality of images with a subset of the ranging data corresponding to the field of view based on a relationship between a timestamp associated with the first image and the second frequency;

determining, by the control module, a command for operating one or more actuators onboard the vehicle in a manner that is influenced by the correlation between the first image and the subset of the ranging data; and

autonomously operating, by the control module, the one or more actuators in accordance with the command.

12. The method of claim 11, further comprising:

determining, by the control module, a sampling time corresponding to alignment of a line of sight of the ranging device with the field of view; and

selecting, by the control module, the first image from among the plurality of images based on a difference between the timestamp and the sampling time.

13. The method of claim 12, further comprising:

obtaining, by the control module, a starting orientation for the scanning of the environment by the ranging device; and

obtaining, by the control module, an orientation for the field of view of the camera, wherein determining the sampling time comprises calculating the sampling time based on a difference between the starting orientation and the orientation for the field of view.

14. The method of claim 13, further comprising receiving, by the control module, an indication from the ranging device at a scan start time in response to the ranging device traversing the reference orientation, wherein calculating the sampling time comprises:

calculating an offset based on the difference and the second frequency; and

adding the offset to the scan start time to obtain the sampling time.

15. The method of claim 11, wherein correlating comprises the assigning depths to the first image based on the subset.

16. The method of claim 11, wherein correlating comprises identifying a portion of the subset corresponding to an object based on the first image.

17. The method of claim 16, further comprising classifying the object as an object type based on the portion of the subset of the ranging data, wherein the determining the command comprises determining the command in a manner that is influenced by the object type.