Object tracking

• US 10,318,822 B2
• Filed: 04/06/2017
• Issued: 06/11/2019
• Est. Priority Date: 04/06/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Gao, Peter, Minster, Gautier, Branson, Elliot, Glaser, Stuart
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US15/480,437
• Publication Number:
  US 20180293445A1
• Time in Patent Office: 796 Days
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/10 including control of change-speed gearings
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2510/10 Change speed gearings
• CPC Class Codes: B60W 2510/18 Braking system
• CPC Class Codes: B60W 2510/20 Steering systems
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2710/10 Change speed gearings
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0297 by controlling means in a control room
• CPC Class Codes: G06Q 50/40 Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: G06T 15/20 : Perspective computation
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/127 : to a central station ; Indicators in a central station
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 1/202 : Dispatching vehicles on the basis of a location, e.g. taxi dispatching

Abstract

Methods and systems are provided for tracking an object. The system includes a data receiving module receiving two dimensional imaging data including an object and height map data correlating ground height and location. A two dimensions to three dimensions transformation module determines a location of the object in three dimensional space based on the two dimensional imaging data and the height map data. A tracking module tracks the object using the location of the object.

Claims

1. An object tracking system for a vehicle having one or more vehicle control components, comprising:

a vehicle control module in communication with the one or more control components, the vehicle control module including;

a non-transitory computer readable medium comprising;

a data storage module configured to, by a processor, store in a data storage device height map data correlating ground height and location and generated from lidar data;

a data receiving module configured to, by a processor, receive two dimensional imaging data generated by a camera and including at least one object;

a two dimensions to three dimensions transformation module configured to, by a processor, determine a location of the camera in the height map data based on a real world location of the camera, project the at least one object of the two dimensional imaging data into the height map data based on the location and ray tracing to determine a distance of the object from the camera, and determine a location of the at least one object in a three dimensional space based on the distance;

a tracking module configured to, by a processor, track at least one object using the location of the at least one object in the three dimensional space and generate tracking data based thereon; and

the vehicle control module being configured to actuate at least one of the one or more control components based, in part, on the tracking data.

2. The object tracking system of claim 1, wherein the two dimensions to three dimensions transformation module is further configured to project the at least one object of the two dimensional imaging data into the height map data to determine a ground intersection of the at least one object and, based on the ground intersection and the correlation of ground height and location in the height map data to determine the distance of the object from the camera.

3. The object tracking system of claim 1, wherein the data storage module is configured to store calibration data including camera pose data, and wherein the two dimensions to three dimensions transformation module is configured to project the at least one object of the two dimensional imaging data into the height map data from the camera pose.

4. The object tracking system of claim 1, comprising an object identification module configured to demarcate the at least one object in the two dimensional imaging data to obtain two dimensional object data, and wherein the two dimensions to three dimensions transformation module is configured to determine the location of the at least one object in the three dimensional space based on the two dimensional object data.

5. The object tracking system of claim 4, wherein the object identification module is configured to determine at least one bounding box as the two dimensional object data.

6. The object tracking system of claim 4, wherein the two dimensions to three dimensions transformation module is configured to transform a bottom of the at least one bounding box to a ground intersection in the height map data in determining the location of the at least one object in three dimensional space.

7. The object tracking system of claim 1, comprising a visual classification module configured to run a neural network to classify the at least one object and to determine dimensions of the at least one object based on the classification, wherein the tracking module is configured to track at least one object using the location of the at least one object and the dimensions of the at least one object.

8. The object tracking system of claim 7, comprising an object identification module configured to demarcate the at least one object to obtain at least one bounding box, wherein the visual classification module is configured to perform bounding box regression on the at least one bounding box using the neural network to obtain at least one regressed bounding box, and wherein the two dimensions to three dimensions transformation module is configured to determine the location of the at least one object in three dimensional space based on the two dimensional imaging data, the height map data and the at least one regressed bounding box.

9. The object tracking system of claim 1, wherein the data receiving module is configured to receive three dimensional imaging data including at least one other object, wherein the tracking module is configured to track the at least one object based on the two dimensional imaging data and the at least one other object based on the three dimensional imaging data.

10. An autonomous vehicle, comprising:

a sensor system comprising at least one camera that generates two dimensional imaging data including at least one object;

a data storage device configured to store height map data correlating ground height and location;

a processor configured to;

determine a location of the at least one camera in the height map data based on a real world location of the autonomous vehicle and a pose of the at least one camera,

project the at least one object of the two dimensional imaging data into the height map data based on the location and ray tracing to determine a distance of the object from the camera,

determine a location of the at least one object in a three dimensional space based on the distance;

track at least one object using the location of the at least one object and to responsively output tracking data; and

an autonomous vehicle control module configured to control one or more vehicle components based, in part, on the tracking data.

11. The autonomous vehicle of claim 10, wherein the processor is configured to transform a height of the at least one object in the two dimensional imaging data to a ground intersection in the height map data and determine the distance based on the ground intersection.

12. The autonomous vehicle of claim 10, wherein the processor is configured to demarcate the at least one object to obtain at least one bounding box and to perform bounding box regression on the at least one bounding box using a neural network to obtain at least one regressed bounding box, and wherein processor is configured to determine the location of the at least one object in the three dimensional space based on the at least one regressed bounding box.

13. The autonomous vehicle of claim 12, wherein the processor is configured to transform a bottom of the at least one bounding box to a ground intersection in the height map data in determining the location of the at least one object in three dimensional space.

14. A method of tracking at least one object, comprising:

storing, via a processor, height map data correlating ground height and real world location;

receiving, via a processor, two dimensional imaging data including at least one object;

determining, via a processor, a location of the camera in the height map data based on a real world location of the camera;

projecting, via a processor, the at least one object of the two dimensional imaging data into the height map data based on the location and ray tracing to determine a distance of the object from the camera;

determining, via a processor, a location of the at least one object in a three dimensional space based on the distance;

tracking, via a processor, at least one object using the location of the at least one object for use in autonomous vehicle control; and

exercising vehicle control based, in part, on the tracking.

15. The method of claim 14, comprising receiving, via a processor, calibration data including camera pose data spatially relating the autonomous vehicle and the at least one camera, determining, via a processor, a camera pose in the height map data based on the location of the vehicle and the camera pose data and ray tracing the at least one object of the two dimensional imaging data into the height map data from the camera pose in the height map data in determining a location of the at least one object in three dimensional space.

16. The method of claim 14, comprising determining, via a processor, a ground intersection in the height map data based on the at least one object in the two dimensional imaging data in determining location of the at least one object in three dimensional space.