OBJECT TRACKING

• US 20180293445A1
• Filed: 04/06/2017
• Published: 10/11/2018
• Est. Priority Date: 04/06/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Gao, Peter, Minster, Gautier, Branson, Elliot, Glaser, Stuart
• Granted Patent: US 10,318,822 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/10 including control of change-speed gearings
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2510/10 Change speed gearings
• CPC Class Codes: B60W 2510/18 Braking system
• CPC Class Codes: B60W 2510/20 Steering systems
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2710/10 Change speed gearings
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0297 by controlling means in a control room
• CPC Class Codes: G06Q 50/40 Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: G06T 15/20 : Perspective computation
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/127 : to a central station ; Indicators in a central station
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 1/202 : Dispatching vehicles on the basis of a location, e.g. taxi dispatching

Abstract

Methods and systems are provided for tracking an object. The system includes a data receiving module receiving two dimensional imaging data including an object and height map data correlating ground height and location. A two dimensions to three dimensions transformation module determines a location of the object in three dimensional space based on the two dimensional imaging data and the height map data. A tracking module tracks the object using the location of the object.

Claims

1. An object tracking system, comprising:

a data receiving module configured to, by a processor, receive two dimensional imaging data including at least one object and to receive height map data correlating ground height and location;

a two dimensions to three dimensions transformation module configured to, by a processor, determine a location of the at least one object in three dimensional space based on the two dimensional imaging data and the height map data; and

a tracking module configured to, by a processor, track at least one object using the location of the at least one object.

2. The object tracking system of claim 1, wherein the two dimensions to three dimensions transformation module is configured to project the at least one object of the two dimensional imaging data into the height map data to determine a ground intersection for the at least one object and, based on the ground intersection and the correlation of ground height and location in the height map data to determine the location of the at least one object.

3. The object tracking system of claim 1, wherein the two dimensional imaging data is received from at least one camera, wherein the data receiving module is configured to receive a real world location of the at least one camera, and wherein the two dimensions to three dimensions transformation module is configured to locate the at least one camera in the height map data based on the real world location of the at least one camera and to project the at least one object of the two dimensional imaging data into the height map data from the at least one camera location in the height map data in determining a location of the at least one object in three dimensional space.

4. The object tracking system of claim 1, wherein the two dimensional imaging data is received from at least one camera, and wherein the data receiving module is configured to receive calibration data including camera pose data, and wherein the two dimensions to three dimensions transformation module is configured to determine camera pose in the height map data and to project the at least one object of the two dimensional imaging data into the height map data from the camera pose in the height map data in determining a location of the at least one object in three dimensional space.

5. The object tracking system of claim 1, comprising an object identification module configured to demarcate the at least one object to obtain two dimensional object data, and wherein the two dimensions to three dimensions transformation module is configured to determine the location of the at least one object in three dimensional space based on the two dimensional imaging data, the height map data and the two dimensional object data.

6. The object tracking system of claim 5, wherein the object identification module is configured to determine at least one bounding box as the two dimensional object data.

7. The object tracking system of claim 5, wherein the two dimensions to three dimensions transformation module is configured to transform a bottom of the at least one bounding box to a ground intersection in the height map data in determining the location of the at least one object in three dimensional space.

8. The object tracking system of claim 1, comprising a visual classification module configured to run a neural network to classify the at least one object and to determine dimensions of the at least one object based on the classification, wherein the tracking module is configured to track at least one object using the location of the at least one object and the dimensions of the at least one object.

9. The object tracking system of claim 8, comprising an object identification module configured to demarcate the at least one object to obtain at least one bounding box, wherein the visual classification module is configured to perform bounding box regression on the at least one bounding box using the neural network to obtain at least one regressed bounding box, and wherein the two dimensions to three dimensions transformation module is configured to determine the location of the at least one object in three dimensional space based on the two dimensional imaging data, the height map data and the at least one regressed bounding box.

10. The object tracking system of claim 1, wherein the tracking module is configured to track the at least one object using the location of the at least one object and to responsively output tracking data, wherein the system comprises a vehicle control module configured to exercise autonomous vehicle control based, in part, on the tracking data.

11. The object tracking system of claim 1, wherein the data receiving module is configured to receive three dimensional imaging data including at least one other object, wherein the tracking module is configured to track the at least one object based on the two dimensional imaging data and the at least one other object based on the three dimensional imaging data.

12. An autonomous vehicle, comprising:

a sensor system comprising at least one camera for obtaining two dimensional imaging data;

a data receiving module configured to receive two dimensional imaging data including at least one object and to receive height map data correlating ground height and location;

a processor configured to;

determine a location of the at least one object in three dimensional space based on the two dimensional imaging data and the height map data; and

track at least one object using the location of the at least one object and to responsively output tracking data; and

an autonomous vehicle control module configured to exercise autonomous vehicle control based, in part, on the tracking data.

13. The autonomous vehicle of claim 12, wherein the sensor system comprises a localization system for determining a location of the autonomous vehicle, wherein the data receiving module is configured to receive calibration data including camera pose data spatially relating the autonomous vehicle and the at least one camera, and wherein the processor is configured to determine camera pose in the height map data based on the location of the autonomous vehicle and the camera pose data and to ray trace the at least one object of the two dimensional imaging data into the height map data from the camera pose in the height map data in determining a location of the at least one object in three dimensional space.

14. The autonomous vehicle of claim 12, wherein the processor is configured to transform a height of the at least one object in the two dimensional imaging data to a ground intersection in the height map data in determining location of the at least one object in three dimensional space.

15. The autonomous vehicle of claim 12, wherein the processor is configured to demarcate the at least one object to obtain at least one bounding box and to perform bounding box regression on the at least one bounding box using a neural network to obtain at least one regressed bounding box, and wherein processor is configured to determine the location of the at least one object in three dimensional space based on the two dimensional imaging data, the height map data and the at least one regressed bounding box.

16. The autonomous vehicle of claim 15, wherein the processor is configured to transform a bottom of the at least one bounding box to a ground intersection in the height map data in determining the location of the at least one object in three dimensional space.

17. A method of tracking at least one object, comprising:

receiving, via a processor, two dimensional imaging data including at least one object and receiving height map data correlating ground height and real world location;

determining, via a processor, a real world location of the at least one object in three dimensional space based on the two dimensional imaging data and the height map data;

tracking, via a processor, at least one object using the location of the at least one object for use in autonomous vehicle control.

18. The method of claim 17, wherein the tracking step comprises providing output tracking data, and the wherein the method comprises exercising autonomous vehicle control based, in part, on the tracking data.

19. The method of claim 17, comprising receiving a real world location of an autonomous vehicle comprising at least one camera for obtaining the two dimensional imaging data, receiving, via a processor, calibration data including camera pose data spatially relating the autonomous vehicle and the at least one camera, determining, via a processor, a camera pose in the height map data based on the location of the vehicle and the camera pose data and ray tracing the at least one object of the two dimensional imaging data into the height map data from the camera pose in the height map data in determining a location of the at least one object in three dimensional space.

20. The method of claim 17, comprising determining, via a processor, a ground intersection in the height map data based on the at least one object in the two dimensional imaging data in determining location of the at least one object in three dimensional space.