Augmenting Real Sensor Recordings With Simulated Sensor Data

• US 20190065933A1
• Filed: 08/31/2017
• Published: 02/28/2019
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Bogdoll, Daniel, Paudel, Shreyasha, Koduri, Tejaswi
• Granted Patent: US 11,487,988 B2
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06F 30/15 Vehicle, aircraft or watercraft design
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08

Abstract

Original sensor data is received from one or more sensors of a vehicle. Free space around the vehicle is identified according to the sensor data, such as by identifying regions where data points have a height below a threshold. A location for an object model is selected from the free space. A plane is fitted to sensor data around the location and the object model is oriented according to an orientation of the plane. Sensing of the object model by a sensor of the vehicle is simulated to obtain simulated data, which is then added to the original sensor data. Sensor data corresponding to objects that would have been obscured by the object model is removed from the original sensor data. Augmented sensor data may be used to validate a control algorithm or train a machine learning model.

Claims

1. A method comprising, by a computing device:

receiving sensor data from a controller of a vehicle;

identifying available space around the vehicle in the sensor data;

placing an object model at a relative location to the vehicle;

simulating sensing of the object model to obtain simulated sensor data; and

adding the simulated sensor data to the sensor data to obtain augmented sensor data.

2. The method of claim 1, comprising:

identifying, by the computing device, a portion of the sensor data that would not be detectable due to placement of the object model at the relative location; and

removing, by the computing device, the portion of the sensor data from the augmented sensor data.

3. The method of claim 2, further comprising:

defining, by the computing device, a shadow cone of the object model with respect to a sensor location on the vehicle; and

searching, by the computing device, for the portion of the sensor data that would not be detectable within the shadow cone.

4. The method of claim 1, further comprising:

identifying, by the computing device, an area of available space around the relative location;

fitting, by the computing device, a plane to the area; and

transforming, by the computing device, the object model according to an orientation of the plane.

5. The method of claim 4, further comprising:

identifying, by the computing device, the area as being covered by the object model;

determining, by the computing device, (a) that the area covers insufficient data points of the sensor data to define the plane; and

in response to determining (a), expanding, by the computing device, the area until the area covers sufficient data points to fit the plane.

6. The method of claim 1, wherein identifying the available space around the vehicle in the sensor data comprises:

evaluating, by the computing device, a height associated data points in the sensor data; and

identifying, by the computing device, as the available space, regions in which the height associated with data points in the regions are less than a threshold height.

7. The method of claim 6, wherein evaluating the height associated with data points in the sensor data comprises defining, by the computing device, the height with respect to a horizontal plane of the vehicle.

8. The method of claim 1, wherein the sensor data includes at least one of LIDAR (light detection and ranging), RADAR (radio detection and ranging), and image data.

9. The method of claim 1, further comprising testing, by the computing device, one of a detection algorithm and a control algorithm according to the augmented sensor data.

10. The method of claim 1, further comprising training, by the computing device, a machine learning model to at least one of detect obstacles and remove objects according to the augmented training data.

11. A system comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:

receive sensor data from a controller of a vehicle;

identify available space around the vehicle in the sensor data;

place an object model at a relative location to the vehicle;

simulate sensing of the object model to obtain simulated sensor data; and

add the simulated sensor data to the sensor data to obtain augmented sensor data.

12. The system of claim 11, wherein the executable code is further effective to cause the one or more processing devices to:

identify a portion of the sensor data that would not be detectable due to placement of the object model at the relative location; and

remove the portion of the sensor data from the augmented sensor data.

13. The system of claim 12, wherein the executable code is further effective to cause the one or more processing devices to:

define a shadow cone of the object model with respect to a sensor location on the vehicle; and

search for the portion of the sensor data that would not be detectable within the shadow cone.

14. The system of claim 11, wherein the executable code is further effective to cause the one or more processing devices to:

identify an area of available space around the relative location;

fit a plane to the area; and

transform the object model according to an orientation of the plane.

15. The system of claim 14, wherein the executable code is further effective to cause the one or more processing devices to:

identify the area as being covered by the object model;

evaluate whether (a) the area covers insufficient data points of the sensor data to define the plane; and

if (a), expand the area until the area covers sufficient data points to fit the plane.

16. The system of claim 11, wherein the executable code is further effective to cause the one or more processing devices to:

evaluate a height associated data points in the sensor data; and

identify, as the available space, regions in which the height associated with data points in the regions are less than a threshold height.

17. The system of claim 16, wherein the executable code is further effective to cause the one or more processing devices to evaluate the height associated with data points in the sensor data by defining the height with respect to a horizontal plane of the vehicle.

18. The system of claim 11, wherein the sensor data includes at least one of LIDAR (light detection and ranging), RADAR (radio detection and ranging), and image data.

19. The system of claim 11, wherein the executable code is further effective to cause the one or more processing devices to test at least one of a detection algorithm and a control algorithm according to the augmented sensor data.

20. The system of claim 11, wherein the executable code is further effective to cause the one or more processing devices to train a machine learning model to at least one of detect obstacles and remove objects according to the augmented training data.