Method and system for virtual sensor data generation with depth ground truth annotation

• US 10,096,158 B2
• Filed: 03/24/2016
• Issued: 10/09/2018
• Est. Priority Date: 03/24/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Groh, Alexander, Micks, Ashley Elizabeth, Nariyambut Murali, Vidya
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US15/079,540
• Publication Number:
  US 20170278303A1
• Time in Patent Office: 929 Days
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/60 of natural phenomena, e.g. rain, snow, water or plants
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/64 Weathering
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/004 Annotating, labelling

Abstract

Methods and systems for generating virtual sensor data for developing or testing computer vision detection algorithms are described. A system and a method may involve generating a virtual environment. The system and the method may also involve positioning a virtual sensor at a first location in the virtual environment. The system and the method may also involve recording data characterizing the virtual environment, the data corresponding to information generated by the virtual sensor sensing the virtual environment. The system and the method may further involves annotating the data with a depth map characterizing a spatial relationship between the virtual sensor and the virtual environment.

Claims

1. A method, comprising:

generating, by a processor, a virtual environment with a virtual sensor therein;

positioning, by the processor, the virtual sensor on a mobile virtual object in the virtual environment;

generating, by the processor, simulation-generated data characterizing the virtual environment as perceived by the virtual sensor as the mobile virtual object and the virtual sensor move around in the virtual environment;

recording, by the processor, the simulation-generated data; and

annotating, by the processor, the data with a depth map characterizing a spatial relationship between the virtual sensor and the virtual environment.

2. The method of claim 1, further comprising:

moving, by the processor, the virtual sensor from the first location to a second location in the virtual environment such that the virtual sensor is configured to sense the virtual environment from the second location, along a path between the first location and the second location, or both.

3. The method of claim 1, wherein the virtual environment comprises a plurality of virtual objects distributed therewithin, each of the virtual objects either stationary or mobile relative to the virtual sensor, and each of the virtual objects sensible by the virtual sensor.

4. The method of claim 3, wherein the spatial relationship comprises distance information of one or more of the plurality of virtual objects with respect to the virtual sensor.

5. The method of claim 3, wherein the virtual sensor comprises a virtual camera, and wherein the data comprises one or more virtual images of the plurality of virtual objects within the virtual environment as perceived by the virtual camera at the first location.

6. The method of claim 1, wherein the positioning of the virtual sensor at the first location comprises disposing a virtual camera at the first location and setting the virtual camera in a first orientation with respect to the virtual environment, and wherein the data corresponds to information generated by the virtual camera sensing the virtual environment from the first location in the first orientation.

7. The method of claim 6, further comprising:

moving, by the processor, the virtual camera from the first location to a second location in the virtual environment such that the virtual camera is configured to sense the virtual environment from the second location; and

setting, by the processor, the virtual camera in a second orientation with respect to the virtual environment such that the virtual camera is configured to sense the virtual environment in the second orientation.

8. The method of claim 1, wherein the data comprises one or more virtual images, wherein each of the one or more virtual images comprises an array of pixels, and wherein each pixel of the array of pixels is associated with one of a plurality of virtual objects in the virtual environment as perceived by the virtual camera at the first location.

9. The method of claim 8, wherein the depth map comprises an array of elements, each element of the array of elements corresponding to a respective pixel of the array of pixels and comprising a respective numerical value, each numerical value corresponding to a distance in the virtual environment between the virtual sensor and a respective one of the plurality of virtual objects associated with the respective pixel.

10. The method of claim 9, further comprising:

calculating, by the processor, the distance between the virtual sensor and the respective one of the plurality of virtual objects associated with the respective pixel based on a coordinate of the first location and a coordinate of the respective one of the plurality of virtual objects in the virtual environment.

11. The method of claim 1, wherein the data further corresponds to one or more ambient parameters configured to account for at least a weather condition and a lighting condition of the virtual environment.

12. The method of claim 1, wherein the generating of the virtual environment comprises employing at least three-dimensional (3D) modeling techniques and animation techniques.

13. A sensor data generation system, comprising:

one or more processors; and

memory operably connected to the one or more processors, the memory storing a plurality of components executable by the one or more processors, the plurality of components comprising;

a virtual environment programmed to include a plurality of virtual objects;

a sensor model programmed to model a sensor in the virtual environment;

a positioning module programmed to set a location and an orientation of the sensor in the virtual environment, wherein at least one of the virtual objects is a mobile virtual object, and wherein the sensor is on the mobile virtual object;

a simulation module programmed to utilize the virtual environment, the sensor model and the positioning module to produce simulation-generated data characterizing the virtual environment as perceived by the sensor as the mobile virtual object and the virtual sensor move around in the virtual environment to model an output of the sensor in a real-world scenario sensing an actual environment similar to or matching the virtual environment; and

an annotation module programmed to annotate the data with a depth map characterizing a spatial relationship between the sensor and one or more of the plurality of virtual objects in the virtual environment.

14. The sensor data generation system of claim 13, wherein the virtual environment comprises a three-dimensional (3D) environment, and wherein each of the plurality of virtual objects is either stationary or mobile relative to the sensor in the 3D environment.

15. The sensor data generation system of claim 13, wherein the sensor comprises a virtual camera, and wherein the data comprises one or more virtual images of the virtual environment as perceived by the virtual camera at the location and in the orientation.

16. The sensor data generation system of claim 15, wherein each of the one or more virtual images comprises an array of pixels, each pixel of the array of pixels associated with a respective one of the plurality of virtual objects as perceived by the virtual camera at the location and in the orientation.

17. The sensor data generation system of claim 16, wherein the depth map comprises an array of elements, each element of the array of elements corresponding to a respective pixel of the array of pixels and comprising a respective numerical value, each numerical value corresponding to a distance in the virtual environment between the sensor and a respective one of the plurality of virtual objects associated with the respective pixel.

18. The sensor data generation system of claim 17, wherein the distance between the sensor and the respective one of the plurality of virtual objects associated with the respective pixel is determined based on a coordinate of the location and a coordinate of the respective one of the plurality of virtual objects in the virtual environment.

19. The sensor data generation system of claim 13, wherein the memory further stores the data and the depth map.

20. The sensor data generation system of claim 13, wherein the plurality of components comprising:

an ambience module programmed to account for at least a weather condition and a lighting condition of the virtual environment.