System and method for digital environment reconstruction

• US 10,417,816 B2
• Filed: 06/18/2018
• Issued: 09/17/2019
• Est. Priority Date: 06/16/2017
• Status: Active Grant

• Current Assignee: Nauto, Inc.
• Original Assignee: Nauto, Inc.
• Inventors: Satzoda, Ravi Kumar, Sathyanarayana, Suchitra, Gleeson-May, Michael
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Greene, Daniel L
• Application Number:
  US16/011,269
• Publication Number:
  US 20180365888A1
• Time in Patent Office: 456 Days
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30192 Weather; Meteorology
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

A method for digital environment reconstruction, including: capturing data using an onboard vehicle system; determining a vehicle event and an event class of the vehicle event; extracting a set of agent parameters from the data; generating a virtual model based on the set of agent parameters; and providing the virtual model to a remote system.

Claims

1. A method for digital environment reconstruction, comprising:

capturing an image sequence within a time interval using a monocular camera of an onboard vehicle system rigidly mounted to a vehicle, wherein the image sequence depicts an exterior environment of the vehicle, and wherein the image sequence defines a set of frames;

determining that a rare vehicle event occurred within the time interval;

transmitting the image sequence to a remote computing system in communication with the onboard vehicle system, in response to determining that the rare vehicle event occurred;

extracting a set of agent parameters from the image sequence at the remote computing system, wherein the set of agent parameters defines an agent position, an agent pose, and an agent trajectory of an agent in the exterior environment, wherein extracting comprises;

simultaneously processing the image sequence at a position classifier and a pose classifier, wherein the position classifier extracts the agent position relative to the monocular camera from each frame of the set of frames, and wherein the pose classifier extracts the agent pose relative to the monocular camera from each frame of the set of frames, and

processing each frame of the set of frames, the agent position corresponding to each frame, and the agent pose corresponding to each frame at a trajectory estimator to generate the agent trajectory;

generating a virtual model of the exterior environment of the vehicle based on the set of agent parameters;

providing the virtual model to a vehicle controller, wherein the vehicle controller is configured to generate vehicle control instructions based on the virtual model.

2. The method of claim 1, wherein the position classifier comprises a first convolutional neural network (CNN), and wherein the pose classifier comprises a second CNN.

3. The method of claim 1, wherein determining the rare vehicle event occurred within the time interval comprises:

determining a vehicle event based on the image sequence;

computing a probability associated with the vehicle event; and

determining the rare vehicle event occurred based on a threshold probability exceeding the probability associated with the vehicle event.

4. The method of claim 1, wherein the set of agent parameters defines an agent identifier, wherein the agent identifier comprises personally identifying information (PII), and wherein extracting the set of agent parameters further comprises:

processing the image sequence at an agent classifier, wherein the agent classifier extracts the agent identifier, and

deleting the PII from the set of agent parameters prior to generating the virtual model, thereby anonymizing the virtual model.

5. The method of claim 1, wherein extracting the agent parameters is performed iteratively, and wherein, for each iteration subsequent to a first iteration, extracting comprises:

simultaneously

     1) processing each frame and the agent position at each frame at the position classifier and

     2) processing each frame and the agent pose at each frame at the pose classifier, and

processing each frame, the agent position corresponding to each frame, the agent pose corresponding to each frame, and the agent trajectory at the trajectory estimator.

6. The method of claim 1, further comprising:

determining an event class of the rare vehicle event,

receiving an event class request from the vehicle controller prior to providing the virtual model to the vehicle controller, wherein the event class request corresponds to the event class of the rare vehicle event, and

providing the virtual model to the vehicle controller in response to receiving the event class request.

7. The method of claim 6, further comprising, in response to the event class request:

generating a set of perturbed parameters based on the set of agent parameters, generating a perturbed virtual model based on the set of perturbed parameters, and providing the perturbed virtual model to the vehicle controller.

8. The method of claim 7, wherein each of the set of agent parameters defines a parameter value, and wherein generating each perturbed parameter of the set of perturbed parameters comprises:

randomly sampling a gaussian distribution to generate the perturbed parameter, wherein the gaussian distribution is centered at the parameter value of an agent parameter of the set of agent parameters.

9. A method for digital environment reconstruction, comprising:

capturing image data using a camera of an onboard vehicle system rigidly mounted to a vehicle, wherein the image data depicts an exterior environment of the vehicle;

determining that a rare vehicle event is depicted by the image data, determining an event class of the rare vehicle event, and tagging the image data with the event class to generate tagged image data;

extracting a set of agent parameters from the image data at a remote computing system, wherein the set of agent parameters defines;

an agent identifier, an agent position, and an agent pose;

wherein the agent identifier, the agent position, and the agent pose are extracted from the image data in parallel;

wherein the agent identifier comprises personally identifying information (PII);

deleting the PII from the set of agent parameters, thereby anonymizing the set of agent parameters, and subsequently combining the set of agent parameters into a datafile;

receiving an event data request from a user system, wherein the event data request corresponds to the event class of the rare vehicle event; and

providing the datafile to the user system in response to the event data request.

10. The method of claim 9, wherein the image data comprises an image sequence defining a set of frames, wherein the agent pose and agent position are extracted from each frame of the set of frames, wherein the set of agent parameters defines an agent trajectory, and further comprising extracting the agent trajectory based on the agent pose and agent position corresponding to each frame of the set of frames.

11. The method of claim 10, further comprising providing the tagged image data to the user system in response to the event data request.

12. The method of claim 10, wherein the agent pose is extracted from each frame of the set of frames by a pose classifier, and the agent position is extracted in parallel from each frame of the set of frames by a position classifier.

13. The method of claim 12, wherein the pose classifier comprises a first convolutional neural network (CNN), and wherein the position classifier comprises a second CNN.

14. The method of claim 12, wherein extracting the agent trajectory comprises providing the image sequence, the agent pose corresponding to each frame, and the agent position corresponding to each frame to a trajectory estimator, and generating the agent trajectory at the trajectory estimator based on the image sequence, the agent pose corresponding to each frame, and the agent position corresponding to each frame.

15. The method of claim 14, wherein extracting the agent parameters is performed iteratively, and iterations subsequent to a first iteration of extracting the agent parameters comprises providing the agent pose, agent position, and agent trajectory extracted in the first iteration to the pose classifier, the position classifier, and the trajectory estimator.

16. The method of claim 9, further comprising generating a set of perturbed parameters based on the set of agent parameters, and combining the set of perturbed parameters and the set of agent parameters into the datafile.

17. The method of claim 16, wherein each of the set of agent parameters defines a parameter value, and wherein generating each perturbed parameter of the set of perturbed parameters comprises randomly sampling a gaussian distribution to generate the perturbed parameter, wherein the gaussian distribution is centered at the parameter value of an agent parameter of the set of agent parameters.

18. The method of claim 16, wherein generating the set of perturbed parameters comprises transforming the set of agent parameters based on a noise parameter.

19. The method of claim 18, wherein transforming the set of agent parameters based on the noise parameter comprises simulating sensor noise based on a model of a weather effect on sensor performance, and transforming the set of agent parameters to emulate the weather effect.

20. The method of claim 19, wherein the sensor model comprises at least one of a rangefinder sensor model and an image sensor model.