Object identification and labeling tool for training autonomous vehicle controllers

• US 10,535,191 B2
• Filed: 02/27/2018
• Issued: 01/14/2020
• Est. Priority Date: 12/21/2017
• Status: Active Grant

• Current Assignee: Luminar Technologies, Inc.
• Original Assignee: Luminar Technologies, Inc.
• Inventors: Sachdeva, Prateek, Trofymov, Dmytro
• Primary Examiner(s): Zhai, Kyle
• Application Number:
  US15/906,529
• Publication Number:
  US 20190197778A1
• Time in Patent Office: 686 Days
• Field of Search: None
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 : Image-based rendering
• CPC Class Codes: G06T 15/30 : Clipping
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/20 : Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/30256 : Lane; Road marking
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 2210/12 : Bounding box
• CPC Class Codes: G06T 2219/004 : Annotating, labelling
• CPC Class Codes: G06T 2219/028 : Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 2219/2016 : Rotation, translation, scaling
• CPC Class Codes: G06T 7/12 : Edge-based segmentation
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06V 10/7788 : the supervisor being a human, e.g. interactive learning with a human teacher
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/08 : Detecting or categorising vehicles
• CPC Class Codes: H04N 13/361 : Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background

Abstract

Techniques for identifying and labeling distinct objects within 3-D images of environments in which vehicles operate, to thereby generate training data used to train models that autonomously control and/or operate vehicles, are disclosed. A 3-D image may be presented from various perspective views (in some cases, dynamically), and/or may be presented with a corresponding 2-D environment image in a side-by-side and/or a layered manner, thereby allowing a user to more accurately identify groups/clusters of data points within the 3-D image that represent distinct objects. Automatic identification/delineation of various types of objects depicted within 3-D images, automatic labeling of identified/delineated objects, and automatic tracking of objects across various frames of a 3-D video are disclosed. A user may modify and/or refine any automatically generated information. Further, at least some of the techniques described herein are equally applicable to 2-D images.

Claims

1. A computer-implemented method for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the method comprising:

displaying, on a user interface of one or more computing devices, a three-dimensional (3-D) image of an environment in which vehicles operate, the 3-D image depicting one or more physical objects located in the environment;

providing, on the user interface, a paint user control for use by a user to indicate areas within images displayed on the user interface;

receiving, via a user activation of the paint user control, an indication of a data point within the 3-D image;

based upon the indicated data point within the 3-D image,

(i) automatically determining, without any additional user input aside from the user activation, boundaries of a depiction of an essentially planar surface area within the 3-D image, the depiction of the essentially planar surface area including the indicated data point and other data points surrounding the indicated data point within the 3-D image, and the essentially planar surface area being a particular type of surface area,

(ii) automatically determining, from a plurality of visual properties, each of which denotes a different type of surface area, a particular visual property corresponding to the particular type of the essentially planar surface area, and

(iii) automatically modifying, by the one or more computing devices and based on the determined boundaries, the particular visual property of the data points included within the boundaries of the depiction of the essentially planar surface area of the particular type;

obtaining, by the one or more computing devices, an indication of a particular label for the automatically determined depiction of the essentially planar surface area of the 3-D image; and

storing, by the one or more computing devices in one or more tangible, non-transitory memories, an indication of an association between data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label, thereby distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image.

2. The method of claim 1, wherein the particular label is included in a plurality of labels of a plurality of areas and/or physical objects, and wherein each label included in the plurality of labels corresponds to a respective visual property.

3. The computer-implemented method of claim 1, wherein the user activation of the paint user control consists of only a single mouse click.

4. The computer-implemented method of claim 1, wherein the automatically determined depiction of the essentially planar surface area within the 3-D image is a first area representing a first essentially planar surface area of a first particular type, and the visual property is a first visual property, and the method further comprises modifying a second area within the 3-D image with a second visual property different than the first visual property, the second area being another subset of the total area of the 3-D image representing a second essentially planar surface area of a second particular type, and the modifying of the second area based on a usage of the paint user control as a virtual paint brush, a virtual paint spill, or another type of usage of the paint user control.

5. The computer-implemented method of claim 4, wherein the usage of the paint user control is as the virtual paint brush, and modifying the second area with the second visual property comprises the user manipulating the virtual paint brush across the second area within the 3-D image.

6. The computer-implemented method of claim 5, wherein a width of the virtual paintbrush is modifiable.

7. The computer-implemented method of claim 1, wherein receiving, via the user activation of the paint user control, the indication of the data point within the 3-D image comprises receiving an indication of the user activation of the paint user control while a cursor or electronic pointer controlled by the paint user control hovers over or points to a location of the data point.

8. The computer-implemented method of claim 1, further comprising receiving an indication of a particular type of object depicted in the 3-D image, the particular type of object corresponding to the essentially planar surface area.

9. The computer-implemented method of claim 1, wherein the essentially planar surface area includes a road.

10. The computer-implemented method of claim 1, wherein the essentially planar surface area includes a ground.

11. The computer-implemented method of claim 1, wherein automatically modifying the particular visual property of the data points included within the depiction of the essentially planar surface area comprises automatically applying the particular visual property to the data points included within the depiction of the essentially planar surface area.

12. The computer-implemented method of claim 11, further comprising receiving, via the paint user control, an indication of a portion of the data points included within the depiction of the essentially planar surface area from which the particular visual property is to be removed.

13. The computer-implemented method of claim 1, wherein automatically modifying the particular visual property of the data points included within the depiction of the essentially planar surface area comprises removing the particular visual property from the data points included within the depiction of the essentially planar surface area of the 3-D image.

14. The computer-implemented method of claim 1, wherein automatically determining the boundaries of the depiction of the essentially planar surface area within the 3-D image comprises automatically determining the boundaries of the depiction of the essentially planar surface area within the 3-D image by utilizing an object identification model that has been trained based on identified objects depicted within a plurality of historical images of one or more environments in which vehicles operate.

15. The computer-implemented method of claim 1, wherein 3-D image of the environment comprises a dataset generated by one or more active sensing devices or systems.

16. The computer-implemented method of claim 15, wherein the one or more active sensing devices or systems include one or more light detection and ranging (LIDAR) devices.

17. The computer-implemented method of claim 15, wherein dataset is a point cloud dataset.

18. The computer-implemented method of claim 1, wherein the 3-D environment image is an interactive, three-dimensional (3-D) image of the environment.

19. The computer-implemented method of claim 1, wherein storing the indication of the association between the data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label comprises storing data indicative of the particular label in association with each pixel or data point included in the automatically determined depiction of the essentially planar surface area of the 3-D image.

20. The computer-implemented method of claim 1, wherein the method further comprises:

receiving, via the user interface, an instruction to hide respective images of one or more specific objects depicted within the 3-D image of the environment; and

based on the received hiding instruction, rendering non-visible the respective images of the one or more specific objects depicted within the 3-D image of the environment while maintaining respective levels of visibility of other objects and/or areas within the 3-D image of the environment, thereby preventing the respective images of the one or more specific objects from being indicated via the paint user control.

21. The computer-implemented method of claim 1, further comprising providing the stored data indicative of the association between the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label as training data for a machine-learning model that generates control signals to autonomously control one or more autonomous operations of a vehicle.

22. The computer-implemented method of claim 1, wherein receiving the indication of the particular label is executed prior to receiving the indication of the data point within the 3-D image.

23. The computer-implemented method of claim 1, wherein receiving the indication of the data point within the 3-D image is executed prior to receiving the indication of the particular label.

24. A system for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the system comprising:

a communication module;

one or more processors; and

one or more non-transitory, tangible memories coupled to the one or more processors and storing computer executable instructions thereon that, when executed by the one or more processors, cause the system to;

display, on a user interface, a three-dimensional (3-D) image of an environment in which vehicles operate, the 3-D image depicting one or more physical objects located in the environment;

provide, on the user interface, a paint user control for use by a user to indicate areas within the 3-D image;

receive, via the communication module, an indication of a user activation of the paint user control at a location of a data point within the 3-D image;

based upon the indicated location of the data point, (i) automatically determine, without any additional user input aside from the user activation, boundaries of a depiction of an essentially planar surface area within the 3-D image, the depiction of the essentially planar surface area including the indicated data point and other data points surrounding the indicated data point within the 3-D image, and the essentially planar surface area being a particular type of surface area, (ii) automatically determine, from a plurality of visual properties, each of which denotes a different type of surface area, a particular visual property corresponding to the particular type of the essentially planar surface area, and (iii) automatically modify, by the one or more computing devices and based on the determined boundaries, the particular visual property of the data points included within the boundaries of the depiction of the essentially planar surface area of the particular type;

obtain an indication of a particular label for the automatically determined depiction of the essentially planar surface area; and

store, in the one or more tangible, non-transitory memories, an indication of an association between data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label, thereby distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image.

25. The system of claim 24, wherein the particular label is included in a plurality of labels of a plurality of areas and/or objects, and wherein each label included in the plurality of labels corresponds to a respective visual property.

26. The system of claim 24, wherein the automatic determination of the depiction of the essentially planar surface area within the 3-D image comprises a utilization of an object identification model that has been trained based on identified objects depicted within a plurality of historical images of one or more environments in which vehicles operate.

27. The system of claim 24, wherein the stored association distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image is included in a set of training data for a machine-learning model that generates control signals to autonomously control one or more operations of a vehicle.