Object identification and labeling tool for training autonomous vehicle controllers

• US 10,169,678 B1
• Filed: 02/27/2018
• Issued: 01/01/2019
• Est. Priority Date: 12/21/2017
• Status: Active Grant

• Current Assignee: Luminar Technologies, Inc.
• Original Assignee: Luminar Technologies, Inc.
• Inventors: Sachdeva, Prateek, Trofymov, Dmytro
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US15/906,676
• Time in Patent Office: 308 Days
• Field of Search: None
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 : Image-based rendering
• CPC Class Codes: G06T 15/30 : Clipping
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/20 : Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/30256 : Lane; Road marking
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 2210/12 : Bounding box
• CPC Class Codes: G06T 2219/004 : Annotating, labelling
• CPC Class Codes: G06T 2219/028 : Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 2219/2016 : Rotation, translation, scaling
• CPC Class Codes: G06T 7/12 : Edge-based segmentation
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06V 10/7788 : the supervisor being a human, e.g. interactive learning with a human teacher
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/08 : Detecting or categorising vehicles
• CPC Class Codes: H04N 13/361 : Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background

Abstract

Techniques for identifying and labeling distinct objects within 3-D images of environments in which vehicles operate, to thereby generate training data used to train models that autonomously control and/or operate vehicles, are disclosed. A 3-D image may be presented from various perspective views (in some cases, dynamically), and/or may be presented with a corresponding 2-D environment image in a side-by-side and/or a layered manner, thereby allowing a user to more accurately identify groups/clusters of data points within the 3-D image that represent distinct objects. Automatic identification/delineation of various types of objects depicted within 3-D images, automatic labeling of identified/delineated objects, and automatic tracking of objects across various frames of a 3-D video are disclosed. A user may modify and/or refine any automatically generated information. Further, at least some of the techniques described herein are equally applicable to 2-D images.

Claims

1. A computer-implemented method for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the method comprising:

displaying, on a first display area of a user interface, a three-dimensional (3-D) video of an environment in which vehicles operate, the 3-D environment video including respective 3-D images of one or more physical objects located in the environment;

displaying, on a second display area of the user interface and in a time-synchronized manner with the 3-D video, a two-dimensional (2-D) video of at least a portion of the environment depicted in the 3-D environment video so that an image of the environment that was obtained at a particular time and that is included in the 2-D video is presented simultaneously on the user interface with an image of the environment that was obtained at the particular time and that is included in the 3-D video;

receiving, via one or more user controls provided by the user interface, an indication of a boundary of a particular physical object depicted within a 3-D environment image included in the 3-D video;

generating data indicative of the boundary of the particular physical object within the 3-D environment image;

receiving an indication of a particular label for the particular physical object;

associating the particular label for the particular physical object with the data indicative of the boundary of the particular physical object within the 3-D environment image, thereby distinguishing a set of data points that are representative of the particular physical object within the 3-D environment image from other data points included in the 3-D environment image; and

storing an indication of the association between the particular label and the data indicative of the boundary of the particular physical object within the 3-D environment image in one or more tangible memories as a part of a training data set utilized to train one or more machine-learning based models, the one or more machine-learning based models used to autonomously operate vehicles.

2. The computer-implemented method of claim 1, wherein:

the 3-D environment image is displayed from a first perspective view on the user interface; and

the method further comprises;

receiving, via the one or more user controls provided by the user interface, a user instruction to change the 3-D environment image from being displayed on the user interface from the first perspective view to being displayed from a second perspective view; and

based on the received user instruction, displaying, on the user interface, the 3-D environment image from the second perspective view, including modifying a perspective view of the image of the particular physical object within the 3-D environment image.

3. The computer-implemented method of claim 2, further comprising:

automatically determining an interim graphical representation of the boundary of the particular physical object depicted within the second perspective view of the 3-D environment image;

displaying the interim graphical representation of the 3-D environment image displayed from the second perspective view;

receiving an indication of a user modification to the interim graphical representation;

altering the interim graphical representation based on the received user modification to generate an altered graphical representation of the boundary of the particular physical object depicted within the second perspective view of the 3-D environment image;

generating, based on the altered graphical representation of the boundary of the particular physical object within the second perspective view of the 3-D environment image, data indicative of the boundary of the particular physical object within the second perspective view of the 3-D environment image; and

storing, in the one or more tangible memories, the data indicative of the boundary of the particular physical object within the second perspective view of the 3-D environment image in association with the data indicative of the particular label descriptive of the particular physical object, the data indicative of the boundary of the particular physical object within the second perspective view of the 3-D environment image generated based upon the second graphical representation.

4. The method of claim 1,

further comprising:

layering at least a portion of a 2-D environment image included in the 2-D video of the environment with the 3-D environment image, the 2-D environment image including a 2-D image of the particular physical object that is depicted in the 3-D environment image, thereby generating a composite image of the particular physical object, the composite image of the particular physical object including a portion of the 2-D image of the particular physical object and a portion of the 3-D environment image in which the particular physical object is depicted; and

displaying, on the user interface, the composite image of the particular physical object within the 3-D environment image; and

wherein the indicated boundary of the particular physical object depicted within the 3-D environment image is an indicated, graphical representation of the boundary of the composite image depicted within the 3-D environment image.

5. The computer-implemented method of claim 4,

wherein the 3-D environment image is a first 3-D environment image included in a first frame of the 3-D video displayed on the user interface, the 2-D environment image is a first 2-D environment image, and the composite image of the particular physical object is a first composite image of the particular physical object;

and

the method further comprises;

layering at least one of a part of the first 2-D environment image or a part of second 2-D environment image in which the particular physical object is depicted with a second 3-D environment image in which the particular physical object is depicted and that is included in a second frame of the 3-D video, thereby generating a second composite image of the particular physical object; and

displaying the second composite image of the particular physical object within the second frame of the 3-D video on the user interface.

6. The computer-implemented method of claim 5, further comprising:

receiving, via the one or more user controls, an indication of a boundary of the particular physical object depicted by the second composite image within the second frame of the 3-D video;

generating data indicative of the boundary of the particular physical object within the second frame of the 3-D video;

associating the data indicative of the boundary of the particular physical object within the second frame of the 3-D video with the particular label, thereby distinguishing a set of data points that are representative of the particular physical object within the second frame of the 3-D video from other data points included in the second frame; and

storing an indication of the association between the particular label and the data indicative of the boundary of the particular physical object within the second frame of the 3-D video in the one or more tangible memories as another part of the training data set.

7. The computer-implemented method of claim 6, wherein:

the indicated, graphical representation of the boundary of the composite image depicted within the first frame of the 3-D video is a first graphical representation of the boundary of the composite image;

the method further comprises automatically determining an interim graphical representation of the boundary of the particular physical object depicted by the second composite image within the second frame of the 3-D video, and displaying the interim graphical representation within the second frame of the 3-D video on the user interface;

receiving the indication of the boundary of the particular physical object depicted by the second composite image within the second frame of the 3-D video comprises receiving, via the one or more user controls, an indication of a user modification to the interim graphical representation displayed within the second frame of the 3-D video on the user interface; and

the method further comprises altering, based upon the received user modification, the interim graphical representation to generate a second graphical representation of the boundary of the particular physical object depicted by the second composite image within the second frame of the 3-D video; and

generating the data indicative of the boundary of the particular physical object within the second frame of the 3-D video is based upon the second graphical representation.

8. The computer-implemented method of claim 4, wherein layering the at least the portion of the 2-D environment image with the 3-D environment image comprises receiving, via the one or more user controls of the user interface, a user instruction to drag-and-drop or copy the at least a portion of the 2-D environment image onto the 3-D environment image.

9. A computer-implemented method for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the method comprising:

displaying, on a user interface, a three-dimensional (3-D) video of an environment in which vehicles operate, the 3-D environment video including respective 3-D images of one or more physical objects located in the environment;

dynamically varying a perspective view from which the 3-D video is displayed on the user interface across multiple perspective views during a presentation of the 3-D video responsive to one or more user instructions that are received, via the user interface, during the presentation of the 3-D video;

receiving, via one or more user controls provided by the user interface, an indication of a boundary of a particular physical object depicted within a 3-D environment image included in the 3-D video;

generating data indicative of the boundary of the particular physical object within the 3-D environment image;

receiving an indication of a particular label for the particular physical object;

associating the particular label for the particular physical object with the data indicative of the boundary of the particular physical object within the 3-D environment image, thereby distinguishing a set of data points that are representative of the particular physical object within the 3-D environment image from other data points included in the 3-D environment image; and

storing an indication of the association between the particular label and the data indicative of the boundary of the particular physical object within the 3-D environment image in one or more tangible memories as a part of a training data set utilized to train one or more machine-learning based models, the one or more machine-learning based models used to autonomously operate vehicles.

10. The method of claim 9, wherein the indication of the boundary of the particular physical object depicted within the 3-D environment image included in the 3-D video is a graphical representation displayed on the 3-D environment image, and the method further comprises varying the graphical representation of the boundary of the particular physical object in conjunction with varying the perspective view from which the 3-D video is displayed on the user interface.

11. The method of claim 10, wherein the 3-D video including the 3-D environment image is displayed from a first perspective view on the user interface, and the method further comprises:

based on the 3-D video being changed to be displayed from a second perspective view, automatically determining an interim graphical representation of the boundary of the particular physical object depicted within the 3-D environment image of the 3-D video displayed from the second perspective view;

displaying the interim graphical representation within the second perspective view of the 3-D environment image;

receiving an indication of a user modification to the interim graphical representation;

altering the interim graphical representation based on the received user modification to generate a refined graphical representation of the boundary the particular physical object depicted within the particular 3-D environment image displayed from the second perspective view; and

modifying, based on the refined graphical representation, the data indicative of the boundary the particular physical object within the 3-D environment image that is associated with the particular label for the particular physical object.

12. The method of claim 9, wherein the indication of the boundary of the particular physical object depicted within the 3-D environment image included in the 3-D video is a graphical representation displayed on the 3-D environment image, the 3-D environment image is included in a first frame of the 3-D video, and the method further comprises varying the graphical representation of the boundary of the particular physical object in concert with varying depictions of the particular physical object across multiple frames of the 3-D video.

13. The method of claim 12, wherein the 3-D environment image is displayed in a first frame of the 3-D video, the graphical representation of the boundary of the particular physical object depicted within the first frame of the 3-D video is a first graphical representation, and the method further comprises:

displaying, on the user interface, a second frame of the 3-D video in which the particular physical object is depicted;

automatically determining an interim graphical representation of the boundary of the particular physical object depicted within the second frame of the 3-D video;

displaying the interim graphical representation within the second frame of the 3-D video;

receiving an indication of a user modification to the interim graphical representation;

changing the interim graphical representation based on the received user modification to generate a second graphical representation of the boundary the particular physical object depicted within the second frame of the 3-D video;

generating, based on the second graphical representation, data indicative of the boundary of the particular physical object within the 3-D environment image displayed in the second frame of the 3-D video; and

storing, as a further part of the training data set, an indication of an association between the particular label corresponding to the particular physical object and the data indicative of the boundary the particular physical object within the 3-D environment image displayed in the second frame of the 3-D video.

14. The computer-implemented method of claim 9, wherein receiving, via the one or more user controls, the indication of the boundary of the particular physical object depicted within the 3-D environment image comprises one of:

(i) receiving, via the one or more user controls, an indication of a shape that surrounds the depiction of the particular physical object within the 3-D environment image, and generating the data indicative of the boundary of the particular physical object within the 3-D environment image based upon the shape;

(ii) receiving, via the one or more user controls, an indication of an area within the 3-D environment image in which the depiction of the particular physical object is entirely included, the indicated area being a subset of a total area of the 3-D environment image; and

modifying a visual property of the indicated area, thereby visually distinguishing the indicated area within the 3-D environment image, wherein generating the data indicative of the boundary of the particular physical object within the 3-D environment image is based upon the indicated area;

or

(iii) receiving, via the one or more user controls, a user selection of a first location within the 3-D environment image that is non-adjacent to a second location within the 3-D environment image, the first location and the second location indicating respective endpoints of a segment of an edge of a traffic lane, the traffic lane being the particular physical object, and based upon the second location and the received user selection of the first location, automatically generating and displaying, within the 3-D environmental image, a marking indicative of the segment of the edge of the traffic lane, wherein generating the data indicative of the boundary of the traffic lane within the 3-D environment image is based upon the marking indicative of the segment of the edge of the traffic lane.

15. A system for identifying and labeling objects within images for training machine-learning based models that are used to operate vehicles, the system comprising:

a user interface;

one or more processors; and

one or more non-transitory, tangible memories coupled to the one or more processors and storing computer executable instructions thereon that, when executed by the one or more processors, cause the system to;

display, on a first display area of the user interface, a three-dimensional (3-D) video of an environment in which vehicles operate, the 3-D environment video including respective 3-D images of one or more physical objects located in the environment;

display, and a second display area of the user interface and in a time-synchronized manner with the 3-D video, a two-dimensional (2-D) video of the environment depicted in the 3-D environment video so that an image of the environment that was obtained at a particular time and that is included in the 2-D video is presented simultaneously on the user interface with an image of the environment that was obtained at the particular time and that is included in the 3-D video

receive, via the user interface, an indication of a boundary of a particular physical object depicted within a 3-D environment image included in the 3-D video, the indication of the boundary of the particular physical object provided by a user via the user interface;

generate data indicative of the boundary of the particular physical object within the 3-D environment image based on the received indication of the boundary of the particular physical object depicted within the 3-D environment image; and

store, in the one or more non-transitory, tangible memories, data indicative of a particular label descriptive of the particular physical object in association with the data indicative of the boundary of the particular physical object within the 3-D environment image, thereby distinguishing a set of data points that are representative of the particular physical object within the 3-D environment image from other data points included in the 3-D environment image.

16. The system of claim 15, wherein the computer executable instructions are further executable by the one or more processors to cause the system to:

receive, via the user interface, an indication of a user selection or a user creation of the particular label; and

generate the data indicative of the particular label based upon the received indication of the user selection or the user creation of the particular label.

17. The system of claim 15, wherein the computer executable instructions are further executable by the one or more processors to cause the system to automatically select or generate the particular label based upon the received indication of the boundary of the particular physical object, the automatic selection or generation performed by the system exclusive of any real-time user input and by using an image processor.

18. The system of claim 15, wherein:

the computer executable instructions are further executable by the one or more processors to cause the system to receive an indication of a user instruction to drag-and-drop or copy at least a portion of the 2-D environment image depicting the particular physical object onto the 3-D environment image, thereby layering at least a part of the 2-D environment image with the 3-D environment image to generate a composite image of the particular physical object within the 3-D environment image, the composite image of the particular physical object including a portion of the 2-D image of the particular physical object and a portion of the 3-D image of the particular physical object; and

the indicated boundary of the particular physical object within the 3-D environment image is an indicated boundary of the composite image within the 3-D environment image.

19. The system of claim 15, wherein the indication of the boundary of the particular physical object depicted within the 3-D environment image includes at least one of:

an indication of a shape that surrounds the depiction of the particular physical object within the 3-D environment image,

an indication of an area within the 3-D environment image in which the depiction of the particular physical object is entirely included, the indicated area being a subset of a total area of the 3-D environment image,

an indication of a surface area, or

an indication of a traffic lane.

20. The system of claim 19, wherein the indication of the boundary of the particular physical object depicted by the composite image within the 3-D environment is either received via one or more user controls provided by the user interface or automatically generated by further computer-executable instructions exclusive of any user input.

21. The system of claim 15, wherein the 3-D environment image is a first 3-D image included in a first frame of a 3-D video displayed on the user interface, the 2-D environment image is a first 2-D image, and the computer executable instructions are further executable by the one or more processors to cause the system to:

receive an indication of a boundary of the particular physical object depicted within a second frame of the 3-D video;

generate, based on the received indication of the boundary of the particular physical object depicted within the second frame of the 3-D video, data indicative of the boundary of the particular physical object within the second frame of the 3-D video; and

store, in the one or more tangible memories, the data indicative of the boundary of the particular physical object within the second frame of the 3-D video in conjunction with the data indicative of the particular label descriptive of the particular physical object, thereby distinguishing the particular physical object within the second frame of the 3-D video.

22. The system of claim 21, wherein the indication of the boundary of the particular physical object within the second frame of the 3-D video is a graphical representation displayed within the second frame of the 3-D video, and an indication of the graphical representation is received via one or more user controls.

23. The system of claim 21, wherein the indication of the boundary of the particular physical object within the second frame of the 3-D video is a graphical representation displayed within the second frame of the 3-D video, and the graphical representation is automatically generated by the system exclusive of any real-time user input.

24. The system of claim 15, wherein the 3-D environment image is displayed on the user interface from a first perspective view, and the computer executable instructions are further executable by the one or more processors to cause the system to:

receive, via the user interface, an indication of a user instruction to change the 3-D environment image from being displayed on the user interface from the first perspective view to being displayed on the user interface from a second perspective view;

based on the received user instruction, display, on the user interface, the 3-D environment image from the second perspective view, the 3-D environment image from the second perspective view including a depiction of the particular physical object from the second perspective view;

automatically determine an interim graphical representation of the boundary of the particular physical object depicted within the 3-D environment image displayed from the second perspective view;

display the interim graphical representation within the 3-D environment image displayed from the second perspective view;

receive an indication of a user modification to the interim graphical representation;

alter the interim graphical representation based on the received user modification to generate a refined graphical representation of the boundary of the particular physical object depicted within the 3-D environment image;

generate, based on the refined graphical representation of the boundary of the particular physical object, data indicative of the boundary of the particular physical object within the 3-D environment image displayed from the second perspective view; and

update, based on the refined graphical representation, the stored data indicative of the boundary of the particular physical object within the 3-D environment image associated with the data indicative of the particular label descriptive of the particular physical object.

25. A system for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the method comprising:

a user interface;

one or more processors; and

one or more non-transitory, tangible memories coupled to the one or more processors and storing computer executable instructions thereon that, when executed by the one or more processors, cause the system to;

display, on the user interface, a three-dimensional (3-D) video of an environment in which vehicles operate, the 3-D environment video including respective 3-D images of one or more physical objects located in the environment;

dynamically vary a perspective view from which the 3-D video is displayed on the user interface across multiple perspective views during a presentation of the 3-D video responsive to one or more user instructions that are received, via the user interface, during the presentation of the 3-D video;

receive, via one or more user controls provided by the user interface, an indication of a boundary of a particular physical object depicted within a 3-D environment image included in the 3-D video;

generate data indicative of the boundary of the particular physical object within the 3-D environment image;

receive an indication of a particular label for the particular physical object;

associate the particular label for the particular physical object with the data indicative of the boundary of the particular physical object within the 3-D environment image, thereby distinguishing a set of data points that are representative of the particular physical object within the 3-D environment image from other data points included in the 3-D environment image; and

store an indication of the association between the particular label and the data indicative of the boundary of the particular physical object within the 3-D environment image in the one or more tangible memories or in another one or more tangible memories as a part of a training data set utilized to train one or more machine-learning based models, the one or more machine-learning based models used to autonomously operate vehicles.

26. The system of claim 25, wherein the computer executable instructions are further executable by the one or more processors to cause the system to:

receive, via the user interface, an indication of a user selection or a user creation of the particular label; and

generate the data indicative of the particular label based upon the received indication of the user selection or the user creation of the particular label.

27. The system of claim 25, wherein the computer executable instructions are further executable by the one or more processors to cause the system to automatically select or generate the particular label based upon the received indication of the boundary of the particular physical object, the automatic selection or generation performed by the system exclusive of any real-time user input and by using an image processor.

28. The system of claim 25, wherein the indication of the boundary of the particular physical object depicted within the 3-D environment image includes at least one of:

an indication of a shape that surrounds the depiction of the particular physical object within the 3-D environment image,

an indication of an area within the 3-D environment image in which the depiction of the particular physical object is entirely included, the indicated area being a subset of a total area of the 3-D environment image,

an indication of a surface area, or

an indication of a traffic lane.

29. The system of claim 25, wherein the 3-D environment image is included in a first frame of the 3-D video, the indication of the boundary of the particular physical object depicted within the first frame of the 3-D video is a first graphical representation, and the computer-executable instructions, when executed by the one or more processors, cause the system further to:

automatically determine an interim graphical representation of the boundary of the particular physical object depicted within a second frame of the 3-D video;

display the interim graphical representation within the second frame of the 3-D video;

receive an indication of a user modification to the interim graphical representation;

change the interim graphical representation based on the received user modification to generate a second graphical representation of the boundary of the particular physical object depicted within the second frame of the 3-D video;

generate, based on the second graphical representation of the boundary of the particular physical object within the second frame of the 3-D video, data indicative of the boundary of the particular physical object within the second frame of the 3-D video; and

store, in the one or more tangible memories, an indication of an association of the data indicative of the boundary of the particular physical object within the second frame of the 3-D video and the particular label descriptive of the particular physical object.

30. The system of claim 25, wherein the 3-D video including the 3-D environment image is displayed from a first perspective view on the user interface, the indication of the boundary of the particular physical object depicted within the 3-D environment image displayed from the first perspective view is a first graphical representation, and the computer-executable instructions, when executed by the one or more processors, cause the system further to:

based on a change of the 3-D video from being displayed on the user interface from the first perspective view to being displayed on the user interface from a second perspective view, automatically determine an interim graphical representation of the boundary of the particular physical object depicted within the 3-D environment image of the 3-D video displayed from the second perspective view;

display the interim graphical representation within the 3-D environment image displayed from the second perspective view;

receive an indication of a user modification to the interim graphical representation;

alter the interim graphical representation based on the received user modification to generate a refined graphical representation of the boundary of the particular physical object depicted within the 3-D environment image displayed from the second perspective view;

generate, based on the refined graphical representation, updated data indicative of the boundary of the particular physical object within the 3-D environment; and

update the stored indication of the association between the particular label and the data indicative of the boundary of the particular physical object within the 3-D environment image based upon the refined graphical representation.