METHOD FOR RENDERING 2D AND 3D DATA WITHIN A 3D VIRTUAL ENVIRONMENT

• US 20190311546A1
• Filed: 04/09/2019
• Published: 10/10/2019
• Est. Priority Date: 04/09/2018
• Status: Active Grant

• Current Assignee: Drive.ai, Inc. (Apple Inc.)
• Original Assignee: Drive.ai, Inc. (Apple Inc.)
• Inventors: Tay, Kah Seng, Sun, Qing, Marion, James Patrick
• Granted Patent: US 10,854,011 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/93 for anti-collision purposes
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light

Abstract

One variation of a method includes: accessing a 2D color image recorded by a 2D color camera and a 3D point cloud recorded by a 3D depth sensor at approximately a first time, the 2D color camera and the 3D depth sensor defining intersecting fields of view and facing outwardly from an autonomous vehicle; detecting a cluster of points in the 3D point cloud representing a continuous surface approximating a plane; isolating a cluster of color pixels in the 2D color image depicting the continuous surface; projecting the cluster of color pixels onto the plane to define a set of synthetic 3D color points in the 3D point cloud, the cluster of points and the set of synthetic 3D color points representing the continuous surface; and rendering points in the 3D point cloud and the set of synthetic 3D color points on a display.

Claims

1. A method for augmenting 3D depth map data with 2D color image data comprising:

accessing a first 2D color image recorded at a first time via a 2D color camera arranged on an autonomous vehicle;

accessing a first 3D point cloud recorded at approximately the first time via a 3D depth sensor arranged on the autonomous vehicle, the 3D depth sensor and the 2D color camera defining intersecting fields of view and facing outwardly from the autonomous vehicle;

detecting a first cluster of points in the first 3D point cloud representing a first continuous surface approximating a first plane;

isolating a first cluster of color pixels in the first 2D color image depicting the first continuous surface;

projecting the first cluster of color pixels onto the first plane to define a first set of synthetic 3D color points in the first 3D point cloud, the first cluster of points and the first set of synthetic 3D color points representing the first continuous surface; and

rendering points in the first 3D point cloud and the first set of synthetic 3D color points on a display.

2. The method of claim 1, further comprising:

passing the first 3D point cloud, augmented with the first set of synthetic 3D color points, through a perception pipeline to identify characteristics of an object represented by the first cluster of points;

based on characteristics of the object, electing a next navigational action to avoid collision with the object; and

autonomously executing the next navigational action.

3. The method of claim 2:

further comprising passing the first 3D point cloud, excluding the first set of synthetic 3D color points, through the perception pipeline to identify characteristics of the object represented by the first cluster of points; and

wherein projecting the first cluster of color pixels onto the first plane to define a first set of synthetic 3D color points in the first 3D point cloud and passing the first 3D point cloud, augmented with the first set of synthetic 3D color points, through a perception pipeline comprise, in response to an initial confidence for characteristics of the object falling below a threshold confidence;

augmenting the first 3D point cloud with the first set of synthetic 3D color points based on the first cluster of color pixels; and

passing the first 3D point cloud, augmented with the first set of synthetic 3D color points, through the perception pipeline to identify characteristics of the object with a revised confidence greater than the initial confidence.

4. The method of claim 1, wherein rendering the first 3D point cloud and the first set of synthetic 3D color points on the display comprises rendering the first 3D point cloud and the first set of synthetic 3D color points on the display arranged inside the autonomous vehicle at approximately the first time.

5. The method of claim 1:

further comprising, at the autonomous vehicle;

autonomously navigating along a route from a pickup location toward a destination location;

in response to failure to perceive an object in a field around the autonomous vehicle at approximately the first time;

transmitting a request for remote assistance to a remote operator portal; and

transmitting the first 3D point cloud, comprising the first cluster of points and the first set of synthetic 3D color points, to the remote operator portal; and

wherein rendering the first 3D point cloud on the display comprises, at the remote operator portal, rendering the first 3D point cloud for a human operator.

6. The method of claim 1,

further comprising interpolating color values of points in the first cluster of points representing the first continuous surface in the first 3D point cloud based on color values stored in the first set of synthetic 3D color points;

and

wherein rendering points in the first 3D point cloud and the first set of synthetic 3D color points on a display comprises rendering the first cluster of points and the first set of synthetic 3D color points in color.

7. The method of claim 1:

further comprising;

isolating a second set of points in the first 3D point cloud outside of the first cluster of points;

deriving correspondence between the second set of points in the first 3D point cloud and a second set of color pixels in the first 2D color image; and

interpolating color values of points in the second set of points based on color values stored in the second set of color pixels; and

wherein rendering points in the first 3D point cloud and the first set of synthetic 3D color points on a display comprises;

rendering the first cluster of points and the first set of synthetic 3D color points in color on the display at a first point density; and

rendering the second set of points in color on the display at a second point density less than the first point density.

8. The method of claim 1:

further comprising, within a sequence of 3D point clouds recorded via the 3D depth sensor prior to the first time;

detecting a first dynamic object, in a field near the autonomous vehicle, moving toward a path of the autonomous vehicle; and

detecting a second dynamic object, in the field, located remotely from the path of the autonomous vehicle;

wherein detecting the first cluster of points in the first 3D point cloud comprises selecting the first cluster of points, in the first 3D point cloud, representing the first continuous surface on the first dynamic object and located approximately on the first plane;

wherein isolating the first cluster of color pixels in the first 2D color image depicting the first continuous surface comprises selecting the first cluster of color pixels in the first 2D color image depicting the first continuous surface on the first dynamic object;

wherein projecting the first cluster of color pixels onto the first plane to define the first set of synthetic 3D color points in the first 3D point cloud comprises augmenting the first 3D point cloud with the first set of synthetic 3D color points depicting the first continuous surface on the first dynamic object; and

wherein rendering the first 3D point cloud on the display comprises rendering the first 3D point cloud comprising;

the first cluster of points and the first set of synthetic 3D color points depicting the first dynamic object; and

a second cluster of points depicting the second dynamic object exclusive of synthetic 3D color points derived from the first 2D color image.

9. The method of claim 1:

further comprising, within a sequence of 3D point clouds recorded via the 3D depth sensor prior to the first time;

detecting a dynamic object, in a field near the autonomous vehicle, located proximal a path of the autonomous vehicle;

detecting a static object in the field; and

selecting the dynamic object for selective augmentation with 2D color data;

wherein detecting the first cluster of points in the first 3D point cloud comprises selecting the first cluster of points, in the first 3D point cloud, representing the first continuous surface on the dynamic object and located approximately on the first plane;

wherein isolating the first cluster of color pixels in the first 2D color image depicting the first continuous surface comprises selecting the first cluster of color pixels in the first 2D color image depicting the first continuous surface on the dynamic object;

wherein projecting the first cluster of color pixels onto the first plane to define the first set of synthetic 3D color points in the first 3D point cloud comprises, in response to selecting the dynamic object for selective augmentation with 2D color data, augmenting the first 3D point cloud with the first set of synthetic 3D color points depicting the first continuous surface on the dynamic object.

10. The method of claim 9, wherein selecting the first cluster of points, in the first 3D point cloud, representing the first continuous surface on the dynamic object and located approximately on the first plane comprises:

isolating a constellation of points depicting the dynamic object in the first 3D point cloud;

interpreting a set of surfaces on the dynamic object represented by the constellation of points; and

selecting the first cluster of points, in the constellation of points, that represent the first continuous surface spanning a largest planar surface, in the set of surfaces, on the dynamic object.

11. The method of claim 9:

further comprising, in response to detecting the static object in the sequence of 3D point clouds;

isolating a second cluster of points in the first 3D point cloud representing the static object;

aggregating groups of static points representing the static object in the sequence of 3D point clouds preceding the first 3D point cloud; and

projecting the groups of static points into the first 3D point cloud based on changes in absolute position of the autonomous vehicle between the first time and times that 3D point clouds in the sequence of 3D point clouds were recorded by the 3D depth sensor; and

wherein rendering the first 3D point cloud on the display comprises rendering the first 3D point cloud comprising;

the first cluster of points and the first set of synthetic 3D color points depicting the dynamic object; and

a second cluster of points and the groups of static points depicting the static object.

12. The method of claim 1, wherein detecting the first cluster of points in the first 3D point cloud comprises:

detecting a set of clusters of points in the first 3D point cloud, each cluster of points in the set of clusters of points representing a discrete surface in a field around the autonomous vehicle and falling approximately on a plane; and

selecting the first cluster of points, in the set of clusters of points in the first 3D point cloud, representing the first continuous surface located proximal a path of the autonomous vehicle.

13. The method of claim 12:

wherein selecting the first cluster of points comprises selecting the first cluster of points representing the first continuous surface corresponding to a first side of an object proximal the path of the autonomous vehicle; and

further comprising;

selecting a second cluster of points, in the set of clusters of points in the first 3D point cloud, representing a second continuous surface approximating a second plane, the second continuous surface adjacent and moving with the first continuous surface;

isolating a second cluster of color pixels in the first 2D color image depicting the second continuous surface; and

projecting the second cluster of color pixels onto the second plane to define a second set of synthetic 3D color points in the first 3D point cloud, the second cluster of points and the second set of synthetic 3D color points representing the second continuous surface; and

wherein rendering points in the first 3D point cloud and the first set of synthetic 3D color points on the display further comprises rendering points in the first 3D point cloud, the first set of synthetic 3D color points, and the second set of synthetic 3D color points on the display.

14. The method of claim 13:

wherein selecting the first cluster of points comprises selecting the first cluster of points representing the first continuous surface corresponding to a side of a vehicle proximal the path of the autonomous vehicle; and

wherein selecting the second cluster of points comprises selecting the second cluster of points representing the second continuous surface corresponding to a rear of the vehicle.

15. The method of claim 1, wherein detecting the first cluster of points in the first 3D point cloud comprises:

detecting a set of clusters of points in the first 3D point cloud, each cluster of points in the set of clusters of points representing a discrete surface in a field around the autonomous vehicle and containing points that fall approximately on a plane; and

selecting the first cluster of points, in the set of clusters of points in the first 3D point cloud, characterized by a density of points less than a threshold density.

16. The method of claim 15, further comprising:

extracting a distance from the autonomous vehicle to the first continuous surface from the first 3D point cloud; and

calculating the threshold density inversely proportional to the distance from the autonomous vehicle to the first continuous surface.

17. The method of claim 1:

wherein detecting the first cluster of points in the first 3D point cloud comprises detecting the first cluster of points in the first 3D point cloud representing the first continuous surface defined by a roadside billboard;

wherein projecting the first cluster of color pixels onto the first plane to define the first set of synthetic 3D color points in the first 3D point cloud comprises augmenting the first 3D point cloud with the first set of synthetic 3D color points depicting the roadside billboard;

further comprising;

extracting iconography on the roadside billboard from the first cluster of color pixels; and

retrieving electronic content related to the roadside billboard based on iconography extracted from the first cluster of color pixels; and

wherein rendering the first 3D point cloud and the first set of synthetic 3D color points on the display comprises;

rendering the first 3D point cloud and the first set of synthetic 3D color points on the display arranged inside the autonomous vehicle at approximately the first time; and

overlaying the electronic content related to the roadside billboard over the first 3D point cloud and the first set of synthetic 3D color points rendered on the display.

18. A method for augmenting 3D depth map data with 2D color image data comprising:

accessing a first 2D color image recorded at a first time via a 2D color camera arranged on an autonomous vehicle;

accessing a first 3D point cloud recorded at approximately the first time via a 3D depth sensor arranged on the autonomous vehicle, the 3D depth sensor and the 2D color camera defining intersecting fields of view and facing outwardly from the autonomous vehicle;

detecting a first cluster of points in the first 3D point cloud representing a first continuous surface approximating a first plane;

isolating a first cluster of color pixels in the first 2D color image depicting the first continuous surface;

projecting the first cluster of color pixels onto the first plane to define a first set of synthetic 3D color points;

compiling the first cluster of points and the first set of synthetic 3D color points, representing the first continuous surface, into a first 3D frame;

detecting characteristics of an object, comprising the continuous surface, based on the first cluster of points and the first set of synthetic 3D color points in the first 3D frame;

based on characteristics of the object, electing a next navigational action; and

autonomously executing the next navigational action.

19. The method of claim 18, further comprising rendering the first 3D frame on a display arranged inside the autonomous vehicle and facing a rider occupying the autonomous vehicle.

20. The method of claim 18:

further comprising, within a sequence of 3D point clouds recorded via the 3D depth sensor prior to the first time;

detecting a dynamic object, in a field near the autonomous vehicle, located proximal a path of the autonomous vehicle;

detecting a static object, in the field, located remotely from the path of the autonomous vehicle; and

selecting the dynamic object for selective augmentation with 2D color data;

wherein detecting the first cluster of points in the first 3D point cloud comprises selecting the first cluster of points, in the first 3D point cloud, representing the first continuous surface on the dynamic object and located approximately on the first plane;

wherein isolating the first cluster of color pixels in the first 2D color image depicting the first continuous surface comprises selecting the first cluster of color pixels in the first 2D color image depicting the first continuous surface on the dynamic object;

wherein projecting the first cluster of color pixels onto the first plane to define the first set of synthetic 3D color points comprises, in response to selecting the dynamic object for selective augmentation with 2D color data;

initializing the first 3D frame with points from the first 3D point cloud; and

augmenting the first 3D frame with the first set of synthetic 3D color points depicting the first continuous surface on the dynamic object.