Augmenting image data based on related 3D point cloud data

• US 8,872,851 B2
• Filed: 09/24/2010
• Issued: 10/28/2014
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: El Choubassi, Maha, Kozintsev, Igor V., Wu, Yi, Gat, Yoram, Haussecker, Horst
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Tseng, Charles
• Application Number:
  US12/890,251
• Publication Number:
  US 20120075342A1
• Time in Patent Office: 1,495 Days
• Field of Search: 345629-641
• US Class Current: 345/633
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 7/11 Region-based segmentation

Abstract

Embodiments of the invention describe processing a first image data and 3D point cloud data to extract a first planar segment from the 3D point cloud data. This first planar segment is associated with an object included in the first image data. A second image data is received, the second image data including the object captured in the first image data. A second planar segment related to the object is generated, where the second planar segment is geometrically consistent with the object as captured in the second image data. This planar segment is generated based, at least in part, on the second image data, the first image data and the first planar segment. Embodiments of the invention may further augment the second image data with content associated with the object. This augmented image may be displayed such that the content is displayed geometrically consistent with the second planar segment.

Claims

1. An article of manufacture comprising a non-transitory machine-readable storage medium that provides instructions that, if executed by the machine, will cause the machine to perform operations comprising:

processing a first image data and 3D point cloud data of objects included in the first image data to extract a 3D data mask of an object included in the first image data from the 3D point cloud data, wherein the 3D data mask comprises less data than the 3D point cloud data of the objects included in the first image data;

generating a planar segment related to the object as captured in a second image data from a real time view of the object based, at least in part, on the second image data, the first image data and the 3D data mask, the second image data captured via an image sensor included in a mobile computing device, the second image data having the object, the planar segment geometrically consistent and within the object as it appears in the second image data and comprising a plurality of clusters, including a first cluster corresponding to the object and a second cluster corresponding to a second co-planar object; and

augmenting, geometrically consistent with the planar segment and within the first cluster, the real-time view of the object with content associated with the object, the content to include static image data or video image data.

2. The article of manufacture of claim 1, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask includes determining the shape and orientation of the 3D data mask based on metadata associated with the first image data, the metadata having geographical information of the first image data or data related to the object captured in the first image data.

3. The article of manufacture of claim 1, wherein generating the planar segment related to the object as captured in the second image data further includes matching the planar segment with the object captured in the second image data based on color data of the object captured in the first image data.

4. The article of manufacture of claim 1 wherein processing the 3D point cloud data to extract the 3D data mask further includes downsampling the 3D point cloud data.

5. The article of manufacture of claim 1, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask from the 3D point cloud data is further based on reliability data associated with data points of the 3D point cloud data.

6. The article of manufacture of claim 1, wherein the 3D point cloud data includes

a first set of 3D point cloud data captured from a first viewpoint;

and

a second set of 3D point cloud data captured from a second viewpoint.

7. The article of manufacture of claim 6, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask from the 3D point cloud data includes

processing the first set of 3D point cloud data and the first image data to extract a first initial planar segment;

processing the second set of 3D point cloud data and the first image data to extract a second initial planar segment; and

combining the first and second initial planar segments to form the 3D data mask.

8. A system comprising:

at least one processor;

a display;

software to be executed via the processor(s) to

process a first image data and 3D point cloud data of objects included in the first image data to extract a 3D data mask of an object included in the first image data from the 3D point cloud data;

wherein the 3D data mask comprises less data than the 3D point cloud data of the objects included in the first image data,

generate a planar segment related to the object as captured in a second image data from a real time view of the object based, at least in part, on the second image data, the first image data and the 3D data mask, the planar segment geometrically consistent and within the object as it appears in the second image data and comprising a plurality of clusters, including a first cluster corresponding to the object and a second cluster corresponding to a second co-planar object,

augment, geometrically consistent with the planar segment and within the first cluster, the real-time view of the object with content associated with the object, the content to include static image data or video image data,

display the real-time view of the object with the augmented content associated with the object on the display; and

an image sensor to capture the real-time view of the object, including the second image data having the object.

9. The system of claim 8, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask includes determining the shape and orientation of the 3D data mask based on metadata associated with the first image data, the metadata to include geographical information of the first image data or data related to the object captured in the first image data.

10. The system of claim 8, wherein generating the planar segment further includes matching the planar segment with the object captured in the second image data based on color data of the object captured in the first image data.

11. The system of claim 8 wherein processing the 3D point cloud data to extract the 3D data mask further includes downsampling the 3D point cloud data.

12. The system of claim 8, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask from the 3D point cloud data is further based on reliability data associated with data points of the 3D point cloud data.

13. The system of claim 8, wherein the 3D point cloud data includes

a first set of 3D point cloud data captured from a first viewpoint;

and

a second set of 3D point cloud data captured from a second viewpoint.

14. The system of claim 13, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask from the 3D point cloud data includes

processing the first set of 3D point cloud data and the first image data to extract a first initial planar segment;

processing the second set of 3D point cloud data and the first image data to extract a second initial planar segment; and

combining the first and second initial planar segments to form the 3D data mask.

15. A method comprising:

processing a first image data and 3D point cloud data of objects included in the first image data to extract a 3D data mask of an object included in the first image data from the 3D point cloud data, wherein the 3D data mask comprises less data than the 3D point cloud data of the objects included in the first image data;

receiving a second image data from a real-time view of the object captured via an image sensor included in a mobile computing device, the second image data to include the object;

generating a planar segment related to the object as captured in a second image data from a real time view of the object based, at least in part, on the second image data, the first image data and the 3D data mask, the planar segment geometrically consistent and within the object as it appears in the second image data and comprising a plurality of clusters, including a first cluster corresponding to the object and a second cluster corresponding to a second co-planar object; and

augmenting, geometrically consistent with the planar segment and within the first cluster, the real-time view of the object with content associated with the object, the content to include static image data or video image data.

16. The method of claim 15, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask includes determining the shape and orientation of the 3D data mask based on metadata associated with the first image data, the metadata having geographical information of the first image data or data related to the object captured in the first image data.

17. The method of claim 15, wherein generating the planar segment related to the object as captured in the second image data further includes matching the planar segment with the object captured in the second image data based on color data of the object captured in the first image data.

18. The method of claim 15, wherein processing the 3D point cloud data to extract the 3D data mask further includes downsampling the 3D point cloud data.

19. The method of claim 15, wherein processing the first image data and the 3D point cloud data to extract the 3D data mask from the 3D point cloud data is further based on reliability data associated with data points of the 3D point cloud data.

20. The method of claim 15, wherein the 3D point cloud data includes

a first set of 3D point cloud data captured from a first viewpoint;

and

a second set of 3D point cloud data captured from a second viewpoint.

21. The method of claim 16, wherein generating the planar segment related to the object as captured in the second image data includes:

determining, from the first image data and from the geographical information of the first image data, that the object comprises a building; and

generating a non-rectangular planar segment as a rectangular segment in response to determining the object comprises a building.

22. The method of claim 21, wherein the 3D data mask is to only include horizontal or vertical planes in response to determining the object comprises a building.