Object digitization

• US 9,208,571 B2
• Filed: 03/02/2012
• Issued: 12/08/2015
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Evertt, Jeffrey Jesus, Clark, Justin Avram, Scavezze, Mike, Deaguero, Joel, Markovic, Relja, Willoughby, Christopher Harley, Sola, Joe, Haley, David
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US13/410,907
• Publication Number:
  US 20120307010A1
• Time in Patent Office: 1,376 Days
• Field of Search: 348/46, 345/419, 345/633
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/268 based on depth image-based rendering [DIBR]
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

Digitizing objects in a picture is discussed herein. A user presents the object to a camera, which captures the image comprising color and depth data for the front and back of the object. For both front and back images, the closest point to the camera is determined by analyzing the depth data. From the closest points, edges of the object are found by noting large differences in depth data. The depth data is also used to construct point cloud constructions of the front and back of the object. Various techniques are applied to extrapolate edges, remove seams, extend color intelligently, filter noise, apply skeletal structure to the object, and optimize the digitization further. Eventually, a digital representation is presented to the user and potentially used in different applications (e.g., games, Web, etc.).

Claims

1. One or more computer storage devices having computer-executable instructions embodied thereon that, when executed, digitize an object captured by at least one camera, the method comprising:

receiving color and depth data associated with at least one image;

identifying a closest point in the at least one image to the at least one camera based on the depth data;

identifying edges of the object in the at least one image by analyzing depths of points moving outwardly from the closest point looking for drastic differences in depths between adjacent points;

generating a point cloud construction of the object using the depth data within the edges; and

using the point cloud construction to generate a digitization of the object.

2. The one or more computer storage devices of claim 1, wherein identifying the closest point in the image to the at least one camera further comprises:

calculating distances between an aperture of the at least one camera and a plurality of points in the image; and

of the distances, selecting a shortest distance.

3. The one or more computer storage devices of claim of claim 2, further comprising:

identifying a feature connected to the object at the shortest distance; and

storing an indication of the feature representing a closest portion of the object to the camera.

4. The one or more computer storage devices of claim 1, further comprising:

receiving color and depth data for at least one second image of the object;

determining a second closest point in the at least one second image;

identifying the object at the second closest point; and

generating a second point cloud construction of the object, as oriented in the at least one second image.

5. The one or more computer storage devices of claim 4, combining the point cloud representation with the second point cloud construction to create a 3D rendition of the object.

6. The one or more computer storage devices of claim 5, further comprising:

identifying a seam between the point cloud construction and second point cloud construction;

determining a filler color to fill a portion of the seam;

filling the portion of the seam with the filler color to create the 3D rendition with a seamless edge between the point cloud construction and the second point cloud construction; and

storing the 3D rendition.

7. The one or more computer storage devices of claim 6, further comprising displaying the 3D rendition on a display device.

8. The one or more computer storage devices of claim 7, further comprising:

using a set of rules to govern movement of different features of the 3D rendition; and

moving the 3D rendition on the display device according to one or more of the rules.

9. The one or more computer storage devices of claim 1, wherein the method further comprises:

performing an image analysis in a spatial region surrounding the closest point in the at least one image;

determining, based on the image analysis, a color difference between two areas in the spatial region;

designating one of the areas as being associated with the object; and

removing another area with a different color than the one of the areas.

10. The one or more computer storage devices of claim 1, further comprising flood-filling outwardly from the closest point until one or more depth edges are reached.

11. The one or more computer storage devices of claim 1, further comprising removing one or more points of the at least one image around areas with fewer than a threshold number of neighbor points associated with the object, resulting in a mask of depth pixels of the object.

12. The one or more computer storage devices of claim 11, further comprising storing the mask of depth pixels in a ring buffer of depth frames in a manner that overwrites at least one depth frame and averages multiple frames together to produce a final depth image.

19. The one or more computer storage devices of claim 1, wherein a drastic difference in depths between adjacent points comprises at least a half a meter.

20. The one or more computer storage devices of claim 1, wherein a drastic difference in depths between adjacent points corresponds to a distance indicating that one of the adjacent points is not part of the captured object.

13. A method for digitizing an object captured by at least one camera, the method comprising:

receiving color and depth data associated with at least one image;

identifying a closest point in the at least one image to the at least one camera based on the depth data;

identifying edges of the object in the at least one image by analyzing depths of points moving outwardly from the closest point looking for drastic differences in depths between adjacent points;

generating a point cloud construction of the object using the depth data within the edges; and

using the point cloud construction to generate a digitization of the object.

14. The method of claim 13, wherein identifying the closest point in the image to the at least one camera further comprises:

calculating distances between an aperture of the at least one camera and a plurality of points in the image;

of the distances, selecting a shortest distance;

identifying a feature connected to the object at the shortest distance; and

storing an indication of the feature representing a closest portion of the object to the camera.

15. The method of claim 13, further comprising:

receiving color and depth data for at least one second image of the object;

determining a second closest point in the at least one second image;

identifying the object at the second closest point;

generating a second point cloud construction of the object, as oriented in the at least one second image;

combining the point cloud representation with the second point cloud construction to create a 3D rendition of the object;

identifying a seam between the point cloud construction and second point cloud construction;

determining a filler color to fill a portion of the seam;

filling the portion of the seam with the filler color to create the 3D rendition with a seamless edge between the point cloud construction and the second point cloud construction; and

storing the 3D rendition.

16. The method of claim 15, further comprising:

using a set of rules to govern movement of different features of the 3D rendition; and

moving the 3D rendition according to one or more of the rules.

17. The method of claim 13, wherein the method further comprises:

performing an image analysis in a spatial region surrounding the closest point in the at least one image;

determining, based on the image analysis, a color difference between two areas in the spatial region;

designating one of the areas as being associated with the object; and

removing another area with a different color than the one of the areas.

18. A computing device comprising:

one or more processors configured to;

receive color and depth data associated with at least one image;

identify a closest point in the at least one image to the at least one camera based on the depth data;

identify edges of the object in the at least one image by analyzing depths of points moving outwardly from the closest point looking for drastic differences in depths between adjacent points;

generate a point cloud construction of the object using the depth data within the edges; and

use the point cloud construction to generate a digitization of the object.