Target digitization, extraction, and tracking

• US 8,542,252 B2
• Filed: 05/29/2009
• Issued: 09/24/2013
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Perez, Kathryn Stone, Kipman, Alex, Burton, Nicholas, Wilson, Andrew, Nehab, Diego Fernandes
• Primary Examiner(s): Richer, Aaron M
• Assistant Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US12/455,192
• Publication Number:
  US 20100302247A1
• Time in Patent Office: 1,579 Days
• Field of Search: 382/102, 382/203, 345/420, 345/649
• US Class Current: 345/649
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques may comprise identifying surfaces, textures, and object dimensions from unorganized point clouds derived from a capture device, such as a depth sensing device. Employing target digitization may comprise surface extraction, identifying points in a point cloud, labeling surfaces, computing object properties, tracking changes in object properties over time, and increasing confidence in the object boundaries and identity as additional frames are captured. If the point cloud data includes an object, a model of the object may be generated. Feedback of the model associated with a particular object may be generated and provided real time to the user. Further, the model of the object may be tracked in response to any movement of the object in the physical space such that the model may be adjusted to mimic changes or movement of the object, or increase the fidelity of the target'"'"'s characteristics.

Claims

1. A method of target extraction from a physical space, the method comprising:

receiving depth data representative of the physical space;

converting the depth data into a point cloud and extracting at least one surface from the point cloud;

identifying a target in the physical space from the at least one surface;

generating a model of the target with respect to the receipt of the depth data;

labeling the target with a unique ID in accordance with a tunable distance threshold and a tunable normal similarity threshold;

identifying color and/or texture associated with the at least one surface based on the extraction of the at least one surface and the point cloud; and

displaying a visual representation of the model including painting the color and/or texture onto the visual representation of the model.

2. The method of claim 1, wherein extracting the at least one surface comprises at least one of computing a surface from the depth data via at least one of a Hough transform, a Fourier transform, a computation of surface normals, or a Curvelet transform.

3. The method of claim 1, further comprising comparing the depth data to a target filter for target recognition, and if the comparison indicates an association of the point cloud with the target filter, labeling the target to associate with the target filter.

4. The method of claim 1, further comprising:

tracking the target in the physical space by evaluating the depth data received that is representative of the target;

comparing the depth data from frame-to-frame to identify temporal information for updating the point cloud; and

updating the model of the target based on the temporal information,

wherein the temporal information comprises differences in the depth data between frames and a second surface extracted from the updated point cloud.

5. The method of claim 4, further comprising using the temporal information to determine a confidence rating for the identified target in the physical space.

6. The method of claim 4, wherein an update rate for updating the model of the target is at least 20 Hz.

7. The method of claim 1, further comprising setting a threshold number for points in a set of associated points that indicates noise, and if the points in the set of associated points is below the threshold number, removing the points.

8. The method of claim 1, further comprising computing a bounding box associated with each identified target and determining at least one of orientation or axis of rotation from the bounding box.

9. The method of claim 1, wherein the target is at least one of a known object, an unknown object, or a human user.

10. A method of texture extraction, the method comprising:

scanning a target in a physical space;

identifying a point cloud associated with the target;

labeling the target with a unique ID in accordance with a tunable distance threshold and a tunable normal similarity threshold;

capturing target characteristics comprising color and/or texture from the scan based on an extraction of at least one surface from the point cloud; and

displaying a visual representation of the target in a virtual space including translating the target characteristics from the scan to the visual representation of the target, the translating comprising painting the color and/or texture onto the visual representation of the target.

11. The method of claim 10, wherein the target characteristics from the scan further comprise dimensions of the target.

12. The method of claim 10, wherein a user identifies the target to be scanned.

13. The method of claim 10, further comprising updating the visual representation of the target based on a second scan of the target in the physical space.

14. The method of claim 13, wherein an update rate for updating the visual representation of the target is at least 20 Hz.

15. The method of claim 10, wherein a user performs a gesture to identify the target to be scanned.

16. The method of claim 10, further comprising comparing the point cloud associated with the target to at least one target filter for target recognition to determine if the target is a known target or an unknown target.

17. The method of claim 16, wherein if the target is a known target, applying features of the known target to the visual representation of the target.

18. A device for extracting a target in a physical space, the device comprising:

a camera component, wherein the camera component receives depth data representative of the target in the physical space; and

a processor, wherein the processor executes computer executable instructions, and wherein the computer executable instructions comprise instructions for;

converting the depth data into a point cloud and extracting at least one surface from the point cloud;

identifying the target in the physical space from the at least one surface;

generating a model of the target with respect to the receipt of the depth data;

labeling the target with a unique ID in accordance with a tunable distance threshold and a tunable normal similarity threshold;

identifying color and/or texture associated with the at least one surface based on the extraction of the at least one surface and the point cloud; and

displaying a visual representation of the model including painting the color and/or texture onto the visual representation of the model.

19. The device of claim 18, wherein extracting the at least one surface comprises at least one of computing a surface from the depth data via at least one of a Hough transform, a Fourier transform, a computation of surface normals, or a Curvelet transform.

20. The device of claim 18, further comprising a display component for displaying the visual representation of the model with respect to the receipt of the depth data.

21. The device of claim 18, wherein the camera component receives the depth data in frames of data.

22. The device of claim 21, wherein the processor executes the computer executable instructions with respect to the receipt of the frames of data.

23. The device of claim 21, wherein an update rate for displaying the visual representation is at least 20 Hz.

24. The device of claim 18, wherein the computer executable instructions further comprise instructions for comparing the depth data to a target filter for target recognition, and if the comparison indicates an association with the target filter, labeling the target to associate with the target filter.

25. The device of claim 18, wherein the computer executable instructions further comprise instructions for:

tracking the target in the physical space by evaluating the depth data received that is representative of the target;

comparing the depth data from frame-to-frame to identify temporal information for updating the point cloud; and

updating the model of the target based on the temporal information,

wherein the temporal information comprises differences in the depth data between frames and a second surface extracted from the updated point cloud.

26. The device of claim 25, wherein the computer executable instructions further comprise instructions for using the temporal information to determine a confidence rating for the identified target in the physical space.

27. The device of claim 25, wherein an update rate for updating the model of the target is at least 20 Hz.

28. The device of claim 25, wherein the computer executable instructions further comprise instructions for using the temporal information to determine a confidence rating for the identified target in the physical space.

29. A method of target extraction, and tracking in a game application, the method comprising:

receiving raw data representative of a target in a physical space, wherein the raw data comprises depth data and RGB data;

interpreting the raw data into a set of points and determining surface normals from the set of points;

performing surface extraction on the target, wherein the surface extraction is determined from the set of points and the surface normals;

labeling the target with a unique ID in accordance with a tunable distance threshold and a tunable normal similarity threshold;

identifying color and/or texture associated with the at least one extracted surface based on an extraction of the at least one surface and the set of points;

performing color and/or texture extraction on the target comprising painting the color and/or texture onto a visual representation of the target; and

tracking the set of points by continuing to receive raw data and comparing the set of points from frame to frame; and

updating the visual representation of the target based on the tracking.