Target digitization, extraction, and tracking

• US 20100302247A1
• Filed: 05/29/2009
• Published: 12/02/2010
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Nehab, Diego Fernandes, Burton, Nicholas, Wilson, Andrew, Kipman, Alex, Perez, Kathryn Stone
• Granted Patent: US 8,542,252 B2
• Time in Patent Office: Days
• US Class Current: 345/440
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques may comprise identifying surfaces, textures, and object dimensions from unorganized point clouds derived from a capture device, such as a depth sensing device. Employing target digitization may comprise surface extraction, identifying points in a point cloud, labeling surfaces, computing object properties, tracking changes in object properties over time, and increasing confidence in the object boundaries and identity as additional frames are captured. If the point cloud data includes an object, a model of the object may be generated. Feedback of the model associated with a particular object may be generated and provided real time to the user. Further, the model of the object may be tracked in response to any movement of the object in the physical space such that the model may be adjusted to mimic changes or movement of the object, or increase the fidelity of the target'"'"'s characteristics.

Claims

1. A method of target extraction from a physical space, the method comprising:

receiving depth data representative of the physical space;

converting the depth data into a point cloud and extracting at least one surface from the point cloud;

identifying a target in the physical space from the at least one surface associated with the point cloud;

generating a model of the target in real time with respect to the receipt of the depth data; and

displaying a visual representation of the model in real time.

2. The method of claim 1, wherein extracting the at least one surface comprises at least one of computing a surface from the depth data via at least one of a Hough transform, a Fourier transform, a computation of surface normals, or a Curvelet transform.

3. The method of claim 1, further comprising labeling the target in the physical space with a unique ID.

4. The method of claim 1, further comprising comparing the depth data to a target filter for target recognition, and if the comparison indicates an association of the point cloud with the target filter, labeling the target to associate with the target filter.

5. The method of claim 1, further comprising:

tracking the target in the physical space by evaluating the depth data received that is representative of the target;

comparing the depth data from frame-to-frame to identify temporal information for updating the point cloud; and

updating the model of the target in real time based on the temporal information,

wherein the temporal information comprises differences in the depth data between frames and a second surface extracted from the updated point cloud.

6. The method of claim 5, further comprising using the temporal information to determine a confidence rating for the identified target in the physical space.

7. The method of claim 5, wherein an update rate for updating the model of the target is at least 20 Hz.

8. The method of claim 1, further comprising setting a threshold number for points in a set of associated points that indicates noise, and if the points in the set of associated points is below the threshold number, removing the points.

9. The method of claim 1, further comprising computing a bounding box associated with each identified target and determining at least one of orientation or axis of rotation from the bounding box.

10. The method of claim 7, further comprising translating the extracted texture to the visual representation of the target associated with the point cloud.

11. The method of claim 1, wherein the target is at least one of a known object, an unknown object, or a human user.

12. A method of texture extraction, the method comprising:

scanning a target in a physical space,;

identifying a point cloud associated with the target,

capturing target characteristics from the scan; and

displaying a visual representation of the target in a virtual space, wherein the target characteristics from the scan are translated to the visual representation of the target.

13. The method of claim 12, wherein the target characteristics from the scan include at least one of RGB data or dimensions of the target.

14. The method of claim 12, wherein a user identifies the target to be scanned.

15. The method of claim 12, further comprising updating the visual representation of the target in real time based on a second scan of the target in the physical space.

16. The method of claim 15, wherein an update rate for updating the visual representation of the target is at least 20 Hz.

17. The method of claim 12, wherein a user performs a gesture to identify the target to be scanned.

18. The method of claim 12, wherein the method comprises comparing the point cloud associated with the target to at least one target filter for target recognition to determine if the target is a known target or an unknown target.

19. The method of claim 18, wherein if the target is a known target, applying features of the known target to the visual representation of the target.

20. A device for extracting a target in a physical space, the device comprising:

a camera component, wherein the camera component receives raw depth data representative of the target in the physical space; and

a processor, wherein the processor executes computer executable instructions, and wherein the computer executable instructions comprise instructions for;

converting the depth data into a point cloud and extracting at least one surface from the point cloud;

identifying the target in the physical space from the at least one surface associated with the point cloud;

generating a model of the target in real time with respect to the receipt of the raw depth data; and

displaying a visual representation of the model in real time.

21. The device of claim 20, wherein extracting the at least one surface comprises at least one of computing a surface from the depth data via at least one of a Hough transform, a Fourier transform, a computation of surface normals, or a Curvelet transform.

22. The device of claim 20, further comprising a display component for displaying the visual representation of the model in real time with respect to the receipt of the raw depth data.

23. The device of claim 20, wherein the camera component receives depth data in frames of data.

24. The device of claim 23, wherein the processor executes the computer executable instructions comprising the instructions in real-time with respect to the receipt of the frames of data.

25. The device of claim 23, wherein an update rate for displaying the visual representation is at least 20 Hz.

26. The device of claim 20, further comprising labeling the target in the physical space with a unique ID.

27. The device of claim 20, further comprising comparing the depth data to a target filter for target recognition, and if the comparison indicates an association with the target filter, labeling the target to associate with the target filter.

28. The device of claim 20, wherein the computer executable instructions further comprise instructions for:

tracking the target in the physical space by evaluating the depth data received that is representative of the target;

comparing the depth data from frame-to-frame to identify temporal information for updating the point cloud; and

updating the model of the target in real time based on the temporal information,

wherein the temporal information comprises differences in the depth data between frames and a second surface extracted from the updated point cloud.

29. The method of claim 28, further comprising using the temporal information to determine a confidence rating for the identified target in the physical space.

30. The method of claim 28, wherein an update rate for updating the model of the target is at least 20 Hz.

31. The device of claim 28, further comprising using the temporal information to determine a confidence rating for the identified target in the physical space.

32. A method of target extraction, and tracking in a game application, the method comprising:

receiving raw data representative of a target in a physical space, wherein the raw data comprises depth data and RGB data;

interpreting the raw data into a set of points and determining surface normals from the set of points;

performing, in real time, surface extraction on the target, wherein the surface extraction is determined from the set of points and the surface normals;

performing, in real time, texture extraction on the target; and

tracking the set of points by continuing to receive raw data and comparing the set of points from frame to frame; and

updating, in real time, a visual representation of the target based on the tracking.