DEPTH SENSOR WITH APPLICATION INTERFACE

• US 20110292036A1
• Filed: 05/02/2011
• Published: 12/01/2011
• Est. Priority Date: 05/31/2010
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Gurman, Amiad, Sali, Erez, Yanir, Tomer, Guendelman, Eran
• Application Number:
  US13/098,497
• Publication Number:
  US 20110292036A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation

Abstract

A method for processing data includes receiving a depth map of a scene containing a body of a humanoid subject. The depth map includes a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location. The depth map is processed in a digital processor to extract a skeleton of at least a part of the body, the skeleton including multiple joints having respective coordinates. An application program interface (API) indicates at least the coordinates of the joints.

Claims

1. A method for processing data, comprising:

receiving a depth map of a scene containing a body of a humanoid subject, the depth map comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location;

processing the depth map in a digital processor to extract a skeleton of at least a part of the body, the skeleton comprising multiple joints having respective coordinates; and

providing an application program interface (API) indicating at least the coordinates of the joints.

2. The method according to claim 1, wherein the skeleton comprises two shoulder joints having different, respective depth values.

3. The method according to claim 2, wherein the different depth values of shoulder joints define a coronal plane of the body that is rotated by at least 10°

relative to the reference plane.

4. The method according to claim 1, wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing respective depth values of the pixels in the depth map.

5. The method according to claim 1, wherein receiving the depth map comprises receiving a sequence of depth maps as the body moves, and wherein processing the depth map comprises tracking movement of one or more of the joints over the sequence, and wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing an indication of gestures formed by the movement of the one or more of the joints.

6. The method according to claim 1, wherein the scene contains a background, and wherein processing the depth map comprises identifying one or more parameters of at least one element of the background, and wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing the one or more parameters of the at least one element of the background.

7. The method according to claim 6, wherein the at least one element of the background comprises a planar element, and wherein the one or more parameters indicate a location and orientation of a plane corresponding to the planar element.

8. The method according to claim 1, wherein the scene contains respective bodies of two or more humanoid subjects, and wherein processing the depth map comprises distinguishing the bodies from one another and assigning a respective label to identify each of the bodies, and wherein the API identifies the coordinates of the joints of each of the bodies with the respective label.

9. The method according to claim 8, wherein distinguishing the bodies comprises identifying an occlusion of a part of one of the bodies in the depth map by another of the bodies, and wherein the API identifies the occlusion.

10. The method according to claim 1, wherein processing the depth map comprises computing a confidence value associated with an identification of an element in the scene, and wherein the API indicates the identification and the associated confidence value.

11. Apparatus for processing data, comprising:

an imaging assembly, which is configured to generate a depth map of a scene containing a body of a humanoid subject, the depth map comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location; and

a processor, which is configured to process the depth map to extract a skeleton of at least a part of the body, the skeleton comprising multiple joints having respective coordinates, and to provide an application program interface (API) indicating at least the coordinates of the joints.

12. The apparatus according to claim 11, wherein the skeleton comprises two shoulder joints having different, respective depth values.

13. The apparatus according to claim 12, wherein the different depth values of shoulder joints define a coronal plane of the body that is rotated by at least 10°

relative to the reference plane.

14. The apparatus according to claim 11, wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing respective depth values of the pixels in the depth map.

15. The apparatus according to claim 11, wherein the imaging assembly is configured to generate a sequence of depth maps as the body moves, and wherein the processor is configured to track movement of one or more of the joints over the sequence, and wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing an indication of gestures formed by the movement of the one or more of the joints.

16. The apparatus according to claim 11, wherein the scene contains a background, and wherein the processor is configured to identify one or more parameters of at least one element of the background, and wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing the one or more parameters of the at least one element of the background.

17. The apparatus according to claim 16, wherein the at least one element of the background comprises a planar element, and wherein the one or more parameters indicate a location and orientation of a plane corresponding to the planar element.

18. The apparatus according to claim 11, wherein the scene contains respective bodies of two or more humanoid subjects, and wherein the processor is configured to distinguish the bodies from one another and to assign a respective label to identify each of the bodies, and wherein the API identifies the coordinates of the joints of each of the bodies with the respective label.

19. The apparatus according to claim 18, wherein the processor is configured to identify an occlusion of a part of one of the bodies in the depth map by another of the bodies, and wherein the API identifies the occlusion.

20. The apparatus according to claim 11, wherein the processor is configured to compute a confidence value associated with an identification of an element in the scene, and wherein the API indicates the identification and the associated confidence value.

23. The product according to claim 12, wherein the different depth values of shoulder joints define a coronal plane of the body that is rotated by at least 10°

relative to the reference plane.

29. The product according to claim 18, wherein the instructions cause the computer to identify an occlusion of a part of one of the bodies in the depth map by another of the bodies, and wherein the API identifies the occlusion.

21. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a depth map of a scene containing a body of a humanoid subject, the depth map comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location, to process the depth map to extract a skeleton of at least a part of the body, the skeleton comprising multiple joints having respective coordinates, and to provide an application program interface (API) indicating at least the coordinates of the joints.

22. The product according to claim 21, wherein the skeleton comprises two shoulder joints having different, respective depth values.

24. The product according to claim 21, wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing respective depth values of the pixels in the depth map.

25. The product according to claim 21, wherein the instructions cause the processor to receive a sequence of depth maps as the body moves and to track movement of one or more of the joints over the sequence, and wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing an indication of gestures formed by the movement of the one or more of the joints.

26. The product according to claim 21, wherein the scene contains a background, and wherein the instructions cause the processor to identify one or more parameters of at least one element of the background, and wherein the API comprises a first interface providing the coordinates of the joints and a second interface providing the one or more parameters of the at least one element of the background.

27. The product according to claim 26, wherein the at least one element of the background comprises a planar element, and wherein the one or more parameters indicate a location and orientation of a plane corresponding to the planar element.

28. The product according to claim 21, wherein the scene contains respective bodies of two or more humanoid subjects, and wherein the instructions cause the processor to distinguish the bodies from one another and to assign a respective label to identify each of the bodies, and wherein the API identifies the coordinates of the joints of each of the bodies with the respective label.

30. The product according to claim 21, wherein the instructions cause the computer to compute a confidence value associated with an identification of an element in the scene, and wherein the API indicates the identification and the associated confidence value.

31. A method for processing data, comprising:

receiving a depth map of a scene comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location;

segmenting the depth map in a digital processor to identify one or more objects in the scene;

generating a label map comprising respective labels identifying the pixels belonging to the one or more objects; and

providing an indication of the label map via an application program interface (API).

32. The method according to claim 31, wherein receiving the depth map comprises receiving a sequence of depth maps as the objects move, and wherein generating the label map comprises updating the label map over the sequence responsively to movement of the objects.

33. The method according to claim 31, wherein at least one of the objects comprises multiple segments, and wherein generating the label map comprises assigning a single label to all of the segments.

34. The method according to claim 31, wherein segmenting the depth map comprises recognizing an occlusion of a part of one of the identified objects in the depth map by another object, and wherein generating the label map comprises identifying the occlusion in the label map.

35. Apparatus for processing data, comprising:

an imaging assembly, which is configured to generate a depth map of a scene comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location; and

a processor, which is configured to segment the depth map to identify one or more objects in the scene, to generate a label map comprising respective labels identifying the pixels belonging to the one or more objects, and to provide an indication of the label map via an application program interface (API).

36. The apparatus according to claim 35, wherein the imaging assembly is configured to generate a sequence of depth maps as the objects move, and wherein the processor is configured to update the label map over the sequence responsively to movement of the objects.

37. The apparatus according to claim 35, wherein at least one of the objects comprises multiple segments, and wherein the processor is configured to assign a single label to all of the segments.

38. The apparatus according to claim 35, wherein the processor is configured to recognize an occlusion of a part of one of the identified objects in the depth map by another object, and to generate the label map so as to identify the occlusion.

39. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive a depth map of a scene comprising a matrix of pixels, each pixel corresponding to a respective location in the scene and having a respective pixel depth value indicative of a distance from a reference plane to the respective location, to segment the depth map to identify one or more objects in the scene, to generate a label map comprising respective labels identifying the pixels belonging to the one or more objects, and to provide an indication of the label map via an application program interface (API).

40. The product according to claim 39, wherein the imaging assembly is configured to generate a sequence of depth maps as the objects move, and wherein the instructions cause the processor to update the label map over the sequence responsively to movement of the objects.

41. The product according to claim 39, wherein at least one of the objects comprises multiple segments, and wherein the instructions cause the processor to assign a single label to all of the segments.

42. The product according to claim 39, wherein the instructions cause the processor to recognize an occlusion of a part of one of the identified objects in the depth map by another object, and to generate the label map so as to identify the occlusion.