Tracking body parts by combined color image and depth processing

• US 20110211754A1
• Filed: 02/28/2011
• Published: 09/01/2011
• Est. Priority Date: 03/01/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Litvak, Shai, Guendelman, Eran, Yanir, Tomer
• Granted Patent: US 8,787,663 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method for image processing includes receiving a depth image of a scene containing a human subject and receiving a color image of the scene containing the human subject. A part of a body of the subject is identified in at least one of the images. A quality of both the depth image and the color image is evaluated, and responsively to the quality, one of the images is selected to be dominant in processing of the part of the body in the images. The identified part is localized in the dominant one of the images, while using supporting data from the other one of the images.

Claims

1. A method for image processing, comprising:

receiving a depth image of a scene containing a human subject;

receiving a color image of the scene containing the human subject;

identifying a part of a body of the subject in at least one of the images;

evaluating a quality of both the depth image and the color image, and selecting, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images; and

localizing the identified part in the dominant one of the images, while using supporting data from the other one of the images.

2. The method according to claim 1, wherein receiving the depth image comprises projecting patterned optical radiation toward the subject, and capturing and processing a two-dimensional image of the patterned optical radiation that is reflected from the subject.

3. The method according to claim 2, wherein projecting the patterned optical radiation comprises projecting a spatial pattern of the optical radiation, and wherein processing the two-dimensional image comprises deriving depth coordinates of the body of the subject based on transverse shifts of the spatial pattern in the two-dimensional image.

4. The method according to claim 2, wherein projecting the patterned optical radiation comprises projecting infrared radiation.[

5. The method according to claim 1, wherein the part of the body comprises a hand of the human subject.

6. The method according to claim 5, wherein localizing the identified part comprises identifying a group of pixels belonging to the hand, fitting a parametric model to the group, and finding an orientation of the hand based on a parameter of the model.

7. The method according to claim 6, wherein fitting the parametric model comprises fitting an ellipse to the group, and wherein finding the orientation comprises finding a direction of an axis of the ellipse.

8. The method according to claim 5, wherein localizing the identified part comprises detecting an initial location of the hand by recognizing, in a sequence of the images, a predetermined focal gesture made by the hand, and then tracking the hand beginning from the initial location.

9. The method according to claim 1, wherein localizing the identified part comprises deriving from the depth image and color image respective matrix data structures, having dimensions corresponding to the dimensions of the depth image and color image, and combining at least a part of the respective matrix data structures in order to find a location of the part of the body.

10. The method according to claim 9, wherein deriving the respective matrix data structures comprises deriving respective masks indicating pixels that are candidates to belong to the part of the body, and wherein combining the respective matrix data structures comprises applying a logical operation to the respective masks.

11. The method according to claim 9, wherein at least one of the matrix data structures is indicative of the pixels in at least one of the images in which significant changes occurred over a sequence of the images, thereby representing motion in the images.

12. The method according to claim 9, wherein at least one of the matrix data structures is indicative of morphological features of the part of the body.

13. The method according to claim 9, wherein deriving the respective matrix data structures comprises deriving from the depth image and color image respective maps containing graded values expressing an ascending monotonic function associated with a property of interest at pixels in the images, and wherein combining the respective matrix data structures comprises applying a mathematical operation to the respective maps.

14. The method according to claim 13, wherein the graded values indicate respective probabilities that the pixels to belong to the part of the body, and wherein applying the mathematical operation comprises multiplying the respective maps together in order to find the location of the part of the body.

15. The method according to claim 1, wherein selecting the one of the images comprises selecting the color image to be the dominant one, and wherein localizing the identified part comprises identifying a range of depths in which the part of the body is located, and filtering the color image using depth coordinates from the depth image to eliminate pixels that are outside the range.

16. The method according to claim 1, wherein selecting the one of the images comprises selecting the depth image to be the dominant one, and wherein localizing and tracking the identified part comprises identifying a range of colors associated with the part of the body, and filtering the depth image using color information from the color image to eliminate pixels having colors that are outside the range.

17. The method according to claim 1, wherein localizing the identified part comprises applying a location and motion filter to a sequence of the depth and color images in order to combine depth- and color-based information from the images.

18. The method according to claim 1, wherein localizing the identified part comprises tracking the identified part over a sequence of the images.

19. The method according to claim 18, wherein tracking the identified part comprises processing at least some of the images in the sequence using both the supporting data and information from one or more previous images in the sequence.

20. Apparatus for image processing, comprising:

an input interface coupled to receive a depth image of a scene containing a human subject and to receive a color image of the scene containing the human subject; and

a processor, which is configured to identify a part of a body of the subject in at least one of the images, to evaluate a quality of both the depth image and the color image, and to select, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images, and to localize the identified part in the dominant one of the images, while using supporting data from the other one of the images.

21. The apparatus according to claim 20, and comprising a imaging assembly, which is configured to project patterned optical radiation toward the subject, and to capture and process a two-dimensional image of the patterned optical radiation that is reflected from the subject in order to generate the depth image.

22. The apparatus according to claim 21, wherein the patterned optical radiation comprises a spatial pattern, and wherein the processor is configured to derive depth coordinates of the body of the subject based on transverse shifts of the spatial pattern in the two-dimensional image.

23. The apparatus according to claim 21, wherein the patterned optical radiation comprises infrared radiation.

24. The apparatus according to claim 20, wherein the part of the body comprises a hand of the human subject.

25. The apparatus according to claim 24, wherein the processor is configured to identify a group of pixels belonging to the hand, to fit a parametric model to the group, and to find an orientation of the hand based on a parameter of the model.

26. The apparatus according to claim 25, wherein the parametric model comprises an ellipse that is fitted to the group of the pixels, and wherein the orientation of the hand is indicated by a direction of an axis of the ellipse.

27. The apparatus according to claim 24, wherein the processor is configured to detect an initial location of the hand by recognizing, in a sequence of the images, a predetermined focal gesture made by the hand, and then to track the hand beginning from the initial location.

28. The apparatus according to claim 20, wherein the processor is configured to derive from the depth image and color image respective matrix data structures, having dimensions corresponding to the dimensions of the depth image and color image, and to combine at least a part of the respective matrix data structures in order to find a location of the part of the body.

29. The apparatus according to claim 28, wherein the respective matrix data structures comprise respective masks indicating pixels that are candidates to belong to the part of the body, and wherein the processor is configured to combine the respective masks using a logical operation.

30. The apparatus according to claim 28, wherein at least one of the matrix data structures is indicative of the pixels in at least one of the images in which significant changes occurred over a sequence of the images, thereby representing motion in the images.

31. The apparatus according to claim 28, wherein at least one of the matrix data structures is indicative of morphological features of the part of the body.

32. The apparatus according to claim 28, wherein the respective matrix data structures comprise respective maps containing graded values expressing an ascending monotonic function associated with a property of interest at pixels in the images, and wherein the processor is configured to combine the respective maps using a mathematical operation.

33. The apparatus according to claim 32, wherein the graded values indicate, for pixels in the images, respective probabilities that the pixels to belong to the part of the body, and wherein the processor is configured to multiply the respective masks together in order to find the location of the part of the body.

34. The apparatus according to claim 20, wherein the processor is configured, upon selecting the color image to be the dominant one, to identify a range of depths in which the part of the body is located, and to filter the color image using depth coordinates from the depth image to eliminate pixels that are outside the range.

35. The apparatus according to claim 20, wherein the processor is configured, upon selecting the depth image to be the dominant one, to identify a range of colors associated with the part of the body, and to filter the depth image using color information from the color image to eliminate pixels having colors that are outside the range.

36. The apparatus according to claim 20, wherein the processor is configured to apply a location and motion filter to a sequence of the depth and color images in order to combine depth- and color-based information from the images.

37. The apparatus according to claim 20, wherein the processor is configured to track the identified part over a sequence of the images.

38. The apparatus according to claim 37, wherein the processor is configured to track the identified part by processing at least some of the images in the sequence using both the supporting data and information from one or more previous images in the sequence.

39. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth image of a scene containing a human subject and to receive a color image of the scene containing the human subject, to identify a part of a body of the subject in at least one of the images, to evaluate a quality of both the depth image and the color image, and to select, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images, and to localize the identified part in the dominant one of the images, while using supporting data from the other one of the images.

40. The product according to claim 39, wherein the depth image is generated by projecting patterned optical radiation toward the subject, and capturing and processing a two-dimensional image of the patterned optical radiation that is reflected from the subject.

41. The product according to claim 40, wherein the patterned optical radiation comprises a spatial pattern, and wherein the instructions cause the computer to derive depth coordinates of the body of the subject based on transverse shifts of the spatial pattern in the two-dimensional image.

42. The product according to claim 40, wherein the patterned optical radiation comprises infrared radiation.

43. The product according to claim 39, wherein the part of the body comprises a hand of the human subject.

44. The product according to claim 43, wherein the instructions cause the computer to identify a group of pixels belonging to the hand, to fit a parametric model to the group, and to find an orientation of the hand based on a parameter of the model.

45. The product according to claim 44, wherein the parametric model comprises an ellipse that is fitted to the group of the pixels, and wherein the orientation of the hand is indicated by a direction of an axis of the ellipse.

46. The product according to claim 43, wherein the instructions cause the computer to detect an initial location of the hand by recognizing, in a sequence of the images, a predetermined focal gesture made by the hand, and then to track the hand beginning from the initial location.

47. The product according to claim 39, wherein the instructions cause the computer to derive from the depth image and color image respective matrix data structures, having dimensions corresponding to the dimensions of the depth image and color image, and to combine at least a part of the respective matrix data structures in order to find a location of the part of the body.

48. The product according to claim 47, wherein the respective matrix data structures comprise respective masks indicating pixels that are candidates to belong to the part of the body, and wherein the instructions cause the computer to combine the respective masks using a logical operation.

49. The product according to claim 47, wherein at least one of the matrix data structures is indicative of the pixels in at least one of the images in which significant changes occurred over a sequence of the images, thereby representing motion in the images.

50. The product according to claim 47, wherein at least one of the matrix data structures is indicative of morphological features of the part of the body.

53. The product according to claim 39, wherein the instructions cause the computer, upon selecting the color image to be the dominant one, to identify a range of depths in which the part of the body is located, and to filter the color image using depth coordinates from the depth image to eliminate pixels that are outside the range.

54. The product according to claim 39, wherein the instructions cause the computer, upon selecting the depth image to be the dominant one, to identify a range of colors associated with the part of the body, and to filter the depth image using color information from the color image to eliminate pixels having colors that are outside the range.

55. The product according to claim 39, wherein the instructions cause the computer to apply a location and motion filter to a sequence of the depth and color images in order to combine depth- and color-based information from the images.

56. The product according to claim 39, wherein the instructions cause the computer to track the identified part over a sequence of the images.

57. The product according to claim 56, wherein the instructions cause the computer to track the identified part by processing at least some of the images in the sequence using both the supporting data and information from one or more previous images in the sequence.

51. The product according to claim 4743, wherein the respective matrix data structures comprise respective maps containing graded values expressing an ascending monotonic function associated with a property of interest at pixels in the images, and wherein the instructions cause the computer to combine the respective maps using a mathematical operation.

52. The product according to claim 51, wherein the graded values indicate, for pixels in the images, respective probabilities that the pixels to belong to the part of the body, and wherein the instructions cause the computer to multiply the respective masks together in order to find the location of the part of the body.

58. A method for image processing, comprising:

receiving a depth image of a scene containing a human subject;

receiving a color image of the scene containing the human subject;

identifying a part of a body of the subject in at least one of the images;

evaluating a quality of both the depth image and the color image, and selecting, responsively to the quality, one of the images to be dominant in processing of the part of the body in the images; and

segmenting the identified part in the dominant one of the images, while using supporting data from the other one of the images.