Analysis of three-dimensional scenes

• US 8,594,425 B2
• Filed: 08/11/2010
• Issued: 11/26/2013
• Est. Priority Date: 05/31/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Gurman, Amiad, Yanir, Tomer, Brand, Michael, Guendelman, Eran
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): PARK, SOO JIN
• Application Number:
  US12/854,187
• Publication Number:
  US 20110293137A1
• Time in Patent Office: 1,203 Days
• Field of Search: 382/103, 382/154, 382/173, 345/419
• US Class Current: 382/173
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/507 from shading G06T7/586 takes precedence
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method for processing data includes receiving a depth map of a scene containing a humanoid form. The depth map is processed so as to identify three-dimensional (3D) connected components in the scene, each connected component including a set of the pixels that are mutually adjacent and have mutually-adjacent depth values. Separate, first and second connected components are identified as both belonging to the humanoid form, and a representation of the humanoid form is generated including both of the first and second connected components.

Claims

1. A method for processing data, comprising:

receiving a temporal sequence of depth maps of a scene containing a stationary background and one or more moving foreground objects that conceal a part of the background, the depth maps comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene; and

using a digital processor, constructing a background model representing the stationary background of the scene by saving, at each pixel, a maximum depth value found at the pixel over the sequence of the depth maps, such that at least a portion of the stationary background that is concealed by the moving foreground objects in a first map in the sequence is added to the background model by finding greater depth values at the corresponding pixels in at least a second depth map in the sequence in which the portion of the stationary background is no longer concealed due to motion of at least one of the foreground objects over the sequence.

2. The method according to claim 1, and comprising generating one or more foreground maps by removing, based on the background model, one or more background objects from the depth maps.

3. The method according to claim 2, wherein at least one of the moving foreground objects is a humanoid figure, and wherein the method comprises extracting the humanoid figure from a foreground map derived from the second depth map together with information relating to the humanoid figure in the first depth map.

4. The method according to claim 3, wherein extracting the humanoid figure comprises detecting an area belonging to the humanoid figure that is touching one of the background objects, and identifying the area as a part of the humanoid figure.

5. The method according to claim 1, and comprising identifying a planar surface in the depth maps.

6. The method according to claim 1, and comprising identifying a shadow of an object in the depth maps.

7. The method according to claim 6, wherein identifying the shadow comprises identifying an area of the depth maps that has no depth value and is adjacent to the object as belonging to the shadow of the object.

8. The method according to claim 7, wherein constructing the background model comprises removing from the background model a background depth value of a given pixel in the shadow when an object depth value of the object casting the shadow is greater than the background depth value of the given pixel.

9. The method according to claim 6, wherein constructing the background model comprises, when a given pixel that is not in an identified shadow has no depth value available over a predefined number of the depth maps in the sequence, removing a background depth value of the given pixel from the background model.

10. The method according to claim 1, wherein constructing the background model comprises updating the background model responsively to changes that appear in the background over the sequence of the depth maps.

11. The method according to claim 10, wherein updating the background model comprises detecting a connected component that is adjacent to the background in a given area of one of the depth maps, and correcting the depth values of the background model in the area in response to detection of the connected component.

12. The method according to claim 10, wherein updating the background model comprises attaching a group of the pixels having foreground depth values to the stationary background when the group of the pixels is not identified as being a part of an moving object over a certain number of the depth maps in the sequence.

13. The method according to claim 1, wherein constructing the background model comprises identifying, in one of the depth maps in the sequence, a foreground connected component adjacent to the background that contains a given pixel in the background model that does not have a background depth value, and assigning a current depth value of the given pixel in the one of the depth maps to be the background depth value of the given pixel.

14. A method for processing data, comprising:

receiving a temporal sequence of depth maps of a scene containing a stationary background and one or more moving foreground objects that conceal a part of the background, the depth maps comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene;

using a digital processor, processing a first depth map in the sequence so as to identify a first portion of the stationary background that is not concealed by the moving foreground objects in the first map;

processing at least a second depth map in the sequence so as to identify at least a second portion of the stationary background, which is different from the first portion due to motion of at least one of the foreground objects over the sequence; and

constructing a background model of the scene by combining at least the first and second portions of the stationary background,

wherein constructing the background model comprises updating the background model responsively to changes that appear in the background over the sequence of the depth maps, and

wherein the depth maps are captured by an imaging assembly, and wherein the method comprises assessing an extent of the changes in the background model between the depth maps in the sequence, and detecting that the imaging assembly has moved when the extent of the changes exceeds a predetermined threshold.

15. Apparatus for processing data, comprising:

an imaging assembly, which is configured to produce a temporal sequence of depth maps of a scene containing a stationary background and one or more moving foreground objects that conceal a part of the background, the depth maps comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene; and

a processor, which is configured to process the depth maps in the sequence so as to construct a background model representing the stationary background of the scene by saving, at each pixel, a maximum depth value found at the pixel over the sequence of the depth maps, such that at least a portion of the stationary background that is concealed by the moving foreground objects in a first map in the sequence is added to the background model by finding greater depth values at the corresponding pixels in at least a second depth map in the sequence in which the portion of the stationary background is no longer concealed due to motion of at least one of the foreground objects over the sequence.

16. The apparatus according to claim 15, wherein the processor is configured to generate one or more foreground maps by removing, based on the background model, one or more background objects from the depth maps.

17. The apparatus according to claim 16, wherein at least one of the moving foreground objects is a humanoid figure, and wherein the processor is configured to extract the humanoid figure from a foreground map derived from the second depth map together with information relating to the humanoid figure in the first depth map.

18. The apparatus according to claim 17, wherein the processor is configured to detect an area belonging to the humanoid figure that is touching one of the background objects, and to identify the area as a part of the humanoid figure.

19. The apparatus according to claim 15, wherein the processor is configured to identify a planar floor in the depth maps.

20. The apparatus according to claim 15, wherein the processor is configured to identify a shadow of an object in the depth maps.

21. The apparatus according to claim 20, wherein the processor is configured to identify an area of the depth maps that has no depth value and is adjacent to the object as belonging to the shadow of the object.

22. The apparatus according to claim 21, wherein the processor is configured to remove from the background model a background depth value of a given pixel in the shadow when an object depth value of the object casting the shadow is greater than the background depth value of the given pixel.

23. The apparatus according to claim 20, wherein the processor is configured to remove from the background model a background depth value of a given pixel that is not in an identified shadow when the given pixel has no depth value available over a predefined number of the depth maps in the sequence.

24. The apparatus according to claim 15, wherein the processor is configured to update the background model responsively to changes that appear in the background over the sequence of the depth maps.

25. The apparatus according to claim 24, wherein the processor is configured to detect a connected component that is adjacent to the background in a given area of one of the depth maps, and to correct the depth values of the background model in the area in response to detection of the connected component.

26. The apparatus according to claim 24, wherein the processor is configured to attach a group of the pixels having foreground depth values to the stationary background when the group of the pixels is not identified as being a part of an moving object over a certain number of the depth maps in the sequence.

27. The apparatus according to claim 15, wherein the processor is configured to identify, in one of the depth maps in the sequence, a foreground connected component adjacent to the background that contains a given pixel in the background model that does not have a background depth value, and to assign a current depth value of the given pixel in the one of the depth maps to be the background depth value of the given pixel.

28. Apparatus for processing data, comprising:

an imaging assembly, which is configured to produce a temporal sequence of depth maps of a scene containing a stationary background and one or more moving foreground objects that conceal a part of the background, the depth maps comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene; and

a processor, which is configured to process a first depth map in the sequence so as to identify a first portion of the stationary background that is not concealed by the moving foreground objects in the first map, and to process at least a second depth map in the sequence so as to identify at least a second portion of the stationary background, which is different from the first portion due to motion of at least one of the foreground objects over the sequence, and to construct a background model of the scene by combining at least the first and second portions of the stationary background,

wherein the processor is configured to update the background model responsively to changes that appear in the background over the sequence of the depth maps, and

wherein the processor is configured to assess an extent of the changes in the background model between the depth maps in the sequence, and to detect that the imaging assembly has moved when the extent of the changes exceeds a predetermined threshold.

29. A computer software product, comprising a tangible, non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a temporal sequence of depth maps of a scene containing a stationary background and one or more moving foreground objects that conceal a part of the background, the depth maps comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene,

wherein the instructions cause the computer to process the depth maps in the sequence so as to construct a background model representing the stationary background of the scene by saving, at each pixel, a maximum depth value found at the pixel over the sequence of the depth maps, such that at least a portion of the stationary background that is concealed by the moving foreground objects in a first map in the sequence is added to the background model by finding greater depth values at the corresponding pixels in at least a second depth map in the sequence in which the portion of the stationary background is no longer concealed due to motion of at least one of the foreground objects over the sequence.

30. The product according to claim 29, wherein the instructions cause the computer to generate one or more foreground maps by removing, based on the background model, one or more background objects from the depth maps.

31. The product according to claim 30, wherein at least one of the moving foreground objects is a humanoid figure, and wherein the instructions cause the computer to extract the humanoid figure from a foreground map derived from the second depth map together with information relating to the humanoid figure in the first depth map.

32. The product according to claim 31, wherein the instructions cause the computer to detect an area belonging to the humanoid figure that is touching one of the background objects, and to identify the area as a part of the humanoid figure.

33. The product according to claim 29, wherein the instructions cause the computer to identify a planar floor in the depth maps.

34. The product according to claim 29, wherein the instructions cause the computer to identify a shadow of an object in the depth maps.

35. The product according to claim 34, wherein the instructions cause the computer to identify an area of the depth maps that has no depth value and is adjacent to the object as belonging to the shadow of the object.

36. The product according to claim 35, wherein the instructions cause the computer to remove from the background model a background depth value of a given pixel in the shadow when an object depth value of the object casting the shadow is greater than the background depth value of the given pixel.

37. The product according to claim 34, wherein the instructions cause the computer to remove from the background model a background depth value of a given pixel that is not in an identified shadow when the given pixel has no depth value available over a predefined number of the depth maps in the sequence.

38. The product according to claim 29, wherein the instructions cause the computer to update the background model responsively to changes that appear in the background over the sequence of the depth maps.

39. The product according to claim 38, wherein the instructions cause the computer to detect a connected component that is adjacent to the background in a given area of one of the depth maps, and to correct the depth values of the background model in the area in response to detection of the connected component.

40. The product according to claim 38, wherein the instructions cause the computer to attach a group of the pixels having foreground depth values to the stationary background when the group of the pixels is not identified as being a part of an moving object over a certain number of the depth maps in the sequence.

41. The product according to claim 29, wherein the instructions cause the computer to identify, in one of the depth maps in the sequence, a foreground connected component adjacent to the background that contains a given pixel in the background model that does not have a background depth value, and to assign a current depth value of the given pixel in the one of the depth maps to be the background depth value of the given pixel.

42. A computer software product, comprising a tangible, non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a temporal sequence of depth maps of a scene containing a stationary background and one or more moving foreground objects that conceal a part of the background, the depth maps comprising a matrix of pixels, at least some of which have respective pixel depth values and correspond to respective locations in the scene,

wherein the instructions cause the computer to process a first depth map in the sequence so as to identify a first portion of the stationary background that is not concealed by the moving foreground objects in the first map, and to process at least a second depth map in the sequence so as to identify at least a second portion of the stationary background, which is different from the first portion due to motion of at least one of the foreground objects over the sequence, and to construct a background model of the scene by combining at least the first and second portions of the stationary background, and

wherein the instructions cause the computer to update the background model responsively to changes that appear in the background over the sequence of the depth maps, and

wherein the instructions cause the computer to assess an extent of the changes in the background model between the depth maps in the sequence, and to detect that the imaging assembly has moved when the extent of the changes exceeds a predetermined threshold.