Systems and methods for removing a background of an image

• US 8,867,820 B2
• Filed: 10/07/2009
• Issued: 10/21/2014
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Peeper, Craig, Lee, Johnny, Leyvand, Tommer, Stachniak, Szymon
• Primary Examiner(s): Seth, Manav
• Application Number:
  US12/575,363
• Publication Number:
  US 20110081044A1
• Time in Patent Office: 1,840 Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may then be discarded to isolate one or more voxels associated with a foreground object such as a human target and the isolated voxels associated with the foreground object may be processed.

Claims

1. A method for processing depth information of a scene, the method comprising:

receiving a depth image;

generating a grid of voxels based on the depth image;

determining whether one or more voxels in the grid are associated with a background by at least comparing a minimum depth value of the voxels in the grid with a reference plate comprising a depth history of the voxels in the grid;

discarding the one or more voxels associated with the background to isolate voxels associated with a foreground object in the depth image; and

processing the non-discarded voxels in the grid.

2. The method of claim 1, further comprising analyzing each of the voxels in the grid of voxels to determine one or more objects in the depth image.

3. The method of claim 2, wherein analyzing each of the voxels in the grid of voxels to determine one or more objects in the depth image comprises:

comparing a value associated with a particular voxel in the grid with corresponding values of voxels adjacent to the particular voxel in the grid; and

identifying the particular voxel and one or more of the adjacent voxels as being part of an object if, based on the comparison, a difference between the value associated with the particular voxel in the grid and the corresponding values is less than a threshold.

4. The method of claim 2, further comprising determining whether to merge an object in the one or more objects with a nearby object in the one or more objects.

5. The method of claim 4, wherein determining whether to merge the object in the one or more objects with the nearby object in the one or more objects comprises:

comparing X-values, Y-values, or depth values associated with voxels in the object with X-values, Y-values, or depth values associated with voxels of the nearby object; and

merging the object with the nearby object if, based on the comparison, the X-values, Y-values, or the depth values associated with the voxels in the object overlap the X-values, Y-values, or depth values associated with the voxels of the nearby object.

6. The method of claim 2, wherein determining whether the one or more voxels in the grid are associated with the background comprises determining whether an object of the one or more objects in the grid is moving.

7. The method of claim 6, wherein the one or more objects in the grid that are moving are not associated with the background.

8. The method of claim 6, wherein determining whether the object of the one or more objects in the grid is moving comprises:

comparing a first value associated with each voxel in the object with a corresponding value of a reference voxel included in a reference plate; and

identifying each voxel in the object as moving when a difference between the first value and the corresponding value exceeds a motion threshold.

9. The method of claim 8, wherein determining whether the object of the one or more objects in the grid is moving further comprises:

calculating a foreground score based on a percentage of moving voxels associated with the object; and

identifying the object as moving when the foreground score exceeds a motion threshold.

10. A method for processing depth information of a scene, the method comprising:

receiving a depth image;

generating a grid of voxels based on the depth image;

analyzing each of the voxels in the grid of voxels to determine one or more objects in the depth image;

determining whether one or more voxels in the grid are associated with a background by at least determining whether an object of the one or more objects in the grid is moving by at least;

comparing a first value associated with each voxel in the object with a corresponding value of a reference voxel included in a reference plate; and

identifying each voxel in the object as moving when a difference between the first value and the corresponding value exceeds a motion threshold;

discarding the one or more voxels associated with the background to isolate voxels associated with a foreground object in the depth image; and

processing the non-discarded voxels in the grid.

11. The method of claim 10, wherein analyzing each of the voxels in the grid of voxels to determine one or more objects in the depth image comprises:

comparing a value associated with a particular voxel in the grid with corresponding values of voxels adjacent to the particular voxel in the grid; and

identifying the particular voxel and one or more of the adjacent voxels as being part of an object if a difference between the value associated with a particular voxel in the grid and the corresponding values is less than a threshold.

12. The method of claim 10, further comprising determining whether to merge an object in the one or more objects with a nearby object in the one or more objects.

13. The method of claim 12, wherein determining whether to merge the object in the one or more objects with the nearby object in the one or more objects comprises:

comparing X-values, Y-values, or depth values associated with voxels in the object with X-values, Y-values, or depth values associated with voxels of the nearby object; and

merging the object with the nearby object if, based on the comparison, the X-values, Y-values, or the depth values associated with the voxels in the object overlap the X-values, Y-values, or depth values associated with the voxels of the nearby object.

14. The method of claim 10, wherein the one or more objects in the grid that are moving are not associated with the background.

15. The method of claim 10, wherein determining whether the object of the one or more objects in the grid is moving further comprises:

calculating a foreground score based on a percentage of moving voxels associated with the object; and

identifying the object as moving when the foreground score exceeds a motion threshold.

16. A computer-readable storage device having stored thereon computer-executable instructions that, when executed by a processor, implement a method for processing depth information of a scene, the method implemented by the computer-executable instructions comprising:

receiving a depth image;

generating a grid of voxels based on the depth image;

determining whether one or more voxels in the grid are associated with a background by at least comparing a minimum depth value of the voxels in the grid with a reference plate comprising a depth history of the voxels in the grid;

discarding the one or more voxels associated with the background to isolate voxels associated with a foreground object in the depth image; and

processing the non-discarded voxels in the grid.

17. The computer-readable storage device of claim 16, wherein the method implemented by the computer-executable instructions further comprises analyzing each of the voxels in the grid of voxels to determine one or more objects in the depth image.

18. The computer-readable storage device of claim 17, wherein analyzing each of the voxels in the grid of voxels to determine one or more objects in the depth image comprises:

comparing a value associated with a particular voxel in the grid with corresponding values of voxels adjacent to the particular voxel in the grid; and

identifying the particular voxel and one or more of the adjacent voxels as being part of an object if, based on the comparison, a difference between the value associated with a particular voxel in the grid and the corresponding values is less than a threshold.

19. The computer-readable storage device of claim 17, wherein the method implemented by the computer-executable instructions further comprises determining whether to merge an object in the one or more objects with a nearby object in the one or more objects.

20. The computer-readable storage device of claim 19, wherein determining whether to merge the object in the one or more objects with the nearby object in the one or more objects comprises:

comparing X-values, Y-values, or depth values associated with voxels in the object with X-values, Y-values, or depth values associated with voxels of the nearby object; and

merging the object with the nearby object if, based on the comparison, the X-values, Y-values, or the depth values associated with the voxels in the object overlap the X-values, Y-values, or depth values associated with the voxels of the nearby object.