Systems and methods for removing a background of an image

• US 10,147,194 B2
• Filed: 05/08/2017
• Issued: 12/04/2018
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Peeper, Craig, Lee, Johnny, Leyvand, Tommer, Stachniak, Szymon
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/589,827
• Publication Number:
  US 20170278251A1
• Time in Patent Office: 575 Days
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

An image such as a depth image of a scene may be received, observed, or captured by a device. A grid of voxels may then be generated based on the depth image such that the depth image may be downsampled. A background included in the grid of voxels may then be discarded to isolate one or more voxels associated with a foreground object such as a human target and the isolated voxels associated with the foreground object may be processed.

Claims

1. A computer-readable storage device having stored thereon computer executable instructions for processing depth information of a scene, the computer executable instructions comprising instructions for:

receiving a depth image of the scene, wherein the depth image comprises one or more objects;

comparing a value associated with voxels in one or more objects in the depth image with a corresponding value of a reference voxel;

identifying voxels as moving when a difference between the value and the corresponding value exceeds a motion threshold;

identifying other voxels as non-moving;

calculating a foreground score based on a percentage of moving voxels associated with one object; and

identifying the one object as moving when the foreground score exceeds a score threshold.

2. The computer-readable storage device of claim 1, the computer executable instructions further comprising instructions for determining whether to merge an object in the one or more objects with a nearby object in the one or more objects.

3. The computer-readable storage device of claim 2, wherein determining whether to merge the object in the one or more objects with the nearby object in the one or more objects comprises:

comparing X-values, Y-values, or depth values associated with voxels in the object with X-values, Y-values, or depth values associated with voxels of the nearby object; and

merging the object with the nearby object if, based on the comparison, the X-values, Y-values, or the depth values associated with the voxels in the object overlap the X-values, Y-values, or depth values associated with the voxels of the nearby object.

4. The computer-readable storage device of claim 1, the computer executable instructions further comprising instructions for:

isolating the one object associated with the voxels identified to be moving; and

processing the one object isolated.

5. The computer-readable storage device of claim 1, the computer executable instructions further comprising instructions for determining that at least one object of the one or more objects is non-moving.

6. The computer-readable storage device of claim 5, the computer executable instructions further comprising instructions for discarding the at least one non-moving object of the one or more objects.

7. A system for processing depth information of a scene, the system comprising:

a capture device comprising an image camera component configured to capture a depth image of the scene; and

a computing device communicatively coupled to the capture device, wherein the computing device comprises a processor and memory storing thereon computer-readable instructions that, when executed by the processor, cause the computing device to perform operations comprising;

receiving a depth image of the scene, wherein the depth image comprises one or more objects;

comparing a value associated with voxels in one or more objects in the depth image with a corresponding value of a reference voxel;

identifying voxels as moving when a difference between the value and the corresponding value exceeds a motion threshold;

identifying other voxels as non-moving;

calculating a foreground score based on a percentage of moving voxels associated with one object; and

identifying the one object as moving when the foreground score exceeds a score threshold.

8. The system of claim 7, further comprising computer-readable instructions that, when executed by the processor, cause the computing device to perform operations comprising determining whether to merge an object in the one or more objects with a nearby object in the one or more objects.

9. The system of claim 8, further comprising computer-readable instructions that, when executed by the processor, cause the computing device to perform operations comprising:

determining whether merge an object in the one or more objects with a nearby object in the one or more objects;

comparing X-values or depth values associated with voxels in one of the object with X-values or depth values associated with voxels of a nearby object; and

merging the object with the nearby object if, based on the comparison, the X-values or the depth values associated with the voxels in the object overlap the X-values or depth values associated with the voxels of the nearby object.

10. The system of claim 7, further comprising computer-readable instructions that, when executed by the processor, cause the computing device to perform operations comprising:

isolating the one object associated with the voxels identified to be moving; and

processing the one object isolated.

11. The system of claim 7, further comprising computer-readable instructions that, when executed by the processor, cause the computing device to perform operations comprising discarding at least one non-moving object of the one or more objects.

12. A method for processing depth information of a scene, the method comprising:

receiving a depth image of the scene, wherein the depth image comprises one or more objects;

comparing a value associated with voxels in one or more objects in the depth image with a corresponding value of a reference voxel;

identifying voxels as moving when a difference between the value and the corresponding value exceeds a motion threshold;

identifying other voxels as non-moving;

calculating a foreground score based on a percentage of moving voxels associated with at least one object; and

identifying the at least one object as moving when the foreground score exceeds a score threshold.

13. The method of claim 12, further comprising determining whether to merge an object in the one or more objects with a nearby object in the one or more objects.

14. The method of claim 13, wherein determining whether to merge the object in the one or more objects with the nearby object in the one or more objects comprises:

comparing X-values, Y-values, or depth values associated with voxels in the object with X-values, Y-values, or depth values associated with voxels of the nearby object; and

merging the object with the nearby object if, based on the comparison, the X-values, Y-values, or the depth values associated with the voxels in the object overlap the X-values, Y-values, or depth values associated with the voxels of the nearby object.

15. The method of claim 12, further comprising:

isolating the at least one object associated with the voxels identified to be moving; and

processing the at least one object isolated.

16. The method of claim 12, further comprising determining that at least one object of the one or more objects is non-moving.

17. The method of claim 16, further comprising discarding at least one non-moving object of the one or more objects.