Three-dimensional background removal for vision system

• US 8,526,734 B2
• Filed: 06/01/2011
• Issued: 09/03/2013
• Est. Priority Date: 06/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Al-Ghosien, Momin, Peeper, Craig
• Primary Examiner(s): COUSO, JOSE L
• Application Number:
  US13/151,087
• Publication Number:
  US 20120309517A1
• Time in Patent Office: 825 Days
• Field of Search: 382/103, 382106-107, 382/154, 382/173, 382294-295, 348 42- 43, 348/169, 345419-420, 345/422, 345581-582, 345/606, 345/629, 345473-474, 715/723
• US Class Current: 382/173
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G01S 17/18 wherein range gates are used
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

A method for controlling a computer system includes acquiring video of a subject, and obtaining from the video a time-resolved sequence of depth maps. A geometric model of the subject is fit to each depth map in the sequence and tracked into a subsequent depth map in the sequence. From the subsequent depth map, a background section is selected for exclusion. The background section is one that lacks coherent motion and is located more than a threshold distance from the coordinates of the geometric model tracked in. Then, a subsequent geometric model of the subject is fit to the depth map with the background section excluded.

Claims

1. A method for controlling a computer system, the method comprising:

acquiring video of a subject in front of a background;

obtaining from the video a time-resolved sequence of depth maps, each depth map including an array of pixels;

fitting a geometric model of the subject to a first depth map in the sequence;

registering coordinates of the geometric model to a second depth map in the sequence;

selecting from the second depth map a background section lacking coherent motion and located more than a threshold distance from the coordinates of the geometric model; and

refitting the geometric model of the subject to the second depth map with the background section excluded, said acquiring obtaining, fitting, registering, selecting and refitting enacted within a computer vision system of the computer system.

2. The method of claim 1 wherein the geometric model includes a skeleton having a plurality of skeletal segments pivotally coupled at a plurality of joints.

3. The method of claim 2 wherein a geometric solid is associated with each skeletal segment.

4. The method of claim 1 wherein acquiring the video comprises acquiring with a depth camera.

5. The method of claim 1 wherein the background section is a static section of the second depth map.

6. The method of claim 1 wherein selecting the background section includes selecting each pixel lacking coherent motion and located deeper than any skeletal segment of the geometric model.

7. The method of claim 1 wherein selecting the background section includes selecting each pixel lacking coherent motion and located deeper, by more than a threshold amount, than any skeletal segment of the geometric model.

8. The method of claim 1 wherein selecting the background section includes selecting each pixel lacking coherent motion and located exterior to any skeletal segment of the geometric model.

9. The method of claim 1 wherein selecting the background section includes labeling as a background pixel each pixel of the background section.

10. The method of claim 1 wherein selecting the background section includes selecting each pixel lacking coherent motion and located behind a plane positioned with reference to one or more joints or skeletal segments of the geometric model.

11. The method of claim 1 wherein selecting the background section comprises:

for each pixel of the second depth map, incrementing a corresponding exclusion counter if that pixel has been static for a predetermined number of frames of the video and is more than a threshold distance from any skeletal segment of the geometric model.

12. The method of claim 11 further comprising including in the background section those pixels for which the corresponding exclusion counter is above a threshold value.

13. The method of claim 1 wherein the first depth map corresponds to an earlier frame in the video than does the second depth map.

14. The method of claim 1 wherein the subject is a human subject or user of the computer system.

15. The method of claim 1 wherein fitting the geometric model to the first depth map comprises:

assigning a body-part designation to each of a plurality of contours of the first depth map; and

fitting a skeletal segment to each of the plurality of contours based partly on the body-part designation assigned to each contour.

16. The method of claim 1 further comprising furnishing input in the computer system based on a position or orientation of at least one skeletal segment of the geometric model.

17. A method for controlling a computer system, the method comprising:

acquiring video of a subject in front of a background;

obtaining from the video a time-resolved sequence of depth maps, each depth map including an array of pixels;

fitting a first skeleton of the subject to a first depth map in the sequence;

registering the first skeleton to a second depth map in the sequence;

for each pixel of the second depth map, incrementing a corresponding exclusion counter if that pixel has been static for a predetermined number of frames of the video and is more than a threshold distance from any skeletal segment of the first skeleton;

selecting as a background section those pixels for which the corresponding exclusion counter is above a threshold value; and

fitting a second skeleton of the subject to the second depth map with the background section excluded, said acquiring obtaining, fitting, registering, incrementing and selecting enacted within a computer vision system of the computer system.

18. The method of claim 17 further comprising furnishing input in the computer system based on a position or orientation of at least one skeletal segment of the second skeleton.

19. A game system comprising:

a vision subsystem configured to obtain from a depth camera a sequence of time-resolved depth maps imaging a player, each depth map including an array of pixels;

a logic subsystem operatively coupled to the vision subsystem; and

a data subsystem holding instructions executable by the logic subsystem to;

fit a first skeleton of the player to non-background pixels of a first depth map in the sequence,

identify as background pixels of a second depth map in the sequence those pixels lacking coherent motion and located outside of a predetermined range of the first skeleton, and

fit a second skeleton of the player to non-background pixels of the second depth map.

20. The method of claim 19, wherein the vision subsystem is configured to project pulsed illumination, or discrete illumination features, onto the player, and wherein the depth camera is configured to resolve a time-of-flight of the pulsed illumination, or a spacing between the discrete illumination features, respectively.