Three-dimensional foreground selection for vision system

• US 9,594,430 B2
• Filed: 06/01/2011
• Issued: 03/14/2017
• Est. Priority Date: 06/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Al-Ghosien, Momin, Peeper, Craig
• Primary Examiner(s): Eisen, Alexander
• Assistant Examiner(s): LAM, NELSON C
• Application Number:
  US13/151,050
• Publication Number:
  US 20120306735A1
• Time in Patent Office: 2,113 Days
• Field of Search: 345418-422, 345473-474, 345581-582, 345/606, 348/42, 348/51, 348169-172, 382/103, 382106-107, 382/154, 382173-174
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method for controlling a computer system includes acquiring video of a subject, and obtaining from the video a time-resolved sequence of depth maps. An area targeting motion is selected from each depth map in the sequence. Then, a section of the depth map bounded by the area and lying in front of a plane is selected. This section of the depth map is used for fitting a geometric model of the subject.

Claims

1. A method for controlling a computer system, the method comprising:

acquiring video of a subject in front of a background;

obtaining from the video a time-resolved sequence of depth maps;

enacting area-wise motion detection on the sequence of depth maps to select from a depth map an area enclosing a moving contour, excluding a non-moving contour, and thereby targeting motion;

defining an axis parallel to an average surface normal of one or more contour gradients within the area;

selecting as foreground a section of the depth map bounded by the area and lying in front of a plane normal to that axis, and associating with the background, pixels of the depth map located behind the plane or outside the area; and

fitting a geometric model of the subject to the section selected as the foreground.

2. The method of claim 1 wherein acquiring the video comprises acquiring with a depth camera.

3. The method of claim 1 wherein the motion targeted is a coherent motion.

4. The method of claim 1 wherein the subject is a human subject or user of the computer system.

5. The method of claim 1 wherein selecting the section includes positioning the plane to intersect the axis at a starting position and advancing the plane along the axis, wherein a section bounded by the area and lying in front of the plane is selected for each position of the plane.

6. The method of claim 5 further comprising halting advance of the plane when the section sufficiently resembles the subject.

7. The method of claim 5 wherein the starting position is determined based on estimated dimensions of the subject.

8. The method of claim 1 wherein the axis is oriented in a direction that the subject is facing.

9. The method of claim 1 further comprising assigning a body-part designation to a contour inside the area, wherein defining the axis comprises defining based on the contour to which the body-part designation is assigned.

10. The method of claim 9 wherein the body-part designation assigns the contour as a head of the subject, and wherein the plane is positioned immediately above the contour.

11. The method of claim 9 wherein the body-part designation assigns the contour as a shoulder of the subject, and wherein the plane is positioned immediately behind the contour.

12. The method of claim 1 further comprising furnishing input in the computer system based on a position or orientation of at least one skeletal segment of the geometric model.

13. A method for controlling a computer system, the method comprising:

acquiring video of a subject in front of a background;

obtaining from the video a time-resolved sequence of depth maps;

selecting from a depth map in the sequence an area targeting motion;

defining an axis parallel to an average surface normal of one or more contour gradients within the area, the axis oriented in a direction that a torso of the subject is facing;

positioning a plane to intersect the axis at a starting position determined based on estimated dimensions of the subject;

advancing the plane along the axis;

for each position of the plane, selecting as foreground a section bounded by the area and lying in front of the plane;

halting advance of the plane when the section sufficiently resembles the subject;

associating with the background, pixels of the depth map located behind the plane or outside the area; and

fitting a geometric model of the subject to the section selected as the foreground.

14. The method of claim 13 wherein the area is a rectangle in a two-dimensional domain of the depth map, the area defining a rectangular box open at two, opposite ends and having four closed faces and four edges all parallel to a depth coordinate.

15. The method of claim 13 wherein the starting position is one to two meters behind a nearest depth value within the area, and wherein advancing the plane includes advancing by intervals of two centimeters or less.

16. The method of claim 13 wherein the area encloses a moving contour and excludes a non-moving contour and is selected based on area-wise motion detection on the sequence of depth maps.

17. A system comprising:

a vision subsystem configured to obtain from a depth camera a sequence of time-resolved depth maps imaging a human subject;

a logic subsystem operatively coupled to the vision subsystem; and

a data subsystem holding instructions executable by the logic subsystem to;

based on area-wise motion detection on the sequence of depth maps, identify in a depth map an area enclosing a moving contour and excluding a non-moving contour;

based on contour analysis inside the area, identify a direction that a torso of the subject is facing, the direction being parallel to an average surface normal of one or more contour gradients within the area;

select as foreground a section of the depth map bounded by the area and lying in front of a plane normal to the direction, and associate with the background, pixels of the depth map located behind the plane or outside the area, and

fit a geometric model of the subject to the section selected as the foreground.

18. The system of claim 17 wherein the vision subsystem is configured to project pulsed illumination, or discrete illumination features, onto the subject, and wherein the depth camera is configured to resolve a time-of-flight of the pulsed illumination, or a spacing between the discrete illumination features, respectively.