Moving object segmentation using depth images

• US 8,401,225 B2
• Filed: 01/31/2011
• Issued: 03/19/2013
• Est. Priority Date: 01/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Newcombe, Richard, Izadi, Shahram, Hilliges, Otmar, Kim, David, Molyneaux, David, Shotton, Jamie Daniel Joseph, Kohli, Pushmeet, Fitzgibbon, Andrew, Hodges, Stephen Edward, Butler, David Alexander
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US13/017,626
• Publication Number:
  US 20120195471A1
• Time in Patent Office: 778 Days
• Field of Search: 382100-107, 348/154, 348/155, 348169-171, 73488-491, 356 27- 31
• US Class Current: 382/100
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

Moving object segmentation using depth images is described. In an example, a moving object is segmented from the background of a depth image of a scene received from a mobile depth camera. A previous depth image of the scene is retrieved, and compared to the current depth image using an iterative closest point algorithm. The iterative closest point algorithm includes a determination of a set of points that correspond between the current depth image and the previous depth image. During the determination of the set of points, one or more outlying points are detected that do not correspond between the two depth images, and the image elements at these outlying points are labeled as belonging to the moving object. In examples, the iterative closest point algorithm is executed as part of an algorithm for tracking the mobile depth camera, and hence the segmentation does not add substantial additional computational complexity.

Claims

1. A computer-implemented method of segmenting a moving object from a background portion of a depth image, comprising:

receiving the depth image from a mobile depth camera, the depth image comprising a depth value at each image element, each depth value being related to a distance from the depth camera to a surface in a scene captured by the depth camera;

retrieving a previous depth image of at least a portion of the scene;

executing an iterative closest point algorithm using the depth image and the previous depth image, wherein the iterative closest point algorithm comprises a determination of a plurality of corresponding points in the depth image and the previous depth image; and

detecting one or more outlying points during the determination of the plurality of corresponding points, and labeling those image elements in the depth image at the outlying points as moving object image elements.

2. A method according to claim 1, wherein the step of executing the iterative closest point algorithm further comprises using the corresponding points to determine a transformation for aligning the depth image with the previous depth image.

3. A method according to claim 2, wherein the step of executing the iterative closest point algorithm further comprises using the transformation to track the location and orientation of the depth camera, and storing the location and orientation in association with the depth image.

4. A method according to claim 1, wherein the step of detecting one or more outlying points during determination of the plurality of corresponding points comprises:

selecting a source point from the previous depth image; and

mapping the source point to a destination point in the depth image.

5. A method according to claim 4, wherein the step of detecting one or more outlying points during determination of the plurality of corresponding points further comprises determining whether a distance between the source point and the destination point is greater than a predefined distance threshold.

6. A method according to claim 4, wherein the step of detecting one or more outlying points during determination of the plurality of corresponding points further comprises:

determining a first surface normal at the source point and a second surface normal at the destination point;

calculating a measure relating to the angle between the first surface normal and second surface normal; and

determining whether the measure is greater than a predefined angle threshold.

7. A method according to claim 4, wherein the step of detecting one or more outlying points during determination of the plurality of corresponding points further comprises:

determining whether a distance between the source point and the destination point is greater than a predefined distance threshold; and

if it is determined that the distance is not greater than the predefined distance threshold;

determining a first surface normal at the source point and a second surface normal at the destination point, calculating an angle between the first surface normal and second surface normal, and determining whether the angle is greater than a predefined angle threshold.

8. A method according to claim 4, wherein the step of detecting one or more outlying points during determination of the plurality of corresponding points further comprises:

accessing a first red-green-blue image corresponding to the depth image and a second red-green-blue image corresponding to the previous depth image; and

determining whether the difference between the red-green-blue value at the source point of the second red-green-blue image and the destination point of the first red-green-blue image is greater than a predefined value.

9. A method according to claim 4, further comprising retrieving a previous capture location and orientation for the previous depth image.

10. A method according to claim 9, wherein step of mapping the source point to the destination point comprises projecting onto the destination point from the previous capture location and orientation and the source point.

11. A method according to claim 1, further comprising executing a segmentation algorithm using the labeled image elements at the outlying points to identify further image elements belonging to the moving object in the depth image.

12. A method according to claim 11, wherein the segmentation algorithm is performed on a red-green-blue image of the scene captured concurrently with the depth image.

13. A method according to claim 11, wherein the segmentation algorithm comprises at least one of:

a GrabCut algorithm;

a machine-learning classifier;

an object boundary detection algorithm; and

a morphological operation.

14. A method according to claim 1, further comprising the step of executing an additional iterative closest point algorithm to determine a transformation for aligning only the moving object in the depth image with the previous depth image.

15. A method according to claim 1, further comprising the steps of identifying the moving object using an object recognition algorithm, and substituting the moving object in the depth image with a high-resolution representation of the moving object.

16. An image segmentation system, comprising:

a communication interface arranged to receive a depth image from a mobile depth camera, the depth image comprising a depth value at each image element, each depth value being related to a distance from the depth camera to a surface in a scene captured by the depth camera;

a memory arranged to store a previous depth image of at least a portion of the scene; and

a processor arranged to;

execute an iterative closest point algorithm using the depth image and the previous depth image, wherein the iterative closest point algorithm comprises a determination of a plurality of corresponding points in the depth image and the previous depth image;

detect one or more outlying points during the determination of the plurality of corresponding points; and

label those image elements in the depth image at the outlying points as moving object image elements.

17. A system according to claim 16, wherein the previous depth image was previously captured by the depth camera, received at the communication interface and stored at the memory.

18. A system according to claim 16, further comprising a graphics processing unit and an associated graphics memory arranged to store a dense 3D model of the scene, and

wherein the processor is arranged to retrieve the previous depth image by requesting the previous depth image from the graphics processing unit, and the graphics processing unit is arranged to generate the previous depth image from the dense 3D model stored on the graphics memory.

19. A system according to claim 18, wherein the graphics processing unit is further arranged to create a further dense 3D model representing only the moving object in the graphics memory and augment the further dense 3D model using the labeled image elements of the moving object.

20. One or more tangible device-readable media with device-executable instructions that, when executed by a computing system, direct the computing system to perform steps comprising:

receiving a first depth image from a mobile depth camera, the first depth image comprising a depth value at each image element, each depth value being related to a distance from the depth camera to a surface in a scene captured by the depth camera;

retrieving a previous depth image of at least a portion of the scene and a capture location and orientation for the previous depth image;

executing an iterative closest point algorithm using the first depth image, the previous depth image and the capture location and orientation to determine a transformation for aligning the first depth image with the previous depth image, wherein the iterative closest point algorithm comprises a determination of a plurality of corresponding points in the first depth image and the previous depth image;

detecting one or more outlying points during the determination of the plurality of corresponding points; and

subsequently processing a set of image elements in the first depth image at the outlying points as a separate object moving relative to the rest of the scene.