Method and system to segment depth images and to detect shapes in three-dimensionally acquired data

• US 9,165,368 B2
• Filed: 08/26/2011
• Issued: 10/20/2015
• Est. Priority Date: 02/08/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Rafii, Abbas, Gokturk, Salih Burak
• Primary Examiner(s): CONWAY, THOMAS A
• Application Number:
  US13/219,349
• Publication Number:
  US 20110311101A1
• Time in Patent Office: 1,516 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A method and system analyzes data acquired by image systems to more rapidly identify objects of interest in the data. In one embodiment, z-depth data are segmented such that neighboring image pixels having similar z-depths are given a common label. Blobs, or groups of pixels with a same label, may be defined to correspond to different objects. Blobs preferably are modeled as primitives to more rapidly identify objects in the acquired image. In some embodiments, a modified connected component analysis is carried out where image pixels are pre-grouped into regions of different depth values preferably using a depth value histogram. The histogram is divided into regions and image cluster centers are determined. A depth group value image containing blobs is obtained, with each pixel being assigned to one of the depth groups.

Claims

1. A method for use in tracking movement of at least one object in a series of images, comprising:

(a) receiving multiple images including image depth data from an array of pixels acquired by three-dimensional imaging system, each image including z-depth data that comprises a z-depth value of each pixel from at least a fraction of optical energy emitted by the imaging system reflected by at least one object, said image depth data generated by comparing emitted optical energy and the fraction of emitted optical energy detected by said system, and for each of the multiple images;

(a1) examining at least some of said image depth data captured during a single data acquisition;

(a2) using a depth value histogram that records the z-depth value of each pixel of one or more regions, forming segments by grouping together those of said pixels whose acquired said data captured at step (a1) have at least one common characteristic selected from a group consisting of depth distance, depth distance change, object size, and object shape, and assigning pixels in the group a single image cluster center having a depth group value, where the single image cluster center is the geometric center of said group of pixels;

(a3) assigning on a per-segment basis a label to pixels comprising each of said segments formed in step (a2), the pixels so labeled enabling recognition of said at least one object with image depth data; and

(b) tracking motion of the at least one object relative to the system.

2. The method of claim 1 wherein said receiving includes acquiring, in said array of pixels, intensity data;

and step (a2) includes forming segments by grouping together pixels based upon their acquired intensity data.

3. The method of claim 1 wherein for each of the multiple images, the method further includes steps:

(a4) providing at least one pre-defined blob;

(a5) examining said segments formed in step (a2) to determine presence in said image data of a blob matching a pre-defined blob provided at step (a4); and

(a6) providing for output a signal commensurate with detection of a recognized pre-defined blob in said image data.

4. The method of claim 1 wherein step (a2) includes forming segments by grouping together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned a single label.

5. The method of claim 1 wherein step (a2) includes forming segments by grouping together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned intermediate values associated with said objects.

6. The method of claim 1 wherein said step (b) comprises tracking movement of said at least one object by noting movement of commonly labeled pixels.

7. The method of claim 1 wherein the method includes tracking movement of at least two objects, and wherein the three-dimension imaging system comprises a time-of-flight imaging system.

8. An image-analyzing system for tracking objects based on image data, the system including:

a three-dimensional imaging system having an emitter of optical energy, an array of pixels receiving at least a fraction of the emitted optical energy reflected by at least one object;

a depth image data acquisition system operable to generate said image depth data by comparing emitted optical energy and detected optical energy by said array of pixels; and

a processor and memory, the memory including code operable to instruct the processor to track movement of the at least one object by analyzing multiple images, the analyzing including, for each image;

(a) examining at least some of said image depth data captured during a single data acquisition;

(b) forming segments by grouping pixels in each image having a depth distance within a threshold distance from each other and assigning pixels in the group a single image cluster center having a depth group value, where the single image cluster center is the geometric center of said group of pixels, and wherein each pixel in the group is assigned the depth group value of the image cluster center;

(c) assigning on a per-segment basis a label to pixels comprising each of said segments formed to enable recognition of said at least one object with a single acquisition of image depth data; and

(d) for the multiple images, tracking movement of said at least one object by noting movement of commonly labeled pixels.

9. The image-analyzing system of claim 8 wherein the memory is operable to store at least one pre-defined blob comprising a set of commonly labeled pixels;

and code instructing the processor to;

compare said segments formed for each image with contents of said memory to determine presence in said image data of a blob matching a pre-defined blob stored in said memory; and

provide an output signal commensurate with detection of a recognized pre-defined blob in said image data.

10. The image-analyzing system of claim 9 further including a capability to track at least two objects.

11. The image-analyzing system of claim 8 wherein said three-dimensional imaging system is selected from a group consisting of (i) a time-counting time-of-flight system, and (ii) a time-of-flight system employing phase-shift z-depth measurement.

12. The image-analyzing system of claim 9, wherein:

said array of pixels is further configured to acquire intensity data in a single acquisition of image depth data; and

said forming segments by grouping is performed based upon at least one pixel characteristic selected from a group consisting of intensity data acquired by said pixels, z-depth data acquired by said pixels, and z-depth acquired data using pre-determined z-depth values.

13. The image-analyzing system of claim 9, wherein said forming segments by grouping includes grouping together pixels that are within a number k of neighboring pixels, where k is selected from a group consisting of k=4 and k=8.

14. The image-analyzing system of claim 9, wherein said forming segments by grouping includes grouping together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned a single label.

15. The image-analyzing system of claim 9, wherein said forming segments by grouping includes grouping together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned intermediate values associated with said objects.

16. The image-analyzing system of claim 9, wherein said forming segments by grouping includes grouping together pixels based upon their acquired z-depth data, and does so in a manner selected from a group consisting of (i) pre-grouping pixels based upon a histogram record of z-depth values of pixels in said array, and (ii) pre-grouping pixels based upon their acquired z-depth data, wherein pixels are pre-grouped based upon a histogram record of z-depth values of pixels in said array, where said histogram is divided into at least two sub-regions and image cluster centers are determined such that each relevant pixel of said pixels in said array is assigned to a z-depth-determined group.

17. A computer readable hardware storage device including instructions causing a microprocessor to track movement of at least one object in a series of images, the instructions comprising:

(a) receiving multiple images including image depth data acquired by three-dimensional imaging system, image depth data comprising a z-depth value of each of a plurality of pixels, and for each of the multiple images;

(a1) examining at least some of said image depth data;

(a2) using a depth value histogram of one or more regions, wherein the depth value histogram records the z-depth values of pixels in the one or more regions, forming segments by grouping together those of pixels in the image data whose acquired said data captured at step (a) have at least one common characteristic selected from a group consisting of depth distance, object size, and object shape, and assigning pixels in the group a single image cluster center having a depth group value, where the single image cluster center is the geometric center of said group of pixels, and wherein each pixel in the group is assigned the depth group value of the image cluster center;

(a3) assigning on a per-segment basis a label to pixels comprising each of said segments formed in step (a2), the pixels so labeled enabling recognition of said at least one object with an acquisition of image depth data; and

(b) tracking motion of the at least one object relative to the system over the multiple images.

18. The tangible computer readable storage device of claim 17 wherein the step of tracking is performed by noting movement of commonly labeled pixels.

19. The tangible computer readable storage device of claim 18 further including instructions comprising:

storing in a memory at least one pre-defined blob comprising a set of commonly labeled pixels;

comparing said segments formed for each image with contents of said memory to determine presence in said image data of a blob matching a pre-defined blob stored in said memory; and

providing an output signal commensurate with detection of a recognized pre-defined blob in said image data.

20. The computer readable storage device of claim 19 further including tracking at least two objects.