METHOD AND SYSTEM TO SEGMENT DEPTH IMAGES AND TO DETECT SHAPES IN THREE-DIMENSIONALLY ACQUIRED DATA

• US 20110311101A1
• Filed: 08/26/2011
• Published: 12/22/2011
• Est. Priority Date: 02/08/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rafii, Abbes, Gokturk, Salih Burak
• Granted Patent: US 9,165,368 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A method and system analyzes data acquired by image systems to more rapidly identify objects of interest in the data. In one embodiment, z-depth data are segmented such that neighboring image pixels having similar z-depths are given a common label. Blobs, or groups of pixels with a same label, may be defined to correspond to different objects. Blobs preferably are modeled as primitives to more rapidly identify objects in the acquired image. In some embodiments, a modified connected component analysis is carried out where image pixels are pre-grouped into regions of different depth values preferably using a depth value histogram. The histogram is divided into regions and image cluster centers are determined. A depth group value image containing blobs is obtained, with each pixel being assigned to one of the depth groups.

Claims

1-20. -20. (canceled)

21. A method for use in tracking movement of at least one object in a series of images, comprising:

(a) receiving multiple images including image depth data from an array of pixels acquired by three-dimensional time-of-flight imaging system, each image including z-depth data from at least a fraction of optical energy emitted by the imaging system reflected by at least one object, said image depth data generated by comparing emitted optical energy and the fraction of emitted optical energy detected by said system, and for each of the multiple images;

(a1) examining at least some of said image depth data captured during a single data acquisition;

(a2) forming segments by grouping together those of said pixels whose acquired said data captured at step (a1) have at least one common characteristic selected from a group consisting of depth distance, depth distance change, object size, and object shape;

(a3) assigning on a per-segment basis a label to pixels comprising each of said segments formed in step (a2), the pixels so labeled enabling recognition of said at least one object with image depth data; and

(b) tracking motion of the at least one object relative to the system.

22. The method of claim 21 wherein said receiving includes acquiring, in said array of pixels, intensity data;

and step (a2) includes forming segments by grouping together pixels based upon their acquired intensity data.

23. The method of claim 21 wherein for each of the multiple images, the method further includes steps:

(a4) providing at least one pre-defined blob;

(a5) examining said segments formed in step (a2) to determine presence in said image data of a blob matching a pre-defined blob provided at step (a4);

and (a6) providing for output a signal commensurate with detection of a recognized pre-defined blob in said image data.

24. The method of claim 21 wherein step (a2) includes forming segments by grouping together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned a single label.

25. The method of claim 21 wherein step (a2) includes forming segments by grouping together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned values intermediate values associated with said objects.

26. The method of claim 21 wherein said step (b) comprises tracking movement of said at least one object by noting movement of commonly labeled pixels.

27. The method of claim 21 wherein the method includes tracking movement of at least two objects.

28. An image-analyzing system for tracking objects based on image data, the system including:

a three-dimensional time-of-flight imaging system having an emitter of optical energy, an array of pixels receiving at least a fraction of the emitted optical energy reflected by at least one object,

a depth image data acquisition system generating said image depth data by comparing emitted optical energy and detected optical energy by said array of pixels,

a processor and memory, the memory including code instructing the processor to track movement of the at least one object by analyzing multiple images, the analyzing including, for each image;

examining at least some of said image depth data captured during a single data acquisition;

forming segments by grouping pixels in each image whose data share at least one common characteristic selected from a group consisting of depth distance, depth distance change, object size and object shape; and

assigning on a per-segment basis a label to pixels comprising each of said segments formed to enable recognition of said at least one object with a single acquisition of image depth data;

and for the multiple images, tracking movement of said at least one object by noting movement of commonly labeled pixels.

29. The image-analyzing system of claim 28, wherein the memory stores at least one pre-defined blob comprising a set of commonly labeled pixels;

and code instructing the processor to;

compare said segments formed for each image with contents of said memory to determine presence in said image data of a blob matching a pre-defined blob stored in said memory; and

provide an output signal commensurate with detection of a recognized pre-defined blob in said image data.

30. The image-analyzing system of claim 29 further including tracking at least two objects.

31. The image-analyzing system of claim 28 wherein said a three-dimensional time-of-flight imaging system is selected from a group consisting of (i) a time-counting time-of-flight system, and (ii) a time-of-flight system employing phase-shift z-depth measurement.

32. The image-analyzing system of claim 29, wherein:

said array of pixels further acquires intensity data in a single acquisition of image depth data; and

said forming segments by grouping is performed based upon at least one pixel characteristic selected from a group consisting of intensity data acquired by said pixels, z-depth data acquired by said pixels, and z-depth acquired data using pre-determined z-depth values.

33. The image-analyzing system of claim 29, wherein said forming step groups together pixels that are within a number k of neighboring pixels, where k is selected from a group consisting of k=4 and k=8.

34. The image-analyzing system of claim 29, wherein said forming step groups together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned a single label.

35. The image-analyzing system of claim 29, wherein said forming step groups together pixels based upon their acquired z-depth data, wherein pixels representing an overlapping portion of objects in said image data are assigned values intermediate values associated with said objects.

36. The image-analyzing system of claim 29, wherein said forming step groups together pixels based upon their acquired z-depth data, and does so in a manner selected from a group consisting of (i) pre-grouping pixels based upon a histogram record of z-depth values of pixels in said array, and (ii) pre-grouping pixels based upon their acquired z-depth data, wherein pixels are pre-grouped based upon a histogram record of z-depth values of pixels in said array, where said histogram is divided into at least two sub-regions and image cluster centers are determined such that each relevant pixel of said pixels in said array is assigned to a z-depth-determined group.

37. A computer readable medium including instructions causing a microprocessor to track movement of at least one object in a series of images, the instructions comprising:

(a) receiving multiple images including image depth data acquired by three-dimensional imaging system, each image depth data, and for each of the multiple images;

(a1)) examining at least some of said image depth data;

(a2) forming segments by grouping together those of pixels in the image data whose acquired said data captured at step (a) have at least one common characteristic selected from a group consisting of depth distance, depth distance change, object size, and object shape;

(a3) assigning on a per-segment basis a label to pixels comprising each of said segments formed in step (a2), the pixels so labeled enabling recognition of said at least one object with an acquisition of image depth data; and

(b) tracking motion of the at least one object relative to the system over the multiple images.

38. The computer readable medium of claim 37 wherein the step of tracking is performed by noting movement of commonly labeled pixels.

39. The computer readable medium of claim 38 further including instructions comprising:

storing in a memory at least one pre-defined blob comprising a set of commonly labeled pixels;

comparing said segments formed for each image with contents of said memory to determine presence in said image data of a blob matching a pre-defined blob stored in said memory; and

providing an output signal commensurate with detection of a recognized pre-defined blob in said image data.

40. The computer readable medium of claim 39 further including tracking at least two objects.