METHOD AND APPARATUS FOR PERFORMING SEGMENTATION OF AN IMAGE

• US 20120207383A1
• Filed: 02/10/2011
• Published: 08/16/2012
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: Golden Edge Holding Corporation
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek, King, Joshua, Cluster, Jordan, Holmes, James Edward
• Granted Patent: US 8,655,093 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

A method and system for segmenting a plurality of images. The method comprises the steps of segmenting the image through a novel clustering technique that is, generating a composite depth map including temporally stable segments of the image as well as segments in subsequent images that have changed. These changes may be determined by determining one or more differences between the temporally stable depth map and segments included in one or more subsequent frames. Thereafter, the portions of the one or more subsequent frames that include segments including changes from their corresponding segments in the temporally stable depth map are processed and are combined with the segments from the temporally stable depth map to compute their associated disparities in one or more subsequent frames. The images may include a pair of stereo images acquired through a stereo camera system at a substantially similar time.

Claims

1. A method for performing segmentation in a stereo image capture device, comprising the steps of:

determining one or more segments present in a first image acquired at a first time;

determining one or more segments present in a second image acquired at the first time;

determining a disparity of one of the one or more segments in the first image based upon a defined overlap metric;

assigning the determined disparity to every pixel comprising the one of the one or more segments in the first image;

assigning the determined disparity to every pixel in a segment in the second image corresponding to the one or more segments in the first image;

wherein one or more of the pixels in the segment in the second image is semi-occluded.

2. A method for performing segmentation of an image, comprising the steps of:

assigning a unique cluster number to each pixel in an image frame, the cluster numbers being assigned in accordance with a predetermined sequence;

connecting two pixels to form a larger cluster, and assigning a cluster number from one of the two unique cluster numbers to the other pixel, if it is determined that the pixels do not differ by more than a predetermined threshold amount in one or more of one or more characteristics.

3. The method of claim 2, wherein the predetermined sequence comprises an ascending number sequence.

4. The method of claim 3, wherein the assigned cluster number is the lower of the two unique cluster numbers.

5. The method of claim 2, wherein the predetermined sequence comprises a descending number sequence.

6. The method of claim 5, wherein the assigned cluster number is the higher of the two unique cluster numbers.

7. The method of claim 2, wherein the step of connecting two pixels is performed for each pair of pixels in the image located spatially adjacent.

8. The method of claim 2, wherein the step of connecting two pixels is performed for each pair of pixels in the image located within a predetermined distance from each other.

9. The method of claim 2, further comprising the steps of:

connecting a third pixel to the formed larger cluster; and

assigning a cluster number from one of the cluster number of the third pixel and the cluster number of all of the pixels in the larger cluster to each of the third pixel and all of the pixels in the larger cluster, if it is determined that the third pixel and the pixels of the larger cluster do not differ by more than a predetermined threshold amount in one or more of one or more characteristics.

10. The method of claim 2, wherein one or more large clusters comprising at least two pixels similar with respect to the one or more characteristics are formed.

11. The method of claim 10, further comprising the step of defining a key pixel for each large cluster from among the at least two pixels therein, and utilizing that key pixel as the unique cluster number.

12. The method of claim 2, further comprising the step of connecting a plurality of clusters via their cluster numbers.

13. The method of claim 12, wherein the plurality of clusters are further connected in accordance with one or more predetermined thresholds.

14. The method of claim 12, further comprising the step of defining a stopping criterion beyond which the cluster does not extend.

15. The method of claim 2, wherein the one or more characteristics comprise values that may be selected from among the group of a red, green and blue color space;

Hue;

Saturation Value;

a single color channel; and

a predetermined colorspace, or combination thereof.

16. The method of claim 2, wherein the one or more characteristics comprise a depth value calculated from a disparity value.

17. The method of claim 2, further comprising the steps of:

segmenting each of the one or more clusters into sub-regions in accordance with one or more characteristics of the segments and pixels thereof; and

calculating the disparity of each of the sub-regions.

18. The method of claim 17, wherein each of the one or more clusters are segmented into the sub regions by breaking up the clusters across rows and columns of pixels in an image.

21. The method of claim 18, further comprising the step of generating a composite segmentation image in accordance with the segmented one or more subsequent images.

22. The method of claim 18, wherein a subsequent depth map for a subsequent image is generated in accordance with the combined segments from the temporally stable depth map and the segments in the subsequent image indicated as having changed.

23. The method of claim 18, wherein a pixel of a segment is determined to have changed from the temporally stable depth map to the subsequent image if the segment is determined to have changed.

24. The method of claim 18, wherein the segment is determined to have changed if a predetermined number of pixels have changed.

25. The method of claim 18, wherein the segment is determined to have changed if the overall color of the segment has changed,

26. The method of claim 18, wherein the segment is determined to have changed if the size of the segment has changed by more than a predetermined amount.

19. A method for segmenting a plurality of images, comprising the steps of:

generating a temporally stable depth map including one or more segments in accordance with an image;

determining one or more differences between one or more of the one or more segments in the temporally stable depth map and one or more corresponding segments included in one or more subsequent images; and

combining the segments of the one or more subsequent images that include determined segments including changes from their corresponding segments in the temporally stable depth map with segments from the temporally stable depth map, to segment the one or more subsequent frames.

20. The method of claim 19, wherein the image is one of a pair of stereo images.

27. A computer program stored in a non-transitory storage medium for performing segmentation in stereo image capture device, the computer program, when implemented on a processor, causes the processor to perform the steps of:

determining one or more segments present in first image acquired at a first time;

determining one or more segments present in a second image acquired at the first time;

determining a disparity of one of the one or more segments in the first image based upon a defined overlap metric;

assigning the determined disparity to every pixel comprising the one of the one or more segments in the first image;

assigning the determined disparity to every pixel in a segment in the second image corresponding to the one or more segments in the first image;

wherein one or more of the pixels in the segment in the second image is semi-occluded.

28. A computer program stored in a non-transitory storage medium for performing segmentation of an image, the computer program, when implemented on a processor, causing the processor to perform the steps of:

assigning a unique cluster number to each pixel in an image frame, the cluster numbers being assigned in accordance with a predetermined sequence;

connecting two pixels to form a larger cluster, and assigning a cluster number from one of the two unique cluster numbers to the other pixel, if it is determined that the pixels do not differ by more than a predetermined threshold amount in one or more of one or more characteristics.

29. The computer program of claim 28, wherein the predetermined sequence comprises an ascending number sequence.

30. The computer program of claim 29, wherein the assigned cluster number is the lower of the two unique cluster numbers.

31. The computer program of claim 28, wherein the predetermined sequence comprises a descending number sequence.

32. The computer program of claim 31, wherein the assigned cluster number is the higher of the two unique cluster numbers.

33. The computer program of claim 28, wherein the step of connecting two pixels is performed for each pair of pixels in the image located spatially adjacent.

34. The computer program of claim 28, wherein the step of connecting two pixels is performed for each pair of pixels in the image located within a predetermined distance from each other.

35. The computer program of claim 28, further comprising the steps of:

connecting a third pixel to the formed larger cluster; and

assigning a cluster number from one of the cluster number of the third pixel and the cluster number of all of the pixels in the larger cluster to each of the third pixel and all of the pixels in the larger cluster, if it is determined that the third pixel and the pixels of the larger cluster do not differ by more than a predetermined threshold amount in one or more of one or more characteristics.

36. The computer program of claim 28, wherein one or more large clusters comprising at least two pixels similar with respect to the one or more characteristics are formed.

37. The computer program of claim 36, further comprising the step of defining a key pixel for each large cluster from among the at least two pixels therein, and utilizing that key pixel as the unique cluster number.

38. The computer program of claim 28, further comprising the step of connecting a plurality of clusters via their cluster numbers.

39. The computer program of claim 38, wherein the plurality of clusters are further connected in accordance with one or more predetermined thresholds.

40. The computer program of claim 39, further comprising the step of defining a stopping criterion beyond which the cluster does not extend.

41. The computer program of claim 28, wherein the one or more characteristics comprise values that may be selected from among the group of a red, green and blue color space;

Hue;

Saturation Value;

a single color channel; and

a predetermined colorspace, or combination thereof.

42. The computer program of claim 28, wherein the one or more characteristics comprise a depth value calculated from a disparity value.

43. The computer program of claim 28, further comprising the steps of:

segmenting each of the one or more clusters into sub-regions in accordance with one or more characteristics of the segments and pixels thereof; and

calculating the disparity of each of the sub-regions.

44. The computer program of claim 43, wherein each of the one or more clusters is segmented into the sub regions by breaking up the clusters across rows and columns of pixels in an image.

45. A computer program stored in a non-transitory recording medium for segmenting a plurality of images, the computer program, when implemented on a processor, causes the processor to perform the steps of:

generating a temporally stable depth map including one or more segments in accordance with an image;

determining one or more differences between one or more of the one or more segments in the temporally stable depth map and one or more corresponding segments included in one or more subsequent images; and

combining the segments of the one or more subsequent images that include determined segments including changes from their corresponding segments in the temporally stable depth map with segments from the temporally stable depth map, to segment the one or more subsequent frames.

46. The computer program of claim 45, wherein the image is one of a pair of stereo images.

47. The computer program of claim 45, further comprising the step of generating a composite segmentation image in accordance with the segmented one or more subsequent images.

48. The computer program of claim 45, wherein a subsequent depth map for a subsequent image is generated in accordance with the combined segments from the temporally stable depth map and the segments in the subsequent image indicated as having changed.

49. The computer program of claim 45, wherein a pixel of a segment is determined to have changed from the temporally stable depth map to the subsequent image if the segment is determined to have changed.

50. The computer program of claim 45, wherein the segment is determined to have changed if a predetermined number of pixels have changed.

51. The computer program of claim 45, wherein the segment is determined to have changed if the overall color of the segment has changed,

52. The computer program of claim 45, wherein the segment is determined to have changed if the size of the segment has changed by more than a predetermined amount.