Apparatus and method for segmenting an image

• US 10,586,334 B2
• Filed: 01/07/2014
• Issued: 03/10/2020
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: Golden Edge Holding Corporation
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek, King, Joshua, Cluster, Jordan, Holmes, James Edward
• Primary Examiner(s): Rush, Eric
• Application Number:
  US14/148,761
• Publication Number:
  US 20140219559A1
• Time in Patent Office: 2,254 Days
• Field of Search: 382100, 382154, 382164, 382173, 382282, 382284, 382286
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

A method and system for segmenting a plurality of images. The method comprises the steps of segmenting the image through a novel clustering technique that is, generating a composite depth map including temporally stable segments of the image as well as segments in subsequent images that have changed. These changes may be determined by determining one or more differences between the temporally stable depth map and segments included in one or more subsequent frames. Thereafter, the portions of the one or more subsequent frames that include segments including changes from their corresponding segments in the temporally stable depth map are processed and are combined with the segments from the temporally stable depth map to compute their associated disparities in one or more subsequent frames. The images may include a pair of stereo images acquired through a stereo camera system at a substantially similar time.

Claims

1. A method for performing segmentation of an image, comprising the steps of:

acquiring one or more images by an image sensor;

classifying each pixel in the acquired one or more images as one of a background pixel, a temporally stable pixel, and a temporally unstable pixel as compared with a prior image frame;

assigning a unique cluster number only to each pixel in the acquired one or more images by a computer processor that are determined to be temporally unstable, the cluster numbers being assigned in accordance with a predetermined sequence;

connecting a first of the pixels determined to be temporally unstable with a second of the pixels determined to be temporally unstable to form a larger cluster and assigning a cluster number from the first of the pixels to the second of the pixels by the processor, if it is determined that the pixels do not differ by more than a predetermined threshold with respect to one or more characteristics; and

connecting the larger cluster with a cluster comprising temporally stable pixels if it is determined that the pixels in the larger cluster do not differ from the pixels in the cluster comprising the temporally stable pixels by more than a predetermined threshold with respect to one or more characteristics.

2. The method of claim 1, wherein the predetermined sequence comprises an ascending number sequence.

3. The method of claim 2, wherein the assigned cluster number is the lower of the two unique cluster numbers.

4. The method of claim 1, wherein the predetermined sequence comprises a descending number sequence.

5. The method of claim 4, wherein the assigned cluster number is the higher of the two unique cluster numbers.

6. The method of claim 1, wherein the step of connecting two pixels is performed for each pair of pixels in the one or more images located spatially adjacent to each other.

7. The method of claim 1, wherein the step of connecting two pixels is performed for each pair of pixels in the one or more images located within a predetermined distance from each other.

8. The method of claim 1, further comprising the steps of:

connecting a third pixel determined to be temporally unstable as compared with the prior image frame to the formed larger cluster; and

assigning a cluster number from one of the cluster number of the third pixel and the cluster number of all of the pixels in the larger cluster to each of the third pixel and all of the pixels in the larger cluster, if it is determined that the third pixel and the pixels of the larger cluster do not differ by more than a predetermined threshold with respect to one or more characteristics thereof.

9. The method of claim 1, wherein one or more large clusters comprising at least two pixels similar to each other with respect to one or more characteristics are formed.

10. The method of claim 9, further comprising the step of defining a key pixel for each larger cluster from among the at least two pixels therein, and utilizing that key pixel as the unique cluster number.

11. The method of claim 1, further comprising the step of connecting a plurality of clusters via their cluster numbers.

12. The method of claim 11, wherein the plurality of clusters are further connected in accordance with one or more predetermined thresholds.

13. The method of claim 11, further comprising the step of defining a stopping criterion beyond which connection of the clusters does not extend.

14. The method of claim 1, further comprising the steps of:

segmenting each cluster into sub-regions in accordance with one or more characteristics of each cluster and pixels thereof; and

calculating a disparity of each of the sub-regions.

15. The method of claim 14, wherein each of the one or more clusters are segmented into the sub regions by breaking up the clusters across rows and columns of pixels in an image.

16. A computer program stored in a non-transitory storage medium for performing segmentation of an image, the computer program, when implemented on a processor, causing the processor to perform the steps of:

receiving one or more images from an image sensor;

classifying each pixel in the received one or more images as one of a background pixel, a temporally stable pixel, and a temporally unstable pixel as compared with a prior image frame;

assigning a unique cluster number only to each pixel in the received one or more images that is determined to be temporally unstable, the cluster numbers being assigned in accordance with a predetermined sequence;

connecting a first of the pixels determined to be temporally unstable with a second of the pixels determined to be temporally unstable to form a larger cluster and assigning a cluster number from the first of the pixels to the second of the pixels, if it is determined that the pixels do not differ by more than a predetermined threshold amount in accordance with one or more characteristics thereof; and

connecting the larger cluster with a cluster comprising temporally stable pixels if it is determined that the pixels in the larger cluster do not differ from the pixels in the cluster comprising the temporally stable pixels by more than a predetermined threshold with respect to one or more characteristics.

17. The computer program of claim 16, wherein the predetermined sequence comprises an ascending number sequence.

18. The computer program of claim 17, wherein the assigned cluster number is the lower of the two unique cluster numbers.

19. The computer program of claim 16, wherein the predetermined sequence comprises a descending number sequence.

20. The computer program of claim 19, wherein the assigned cluster number is the higher of the two unique cluster numbers.

21. The computer program of claim 16, wherein the step of connecting two pixels is performed for each pair of pixels in the one or more images located spatially adjacent to each other.

22. The computer program of claim 16, wherein the step of connecting two pixels is performed for each pair of pixels in the one or more images located within a predetermined distance from each other.

23. The computer program of claim 16, further comprising the steps of:

connecting a third pixel determined to be temporally unstable as compared with the prior image frame to the formed larger cluster; and

assigning a cluster number from one of the cluster number of the third pixel and the cluster number of all of the pixels in the larger cluster to each of the third pixel and all of the pixels in the larger cluster, if it is determined that the third pixel and the pixels of the larger cluster do not differ by more than a predetermined threshold with respect to one or more characteristics thereof.

24. The computer program of claim 16, wherein one or more large clusters comprising at least two pixels similar to each other with respect to one or more characteristics are formed.

25. The computer program of claim 24, further comprising the step of defining a key pixel for each larger cluster from among the at least two pixels therein, and utilizing that key pixel as the unique cluster number.

26. The computer program of claim 16, further comprising the step of connecting a plurality of clusters via their cluster numbers.

27. The computer program of claim 26, wherein the plurality of clusters are further connected in accordance with one or more predetermined thresholds.

28. The computer program of claim 27, further comprising the step of defining a stopping criterion beyond which connection of the clusters does not extend.

29. The computer program of claim 16, further comprising the steps of:

segmenting each cluster into sub-regions in accordance with one or more characteristics of each cluster and pixels thereof; and

calculating a disparity of each of the sub-regions.

30. The computer program of claim 29, wherein each of the one or more clusters is segmented into the sub regions by breaking up the clusters across rows and columns of pixels in an image.