Apparatus and method for determining disparity of textured regions

• US 9,723,296 B2
• Filed: 01/22/2014
• Issued: 08/01/2017
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: Golden Edge Holding Corporation
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek, King, Joshua, Holmes, James Edward, Cluster, Jordan
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US14/160,592
• Publication Number:
  US 20140132732A1
• Time in Patent Office: 1,287 Days
• Field of Search: None
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method and system to determine the disparity associated with one or more textured regions of a plurality of images is presented. The method comprises the steps of breaking up the texture into its color primitives, further segmenting the textured object into any number of objects comprising such primitives, and then calculating a disparity of these objects. The textured objects emerge in the disparity domain, after having their disparity calculated. Accordingly, the method is further comprised of defining one or more textured regions in a first of a plurality of images, determining a corresponding one or more textured regions in a second of the plurality of images, segmenting the textured regions into their color primitives, and calculating a disparity between the first and second of the plurality of images in accordance with the segmented color primitives.

Claims

1. A method to determine a disparity associated with one or more textured regions of a plurality of images, comprising the steps of:

segmenting one or more textured regions from a first and second image of a first plurality of images into their color primitives, at least two of the one or more segmented textured regions being discontinuous in space resulting in at least two regions, each having two or more distinct portions associated therewith;

determining one or more changes in the one or more segmented textured regions over time;

calculating a disparity between the first and second images only for one or more of the segmented textured regions determined to have one or more changes therein in accordance with the segmented color primitives; and

combining two or more of the one or more segmented textured regions to define a single texture in accordance with the calculated disparity and the determined changes over time of the one or more segmented textured regions.

2. The method of claim 1, wherein the steps of segmenting and calculating are performed repetitively until the one or more textured regions are fully segmented.

3. The method of claim 1, further comprising the step of combining two or more of the segmented primitives to define one or more textured segments.

4. The method of claim 1, further comprising the steps of:

generating a first disparity map based upon the calculated disparity;

acquiring at least a third image and a fourth image from a second plurality of images;

determining one or more portions comprising a difference between one of the first and second images and a corresponding one of the third and fourth images;

generating a disparity map update for the one or more determined portions comprising a difference from the second plurality of images; and

generating a disparity map by combining the disparity map update and the first disparity map.

5. The method of claim 4, wherein combining the disparity map update and the first disparity map further comprises the steps of:

identifying one or more locations in the first disparity map corresponding to the determined one or more portions; and

substituting for each identified location in the first disparity map a corresponding portion of the disparity map update.

6. A method for generating a disparity map, comprising the steps of:

defining one or more textured regions in a first of a plurality of images acquired at a first time;

segmenting the one or more textured regions into their color primitives, at least two of the one or more segmented textured regions being discontinuous in space resulting in at least two regions, each having two or more distinct portions associated therewith;

determining a corresponding one or more segmented textured regions in a second of the plurality of images acquired at the first time;

determining one or more changes in the one or more segmented textured regions over time;

calculating a disparity only between the segmented textured regions in the first and second of the plurality of images for one or more of the segmented textured regions determined to have one or more changes therein over time in accordance with the segmented color primitives; and

combining two or more of the one or more segmented textured regions to define a single texture in accordance with the calculated disparity and the determined changes over time of the one or more segmented textured regions.

7. The method of claim 6, further comprising the steps of:

determining one or more objects that change between the plurality of images acquired at the first time and a corresponding plurality of images acquired at a second time;

generating disparity map update segments corresponding to the determined one or more objects; and

combining the first disparity map and the disparity map update segments.

8. The method of claim 7, wherein the step of combining the first disparity map and the disparity map update segments further comprises the steps of:

determining a three-dimensional position of the objects in the images acquired at the second time; and

replacing these positions in the first disparity map with the corresponding disparity map updated segments.

9. The method of claim 8, wherein the objects comprise portions of an image that have changed.

10. A system to determine a disparity associated with one or more textured regions of a plurality of images, comprising:

a stereo camera for acquiring a first plurality of images at a first time and a second plurality of images at a second time; and

a processor for segmenting one or more textured regions from a first and second image of the first plurality of images into their color primitives, at least two of the one or more segmented textured regions being discontinuous in space resulting in at least two regions, each having two or more distinct portions associated therewith, determining one or more changes in the one or more segmented textured regions over time;

for calculating a disparity between the first and second images only for one or more of the segmented textured regions determined to have one or more changes therein over time in accordance with the segmented color primitives, and combining two or more of the one or more segmented textured regions to define a single texture in accordance with the calculated disparity and the determined changes over time of the one or more segmented textured regions.

11. The system of claim 10, wherein the processor is further adapted to generate a first disparity map based upon the calculated disparity, determine one or more portions comprising a difference between one of the first plurality of images and a corresponding one of the second plurality of images, generating a disparity map update for the one or more determined portions in accordance with a first and second images from the second plurality of images, and generating a disparity map by combining the disparity map update and the first disparity map.

12. The system of claim 11, wherein the processor is further adapted to combine the disparity map update and the first disparity map by identifying one or more three-dimensional locations in the first disparity map corresponding to the determined one or more portions, and substituting for each identified location in the first disparity map a corresponding portion of the disparity map update.

13. A system for generating a disparity map, comprising:

a stereo camera for acquiring right and left images at a first time and corresponding right and left images at a second time; and

a processor for defining one or more textured regions in one of the right and left images acquired at the first time, segmenting the one or more textured regions into their color primitives, at least two of the one or more segmented textured regions being discontinuous in space resulting in at least two regions, each having two or more distinct portions associated therewith, determining a corresponding one or more segmented textured regions in the other of the right and left images acquired at the first time, determining one or more changes in the one or more segmented textured regions over time, calculating a disparity between the segmented textured regions in the first and second of the right and left images acquired at the first time only for one or more of the segmented textured regions determined to have one or more changes therein in accordance with the segmented color primitives, and combining two or more of the one or more segmented textured regions to define a single texture in accordance with the calculated disparity and the determined changes over time of the one or more segmented textured regions.

14. The system of claim 13, wherein the processor is further adapted to generate a first disparity map based upon the calculated disparity, determining one or more objects that change position between one of the left and right images acquired at the first time and a corresponding one of the left and right images acquired at the second time, generating disparity map update segments corresponding to the objects, and combining the first disparity map and the disparity map update segments.

15. The system of claim 14, wherein the processor is further adapted to combine the first disparity map and the disparity map update segments by determining a three-dimensional position of the objects in the left and right images acquired at the second time, and replacing these positions in the first disparity map with the corresponding disparity map update segments.

16. The system of claim 15, wherein the objects comprise portions of an image that have changed.

17. The system of claim 14, wherein the step of determining one or more objects that change position between one of the left and right images acquired at the first time and a corresponding one of the left and right images acquired at the second time further comprises calculating a difference result, such difference result being stored in a compressed format of (1-bit/disparity) for 32, 64, or 128-bit values, or any other combination of values.

18. The system of claim 17, wherein a bit size of an image used for storing the difference results is dependent upon the number of disparity levels employed.

19. The system of claim 18, wherein the bit size of the image is dependent upon the number of disparity levels employed.

20. The system of claim 17, wherein the difference results are calculated only for segments determined to be temporally unstable.

21. The system of claim 17, further comprising the steps of:

combining one or more determined temporally unstable segments with one or more temporally stable segments; and

re-computing disparity based upon the combined segments.

22. The system of claim 17, further comprising the steps of

combining one or more determined temporally unstable segments with one or more temporally stable segments;

and

assigning a disparity of one of the segments to the combined segment.