Depth Estimate Determination, Systems and Methods

• US 20120163672A1
• Filed: 12/20/2011
• Published: 06/28/2012
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: Qutbluebox Trust
• Original Assignee: Qutbluebox Trust
• Inventors: McKinnon, David
• Granted Patent: US 9,177,381 B2
• Time in Patent Office: Days
• US Class Current: 382/106
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Systems and methods for generating pixel based depth estimates are disclosed. An image processing system operating as depth analysis engine generates an estimated depth associated with a pixel based on a reference image and other related images. A current depth estimate is refined based on neighboring pixels and calculated consistency scores. Further, depth estimates can be levered in object or scene recognition to trigger or initiate an action taken by a computing device.

Claims

1. A method for determining a depth estimate for at least one pixel in a reference image, the reference image being one of a plurality of images of a three dimensional scene, the method including:

a) providing access to a depth analysis engine comprising a device interface;

b) obtaining, by the depth analysis engine, a plurality of images including a reference image representative of a scene via the device interface;

c) determining, by the depth analysis engine, current depth estimates for current pixels in a current resolution of the reference image; and

d) deriving, by the depth analysis engine, a refined depth estimate by refining the current depth estimate for at least one current pixel via;

i) determining a plurality of candidate depths at least in part using the current depth estimates of the neighboring pixels and not that of the current pixel;

ii) determining consistency scores for at least some of the candidate depths, the consistency score being indicative of the similarity of the current pixel with at least one candidate pixel in other ones of the plurality of images;

iii) updating the current depth estimate of the current pixel using the consistency scores; and

e) configuring a device to initiate an action as a function of the refined current depth estimate.

2. A method according to claim 1, wherein the method includes, refining the current depth estimate by:

a) selecting a candidate depth having an optimum consistency score; and

b) refining the current depth estimate using the selected candidate depth.

3. A method according to claim 2, wherein the method includes, refining the current depth estimate by replacing the current depth estimate with:

a) the selected candidate depth; and

b) a depth interpolated from the selected candidate depth and two or more neighboring candidate depths using the respective consistency scores.

4. A method according to any one of the claims 1, wherein the method includes, determining the plurality of candidate depths by:

a) comparing the current pixel to a plurality of neighboring current pixels;

b) selecting at least one neighboring current pixel referenced on a similarity to the current pixel; and

c) determining the candidate depths to include the current depth estimates of the current pixel and the at least one selected neighboring current pixel.

5. A method according to claim 4, wherein the method includes, determining the plurality of candidate depths to include the current depth estimates and adjacent depth estimates of the current pixel and the at least one selected neighboring current pixel.

6. A method according to any one of the claims 1, wherein the previous and current depth estimates are assigned to previous and current depth planes respectively, the number of current depth planes being greater than the number of previous depth planes.

7. A method according to any one of the claims 1, wherein the candidate depths are at least one of:

a) a plurality of discrete depths; and

b) a depth range.

8. A method according to any one of the claims 1, wherein the method includes iteratively refining the current depth estimates of current pixels in the current resolution of the reference image.

9. A method according to claim 8, wherein the method includes iteratively refining the current depth estimates by:

a) refining the current depth estimate of a plurality of current pixels; and

b) further refining the current depth estimate of at least one of the plurality of current pixels.

10. A method according to any one of the claims 1, wherein the method includes, determining the current depth estimates using previous depth estimates for equivalent previous pixels in a previous resolution of the reference image, the previous resolution being lower than the current resolution

11. A method according to any one of the claims 10, wherein a current pixel is equivalent to a previous pixel if the current and previous pixels are at the same location in the reference image.

12. A method according to any one of the claims 10, wherein the method includes, determining first depth estimates for first pixels in a first resolution of the reference image using a parallax method, the first resolution being a lowest resolution.

13. A method according to claim 14, wherein the method includes, determining the first depth estimates using a global parallax method.

14. A method according to any one of the claims 1, wherein the method includes determining current depth estimates for current pixels by successively increasing current resolutions of the reference image using previous depth estimates of equivalent previous pixels from a previous lower resolution of the reference image.

15. A method according to claim 14, wherein the method includes, refining the current depth estimates for current resolutions of the image using a parallax local method, wherein the local method is performed for limited candidate depths determined from previous depth estimates of equivalent previous pixels from a previous lower resolution of the reference image.

16. A method according to any one of the claims 1, wherein the method includes:

a) in an electronic processing device operating as the depth analysis engine, determining the current depth estimates; and

b) in the electronic processing device operating as the depth analysis engine, refining the current depth estimate for at least one current pixel by;

i) selecting the plurality of candidate depths;

ii) calculating the consistency scores; and

iii) updating the current depth estimate of the current pixel.

17. A method according to claim 16, wherein the method includes, storing in a store, at least one of:

a) image data; and

b) depth estimate data.

18. A method according to claim 17, wherein the method includes updating a current depth estimate by updating depth estimate data stored in a store.

19. A method for determining a depth estimate for at least one pixel in a reference image, the reference image being one of a plurality of images of a three dimensional scene, the method including:

a) in an depth analysis device, determining current depth estimates for current pixels in a current resolution of the reference image; and

b) in the depth analysis device, refining the current depth estimate for at least one current pixel by;

i) selecting a plurality of candidate depths at least in part using the current depth estimates of the current pixel and at least one neighboring current pixel;

ii) calculating consistency scores for at least some of the candidate depths, the consistency score being indicative of the similarity of the current pixel with at least one candidate pixel in other ones of the plurality of images; and

iii) updating the current depth estimate of the current pixel using the consistency scores.

20. A depth analysis system, comprising:

a device interface configured to communicatively couple to a device and receive image data from the device;

an image database coupled with the device interface and storing the image data, the image data representative of a scene comprising a plurality of images, including a reference image; and

a depth analysis engine coupled with the image database and configured to;

a) determine current depth estimates for current pixels in a current resolution of the reference image; and

b) refine the current depth estimate for at least one current pixel by;

i) selecting a plurality of candidate depths at least in part using the current depth estimates of the current pixel and at least one neighboring current pixel;

ii) calculating consistency scores for at least some of the candidate depths, the consistency score being indicative of the similarity of the current pixel with at least one candidate pixel in other ones of the plurality of images;

iii) updating the current depth estimate of the current pixel using the consistency scores; and

c) configure the device to initiate an action based on the updated current depth estimate.