Depth estimate determination, systems and methods

• US 9,177,381 B2
• Filed: 12/20/2011
• Issued: 11/03/2015
• Est. Priority Date: 12/22/2010
• Status: Expired due to Fees

• Current Assignee: Qutbluebox Trust
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: McKinnon, David
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Kholdebarin, Iman K
• Application Number:
  US13/330,883
• Publication Number:
  US 20120163672A1
• Time in Patent Office: 1,414 Days
• Field of Search: 382/106, 382/154
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Systems and methods for generating pixel based depth estimates are disclosed. An image processing system operating as depth analysis engine generates an estimated depth associated with a pixel based on a reference image and other related images. A current depth estimate is refined based on neighboring pixels and calculated consistency scores. Further, depth estimates can be levered in object or scene recognition to trigger or initiate an action taken by a computing device.

Claims

1. A method for determining a depth estimate for at least one pixel in a reference image, the reference image being one of a plurality of images of a three dimensional scene, the method including:

a) providing access to a depth analysis engine comprising a device interface;

b) obtaining, by the depth analysis engine, a plurality of images including a reference image representative of a scene via the device interface, wherein each of the plurality of images comprises a different perspective of the scene;

c) determining first depth estimates for first pixels in a first resolution of the reference image using a parallax method, the first resolution being a lowest resolution;

d) determining, by the depth analysis engine, current depth estimates for current pixels in a current resolution of the reference image by using previous depth estimates for equivalent previous pixels in a previous resolution of the reference image, the previous resolution being lower than the current resolution; and

e) deriving, by the depth analysis engine, a refined depth estimate by refining the current depth estimate for at least one current pixel via;

i) determining a plurality of candidate depths at least in part using the current depth estimates of the neighboring pixels and not that of the current pixel;

ii) determining consistency scores for at least some of the candidate depths, the consistency score being indicative of the similarity of the current pixel with at least one candidate pixel in a first image and a second, different image from the plurality of images;

iii) updating the current depth estimate of the current pixel using the consistency scores; and

f) configuring a device to initiate an action as a function of the refined current depth estimate.

2. A method according to claim 1, wherein the method includes, refining the current depth estimate by:

a) selecting a candidate depth having an optimum consistency score; and

b) refining the current depth estimate using the selected candidate depth.

3. A method according to claim 2, wherein the method includes, refining the current depth estimate by replacing the current depth estimate with:

a) the selected candidate depth; and

b) a depth interpolated from the selected candidate depth and two or more neighboring candidate depths using the respective consistency scores.

4. A method according to claim 1, wherein the method includes, determining the plurality of candidate depths by:

a) comparing the current pixel to a plurality of neighboring current pixels;

b) selecting at least one neighboring current pixel referenced on a similarity to the current pixel; and

c) determining the candidate depths to include the current depth estimates of the current pixel and the at least one selected neighboring current pixel.

5. A method according to claim 4, wherein the method includes, determining the plurality of candidate depths to include the current depth estimates and adjacent depth estimates of the current pixel and the at least one selected neighboring current pixel.

6. A method according to claim 1, wherein the previous and current depth estimates are assigned to previous and current depth planes respectively, said previous and current depth planes defining a number of previous depth planes and a number of current depth planes wherein the number of current depth planes being greater than the number of previous depth planes.

7. A method according to claim 1, wherein the candidate depths are at least one of:

a) a plurality of discrete depths; and

b) a depth range.

8. A method according to claim 1, wherein the method includes iteratively refining the current depth estimates of current pixels in the current resolution of the reference image.

9. A method according to claim 8, wherein the method includes iteratively refining the current depth estimates by:

a) refining the current depth estimate of a plurality of current pixels; and

b) further refining the current depth estimate of at least one of the plurality of current pixels.

10. A method according to claim 1, wherein a current pixel is equivalent to a previous pixel if the current and previous pixels are at the same location in the reference image.

11. A method according to claim 1, wherein the method includes, determining the first depth estimates using a global parallax method.

12. A method according to claim 1, wherein the method includes, refining the current depth estimates for current resolutions of the image using a parallax local method, wherein the local method is performed for limited candidate depths determined from previous depth estimates of equivalent previous pixels from a previous lower resolution of the reference image.

13. A method according to claim 1, wherein the method includes:

a) in an electronic processing device operating as the depth analysis engine, determining the current depth estimates; and

b) in the electronic processing device operating as the depth analysis engine, refining the current depth estimate for at least one current pixel by;

i) selecting the plurality of candidate depths;

ii) calculating the consistency scores; and

iii) updating the current depth estimate of the current pixel.

14. A method according to claim 13, wherein the method includes, storing in a store, at least one of:

a) image data; and

b) depth estimate data.

15. A method according to claim 14, wherein the method includes updating a current depth estimate by updating depth estimate data stored in a store.

16. The method of claim 1, wherein the at least one candidate pixel in the first image and the second image each corresponds to a neighboring current pixel from the at least one current pixel associated with a candidate depth from the at least some of the candidate depths.

17. A method for determining a depth estimate for at least one pixel in a reference image, the reference image being one of a plurality of images of a three dimensional scene, the method including:

a) providing access to a depth analysis engine comprising a device interface;

b) obtaining, by the depth analysis engine, a plurality of images including a reference image representative of a scene via the device interface, wherein each of the plurality of images comprises a different perspective of the scene;

c) determining, by the depth analysis engine, current depth estimates for current pixels in a current resolution of the reference image by successively increasing current resolutions of the reference image using previous depth estimates of equivalent previous pixels from a previous lower resolution of the reference image; and

d) deriving, by the depth analysis engine, a refined depth estimate by refining the current depth estimate for at least one current pixel via;

i) determining a plurality of candidate depths at least in part using the current depth estimates of the neighboring pixels and not that of the current pixel;

ii) determining consistency scores for at least some of the candidate depths, the consistency score being indicative of the similarity of the current pixel with at least one candidate pixel in a first image and a second, different image from the plurality of images;

iii) updating the current depth estimate of the current pixel using the consistency scores; and

e) configuring a device to initiate an action as a function of the refined current depth estimate.

18. A method according to claim 17, wherein the method includes, refining the current depth estimate by:

a) selecting a candidate depth having an optimum consistency score; and

b) refining the current depth estimate using the selected candidate depth.

19. A method according to claim 18, wherein the method includes, refining the current depth estimate by replacing the current depth estimate with:

a) the selected candidate depth; and

b) a depth interpolated from the selected candidate depth and two or more neighboring candidate depths using the respective consistency scores.

20. A method according to claim 17, wherein the method includes, determining the plurality of candidate depths by:

a) comparing the current pixel to a plurality of neighboring current pixels;

b) selecting at least one neighboring current pixel referenced on a similarity to the current pixel; and

c) determining the candidate depths to include the current depth estimates of the current pixel and the at least one selected neighboring current pixel.

21. A method according to claim 20, wherein the method includes, determining the plurality of candidate depths to include the current depth estimates and adjacent depth estimates of the current pixel and the at least one selected neighboring current pixel.

22. A method according to claim 17, wherein the previous and current depth estimates are assigned to previous and current depth planes respectively, said previous and current depth planes defining a number of previous depth planes and a number of current depth planes wherein the number of current depth planes being greater than the number of previous depth planes.

23. A method according to claim 17, wherein the candidate depths are at least one of:

a) a plurality of discrete depths; and

b) a depth range.

24. A method according to claim 17, wherein the method includes iteratively refining the current depth estimates of current pixels in the current resolution of the reference image.

25. A method according to claim 24, wherein the method includes iteratively refining the current depth estimates by:

a) refining the current depth estimate of a plurality of current pixels; and

b) further refining the current depth estimate of at least one of the plurality of current pixels.

26. A method according to claim 17, wherein a current pixel is equivalent to a previous pixel if the current and previous pixels are at the same location in the reference image.

27. A method according to claim 17, wherein the method includes, refining the current depth estimates for current resolutions of the image using a parallax local method, wherein the local method is performed for limited candidate depths determined from previous depth estimates of equivalent previous pixels from a previous lower resolution of the reference image.

28. A method according to claim 17, wherein the method includes:

a) in an electronic processing device operating as the depth analysis engine, determining the current depth estimates; and

b) in the electronic processing device operating as the depth analysis engine, refining the current depth estimate for at least one current pixel by;

i) selecting the plurality of candidate depths;

ii) calculating the consistency scores; and

iii) updating the current depth estimate of the current pixel.

29. A method according to claim 28, wherein the method includes, storing in a store, at least one of:

a) image data; and

b) depth estimate data.

30. A method according to claim 29, wherein the method includes updating a current depth estimate by updating depth estimate data stored in a store.

31. A method according to claim 17, wherein the at least one candidate pixel in the first image and the second image each corresponds to a neighboring current pixel from the at least one current pixel associated with a candidate depth from the at least some of the candidate depths.