Autofocus for stereo images

• US 9,485,495 B2
• Filed: 08/08/2011
• Issued: 11/01/2016
• Est. Priority Date: 08/09/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Atanassov, Kalin M., Goma, Sergiu R., Ramachandra, Vikas
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US13/205,481
• Publication Number:
  US 20120033051A1
• Time in Patent Office: 1,912 Days
• Field of Search: 348 42- 60, 382/107
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Present embodiments contemplate systems, apparatus, and methods to determine an appropriate focal depth for a sensor based upon a pair of stereoscopic images. Particularly, certain of the embodiments contemplate determining keypoints for each image, identifying correlations between the keypoints, and deriving object distances from the correlations. These distances may then be used to select a proper focal depth for one or more sensors.

Claims

1. A method in an electronic device for determining a focal depth for an image sensor, comprising:

receiving a first image in the visible spectrum, the first image associated with a first viewpoint;

receiving a second image in the visible spectrum, the second image associated with a second viewpoint;

determining a first plurality of keypoints based on the first image;

correlating keypoints from the first plurality of keypoints with positions in the second image;

determining a plurality of disparities associated with each of the first plurality of keypoints to form a sparse depth map; and

determining a focal depth for capturing a third image in the visible spectrum based upon the sparse depth map, the position of the first viewpoint and the position of the second viewpoint.

2. The method of claim 1, further comprising determining a second plurality of keypoints based on the second image.

3. The method of claim 2, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises correlating keypoints from the first plurality of keypoints with keypoints from the second plurality of keypoints.

4. The method of claim 1, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises iterating over pixels within a search range in the second image.

5. The method of claim 4, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises determining the mean square error between pixels in the first image and the second image.

6. The method of claim 1, wherein determining a first plurality of keypoints based on the first image comprises determining Scale Invariant Feature Transform (SIFT) keypoints based on the first image.

7. The method of claim 1, wherein determining a first plurality of keypoints based on the first image comprises sub-sampling the first image, applying a high-pass filter to the first image, calculating the power of the first image, and thresholding the first image.

8. The method of claim 1, wherein correlating keypoints from the first plurality of keypoints with positions in the second image occurs in realtime.

9. The method of claim 1, wherein the electronic device comprises a mobile phone.

10. A non-transitory computer readable medium comprising instructions configured to cause a computer to perform the steps of:

receiving a first image in the visible spectrum, the first image associated with a first viewpoint;

receiving a second image in the visible spectrum, the second image associated with a second viewpoint;

determining a first plurality of keypoints based on the first image;

correlating keypoints from the first plurality of keypoints with positions in the second image;

determining a plurality of disparities associated with each of the first plurality of keypoints to form a sparse depth map; and

determining a focal depth for capturing a third image in the visible spectrum based upon the sparse depth map, the position of the first viewpoint and the position of the second viewpoint.

11. The non-transitory computer readable medium of claim 10, further comprising determining a second plurality of keypoints based on the second image.

12. The non-transitory computer readable medium of claim 11, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises correlating keypoints from the first plurality of keypoints with keypoints from the second plurality of keypoints.

13. The non-transitory computer readable medium of claim 10, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises iterating over pixels within a search range in the second image.

14. The non-transitory computer readable medium of claim 13, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises determining the mean square error between pixels in the first image and the second image.

15. The non-transitory computer readable medium of claim 10, wherein determining a first plurality of keypoints based on the first image comprises determining Scale Invariant Feature Transform (SIFT) keypoints based on the first image.

16. The non-transitory computer readable medium of claim 10, wherein determining a first plurality of keypoints based on the first image comprises sub-sampling the first image, applying a high-pass filter to the first image, calculating the power of the first image, and thresholding the first image.

17. The non-transitory computer readable medium of claim 10, wherein correlating keypoints from the first plurality of keypoints with positions in the second image occurs in realtime.

18. The non-transitory computer readable medium of claim 10, wherein the computer is located in a mobile phone.

19. A system for focusing a stereoscopic capture device, the system comprising:

a first image sensor configured to generate a first image in the visible spectrum, the first image associated with a first viewpoint;

a second image sensor configured to generate a second image in the visible spectrum, the second image associated with a second viewpoint;

a feature generation module configured to determine a first plurality of keypoints based on the first image;

a keypoint correlation module configured to correlate keypoints from the first plurality of keypoints with positions in the second image;

a disparity determination module configured to determine a plurality of disparities associated with each of the first plurality of keypoints to form a sparse depth map; and

a depth determination module configured to determine a focal depth for capturing a third image in the visible spectrum based upon the sparse depth map, the position of the first viewpoint and the position of the second viewpoint.

20. The system of claim 19, wherein the feature generation module is configured to determine a second plurality of keypoints based on the second image.

21. The system of claim 20, wherein the module configured to correlate keypoints is configured to correlate keypoints from the first plurality of keypoints with keypoints from the second plurality of keypoints.

22. The system of claim 19, wherein the module configured to correlate keypoints is configured to iterate over pixels within a search range in the second image.

23. The system of claim 22, wherein correlating keypoints from the first plurality of keypoints with positions in the second image comprises determining the mean square error between pixels in the first image and the second image.

24. The system of claim 19, wherein the feature generation module is configured to determine Scale Invariant Feature Transform (SIFT) keypoints based on the first image.

25. The system of claim 19, wherein the feature generation module is configured to sub-sample the first image, apply a high-pass filter to the first image, calculate the power of the first image, and threshold the first image.

26. The system of claim 19, wherein the module configured to correlate keypoints correlates keypoints from the first plurality of keypoints with positions in the second image in realtime.

27. The system of claim 19, wherein the stereoscopic capture device is located on a mobile phone.

28. The system of claim 19, wherein the module configured to determine a focal depth comprises a disparity histogram.

29. A system for focusing a stereoscopic capture device, the system comprising:

means for receiving a first image in the visible spectrum, the first image associated with a first viewpoint;

means for receiving a second image in the visible spectrum, the second image associated with a second viewpoint;

means for determining a first plurality of keypoints based on the first image;

means for correlating keypoints from the first plurality of keypoints with positions in the second image;

means for determining a plurality of disparities associated with each of the first plurality of keypoints to form a sparse depth map; and

means for determining a focal depth for capturing a third image in the visible spectrum based upon the sparse depth map, the position of the first viewpoint and the position of the second viewpoint.

30. The system of claim 29, wherein the means for receiving a first image comprises a first sensor, the means for receiving a second image comprises a second sensor, the means for determining a first plurality of keypoints comprises a feature generation module, the means for correlating comprises a keypoint correlation module, the means for determining a plurality of disparities comprises a disparity determination module, and the means for determining a focal depth comprises a depth determination module.

31. The system of claim 29, wherein the means for determining a first plurality of keypoints is configured to determine a second plurality of keypoints based on the second image.

32. The system of claim 31, wherein the means for correlating keypoints from the first plurality of keypoints with positions in the second image is configured to correlate keypoints from the first plurality of keypoints with keypoints from the second plurality of keypoints.

33. The system of claim 29, wherein the means for correlating keypoints from the first plurality of keypoints with positions in the second image is configured to iterate over pixels within a search range in the second image.

34. The system of claim 33, wherein the means for correlating keypoints from the first plurality of keypoints with positions in the second image is configured to determine the mean square error between pixels in the first image and the second image.

35. The system of claim 29, wherein the means for determining a first plurality of keypoints is configured to determine Scale Invariant Feature Transform (SIFT) keypoints based on the first image.

36. The system of claim 29, wherein the means for determining a first plurality of keypoints is configured to sub-sample the first image, apply a high-pass filter to the first image, calculate the power of the first image, and threshold the first image.

37. The system of claim 29, wherein the means for correlating keypoints correlates the keypoints from the first plurality of keypoints with positions in the second image in realtime.

38. The system of claim 29, wherein the stereoscopic capture device is located on a mobile phone.