INERTIAL SENSOR AIDED INSTANT AUTOFOCUS

• US 20140029837A1
• Filed: 07/30/2012
• Published: 01/30/2014
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Venkatraman, Subramaniam, Hung, Szepo Robert, Velarde, Ruben M., Puig, Carlos Manuel
• Granted Patent: US 9,025,859 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/676 Bracketing for image capture at varying focusing conditions
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

The disclosure is directed to creating an inertial sensor aided depth map of a scene. An embodiment of the disclosure captures at least a first image and a second image during movement of a device caused by a user while framing or recording the scene, compensates for rotation between the first image and the second image, calculates an amount of translation of the device between the first image and the second image, calculates a pixel shift of a plurality of key points of the first image and the second image, and estimates a depth to one or more of the plurality of key points of the first image and the second image.

Claims

1. A method for creating an inertial sensor aided depth map of a scene, comprising:

capturing at least a first image and a second image during movement of a device caused by a user while framing or recording the scene;

compensating for rotation between the first image and the second image;

calculating an amount of translation of the device between the first image and the second image;

calculating a pixel shift of a plurality of key points of the first image and the second image; and

estimating a depth to one or more of the plurality of key points of the first image and the second image.

2. The method of claim 1, further comprising:

identifying the plurality of key points of the first image and the second image.

3. The method of claim 1, wherein the calculating the amount of translation is based on inertial sensors of the device.

4. The method of claim 1, wherein the estimating comprises estimating an absolute depth from the device to the one or more of the plurality of key points or a relative depth of the one or more of the plurality of key points with respect to each other.

5. The method of claim 1, further comprising:

estimating an amount of rotation of the device between the first image and the second image.

6. The method of claim 5, wherein the estimating the amount of rotation is based at least in part on data from a gyroscope.

7. The method of claim 6, further comprising:

periodically calibrating the gyroscope based on periods of time during which movement of the device is below a threshold.

8. The method of claim 5, wherein the estimating the amount of rotation is based at least in part on a comparison of tracking features of each of the first image and the second image.

9. The method of claim 1, wherein the calculating the amount of translation is based at least in part on data from an accelerometer.

10. The method of claim 9, further comprising:

periodically calibrating the accelerometer based on periods of time during which the movement of the device is below a threshold.

11. The method of claim 10, wherein the periodically calibrating comprises determining when the movement of the device is below the threshold by comparing consecutive images recorded by the device.

12. The method of claim 1, wherein the calculating the amount of translation comprises integrating a gravity-compensated acceleration to obtain the amount of translation of the device between the first image and the second image.

13. The method of claim 1, wherein the capturing is performed within a given time and a given range of motion.

14. The method of claim 1, wherein the plurality of key points are a plurality of key points of each of the first image and the second image.

15. The method of claim 1, further comprising:

determining a focus setting based on the estimating.

16. The method of claim 15, wherein the focus setting is a focal length or a lens position that is adjusted based on the estimated depth.

17. The method of claim 15, wherein the determining is based on a facial recognition algorithm.

18. The method of claim 15, wherein the determining is based on a user selected region of the first image or the second image.

19. The method of claim 1, further comprising:

obtaining an orientation of the device with respect to a vertical axis using an accelerometer; and

estimating a driving current for a lens of the device based on the obtaining.

20. The method of claim 1, where the device is one of a wireless device with an integrated camera, a smartphone, a camera, or a video camcorder.

21. An apparatus for creating an inertial sensor aided depth map of a scene, comprising:

logic configured to capture at least a first image and a second image during movement of a device caused by a user while framing or recording the scene;

logic configured to compensate for rotation between the first image and the second image;

logic configured to calculate an amount of translation of the device between the first image and the second image;

logic configured to calculate a pixel shift of a plurality of key points of the first image and the second image; and

logic configured to estimate a depth to one or more of the plurality of key points of the first image and the second image.

22. The apparatus of claim 21, further comprising:

logic configured to identify the plurality of key points of the first image and the second image.

23. The apparatus of claim 21, wherein the calculating the amount of translation is based on inertial sensors of the device.

24. The apparatus of claim 21, wherein the estimating comprises estimating an absolute depth from the device to the one or more of the plurality of key points or a relative depth of the one or more of the plurality of key points with respect to each other.

25. The apparatus of claim 21, further comprising:

logic configured to estimate an amount of rotation of the device between the first image and the second image.

26. The apparatus of claim 25, wherein the logic configured to estimate the amount of rotation is based at least in part on data from a gyroscope.

27. The apparatus of claim 26, further comprising:

logic configured to periodically calibrate the gyroscope based on periods of time during which movement of the device is below a threshold.

28. The apparatus of claim 25, wherein the logic configured to estimate the amount of rotation is based at least in part on a comparison of tracking features of each of the first image and the second image.

29. The apparatus of claim 23, wherein the logic configured to calculate the amount of translation is based at least in part on data from an accelerometer.

30. The apparatus of claim 29, further comprising:

logic configured to periodically calibrate the accelerometer based on periods of time during which the movement of the device is below a threshold.

31. The apparatus of claim 30, wherein the logic configured to periodically calibrate comprises determining when the movement of the device is below the threshold by comparing consecutive images recorded by the device.

32. The apparatus of claim 23, wherein the logic configured to calculate the amount of translation comprises logic configured to integrate a gravity-compensated acceleration to obtain the amount of translation of the device between the first image and the second image.

33. The apparatus of claim 23, wherein the logic configured to capture is performed within a given time and a given range of motion.

34. The apparatus of claim 23, wherein the plurality of key points are a plurality of key points of each of the first image and the second image.

35. The apparatus of claim 23, further comprising:

logic configured to determine a focus setting based on the estimating.

36. The apparatus of claim 35, wherein the focus setting is a focal length or a lens position that is adjusted based on the estimated depth.

37. The apparatus of claim 35, wherein the determining is based on a facial recognition algorithm.

38. The apparatus of claim 35, wherein the determining is based on a user selected region of the first image or the second image.

39. The apparatus of claim 21, further comprising:

logic configured to obtain an orientation of the device with respect to a vertical axis using an accelerometer; and

logic configured to estimate a driving current for a lens of the device based on the obtaining

40. The apparatus of claim 21, where the device is one of a wireless device with an integrated camera, a smartphone, a camera, or a video camcorder.

41. An apparatus for creating an inertial sensor aided depth map of a scene, comprising:

means for capturing at least a first image and a second image during movement of a device caused by a user while framing or recording the scene;

means for compensating for rotation between the first image and the second image;

means for calculating an amount of translation of the device between the first image and the second image;

means for calculating a pixel shift of a plurality of key points of the first image and the second image; and

means for estimating a depth to one or more of the plurality of key points of the first image and the second image.

42. A non-transitory computer-readable medium for creating an inertial sensor aided depth map of a scene, comprising:

at least one instruction to capture at least a first image and a second image during movement of a device caused by a user while framing or recording the scene;

at least one instruction to compensate for rotation between the first image and the second image;

at least one instruction to calculate an amount of translation of the device between the first image and the second image;

at least one instruction to calculate a pixel shift of a plurality of key points of the first image and the second image; and

at least one instruction to estimate a depth to one or more of the plurality of key points of the first image and the second image. QC 121479