VIDEO ROLLING SHUTTER CORRECTION FOR LENS MOVEMENT IN OPTICAL IMAGE STABILIZATION CAMERAS

• US 20150350550A1
• Filed: 11/18/2014
• Published: 12/03/2015
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Thivent, Damien J., Zhou, Jianping, Toft, Rolf
• Granted Patent: US 9,357,132 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/6842 by controlling the scanning position, e.g. windowing
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position

Abstract

In some embodiments, a method for compensating for lens motion includes estimating a starting position of a lens assembly associated with captured pixel data. The captured pixel data is captured from an image sensor. In some embodiments, the method further includes calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data. The lens movement is mapped into pixel movement associated with the captured pixel data. A transform matrix is adjusted to reflect at least the pixel movement. A limit factor associated with the position data is calculated. The captured pixel data is recalculated using the transform matrix and the limit factor.

Claims

1. A mobile device having a camera module, the mobile device having a camera module comprising:

a lens assembly mechanically coupled to a base of the camera module, wherein

the lens assembly comprises one or more lenses subject to optical image stabilization;

an image sensor mechanically coupled to the base of the camera module;

one or more position sensors, wherein

at least one of the position sensors is coupled to the lens assembly; and

a non-transitory storage medium storing program instructions, wherein the program instructions are computer-executable to implement;

estimating a starting position of the lens assembly associated with captured pixel data, wherein

the captured pixel data is captured from the image sensor,

calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data,

mapping the lens movement into pixel movement associated with the captured pixel data,

adjusting a transform matrix to reflect at least the pixel movement,

calculating a limit factor associated with the position data, and

recalculating the captured pixel data using the transform matrix and the limit factor.

2. The mobile device of claim 1, wherein

the one or more position sensors further comprise a Hall sensor coupled to the lens assembly.

3. The mobile device of claim 1, wherein

the one or more position sensors further comprise a gyroscopic sensor coupled to a multifunction device housing the camera module.

4. The mobile device of claim 1, wherein

the captured pixel data represents data from a single row of sensor elements of the image sensor.

5. The mobile device of claim 1, wherein

the captured pixel data represents data from a first row of sensor elements of the image sensor, and

the non-transitory storage medium storing program instructions further comprise program instruction computer-executable to implement

repeating the estimating, calculating lens movement, mapping adjusting, calculating a limit factor and recalculating for second pixel data, wherein the second pixel data represents data from a second row of sensor elements of the image sensor.

6. The camera module of claim 1, wherein

the program instructions computer-executable to implement recalculating the captured pixel data using the transform matrix and the limit factor further comprise

program instructions computer-executable to implement correcting the captured pixel data to adjust for motion of the camera module.

7. The camera module of claim 1, wherein

the captured pixel data represents light that has transited the one or more lenses of the lens assembly.

8. A method, the method comprising:

estimating a starting position of a lens assembly associated with captured pixel data, wherein

the captured pixel data is captured from an image sensor;

calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data;

mapping the lens movement into pixel movement associated with the captured pixel data;

adjusting a transform matrix to reflect at least the pixel movement;

calculating a limit factor associated with the position data; and

recalculating the captured pixel data using the transform matrix and the limit factor.

9. The method of claim 8, wherein

the recalculating the captured pixel data using the transform matrix and the limit factor further comprises adjusting spatial assignment of the pixel data relative to other pixel data from the image sensor to compensate for movement of the lens assembly.

10. The method of claim 8, wherein

the recalculating the captured pixel data using the transform matrix and the limit factor further comprises

correcting the captured pixel data to adjust for motion of the camera module.

11. The method of claim 8, wherein

the captured pixel data represents light that has transited the one or more lenses of the lens assembly.

12. The method of claim 8, further comprising:

repeating the estimating, calculating lens movement, mapping adjusting, calculating a limit factor and recalculating for second pixel data, wherein

the second pixel data represents data from a second row of sensor elements of the image sensor.

13. The method of claim 8, wherein

the calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data further comprises

calculating from the starting position and position data received from one or more Hall sensors lens movement associated with the captured pixel data.

14. The method of claim 8, wherein

the calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data further comprises

calculating from the starting position and position data received from one or more Hall sensors lens movement associated with the captured pixel data.

15. A non-transitory computer-readable storage medium storing program instructions,

wherein the program instructions are computer-executable to implement:

estimating a starting position of a lens assembly associated with captured pixel data, wherein

the captured pixel data is captured from an image sensor;

calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data;

mapping the lens movement into pixel movement associated with the captured pixel data;

adjusting a transform matrix to reflect at least the pixel movement;

calculating a limit factor associated with the position data; and

recalculating the captured pixel data using the transform matrix and the limit factor.

16. The non-transitory computer-readable storage medium of claim 15, wherein

the program instructions computer-executable to implement recalculating the captured pixel data using the transform matrix and the limit factor further comprise program instructions computer-executable to implement adjusting spatial assignment of the pixel data relative to other pixel data from the image sensor to compensate for movement of the lens assembly.

17. The non-transitory computer-readable storage medium of claim 15, wherein

the program instructions computer-executable to implement recalculating the captured pixel data using the transform matrix and the limit factor further comprise program instructions computer-executable to implement correcting the captured pixel data to adjust for motion of the camera module.

18. The non-transitory computer-readable storage medium of claim 15, wherein

the captured pixel data represents light that has transited the one or more lenses of the lens assembly.

19. The non-transitory computer-readable storage medium of claim 15, further comprising:

program instructions computer-executable to implement repeating the estimating, calculating lens movement, mapping adjusting, calculating a limit factor and recalculating for second pixel data, wherein

the second pixel data represents data from a second row of sensor elements of the image sensor.

20. The non-transitory computer-readable storage medium of claim 15, wherein

the program instructions computer-executable to implement calculating from the starting position and position data received from the one or more position sensors lens movement associated with the captured pixel data further comprise

program instructions computer-executable to implement calculating from the starting position and position data received from one or more Hall sensors lens movement associated with the captured pixel data.