Using inertial sensors to provide smoothed exposure and white balance adjustments for video and photographic applications

• US 9,357,136 B2
• Filed: 02/04/2014
• Issued: 05/31/2016
• Est. Priority Date: 02/04/2014
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Noorkami, Maneli, Vaquero, Daniel A.
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): COWAN, EUEL K
• Application Number:
  US14/172,258
• Publication Number:
  US 20150222799A1
• Time in Patent Office: 847 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control

Abstract

A first image frame is gathered. The first image frame, a first white balance value, and a first exposure value are stored. Information is sensed from an inertial sensor. A second image frame and intermediate image frames are gathered. A distance traversed by a camera is determined. In response to one or more of: the distance exceeding a predetermined threshold, or a new white balance value significantly differing from a previous white balance value, or a new exposure value significantly differing from a previous exposure value, a second white balance value and a second exposure value are determined for the second image frame. A smoothed white balance value and a smoothed exposure value are determined for the intermediate image frames.

Claims

1. A method comprising:

gathering a first image frame;

storing in a memory buffer the first image frame, a first white balance value for the first image frame, and a first exposure value for the first image frame;

sensing information from one or more inertial sensors in a camera;

gathering a second image frame and gathering one or more intermediate image frames after the gathering of the first image frame and prior to the gathering of the second image frame;

using the sensed information from the one or more inertial sensors to determine a distance traversed by the camera from the gathering of the first image frame to the gathering of the second image frame and,

in response to the distance exceeding a predetermined threshold,

determining a second white balance value and a second exposure value for the second image frame; and

determining, estimating, or calculating a smoothed white balance value and a smoothed exposure value for the one or more intermediate image frames;

else in response to the distance not exceeding the predetermined threshold, utilizing the first white balance value and the first exposure value for the one or more intermediate image frames.

2. The method of claim 1 wherein the determining, estimating, or calculating is performed by interpolating the first white balance value and the second white balance value, and by interpolating the first exposure value and the second exposure value.

3. The method of claim 2 wherein the interpolating of the first white balance value and the second white balance value is performed so as to reduce a white balance fluctuation between the first white balance value and the second white balance value for the one or more intermediate image frames.

4. The method of claim 2 wherein the interpolating of the first exposure value and the second interpolation value is performed so as to reduce an exposure fluctuation between the first exposure value and the second exposure value for the one or more intermediate image frames.

5. The method of claim 1 further comprising determining whether or not the first white balance value is within a specified threshold of the second white balance value and, in response to the first white balance value not being within the specified threshold of the second white balance value, determining the second white balance value and the second exposure value for the second image frame.

6. The method of claim 1 further comprising determining whether or not the first exposure value is within a specified threshold of the second exposure value and, in response to the first exposure value not being within the specified threshold of the second exposure value, determining the second white balance value and the second exposure value for the second image frame.

7. The method of claim 1 further comprising using the sensed information to update a motion accumulator specifying a total distance traversed by the camera from the gathering of the first image frame.

8. A method comprising:

gathering a first image frame;

storing in a memory buffer the first image frame, a first white balance value for the first image frame, and a first exposure value for the first image frame;

gathering a second image frame and gathering one or more intermediate image frames after the gathering of the first image frame and prior to the gathering of the second image frame;

storing in a memory buffer the second image frame, a second white balance value for the second image frame, and a second exposure value for the second image frame;

determining whether or not the first white balance value is within a first specified threshold of the second white balance value; and

determining whether or not the first exposure value is within a second specified threshold of the second exposure value; and

in response to the first white balance value not being within the first specified threshold of the second white balance value, or in response to the first exposure value not being within the second specified threshold of the second exposure value, then determining, estimating, or calculating a smoothed white balance value and a smoothed exposure value for the one or more intermediate image frames;

else in response to the first white balance value being within the first specified threshold of the second white balance value and the first exposure value being within the second specified threshold of the second exposure value, then utilizing the first white balance value and the first exposure value for the one or more intermediate image frames.

9. The method of claim 8 wherein the determining, estimating, or calculating is performed by interpolating the first white balance value and the second white balance value, and by interpolating the first exposure value and the second exposure value.

10. The method of claim 9 wherein the interpolating of the first white balance value and the second white balance value is performed so as to reduce a white balance fluctuation between the first white balance value and the second white balance value for the one or more intermediate image frames.

11. The method of claim 9 wherein the interpolating of the first exposure value and the second interpolation value is performed so as to reduce an exposure fluctuation between the first exposure value and the second exposure value for the one or more intermediate image frames.

12. An apparatus comprising at least one processor, and at least one memory including computer program code, the at least one memory and the computer program code configured, with the at least one processor, to cause, at least in part, the apparatus to perform:

gathering a first image frame;

storing in a memory buffer the first image frame, a first white balance value for the first image frame, and a first exposure value for the first image frame;

sensing information from one or more inertial sensors in a camera;

gathering a second image frame and gathering one or more intermediate image frames after the gathering of the first image frame and prior to the gathering of the second image frame;

using the sensed information from the one or more inertial sensors to determine a distance traversed by the camera from the gathering of the first image frame to the gathering of the second image frame and,

in response to the distance exceeding a predetermined threshold,

determining a second white balance value and a second exposure value for the second image frame; and

determining, estimating, or calculating a smoothed white balance value and a

smoothed exposure value for the one or more intermediate image frames;

else in response to the distance not exceeding the predetermined threshold, utilizing the first white balance value and the first exposure value for the one or more intermediate image frames.

13. The apparatus of claim 12, wherein the apparatus performs the determining, estimating, or calculating by interpolating the first white balance value and the second white balance value, and by interpolating the first exposure value and the second exposure value.

14. The apparatus of claim 13, wherein the apparatus performs the interpolating of the first white balance value and the second white balance value so as to reduce a white balance fluctuation between the first white balance value and the second white balance value for the one or more intermediate image frames.

15. The apparatus of claim 13, wherein the apparatus performs the interpolating of the first exposure value and the second interpolation value so as to reduce an exposure fluctuation between the first exposure value and the second exposure value for the one or more intermediate image frames.

16. The apparatus of claim 12, wherein the apparatus determines whether or not the first white balance value is within a specified threshold of the second white balance value and, in response to the first white balance value not being within the specified threshold of the second white balance value, the apparatus determines the second white balance value and the second exposure value for the second image frame.

17. The apparatus of claim 12, wherein the apparatus determines whether or not the first exposure value is within a specified threshold of the second exposure value and, in response to the first exposure value not being within the specified threshold of the second exposure value, the apparatus determines the second white balance value and the second exposure value for the second image frame.

18. The apparatus of claim 12, wherein the apparatus uses the sensed information to update a motion accumulator specifying a total distance traversed by the camera from the gathering of the first image frame.

19. An apparatus comprising at least one processor, and at least one memory including computer program code, the at least one memory and the computer program code configured, with the at least one processor, to cause, at least in part, the apparatus to perform:

gathering a first image frame;

storing in a memory buffer a the first image frame, a first white balance value for the first image frame, and a first exposure value for the first image frame;

gathering a second image frame and gathering one or more intermediate image frames after the gathering of the first image frame and prior to the gathering of the second image frame;

storing in a memory buffer the second image frame, a second white balance value for the second image frame, and a second exposure value for the second image frame;

determining whether or not the first white balance value is within a first specified threshold of the second white balance value; and

determining whether or not the first exposure value is within a second specified threshold of the second exposure value;

in response to the first white balance value not being within the first specified threshold of the second white balance value, or in response to the first exposure value not being within the second specified threshold of the second exposure value, then determining, estimating, or calculating a smoothed white balance value and a smoothed exposure value for the one or more intermediate image frames;

else in response to the first white balance value being within the first specified threshold of the second white balance value and the first exposure value being within the second specified threshold of the second exposure value, then utilizing the first white balance value and the first exposure value for the one or more intermediate image frames.

20. The apparatus of claim 19, wherein the apparatus performs the determining, estimating, or calculating by interpolating the first white balance value and the second white balance value, and by interpolating the first exposure value and the second exposure value.

21. The apparatus of claim 20, wherein the apparatus performs the interpolating of the first white balance value and the second white balance value so as to reduce a white balance fluctuation between the first white balance value and the second white balance value for the one or more intermediate image frames.

22. The apparatus of claim 20, wherein the interpolating of the first exposure value and the second interpolation value is performed so as to reduce an exposure fluctuation between the first exposure value and the second exposure value for the one or more intermediate image frames.

23. A computer program product comprising code stored in a non-transitory computer readable memory for performing at least the following when executed on a computer device:

gathering a first image frame;

storing in a memory buffer the first image frame, a first white balance value for the first image frame, and a first exposure value for the first image frame;

sensing information from one or more inertial sensors in a camera;

gathering a second image frame and gathering one or more intermediate image frames after the gathering of the first image frame and prior to the gathering of the second image frame;

using the sensed information from the one or more inertial sensors to determine a distance traversed by the camera from the gathering of the first image frame to the gathering of the second image frame and,

in response to the distance exceeding a predetermined threshold,

determining a second white balance value and a second exposure value for the second image frame; and

determining, estimating, or calculating a smoothed white balance value and a smoothed exposure value for the one or more intermediate image frames;

else in response to the distance not exceeding the predetermined threshold, utilizing the first white balance value and the first exposure value for the one or more intermediate image frames.

24. A computer program product comprising code stored in a non-transitory computer readable memory for performing at least the following when executed on a computer device:

gathering a first image frame;

storing in a memory buffer the first image frame, a first white balance value for the first image frame, and a first exposure value for the first image frame;

gathering a second image frame and gathering one or more intermediate image frames after the gathering of the first image frame and prior to the gathering of the second image frame;

storing in a memory buffer the second image frame, a second white balance value for the second image frame, and a second exposure value for the second image frame;

determining whether or not the first white balance value is within a first specified threshold of the second white balance value; and

determining whether or not the first exposure value is within a second specified threshold of the second exposure value; and

in response to the first white balance value not being within the first specified threshold of the second white balance value, or in response to the first exposure value not being within the second specified threshold of the second exposure value, then determining, estimating, or calculating a smoothed white balance value and a smoothed exposure value for the one or more intermediate image frames;

else in response to the first white balance value being within the first specified threshold of the second white balance value and the first exposure value being within the second specified threshold of the second exposure value, then utilizing the first white balance value and the first exposure value for the one or more intermediate image frames.