Method and apparatus for optimizing image quality based on scene content

• US 8,830,360 B1
• Filed: 08/25/2011
• Issued: 09/09/2014
• Est. Priority Date: 08/25/2010
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Burt, Peter Jeffrey, Piacentino, Michael Raymond, van der Wal, Gooitzen Siemen, Levine, Peter Alan, Vogelsong, Thomas Lee, Tower, John Robertson, Chai, Sek Meng, Zhang, David Chao
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Haliyur, Padma
• Application Number:
  US13/218,152
• Time in Patent Office: 1,111 Days
• Field of Search: 348/241
• US Class Current: 348/241
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

A method and apparatus for optimizing image quality based on scene content comprising a sensor for generating a sequence of frames where each frame in the sequence of frames comprises content representing a scene and a digital processor, coupled to the sensor, for performing scene content analysis and for establishing a window defining a number of input frames from the sensor and processed output frames, and for aligning and combining the number of frames in the window to form an output frame, wherein sensor parameters and frame combination parameters are adjusted based on scene content.

Claims

1. An apparatus for optimizing image quality based on scene content comprising:

a sensor for generating a sequence of frames, where each frame in the sequence of frames comprises content representing a scene; and

a digital processor, coupled to the sensor, for establishing a window that defines a number of input frames, a subset of frames from the sequence of frames generated by the sensor, for performing scene content analysis on the number of input frames in the window, and for aligning and combining the number of frames according to a respective weight of each of the number of frames in the window to form one or more output frames as the sequence of frames are generated from the sensor, wherein the respective weight is computed based on at least one of noise levels, light levels, amount of motion in a frame, and focus settings,

wherein sensor parameters and combining parameters are adjusted based on scene content and the number of input frames in the window is dynamically adjusted based on the scene content in one or more of the input frames.

2. The apparatus of claim 1 wherein the scene content upon which the sensor parameters and the combining parameters are adjusted is at least one of scene brightness, amount of motion in the scene, focus and scene noise as detected in the sequence of frames by the digital processor,

wherein the combining parameters comprise at least one of the number of input frames from the sensor and processed output frames to be combined, a method of frame combination comprising at least one of average or selection, and the respective weight of each of the number of input frames used in frame combination.

3. The apparatus of claim 1 wherein the digital processor further performs post processing comprising at least one of contrast normalization, motion adaptive deblurring, spatial noise reduction and feature enhancement on the output frame.

4. The apparatus of claim 2 wherein the digital processor further comprises:

a motion module configured for analyzing motion at least one of globally or locally within the sequence of frames;

a windowing module configured for establishing the window of the number of the input frames from the sensor and processed output frames;

an alignment module, coupled to the windowing module, configured for aligning the number of frames in the window; and

a combination module, coupled to the alignment module, configured for assigning the respective weight to each frame at least one of globally or locally based on scene contents and temporal occurrence of each frame and the combination module is configured for the combining of the aligned number of frames using their respective weight and generating the output frame.

5. The apparatus of claim 4 wherein a filter is applied to frames in the window of frames in order to enhance signal and reduce noise in individual frames prior to motion analysis.

6. The apparatus of claim 4 wherein a sensor motion model is generated based on at least one of frame motion analysis and an external inertial measurement unit, comprising acceleration and velocity information, and use of the sensor motion model to align images based on predicted motion.

7. The apparatus of claim 1 wherein the digital processor further controls parameters associated with the sensor by providing a feedback control to the sensor based upon content within the sequence of frames,

wherein the controlled parameters comprise at least one of a frame rate, a gain level, sensor integration time, optics aperture size and focus.

8. The apparatus of claim 4 wherein the alignment is performed as a Finite Impulse Response filter, and the window is advanced by at least one frame in the sequence of frames wherein the combining the number of frames overlaps with a previous combining by at least one frame and the output frame is aligned with the number of frames currently in the window.

9. The apparatus of claim 4 wherein the alignment is performed as an Infinite Impulse Response filter and the window includes both input frames from the sensor and processed output frames.

10. The apparatus of claim 4 wherein the alignment is performed at least one of globally wherein all portions of each of the subset of frames are aligned across the subset of frames in the window or locally where one area of the subset of frames is aligned across the subset of frames in the window.

11. A computer implemented method for optimizing image quality based on scene content comprising:

generating, from a sensor, a sequence of frames where each input frame in the sequence of frames comprising content representing a scene;

establishing a window defining a number of input frames, a subset of frames from the generated sequence of frames;

performing scene content analysis on the number of input frames in the window;

aligning each frame in the number of frames, based on the content of each frame; and

combining the aligned number of frames according to a respective weight of each of the number of frames to form one or more output frames, wherein the respective weight is computed based on at least one of noise levels, light levels, amount of motion in a frame, and focus settings,

wherein sensor parameters and combining parameters are adjusted based on scene content and the number of input frames in the window is dynamically adjusted based on the scene content in one or more of the input frames.

12. The method of claim 11 wherein the scene content upon which the sensor and combining parameters are adjusted is at least one of scene brightness, amount of motion in the scene, focus and scene noise as detected in the sequence of frames,

wherein the combining parameters comprise at least one of the number of input frames from the sensor and processed output frames combined, a method of frame combination comprising at least one of average or selection, and the respective weight of each of the number of input frames used in frame combination.

13. The method of claim 11 further comprising performing at least one of contrast normalization, motion adaptive deblurring, spatial noise reduction and feature enhancement on the output frame.

14. The method of claim 12 further comprising:

analyzing motion at least one of globally or locally within the sequence of frames;

establishing the window of the number of the input frames from the sensor and processed output frames;

aligning the number of frames in the window; and

assigning the respective weight to each frame at least one of globally or locally based on scene contents and temporal occurrence of each frame and for the combining of the aligned number of frames using their respective weight and generating the output frame.

15. The method of claim 14 wherein a filter is applied to frames in the window of frames in order to enhance signal and reduce noise in individual frames prior to motion analysis.

16. The method of claim 14 wherein a sensor motion model is generated based on at least one of global frame motion analysis and an external inertial measurement unit, comprising acceleration and velocity information, and use of the sensor motion model to align images based on predicted motion.

17. The method of claim 11 further comprising controlling parameters associated with the sensor by providing a feedback control to the sensor based upon content within the sequence of frames,

wherein the controlled parameters comprise at least one of a frame rate, a gain level, sensor integration time, optics aperture size and focus.

18. The method of claim 14 wherein the alignment is performed as a Finite Impulse Response filter, and the window is advanced by at least one image in the sequence of frames wherein the combining the number of frames overlaps with a previous combining by at least one frame and the output frame is aligned with the number of frames currently in the window.

19. The method of claim 14 wherein the alignment is performed as an Infinite Impulse Response filter and the window includes both input frames from the sensor and processed output frames.

20. The method of claim 14 wherein the alignment is performed at least one of globally wherein all portions of each of the subset of frames are aligned across the subset of frames in the window or locally where one area of the subset of frames is aligned across the subset of frames in the window.