Focus detection

• US 9,715,721 B2
• Filed: 12/18/2015
• Issued: 07/25/2017
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Wei, Jianing, Tu, Xue, Matsui, Akira, Li, Pingshan, Berestov, Alexander, Mizukura, Takami, Shuda, Tomonori
• Primary Examiner(s): Liu, Li
• Application Number:
  US14/975,315
• Publication Number:
  US 20170178296A1
• Time in Patent Office: 585 Days
• CPC Class Codes: G03B 13/36 Autofocus systems
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

Focus detection is to determine whether an image is in focus or not. Focus detection is able to be used for improving camera autofocus performance. Focus detection by using only one feature does not provide enough reliability to distinguish in-focus and slightly out-of-focus images. A focus detection algorithm of combining multiple features used to evaluate sharpness is described herein. A large image data set with in-focus and out-of-focus images is used to develop the focus detector for separating the in-focus images from out-of-focus images. Many features such as iterative blur estimation, FFT linearity, edge percentage, wavelet energy ratio, improved wavelet energy ratio, Chebyshev moment ratio and chromatic aberration features are able to be used to evaluate sharpness and determine big blur images.

Claims

1. A method programmed in a non-transitory memory of a device comprising:

a. acquiring content, wherein the content includes one or more images, wherein the content includes at least one small blur image;

b. determining if the content includes one or more big blur images;

c. removing the one or more big blur images, when the one or more big blur images are determined; and

d. determining in-focus images of small blur images without the one or more big blur images, wherein determining the in-focus images of the small blur images utilizes thresholds set for iteration number difference, combined chromatic features and combined non-chromatic features.

2. The method of claim 1 wherein the one or more big blur images are far from an in-focus position such that the one or more big blur images are at least 10 depth of field away.

3. The method of claim 1 wherein the one or more big blur images are determined using criteria selected from iterative blur estimation, FFT linearity, edge percentage, wavelet energy ratio, improved wavelet energy ratio, Chebyshev moment ratio and chromatic aberration features.

4. The method of claim 1 wherein determining if the content includes the one or more big blur images includes utilizing chromatic aberration features including computing a wavelet energy ratio for a first channel of the one or more images, computing a wavelet energy ratio for a second channel of the one or more images, computing a difference of the wavelet energy ratio for the first channel and the wavelet energy ratio of the second channel and comparing the difference with a threshold.

5. The method of claim 4 wherein if the difference is below the threshold, then the one or more images are in focus.

6. The method of claim 1 wherein determining if the content includes the one or more big blur images includes computing a fast fourier transform of an area, computing a radial average of a magnitude of the fast fourier transform coefficients around frequency 0, computing a logarithm of magnitude and frequency, computing a linear regression, calculating an error between the linear regression result and a fast fourier transform coefficient curve for measuring linearity and combining a linearity error with a slope of the linear regression for focus detection.

7. The method of claim 1 wherein determining if the content includes the one or more big blur images includes computing a Chebyshev moment ratio.

8. The method of claim 1 wherein the device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart phone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an portable music player, a tablet computer, a video player, a DVD writer/player, a high definition video writer/player, a television and a home entertainment system.

9. A system comprising:

a. a sensor configured for capturing content from a scene, wherein the content includes one or more images, wherein the content includes at least one small blur image; and

b. a computing module configured for;

i. determining if the content includes one or more big blur images;

ii. removing the one or more big blur images, when the one or more big blur images are determined; and

iii. determining in-focus images of small blur images without the one or more big blur images, wherein determining the in-focus images of the small blur images utilizes thresholds set for iteration number difference, combined chromatic features and combined non-chromatic features.

10. The system of claim 9 wherein the one or more big blur images are far from an in-focus position such that the one or more big blur images are at least 10 depth of field away.

11. The system of claim 9 wherein the one or more big blur images are determined using criteria selected from iterative blur estimation, FFT linearity, edge percentage, wavelet energy ratio, improved wavelet energy ratio, Chebyshev moment ratio and chromatic aberration features.

12. The system of claim 9 wherein determining if the content includes the one or more big blur images includes utilizing chromatic aberration features including computing a wavelet energy ratio for a first channel of the one or more images, computing a wavelet energy ratio for a second channel of the one or more images, computing a difference of the wavelet energy ratio for the first channel and the wavelet energy ratio of the second channel and comparing the difference with a threshold.

13. The system of claim 12 wherein if the difference is below the threshold, then the one or more images are in focus.

14. The system of claim 9 wherein determining if the content includes the one or more big blur images includes computing a fast fourier transform of an area, computing a radial average of a magnitude of the fast fourier transform coefficients around frequency 0, computing a logarithm of magnitude and frequency, computing a linear regression, calculating an error between the linear regression result and a fast fourier transform coefficient curve for measuring linearity and combining a linearity error with a slope of the linear regression for focus detection.

15. The system of claim 9 wherein determining if the content includes the one or more big blur images includes computing a Chebyshev moment ratio.

16. A camera device comprising:

a. a sensor for capturing content from a scene, wherein the content includes one or more images, wherein the content includes at least one small blur image; and

b. a memory for storing an application, the application for;

i. determining if the content includes one or more big blur images;

ii. removing the one or more big blur images, when the one or more big blur images are determined; and

iii. determining in-focus images of small blur images without the one or more big blur images, wherein determining the in-focus images of the small blur images utilizes thresholds set for iteration number difference, combined chromatic features and combined non-chromatic features; and

c. a processor for processing the application.

17. The camera device of claim 16 wherein the one or more big blur images are far from an in-focus position such that the one or more big blur images are at least 10 depth of field away.

18. The camera device of claim 16 wherein the one or more big blur images are determined using criteria selected from iterative blur estimation, FFT linearity, edge percentage, wavelet energy ratio, improved wavelet energy ratio, Chebyshev moment ratio and chromatic aberration features.

19. The camera device of claim 16 wherein determining if the content includes the one or more big blur images includes utilizing chromatic aberration features including computing a wavelet energy ratio for a first channel of the one or more images, computing a wavelet energy ratio for a second channel of the one or more images, computing a difference of the wavelet energy ratio for the first channel and the wavelet energy ratio of the second channel and comparing the difference with a threshold.

20. The camera device of claim 19 wherein if the difference is below the threshold, then the one or more images are in focus.

21. The camera device of claim 16 wherein determining if the content includes the one or more big blur images includes computing a fast fourier transform of an area, computing a radial average of a magnitude of the fast fourier transform coefficients around frequency 0, computing a logarithm of magnitude and frequency, computing a linear regression, calculating an error between the linear regression result and a fast fourier transform coefficient curve for measuring linearity and combining linearity error with a slope of the linear regression for focus detection.

22. The camera device of claim 16 wherein determining if the content includes the one or more big blur images includes computing a Chebyshev moment ratio.