Shooting device and shooting method to suppress blur in an image

• US 10,542,217 B2
• Filed: 11/11/2016
• Issued: 01/21/2020
• Est. Priority Date: 11/26/2015
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Inventors: Yamada, Hideshi, Iso, Daisuke, Kurita, Teppei, Li, Shun
• Primary Examiner(s): Berhan, Ahmed A
• Application Number:
  US15/773,695
• Publication Number:
  US 20180324358A1
• Time in Patent Office: 1,166 Days
• CPC Class Codes: G03B 2207/005 involving control of motion blur
• CPC Class Codes: G03B 5/00 Adjustment of optical system relative to image or object surface other than for focusing
• CPC Class Codes: G03B 7/00 Control of exposure by setting shutters, diaphragms or filters, separately or conjointly control of exposure in television cameras by means of circuitry influencing the optical part of the camera H04N23/75
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

The present disclosure relates to a shooting device, a shooting method, and a program capable of suppressing shooting of an image including large blurring due to motion of a camera such that an image with less blurring may be shot. The camera motion is detected, a distribution degree of a trajectory of the camera motion based on a camera motion detection result is calculated to be compared with a predetermined threshold, and start and finish of exposure is controlled on the basis of a comparison result. The present disclosure may be applied to the shooting device.

Claims

1. A shooting device, comprising:

a central processing unit (CPU) configured to;

detect camera motion;

calculate a distribution degree of a trajectory of the camera motion based on the camera motion, wherein

a first principal component of covariance is calculated as the distribution degree of the trajectory of the camera motion;

compare the distribution degree of the trajectory of the camera motion with a determined threshold; and

control start and finish of exposure based on a comparison result of the comparison.

2. The shooting device according to claim 1, wherein the CPU is further configured to:

generate a point spread function (PSF) image for the camera motion as the distribution degree of the trajectory of the camera motion based on the camera motion, and

execute frequency analysis of the PSF image to compare with the determined threshold.

3. The shooting device according to claim 1, wherein the CPU is further configured to:

approximate detection results of the camera motion in a certain range retroactive to a determined time from current time by a polynomial as the distribution degree of the trajectory of the camera motion based on the camera motion; and

extrapolate the camera motion in future time based on the polynomial.

4. The shooting device according to claim 1, wherein

the CPU further includes a gyroscope sensor, an acceleration sensor, a geomagnetic sensor, an altitude sensor, a vibration sensor, and a motion capture,

the motion capture is configured to track a marker of an object from a sub camera,

the sub camera detects motion of a first camera,

the first camera is different from a second camera that is used to measure the camera motion.

5. The shooting device according to claim 1, wherein the CPU is further configured to finish the exposure based on:

the exposure that is longer than a minimum exposure time,

the exposure that is not longer than a maximum exposure time after the exposure is started, and

blur amount is larger than determined blur amount based on the comparison result.

6. The shooting device according to claim 1,

wherein the CPU is further configured to finish the exposure based on a maximum exposure time that elapses after the exposure is started.

7. The shooting device according to claim 1, wherein

the CPU is further configured to delay a time at which the exposure is started until a blur amount is smaller than a determined blur by the comparison result, and

the delay is based on the blur amount that is larger than the determined blur by the comparison result at a time of start of the exposure.

8. The shooting device according to claim 1, wherein the CPU is further configured to finish the exposure in consideration of a signal to noise ratio (SNR) based on the comparison result.

9. The shooting device according to claim 1, wherein

the CPU is further configured to remove noise of an image based on integration of a plurality of images taken at a determined interval by the shooting device, and

the shooting device is a camera.

10. The shooting device according to claim 9,

wherein the CPU is further configured to:

integrate the image, in which a blur amount is smaller than a determined blur, of the plurality of images, wherein the plurality of images is taken by the camera; and

remove the removes noise of the image.

11. The shooting device according to claim 9, wherein

the CPU is further configured to:

integrate the plurality of images taken by the camera by addition of weights based on an exposure time; and

remove the noise of the image based on the integration.

12. The shooting device according to claim 9, wherein

the CPU is further configured to:

integrate the plurality of images taken by the camera by addition of weights based on a direction of blur of each image of the plurality of images; and

remove the noise of the image based on the integration.

13. The shooting device according to claim 9, wherein the CPU is further configured to:

integrate the plurality of images taken by the camera by addition of an equal weight; and

remove the noise of the image based on the integration.

14. The shooting device according to claim 9,

wherein the CPU is further configured to:

apply fast Fourier transform (FFT) on the plurality of images taken by the camera;

collect a component of determined amplitude for each frequency component of a plurality of frequency components; and

apply inverse FFT to generate the image, such that the noise of the image is removed.

15. The shooting device according to claim 14, wherein the CPU is further configured to:

apply the FFT on the plurality of images taken by the cameral;

collect a component of amplitude of a maximum value for each frequency component of the plurality of frequency components; and

apply the inverse FFT to generate the image, such that the noise of the image is removed.

16. A shooting method, comprising:

detecting camera motion;

calculating a distribution degree of a trajectory of the camera motion based on the camera motion, wherein

a first principal component of covariance is calculated as the distribution degree of the trajectory of the camera motion;

compare the distribution degree of the trajectory of the camera motion with a determined threshold; and

controlling start and finish of exposure based on a comparison result of the comparison.

17. A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by a computer, cause the computer to execute operations, the operations comprising:

detecting camera motion;

calculating a distribution degree of a trajectory of the camera motion based on the camera motion, wherein

a first principal component of covariance is calculated as the distribution degree of the trajectory of the camera motion;

compare the distribution degree of the trajectory of the camera motion with a determined threshold; and

controlling start and finish of exposure based on a comparison result of the comparison.

18. A shooting device, comprising:

a central processing unit (CPU) configured to;

detect camera motion;

calculate a distribution degree of a trajectory of the camera motion based on the camera motion, wherein

a point spread function (PSF) image for the camera motion is generated as the distribution degree of the trajectory of the camera motion based on the camera motion;

execute frequency analysis of the PSF image;

compare a frequency component in the PSF image with a determined threshold based on the frequency analysis; and

control start and finish of exposure based on a comparison result of the comparison.

19. A shooting device, comprising:

a central processing unit (CPU) configured to;

detect camera motion;

calculate a distribution degree of a trajectory of the camera motion based on the camera motion;

compare the distribution degree of the trajectory of the camera motion with a determined threshold;

control start and finish of exposure based on a comparison result of the comparison; and

remove noise of an image by integrating a plurality of images taken at a determined interval by the shooting device,

wherein the plurality of images is integrated while an equal weight is added, to remove the noise of the image.