Image taking apparatus with shake correction, image processing apparatus with shake correction, image processing method with shake correction, and image processing program with shake correction

• US 8,228,390 B2
• Filed: 10/25/2007
• Issued: 07/24/2012
• Est. Priority Date: 11/30/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Okada, Miyuki, Nomura, Hirofumi, Tsujimura, Takashi
• Primary Examiner(s): VU, NGOC YEN T
• Application Number:
  US11/924,158
• Publication Number:
  US 20080186386A1
• Time in Patent Office: 1,734 Days
• Field of Search: 348/208.14
• US Class Current: 348/208.6
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8047 using transform coding
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal

Abstract

In an image taking apparatus, an imaging unit senses an image of a subject, and a face detection unit detects a face in the image sensed by the imaging unit. A motion detection unit detects an amount of face motion indicating an amount of shake of an area of the face detected by the face detection unit in the image sensed by the imaging unit and also detects an amount of background motion indicating an amount of shake of an area other than the face area. A shake correction unit corrects the shake of the image sensed by the imaging unit according to the amount of face motion and the amount of background motion detected by the motion detection unit.

Claims

1. An image taking apparatus comprising:

an imaging sensing unit configured to sense an image of a subject;

a face detection unit configured to detect a face in the image sensed by the imaging unit;

a motion detection unit configured to detect, via a processor, an amount of face motion indicating an amount of shake of an area of the face detected, by the face detection unit, in the image sensed by the imaging unit and also detecting an average amount of background motion indicating an amount of shake of an area other than the face area, based on shake contributions of an entirety of the area other than the face area; and

a shake correction unit configured to correct shake of the image sensed by the imaging unit according to the amount of face motion and the average amount of background motion detected by the motion detection unit.

2. The image taking apparatus according to claim 1, further comprising:

a correction value calculation unit configured to calculate a correction value of the shake of the image to be corrected by the shake correction unit, in accordance with the amount of face motion and the average amount of background motion.

3. The image taking apparatus according to claim 2, wherein the correction value calculation unit calculates the correction value in accordance with the amount of face motion, the average amount of background motion and predetermined weighting factors.

4. The image taking apparatus according to claim 3, wherein the correction value calculation unit calculates the correction value by applying a greater weighting factor to the amount of face motion than to the average amount of background motion.

5. The image taking apparatus according to claim 3, wherein

the face detection unit further detects feature information associated with the face detected in the image, and

the correction value calculation unit calculates the weighting factors in accordance with the feature information associated with the face.

6. The image taking apparatus according to claim 5, wherein the feature information associated with the face includes one or more pieces of information selected from a group consisting of information indicating a face area size, information indicating coordinates of the face area, information indicating the likelihood of being a face, information indicating the degree of facing forward, information indicating the degree of tilt of the face, information indicating the degree of smiling, information indicating the degree of being serious, and information indicating the degree of closing of eyes.

7. The image taking apparatus according to claim 5, wherein when calculating the correction value, the correction value calculation unit increases the weighting factor for blocks located close to the face.

8. The image taking apparatus according to claim 5, further comprising:

a camera control value detection configured to detect the degree of whether focus and/or exposure are set optimally for the face detected by the face detection unit, based on the feature information associated with the face,

wherein the correction value calculation unit calculates the correction value for the face in accordance with the degree of whether focus and/or exposure are set optimally for the face.

9. The image taking apparatus according to claim 3, further comprising:

a specific person storage unit configured to store specific person information associated with a face of a specific person; and

a specific person identification unit configured to determine whether the face is identical to the face of the specific person,

wherein the correction value calculation unit

calculates the correction value such that when the specific person identification unit determines that the face is identical to the face of the specific person, and

modifies the weighting factor for each area depending on whether the area includes the face of the specific person.

10. The image taking apparatus according to claim 2, wherein the correction value calculation unit calculates a first average value indicating the average of the amount of face motion, calculates the correction value for the area of the face in accordance with the first average value, and calculates the correction value for areas other than the area of the face in accordance with the average amount of background motion.

11. The image taking apparatus according to claim 10, wherein when a plurality of faces are detected, the correction value calculation unit calculates an average value of the amount of face motion for each of the faces and calculates the correction value in accordance with the calculated average value.

12. The image taking apparatus according to claim 1, wherein the motion detection unit divides the image into a plurality of areas such that the face is included in one of the areas, detects the amount of face motion from the area including the face, and detects the average amount of background motion from areas other than the area including the face.

13. The image taking apparatus according to claim 1, further comprising:

a vibration detection unit configured to detect an amount of shake due to a vibration of the image taking apparatus,

wherein the shake correction unit corrects the shake of the image in accordance with the amount of face motion and the average amount of background motion and the amount of shake.

14. The image taking apparatus according to claim 1, further comprising:

a setting unit configured to set whether to perform detection of a face in the image.

15. The image taking apparatus according to claim 1, wherein the shake correction unit corrects the shake of the image such that the area of the face is moved toward the center of the image.

16. The image taking apparatus according to claim 1, wherein the motion detection unit detects the amount of face motion of the area of the face, in the image on the basis of the correlation between the face from a current image and a face detected from a previous image detected by the imaging unit a predetermined time before detecting the current image.

17. An image processing apparatus comprising:

an image input configured to input an image;

a face detection unit configured to detect a face in the image input via the image input unit;

a motion detection unit configured to detect, via a processor, an amount of face motion indicating an amount of shake of an area of the face detected, by the face detection unit, in the image input via the image input unit and also detecting an average amount of background motion indicating an amount of shake of an area other than the face area based on shake contributions of an entirety of the area other than the face area; and

a shake correction unit configured to correct the shake of the image input via the image input unit according to the amount of face motion and the average amount of background motion detected by the motion detection unit.

18. A method of processing an image, comprising:

inputting an image;

detecting a face in the image;

detecting, by a processor, an amount of face motion indicating an amount of shake of an area of the face in the image, and detecting an average amount of background motion indicating an amount of shake of an area other than the face area based on shake contributions of an entirety of the area other than the face area; and

correcting the shake of the image according to the amount of face motion and the average amount of background motion.

19. The method of processing an image according to claim 18, further comprising calculating the correction value of the shake of the image to be corrected, in accordance with the amount of face motion and the average amount of background motion detected.

20. The method of processing an image according to claim 19, wherein calculating the correction value is in accordance with the amount of face motion and the average amount of background motion detected and in accordance with predetermined weighting factors.

21. The method of processing an image according to claim 20, wherein calculating the correction value includes applying a greater weighting factor to the amount of face motion detected than to the average amount of background motion detected.

22. The method of processing an image according to claim 20, wherein

detecting a face in the image includes detecting feature information associated with the face detected in the image;

and

calculating the correction value includes calculating the weighting factors in accordance with the feature information associated with the face detected.

23. The method of processing an image according to claim 22, wherein the feature information associated with the face includes one or more pieces of information selected from a group consisting of information indicating a face area size, information indicating coordinates of the face area, information indicating the likelihood of being a face, information indicating the degree of facing forward, information indicating the degree of tilt of the face, information indicating the degree of smiling, information indicating the degree of being serious, and information indicating the degree of closing of eyes.

24. The method of processing an image according to claim 22, wherein calculating the correction value includes increasing the weighting factor for blocks located close to the face detected.

25. The method of processing an image according to claim 22, further comprising detecting the degree of whether focus and/or exposure are set optimally for the face detected, based on the feature information associated with the face,

wherein calculating the correction value for the face is in accordance with the detected degree of whether focus and/or exposure are set optimally for the face.

26. The method of processing an image according to claim 20, further comprising

storing specific person information associated with a face of a specific person;

and

determining whether the face detected is identical to the face of the specific person stored, wherein

calculating the correction value includes modifying the weighting factor for each area depending on whether the area includes the face of the specific person if it is determined that the face detected is identical to the face of the specific person stored.

27. The method of processing an image according to claim 19, wherein calculating the correction value includes calculating a first average value indicating the average of the amounts of face motion detected, calculating the correction value for the area of the face detected in accordance with the first average value, and calculating the correction value for areas other than the area of the face detected in accordance with the average amount of background motion detected.

28. The method of processing an image according to claim 27, wherein, if a plurality of faces is detected, calculating the correction value is in accordance with includes calculating the average value of the amounts of face motion for each of the faces detected and calculating the correction value in accordance with the calculated average value.

29. The method of processing an image according to claim 18, wherein detecting an amount of face motion includes dividing the image inputted into a plurality of areas such that the face detected is included in one of the areas, detecting the amount of face motion from the area including the face, and detecting the average amount of background motion from areas other than the area including the face.

30. The method of processing an image according to claim 18, further comprising detecting an amount of shake due to a vibration of the image taking apparatus,

wherein correcting the shake of the image is in accordance with the amount of face motion and the average amount of background motion detected and the amount of shake due to a vibration detected.

31. The method of processing an image according to claim 18, further comprising setting whether detection of a face in the image sensed should be performed.

32. The method of processing an image according to claim 18, wherein correcting the shake of the image inputted moves the area of the face detected toward the center of the image.

33. The method of processing an image according to claim 18, wherein detecting an amount of face motion of an area of the face is based on a correlation between the face detected from the current image and a face detected from a previous image a predetermined time before the current image.

34. A non-transitory computer-readable medium encoded thereon with a program that, when executed by a computer, causes the computer to perform a method of image processing comprising:

inputting an image;

detecting a face in the image;

detecting an amount of face motion indicating an amount of shake of an area of the face in the image, and detecting an average amount of background motion indicating an amount of shake of an area other than the face area based on shake contributions of an entirety of the area other than the face area; and

correcting the shake of the image according to the amount of face motion and the average amount of background motion.

35. An image taking apparatus comprising:

imaging means for sensing an image of a subject;

face detection means for detecting a face in the image sensed by the imaging means;

motion detection means for detecting an amount of face motion indicating an amount of shake of an area of the face detected, by the face detection means, in the image sensed by the imaging means and also detecting an average amount of background motion indicating an amount of shake of an area other than the face area based on shake contributions of an entirety of the area other than the face area; and

shake correction means for correcting the shake of the image sensed by the imaging means according to the amount of face motion and the average amount of background motion detected by the motion detection means.

36. An image processing apparatus comprising:

image input means for inputting an image;

face detection means for detecting a face in the image input via the image input means;

motion detection means for detecting an amount of face motion indicating an amount of shake of an area of the face detected, by the face detection unit, in the image input via the image input means and also detecting an average amount of background motion indicating an amount of shake of an area other than the face area based on shake contributions of an entirety of the area other than the face area; and

shake correction means for correcting the shake of the image input via the image input means according to the amount of face motion and the average amount of background motion detected by the motion detection means.