Image processing device and image processing method

• US 8,482,651 B2
• Filed: 01/10/2008
• Issued: 07/09/2013
• Est. Priority Date: 02/15/2007
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Asukai, Masamichi, Tsuruta, Masaaki, Ito, Taiji, Sako, Yoichiro
• Primary Examiner(s): OSINSKI, MICHAEL S
• Application Number:
  US12/449,230
• Publication Number:
  US 20100066840A1
• Time in Patent Office: 2,007 Days
• Field of Search: 348/207.99, 348/207.1, 348/208.99, 348/208.1, 348/208.13, 348/218.1, 348/222.1, 34823199-2313, 348/239, 348/241, 34833301-33312, 382/103, 382274-279, 382282-283
• US Class Current: 348/333.12
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

Among pickup image data items of multiple frames picked up at different time instants, an image of a certain frame is regarded as a synthesizing base image. The synthesizing base image and an image of another frame are used to perform synthesizing processing. Specifically, an object-of-synthesis area is designated in the frame image regarded as the synthesizing base image, and processing of replacing pixel data in the object-of-synthesis area with pixel data in another frame is carried out. For example, if a person having his/her eyes closed appears in the synthesizing base image that is a group-photograph image, the part of the person'"'"'s face is regarded as the object-of-synthesis area. As another frame, an image in which the person has his/her eyes left open is searched. Pixel data in the facial image part showing the open eyes is fetched and synthesized with the synthesizing base image. Thus, a group photograph in which all people have their eyes left open is produced.

Claims

1. An image processing method comprising:

acquiring pickup image data items of multiple frames, which are picked up at different times, as pickup image data items to be used for synthesizing processing;

designating one or more object-of-synthesis areas in a selected frame out of the acquired multiple frames;

providing an indication to a user that synthesizing processing may be performed on a first object-of-synthesis area out of the one or more object-of-synthesis areas; and

when receiving input from the user indicating that synthesizing processing should be performed on the first object-of-synthesis area, producing synthetic image data by performing synthesizing processing of replacing pixel data in the first object-of-synthesis area with pixel data in another frame out of the multiple frames; and

performing preserving processing on the synthetic image data, wherein prior to performing synthesizing processing, the user manipulates at least one of the one or more object-of-synthesis areas by selecting said at least one of the one or more object-of-synthesis areas within the selected frame.

2. The image processing device method according to claim 1, wherein acquiring pickup image data items comprises acquiring pickup image data of at least one frame, which is included in the pickup image data items to be used for the synthesizing processing, through imaging by an imaging unit.

3. The image processing method according to claim 2, wherein acquiring pickup image data items comprises acquiring pickup image data items of multiple frames, which are used for the synthesizing processing, through imaging by the imaging unit.

4. The image processing method according to claim 2, wherein the imaging unit is formed using a CCD sensor or a CMOS sensor as an image sensor.

5. The image processing method according to claim 1, wherein acquiring pickup image data items comprises receiving and acquiring pickup image data of at least one frame, which is included in the pickup image data items to be used for the synthesizing processing, through communication with external equipment by a communication unit.

6. The image processing method according to claim 1, wherein acquiring pickup image data items comprises acquiring pickup image data of at least one frame, which is included in the pickup image data items to be used for the synthesizing processing, through reproduction from a portable recording medium by a reproduction unit.

7. The image processing method according to claim 1, wherein the preserving processing means includes a recording unit to record information in a recording medium, and is adapted to perform the performing preserving processing comprises recording the synthetic image data in a recording medium using a recording unit.

8. The image processing method according to claim 1, wherein performing preserving processing comprises transmitting the synthetic image data to external equipment using a communication unit.

9. The image processing method according to claim 1, wherein designating one or more object-of-synthesis areas comprises image analyzing processing.

10. The image processing method according to claim 9, wherein designating one or more object-of-synthesis areas comprises performing a discrimination of all or part of the opening or closing of the eyes in a facial image of a person contained in a frame image, the facial expression, the direction of a face, and the color of the eyes through image analyzing processing, and designates the one or more object-of-synthesis areas on the basis of a result of the discrimination.

11. The image processing method according to claim 1, wherein

designating one or more object-of-synthesis areas is responsive to entry performed at an operating input means.

12. The image processing method according to claim 1, wherein designating one or more object-of-synthesis areas comprises determining a frame to be used as another frame through image analyzing processing performed on each of the multiple frames, and fetch pixel data, which is synthesized with at least one of the one or more object-of-synthesis areas, from the determined frame.

13. The image processing method according to claim 1, wherein designating one or more object-of-synthesis areas comprises:

selecting a frame to be used as another frame from among the multiple frames responsive to entry performed at an operating input means, and

fetching pixel data, which is synthesized with at least one of the one or more object-of-synthesis areas, from the selected frame.

14. The image processing method according to claim 1, further comprising performing outputting processing on the synthetic image data.

15. The image processing method according to claim 14, wherein performing outputting processing on the synthetic image data comprises displaying or outputting the synthetic image data on or to a display unit.

16. The image processing method according to claim 14, wherein performing outputting processing on the synthetic image data comprises transmission or output of the synthetic image data to external equipment via a communication unit.

17. The image processing method according to claim 14, wherein performing outputting processing on the synthetic image data comprises recording the synthetic image data in a portable recording medium via a recording unit.