Image sensing apparatus and method for synthesizing a composite image

• US 7,352,393 B2
• Filed: 04/11/2006
• Issued: 04/01/2008
• Est. Priority Date: 06/22/1998
• Status: Expired due to Fees

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Sakamoto, Koichi
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): JERABEK, KELLY L
• Application Number:
  US11/401,363
• Publication Number:
  US 20060187321A1
• Time in Patent Office: 721 Days
• Field of Search: 348/218.1, 348/239, 348/220.1, 348584-586, 382282-284
• US Class Current: 348/239
• CPC Class Codes: H04N 1/3871 the composed originals being of different kinds, e.g. low- and high-resolution originals
• CPC Class Codes: H04N 1/6086 by scene illuminant, i.e. conditions at the time of picture capture, e.g. flash, optical filter used, evening, cloud, daylight, artificial lighting, white point measurement, colour temperature
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/74 for obtaining special effects H04N9/65 - H04N9/73 take precedence

Abstract

A composite image having a natural appearance is obtained when a synthesis target image is combined with a background image. By photographing a subject under different photographic conditions, a plurality of synthesis target images are obtained and the resulting image data is stored beforehand in a synthesis target image memory. A background image is acquired by photography and a suitable synthesis target image to be combined with the background image is selected by a circuit which searches synthesis target images. A color correction is applied to the selected synthesis target image by an image processing circuit and the synthesis target image following the color correction is combined with the background image. Since a synthesis target image suitable for combination with a background image is selected and then is subjected to a color correction and image synthesis processing, a composite image having a natural appearance is obtained.

Claims

1. An image sensing apparatus, comprising:

synthesis target information storage means for storing, in a storage device, data representing a synthesis target image of a foreground subject and for separately storing, also in the storage device, photographic conditions prevailing at a time of photography of the synthesis target image;

image sensing means for photographing background and outputting data representing a background image;

image adjustment processing means for applying image adjustment processing for at least one of a color adjustment and a luminance adjustment to the synthesis target image represented by the synthesis target image data, which has been stored in the storage device by the synthesis target information storage means, based upon the photographic conditions of the synthesis target image that have been stored in the storage device by the synthesis target information storage means and photographic conditions of the background image sensed by the image sensing means; and

image synthesis means for combining the synthesis target image data that has been subjected to the image adjustment processing by the image adjustment processing means with the background image data output from the image sensing means, and outputting composite image data representing a composite image.

2. The image sensing apparatus according to claim 1, wherein the synthesis target image data that has been stored in the synthesis target image information storage means was obtained by strobe flash photography.

3. The image sensing apparatus according to claim 1, further comprising image adjustment command input means for inputting an image adjustment command for at least one of a color adjustment and a luminance adjustment,

wherein the image adjustment processing means applies image adjustment processing for at least one of a color adjustment and a luminance adjustment to a synthesis target image based upon an image adjustment command input from the image adjustment command input means, and

wherein the image synthesis means combines the synthesis target image data, which has been subjected to image adjustment processing by the image adjustment means, with the background image data.

4. The image sensing apparatus according to claim 1, further comprising:

a display unit for displaying images;

background image display control means for controlling the display unit so as to display the background image sensed by the image sensing means; and

composite image display control means for controlling the display unit so as to display the composite image synthesized by the image synthesis means.

5. The image sensing apparatus according to claim 4, further comprising:

adjustment command input means for inputting an adjustment command for at least one of position and size adjustment of a synthesis target image being displayed on the display unit; and

adjustment means for applying an adjustment for at least one of position and size adjustment of the synthesis target image based upon the adjustment command input from the adjustment command input means.

7. The image sensing apparatus according to claim 1, wherein the image adjustment processing means performs the color adjustment prior to performing the luminance adjustment of the synthesis target image.

8. The image sensing apparatus according to claim 1, wherein the photographic conditions include photographic circumstances and strobe information.

9. The image sensing apparatus according to claim 1, wherein the photographic conditions of the synthesis target image prevailing at the time of photography of the synthesis target image are predetermined.

6. An image sensing method, comprising:

storing, in a storage device, data representing a synthesis target image of a foreground subject obtained by photography;

separately storing, in the storage device, photographic conditions prevailing at a time of photography of the synthesis target image;

obtaining data representing a background image by photographing a background;

applying image adjustment processing for at least one of a color adjustment and a luminance adjustment to the synthesis target image represented by the synthesis target image data stored in the storage device based upon the photographic conditions of the synthesis target image stored in the storage device and photographic conditions prevailing at the time of photography of the background; and

combining the synthesis target image data that has been subjected to the image adjustment processing and the background image data, and outputting composite image data representing a composite image.

10. The image sensing method according to claim 6, wherein the photographic conditions of the synthesis target image prevailing at the time of photography of the synthesis target image are predetermined.

11. The image sensing method according to claim 6, further comprising:

obtaining the synthesis target image data through strobe flash photography.

12. The image sensing method of claim 6, further comprising:

receiving an image adjustment command from a user for adjusting at least one of a color adjustment and luminance adjustment; and

applying image adjustment processing for the at least one of a color adjustment and a luminance adjustment to a synthesis target image represented by the selected synthesis target image data based upon the received image adjustment command.

13. The image sensing method of claim 6, further comprising:

displaying images on a display unit;

controlling said display unit so as to display the background image; and

controlling said display unit so as to display the composite image.

14. The image sensing method of claim 13, further comprising:

receiving an adjustment command for adjusting at least one of a position and a size of a synthesis target image being displayed on the display unit; and

applying an adjustment for at least one of position and size adjustment of the synthesis target image based upon the received adjustment command.

15. An image sensing method, comprising:

storing, in a synthesis target image memory, synthesis target image data of a foreground subject obtained by photography under a predetermined lighting condition;

obtaining background image data through photographing a background and storing the background image data to a background image memory;

obtaining background lighting condition corresponding to the background image data at the time the background is photographed;

automatically determining, without user input, whether a first image adjustment processing on the synthesis target image data is necessary based on the predetermined lighting condition and the background lighting condition;

performing the first image adjustment processing on the synthesis target image data when it is automatically determined that the image adjustment processing on the synthesis target image data is necessary;

combining the synthesis target image data that have been processed through the first image adjustment processing and the background image data to generate composite image data representing a composite image;

allowing a user to determine whether a second image adjustment processing is necessary based on the composite image;

performing the second image adjustment processing on the synthesis target image data when the user determines that the image adjustment processing on the synthesis target image data is necessary; and

combining the synthesis target image data that have been processed through the second image adjustment processing and the background image data to generate composite image data representing a composite image.

16. The image sensing method according to claim 15, further comprising:

separately storing, in the synthesis target image memory, the predetermined lighting condition or separately storing, in the background image memory, the background lighting condition,

wherein the step of automatically determining, without user input, whether the first image adjustment processing on the synthesis target image data is necessary includes;

comparing the predetermined lighting condition stored in the synthesis target image memory with the background lighting condition stored in the background image memory.

17. The image sensing method according to claim 16, wherein the predetermined lighting condition and the background lighting condition include any one or more of a strobe photography, frontlit photography, cloudy-weather photography, backlit photography, color temperature, brightness, contrast, and luminance.

18. The image sensing method according to claim 15,

wherein the first image adjustment processing on the synthesis target image data includes any one or more of a contrast correction processing, a color correction processing, and a luminance correction processing, and

wherein the second image adjustment processing on the synthesis target image data also includes any one or more of the contrast correction processing, the color correction processing, and the luminance correction processing.

19. The image sensing method according to claim 18, wherein the contrast correction processing includes the steps of:

determining if contrast correction values have been set manually;

performing contrast correction on the synthesis target image data based on the manually set contrast correction values when it is determined that the contrast correction values have been set manually;

calculating the contrast correction values based on the predetermined lighting condition and the background lighting condition when it is determined that the contrast correction values have not been set manually; and

performing the contrast correction on the synthesis target image data based on the calculated contrast correction values when it is determined that the contrast correction values have not been set manually.

20. The image sensing method according to claim 19, wherein the step of calculating the contrast correction values includes calculating the contrast correction values based on a difference in brightness between the predetermined lighting condition and the background lighting condition.

21. The image sensing method according to claim 18, wherein the color correction processing includes the steps of:

converting RGB color values of the synthesis target image data to corresponding XYZ tristimulus values;

calculating correction coefficients based on color temperature data in the predetermined lighting condition and on color temperature data in the background lighting condition;

performing color balance correction on the XYZ tristimulus values; and

converting the corrected XYZ tristimulus values to corrected RGB color values.

22. The image sensing method according to claim 18, wherein the luminance correction processing includes the steps of:

determining if luminance correction values have been set manually;

performing luminance correction on the synthesis target image data based on the manually set luminance correction values when it is determined that the luminance correction values have been set manually;

calculating the luminance correction values based on the predetermined lighting condition and the background lighting condition when it is determined that the luminance correction values have not been set manually; and

performing the luminance correction on the synthesis target image data based on the calculated luminance correction values when it is determined that the luminance correction values have not been set manually.

23. The image sensing method according to claim 22, wherein the step of calculating the luminance correction values includes:

calculating the luminance correction values based on

a difference in brightness between the predetermined lighting condition and the background lighting condition, or

whether the background image was frontlit or backlit, or both.

24. The image sensing method according to claim 18, wherein the step of automatically determining whether the first image adjustment processing on the synthesis target image data is necessary includes:

determining whether the contrast correction processing is necessary;

performing the contrast correction processing when it is determined that the contrast correction processing is necessary;

determining whether the color correction processing is necessary;

performing the color correction processing when it is determined that the color correction processing is necessary;

determining whether the luminance correction processing is necessary; and

performing the luminance correction processing when it is determined that the luminance correction processing is necessary.

25. The image sensing method according to claim 24, wherein the step of determining whether the contrast correction processing is necessary includes:

determining a first condition of whether a frontlit or backlit information of the predetermined lighting condition matches a frontlit or backlit information of the background lighting condition;

determining a second condition of whether a strobe information of the predetermined lighting condition matches a strobe information of the background lighting condition;

determining that the contrast correction processing is necessary when it is determined that one or both of the first and second conditions are not true; and

determining that the contrast correction processing is not necessary when it is determined that both of the first and second conditions are true.

26. The image sensing method according to claim 24, wherein the step of determining whether the color correction processing is necessary includes:

determining whether a difference between a color temperature of the predetermined lighting condition and a color temperature of the background lighting condition is within a predetermined allowable color difference range;

determining that the color correction processing is necessary when it is determined that the difference between the color temperature of the predetermined lighting condition and the color temperature of the background lighting condition is not within the predetermined allowable color difference range; and

determining that the color correction processing is not necessary when it is determined that the difference between the color temperature of the predetermined lighting condition and the color temperature of the background lighting condition is within the predetermined allowable color difference range.

27. The image sensing method according to claim 24, wherein the step of determining whether the luminance correction processing is necessary includes:

determining whether a difference between a luminance of the predetermined lighting condition and a luminance of the background lighting condition is within a predetermined allowable luminance difference range;

determining that the luminance correction processing is necessary when it is determined that the difference between the luminance of the predetermined lighting condition and the luminance of the background lighting condition is not within the predetermined allowable luminance difference range; and

determining that the luminance correction processing is not necessary when it is determined that the difference between the luminance of the predetermined lighting condition and the luminance of the background lighting condition is within the predetermined allowable luminance difference range.

28. The image sensing method according to claim 18, wherein the step of allowing the user to determine whether the second image adjustment processing on the synthesis target image data is necessary includes:

allowing the user the manual set contrast correction values;

performing the contrast correction processing when the contrast correction values are manually set;

allowing the user the manual set color balance values;

performing the color correction processing when the color balance values are manually set;

allowing the user the manual set luminance correction values;

performing the luminance correction processing when the luminance correction values are manually set.