Image synthesizing apparatus, image synthesizing method, and image synthesizing program

• US 8,682,104 B2
• Filed: 03/05/2012
• Issued: 03/25/2014
• Est. Priority Date: 03/11/2011
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sugiyama, Masatoshi, Shiono, Toru, Fujii, Shinichi
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US13/411,662
• Publication Number:
  US 20120230606A1
• Time in Patent Office: 750 Days
• Field of Search: 382/106, 382/107, 382/173, 382/284, 382/291, 382/298, 382/305, 382/312, 382/190, 348/36, 348/239, 345/629, 345/660, 358/1.2, 358/528, 358/540, 375/240.16
• US Class Current: 382/284
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/704 Pixels specially adapted for focusing, e.g. phase difference pixel sets
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

An image synthesizing apparatus includes a subject region determination unit that determines a region of a nearby subject for each of taken images based on subject distance information of a plurality of points generated by distance measurement made during imaging and an image synthesizing unit that creates a synthetic image by cutting out an image from each of the taken images based on a position of the region of the nearby subject and combining the cut images of the taken images.

Claims

1. An image synthesizing apparatus comprising:

a subject region determination unit that determines a region of a nearby subject for each of taken images based on subject distance information of a plurality of points generated by distance measurement made during imaging; and

an image synthesizing unit that creates a synthetic image by cutting out an image from each of the taken images based on a position of the region of the nearby subject and combining the cut image for each of the taken images.

2. The image synthesizing apparatus according to claim 1, wherein the image synthesizing unit cuts out an image without dividing the region of the nearby subject.

3. The image synthesizing apparatus according to claim 1, wherein the image synthesizing unit cuts out an image at the center for a region other than the region of the nearby subject.

4. The image synthesizing apparatus according to claim 1, wherein the image synthesizing unit determines whether the region of the nearby subject is redundantly present in taken images based on the subject distance information and, when the region of the nearby subject is redundantly present, the image synthesizing unit cuts out the image in the region of the nearby subject from a taken image in which the region of the nearby subject is closest to the center of the taken image, the taken image being one of the taken images.

5. The image synthesizing apparatus according to claim 4, wherein the image synthesizing unit cuts out the image in the region of the nearby subject from a taken image in which the region of the nearby subject is closest to the center of the taken image in a direction of combination.

6. The image synthesizing apparatus according to claim 4, wherein the image synthesizing unit determines whether the region of the nearby subject is redundantly present in taken images based on the correlation of subject distance information corresponding to the determined region of the nearby subject.

7. The image synthesizing apparatus according to claim 1, further comprising:

an enlargement/reduction unit that estimates a duplicate region in taken images based on the subject distance information and enlarges or reduces the taken images according to difference in the subject distance information in the estimated duplicate region;

wherein the image synthesizing unit creates the synthetic image using the taken images enlarged or reduced according to the difference in the subject distance information.

8. The image synthesizing apparatus according to claim 7, wherein the enlargement/reduction unit estimates the duplicate region in taken images based on the correlation of the subject distance information.

9. The image synthesizing apparatus according to claim 1, further comprising:

a motion vector detecting unit that detects a motion vector between two consecutive taken images based on the subject distance information;

wherein the image synthesizing unit creates the synthetic image based on the detected motion vector.

10. The image synthesizing apparatus according to claim 9, wherein the motion vector detecting unit performs first detection for a motion vector between taken images based on the subject distance information, sets a search range based on the result of the first detection, and performs second detection for the motion vector using image data in the search range.

11. The image synthesizing apparatus according to claim 1, wherein the subject region determination unit makes comparison between a preset threshold and the subject distance information and determines the region of the nearby subject based on the result of the comparison.

12. An image synthesizing method comprising:

determining a region of a nearby subject for each of taken images based on subject distance information of a plurality of points generated by distance measurement made during imaging;

cutting out an image from each of the taken images based on a position of the region of the nearby subject; and

combining the cut image for each of the taken images to create a synthetic image.

13. The method according to claim 12, wherein the act of cutting further comprises cutting out an image without dividing the region of the nearby subject.

14. The method according to claim 12, wherein the act of cutting further comprises cutting out an image at the center for a region other than the region of the nearby subject.

15. The method according to claim 12, wherein:

the act of determining further comprises determining whether the region of the nearby subject is redundantly present in taken images based on the subject distance information; and

when the region of the nearby subject is redundantly present, the act of cutting further comprises cutting out the image in the region of the nearby subject from a taken image in which the region of the nearby subject is closest to the center of the taken image, the taken image being one of the taken images.

16. The method according to claim 15, wherein the act of cutting further comprises cutting out the image in the region of the nearby subject from a taken image in which the region of the nearby subject is closest to the center of the taken image in a direction of combination.

17. The image synthesizing method of claim 15, wherein the act of determining further comprises determining whether the region of the nearby subject is redundantly present in taken images based on the correlation of subject distance information corresponding to the determined region of the nearby subject.

18. The method according to claim 12, wherein:

the method further comprises acts of estimating a duplicate region in taken images based on the subject distance information, and enlarging or reducing the taken images according to difference in the subject distance information in the estimated duplicate region; and

the act of combining further comprises creating the synthetic image using the taken images enlarged or reduced according to the difference in the subject distance information.

19. The method according to claim 18, wherein the act of estimating further comprises estimating the duplicate region in taken images based on the correlation of the subject distance information.

20. The method according to claim 12, wherein:

the method further comprises detecting a motion vector between two consecutive taken images based on the subject distance information; and

the act of combining further comprises creating the synthetic image based on the detected motion vector.

21. The method according to claim 20, wherein the act of detecting further comprises:

performing first detection for a motion vector between taken images based on the subject distance information;

setting a search range based on the result of the first detection; and

performing second detection for the motion vector using image data in the search range.

22. The method according to claim 12, wherein the act of determining further comprises acts of making a comparison between a preset threshold and the subject distance information, and determining the region of the nearby subject based on the result of the comparison.

23. A non-transient computer-readable medium having instructions encoded thereon which, when executed by at least one processor, cause the at least one processor to execute a process for image synthesis creating a synthetic image with a wide angle of view from a plurality of taken images, the process comprising:

determining a region of a nearby subject for each of taken images based on subject distance information of a plurality of points generated by distance measurement made during imaging;

cutting out an image from each of the taken images based on a position of the region of the nearby subject; and

combining the cut image for each of the taken images to create a synthetic image.

24. The computer-readable medium according to claim 23, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of cutting further comprises cutting out an image without dividing the region of the nearby subject.

25. The computer-readable medium according to claim 23, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of cutting further comprises cutting out an image at the center for a region other than the region of the nearby subject.

26. The computer-readable medium according to claim 23, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that:

the act of determining further comprises determining whether the region of the nearby subject is redundantly present in taken images based on the subject distance information; and

when the region of the nearby subject is redundantly present, the act of cutting further comprises cutting out the image in the region of the nearby subject from a taken image in which the region of the nearby subject is closest to the center of the taken image, the taken image being one of the taken images.

27. The computer-readable medium according to claim 26, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of cutting further comprises cutting out the image in the region of the nearby subject from a taken image in which the region of the nearby subject is closest to the center of the taken image in a direction of combination.

28. The computer-readable medium according to claim 26, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of determining further comprises determining whether the region of the nearby subject is redundantly present in taken images based on the correlation of subject distance information corresponding to the determined region of the nearby subject.

29. The computer-readable medium according to claim 23, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that:

the process further comprises acts of estimating a duplicate region in taken images based on the subject distance information, and enlarging or reducing the taken images according to difference in the subject distance information in the estimated duplicate region; and

the act of combining further comprises creating the synthetic image using the taken images enlarged or reduced according to the difference in the subject distance information.

30. The computer-readable medium according to claim 29, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of estimating further comprises estimating the duplicate region in taken images based on the correlation of the subject distance information.

31. The computer-readable medium according to claim 23, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that:

the process further comprises detecting a motion vector between two consecutive taken images based on the subject distance information; and

the act of combining further comprises creating the synthetic image based on the detected motion vector.

32. The computer-readable medium according to claim 31, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of detecting further comprises:

performing first detection for a motion vector between taken images based on the subject distance information;

setting a search range based on the result of the first detection; and

performing second detection for the motion vector using image data in the search range.

33. The computer-readable medium according to claim 23, further comprising additional instructions which, when executed by the at least one processor, cause the at least one processor to perform the process so that the act of determining further comprises acts of making a comparison between a preset threshold and the subject distance information, and determining the region of the nearby subject based on the result of the comparison.