METHOD AND APPARATUS FOR FUSION OF IMAGES

• US 20150147000A1
• Filed: 05/30/2013
• Published: 05/28/2015
• Est. Priority Date: 06/15/2012
• Status: Active Grant

• Current Assignee: Thomson Licensing (Vantiva SA)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Borsum, Malte, Kochale, Axel, Salvador Marcos, Jordi
• Granted Patent: US 9,576,403 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method and an apparatus for improving a main image by fusing the richer information contained in a secondary image are described. A 3D structure of objects contained in the secondary image is retrieved and a parallax-corrected version of the secondary image is generated using the 3D structure. For this purpose a camera pose for which a projection of the 3D structure of the objects contained in the secondary image best resembles the perspective in the main image is determined and the parallax-corrected version of the secondary image is synthesized based on the determined camera pose. The parallax-corrected version of the secondary image is then fused with the main image.

Claims

1-7. -7. (canceled)

8. A method for fusing a secondary image with a main image, the method comprising:

retrieving a 3D structure of objects contained in the secondary image;

generating a parallax-corrected version of the secondary image using the 3D structure by determining a camera pose for which a projection of the 3D structure of the objects contained in the secondary image best resembles the perspective in the main image and synthesizing the parallax-corrected version of the secondary image based on the determined camera pose; and

fusing the parallax-corrected version of the secondary image with the main image.

9. The method according to claim 8, wherein the 3D structure of the secondary image is retrieved from depth information or a light-field description provided for the secondary image.

10. The method according to claim 8, wherein the parallax-corrected version of the secondary image is synthesized by performing a photo-realistic warping of the secondary image.

11. The method according to claim 8, further comprising interpolating a boundary region between the main image and the fused parallax-corrected version of the secondary image.

12. The method according to claim 8, wherein the main image is a panoramic image and the secondary image is a more detailed image of an object in the panoramic image.

13. The method according to claim 8, wherein the main image and the secondary image are two images of a set of images.

14. An apparatus for fusing a secondary image with a main image, the apparatus comprising:

a 3D structure retrieval stage configured to retrieve a 3D structure of objects contained in the secondary image;

a parallax correction stage configured to generate a parallax-corrected version of the secondary image using the 3D structure by determining a camera pose for which a projection of the 3D structure of the objects contained in the secondary image best resembles the perspective in the main image and to synthesize the parallax-corrected version of the secondary image based on the determined camera pose; and

an image fusion stage configured to fuse the parallax-corrected version of the secondary image with the main image.

15. The apparatus according to claim 14, wherein the 3D structure retrieval stage is configured to retrieve the 3D structure of the secondary image from depth information or a light-field description provided for the secondary image.

16. The apparatus according to claim 14, wherein the parallax correction stage is configured to synthesize the parallax-corrected version of the secondary image by performing a photo-realistic warping of the secondary image.

17. The apparatus according to claim 14, wherein the image fusion stage is configured to interpolate a boundary region between the main image and the fused parallax-corrected version of the secondary image.

18. The apparatus according to claim 14, wherein the main image is a panoramic image and the secondary image is a more detailed image of an object in the panoramic image.

19. The apparatus according to claim 14, wherein the main image and the secondary image are two images of a set of images.