Stereoscopic (3D) panorama creation on handheld device

• US 10,080,006 B2
• Filed: 09/09/2010
• Issued: 09/18/2018
• Est. Priority Date: 12/11/2009
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Bigioi, Petronel, Susanu, George, Barcovschi, Igor, Stec, Piotr, Murray, Larry, Drimbarean, Alexandru, Corcoran, Peter
• Primary Examiner(s): Mung, On S
• Application Number:
  US12/879,003
• Publication Number:
  US 20110141227A1
• Time in Patent Office: 2,931 Days
• Field of Search: 348 36, 348 46, 348 51, 348 53, 348 56
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597

Abstract

A technique of generating a stereoscopic panorama image includes panning a portable camera device, and acquiring multiple image frames. Multiple at least partially overlapping image frames are acquired of portions of the scene. The method involves registering the image frames, including determining displacements of the imaging device between acquisitions of image frames. Multiple panorama images are generated including joining image frames of the scene according to spatial relationships and determining stereoscopic counterpart relationships between the multiple panorama images. The multiple panorama images are processed based on the stereoscopic counterpart relationships to form a stereoscopic panorama image.

Claims

1. A method for generating a stereoscopic panorama image, the method comprising:

while a portable imaging device is being panned across a scene, automatically acquiring multiple image frames of portions of said scene;

wherein the multiple image frames are at least partially overlapping;

wherein the multiple image frames are automatically acquired using an optic and imaging sensor of the portable imaging device;

extracting a first set of image pieces from a first subset of the multiple image frames;

constructing a first panoramic image from the first set of image pieces;

wherein each piece in the first set of image pieces is from a different frame than each other image piece in the first set of image pieces;

extracting a second set of image pieces from a second subset of the multiple image frames;

constructing a second panoramic image from the second set of image pieces;

wherein no two pieces in the second set of image pieces are from the same frame;

wherein the first subset and said second subset are distinct and have no image frames in common;

wherein each image piece in the first set of image pieces, from which the first panoramic image is constructed, has a corresponding displaced image piece in the second set of image pieces, from which the second panoramic image is constructed;

determining displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces;

wherein the determined displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces are constant;

forming a stereoscopic image using the first panoramic image and the second panoramic image; and

storing, transmitting or displaying said stereoscopic panorama image, or combinations thereof.

2. The method of claim 1, wherein determining displacements comprises measuring the displacements.

3. The method of claim 1, wherein determining displacements comprises determining displacements in at least two dimensions.

4. The method of claim 3, wherein the at least two dimensions comprise two dimensions orthogonal to a depth dimension.

5. The method of claim 4, wherein determining displacements comprises determining displacements in three dimensions including the depth dimension.

6. The method of claim 1, wherein determining displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces comprises:

pairing image frames having relative displacements within a first predetermined range configured to provide a selected stereoscopic effect;

combining paired image frames to form component stereoscopic images; and

retaining paired image frames having overlap with adjacent paired image frames within a second predetermined range.

7. The method of claim 6, wherein the first predetermined range comprises 5-7.5 cm.

8. The method of claim 1, wherein the partially overlapping comprises between 10% and 50% overlap.

9. The method of claim 1, further comprising interleaving joining of pairs of image frames with acquiring or generating, or both, of next image frames.

10. The method of claim 6, wherein the first predetermined range is not greater than 15% of a panning radius of the portable imaging device.

11. A portable camera-enabled device capable of in-camera generation of a panorama image, comprising:

a lens;

an image sensor;

a processor; and

a non-transitory processor readable medium having code embedded therein for programming the processor to perform a stereoscopic panorama image generation method that comprises;

while a portable imaging device is being panned across a scene, automatically acquiring multiple image frames of portions of said scene;

wherein the multiple image frames are at least partially overlapping;

wherein the multiple image frames are automatically acquired using an optic and imaging sensor of the portable imaging device;

extracting a first set of image pieces from a first subset of the multiple image frames;

constructing a first panoramic image from the first set of image pieces;

wherein each piece in the first set of image pieces is from a different frame than each other image piece in the first set of image pieces;

extracting a second set of image pieces from a second subset of the multiple image frames;

constructing a second panoramic image from the second set of image pieces;

wherein no two pieces in the second set of image pieces are from the same frame;

wherein the first subset and said second subset are distinct and have no image frames in common;

wherein each image piece in the first set of image pieces, from which the first panoramic image is constructed, has a corresponding displaced image piece in the second set of image pieces, from which the second panoramic image is constructed;

determining displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces;

wherein the determined displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces are constant;

forming a stereoscopic image using the first panoramic image and the second panoramic image; and

storing, transmitting or displaying a stereoscopic panorama image, or combinations thereof.

12. The device of claim 11, wherein determining displacements comprises measuring the displacements.

13. The device of claim 11, wherein determining displacements comprises determining displacements in at least two dimensions.

14. The device of claim 13, wherein the at least two dimensions comprise two dimensions orthogonal to a depth dimension.

15. The device of claim 14, wherein determining displacements comprises determining displacements in three dimensions including the depth dimension.

16. The device of claim 11, wherein determining the displacement between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces comprises:

pairing image frames having relative displacements within a first predetermined range configured to provide a selected stereoscopic effect;

combining paired image frames to form component stereoscopic images; and

retaining paired image frames having overlap with adjacent paired image frames within a second predetermined range.

17. The device of claim 16, wherein the first predetermined range comprises 5-7.5 cm.

18. The device of claim 11, wherein the partially overlapping comprises between 10% and 50% overlap.

19. The device of claim 11, wherein the non-transitory processor readable medium further comprises embedded code configured to program the processor to interleave joining of pairs of image frames with acquiring or generating, or both, of next frames.

20. The device of claim 16, wherein the first predetermined range is not greater than 15% of a panning radius of the portable camera-enabled device.

21. One or more non-transitory computer-readable storage media having code embedded therein for programming a processor to generate a stereoscopic panorama image using a portable imaging device, said code being configured to program the processor to:

during panning of the portable imaging device across a scene, automatically acquire multiple image frames of portions of said scene;

wherein the multiple image frames are at least partially overlapping;

wherein the multiple image frames are automatically acquired using an optic and imaging sensor of the portable imaging device;

extracting a first set of image pieces from a first subset of the multiple image frames;

constructing a first panoramic image from the first set of image pieces;

wherein each piece in the first set of image pieces is from a different frame than each other image piece in the first set of image pieces;

extracting a second set of image pieces from a second subset of the multiple image frames;

constructing a second panoramic image from the second set of image pieces;

wherein no two pieces in the second set of image pieces are from the same frame;

wherein the first subset and said second subset are distinct and have no image frames in common;

wherein each image piece in the first set of image pieces, from which the first panoramic image is constructed, has a corresponding displaced image piece in the second set of image pieces, from which the second panoramic image is constructed;

determining displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces;

wherein the determined displacements between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces are constant;

forming a stereoscopic image using the first panoramic image and the second panoramic image; and

storing, transmitting or displaying said stereoscopic panorama image, or combinations thereof.

22. The one or more non-transitory computer-readable storage media of claim 21, wherein determining displacements comprises measuring the displacements.

23. The one or more non-transitory computer-readable storage media of claim 21, wherein determining displacements comprises determining displacements in at least two dimensions.

24. The one or more non-transitory computer-readable storage media of claim 23, wherein the at least two dimensions comprise two dimensions orthogonal to a depth dimension.

25. The one or more non-transitory computer-readable storage media of claim 24, wherein determining displacements comprises determining displacements in three dimensions including the depth dimension.

26. The one or more non-transitory computer-readable storage media of claim 21, wherein determining displacement between each image piece in the first set of image pieces and the corresponding displaced image piece in the second set of image pieces comprises:

pairing image frames having relative displacements within a first predetermined range configured to provide a selected stereoscopic effect;

combining paired image frames to form component stereoscopic images; and

retaining paired image frames having overlap with adjacent paired image frames within a second predetermined range.

27. The one or more non-transitory computer-readable storage media of claim 26, wherein the first predetermined range comprises 5-7.5 cm.

28. The one or more non-transitory computer-readable storage media of claim 21, wherein the partially overlapping comprises between 10% and 50% overlap.

29. The one or more non-transitory computer-readable storage media of claim 21, wherein the processor further comprises embedded code configured to program the processor to interleave joining of pairs of image frames with acquiring or generating, or both, of next frames.

30. The one or more non-transitory computer-readable storage media of claim 26, wherein the first predetermined range is not greater than 15% of a panning radius of the portable imaging device.