Stereoscopic panorama image creating apparatus, stereoscopic panorama image creating method, stereoscopic panorama image reproducing apparatus, stereoscopic panorama image reproducing method, and recording medium

• US 8,687,041 B2
• Filed: 11/20/2012
• Issued: 04/01/2014
• Est. Priority Date: 12/24/2010
• Status: Expired due to Fees

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Wakabayashi, Satoru
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): RAHAMAN, MOHAMMED S
• Application Number:
  US13/682,292
• Publication Number:
  US 20130076856A1
• Time in Patent Office: 497 Days
• Field of Search: 348/36, 348/42
• US Class Current: 348/36
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G03B 37/02 with scanning movement of lens or cameras
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

In a stereoscopic panorama image creating method according to an embodiment of the presently disclosed subject matter, a plurality of stereoscopic images formed of left images and right images continuously shot with a stereoscopic imaging apparatus 1 being moved in one direction, the left images of the stereoscopic images are synthesized together and the right images thereof are synthesized together to create a stereoscopic panorama image, corresponding points of the stereoscopic images are detected, an image shift amount for setting a parallax of each corresponding point at a predetermined value is calculated, and information about the corresponding point and the image shift amount are recorded in association with the stereoscopic panorama image.

Claims

1. A stereoscopic panorama image creating apparatus, comprising:

a stereoscopic image obtaining unit configured to obtain a plurality of stereoscopic images formed of left images and right images shot by a stereoscopic imaging apparatus, the stereoscopic images continuously shot with the stereoscopic imaging apparatus being swung in a constant direction;

a stereoscopic panorama image creating unit configured to create a stereoscopic panorama image formed of left and right panorama images by synthesizing the left images together and synthesizing the right images together from among the plurality of stereoscopic images obtained by the stereoscopic image obtaining unit;

a corresponding point detecting unit configured to detect a plurality of corresponding points for parallax adjustment, the corresponding points having features matched between the left images and the right images of the plurality of stereoscopic images obtained by the stereoscopic image obtaining unit or between the left and right panorama images of the created stereoscopic panorama image;

an image shift amount calculating unit configured to calculate a plurality of image shift amounts for setting parallaxes of the detected plurality of corresponding points each at a predetermined parallax amount; and

a recording unit configured to record the created stereoscopic panorama image in a recording medium, the recording unit configured to record position information of the plurality of corresponding points for parallax adjustment on the stereoscopic panorama image and the calculated plurality of image shift amounts as being associated with each other in the recording medium as auxiliary information of the stereoscopic panorama image;

a reading unit configured to read a stereoscopic panorama image and auxiliary information of the stereoscopic panorama image from the recording medium;

a reproducing unit configured to cause the read stereoscopic panorama image to be magnified with a predetermined magnification, the reproducing unit configured to cause the magnified stereoscopic panorama image to be reproduced by scrolling on a stereoscopic display or reproduced by frame advance for each of a plurality of frames obtained by dividing the stereoscopic panorama image, automatically or with a manual instruction input; and

a parallax adjusting unit configured to select one image shift amount from among a plurality of image shift amounts recorded as the auxiliary information of the stereoscopic panorama image, the image shift amount corresponding to a stereoscopic image in a screen of the stereoscopic display, the parallax adjusting unit configured to perform a parallax adjustment of the stereoscopic image based on the selected image shift amount, when the stereoscopic panorama image is reproduced by scrolling or frame advance by the reproducing unit.

2. The stereoscopic panorama image creating apparatus according to claim 1, wherein

the corresponding point detecting unit detects a set of corresponding points for each set of the left image and the right image of each of the stereoscopic images obtained by the stereoscopic image obtaining unit, and

the image shift amount calculating unit calculates the image shift amounts each for setting a parallax between the detected corresponding points at the predetermined parallax amount.

3. The stereoscopic panorama image creating apparatus according to claim 2, wherein

the corresponding point detecting unit detects the set of corresponding points for each set of the left image and the right image of each of the stereoscopic images in a portion near a center of the stereoscopic image.

4. The stereoscopic panorama image creating apparatus according to claim 3, wherein

when necessary corresponding points are not detected by the corresponding point detecting unit in the portion near the center of the stereoscopic image, the image shift amount calculating unit sets image shift amounts set in advance as calculation values or calculates the image shift amounts by interpolation between adjacent image shift amounts.

5. The stereoscopic panorama image creating apparatus according to claim 1, wherein

the stereoscopic panorama image creating unit creates the stereoscopic panorama image by piecing slit images each in a strip shape together, the slit images being in a portion near a center from among the plurality of stereoscopic images obtained by the stereoscopic image obtaining unit.

6. The stereoscopic panorama image creating apparatus according to claim 1, further comprising

an object detecting unit configured to detect a necessary object included in the plurality of stereoscopic images obtained by the stereoscopic image obtaining unit or the created stereoscopic panorama image, wherein

the corresponding point detecting unit detects a corresponding point for each object detected by the object detecting unit.

7. The stereoscopic panorama image creating apparatus according to claim 1, further comprising

an interpolation computing unit configured to calculate continuous pixel shift amounts by interpolating pixel shift amounts between the corresponding points based on the plurality of image shift amounts calculated by the image shift amount calculating unit, wherein

the recording unit records the plurality of image shift amounts calculated by the image shift amount calculating unit and the pixel shift amounts calculated by the interpolation computing unit in the recording medium.

8. The stereoscopic panorama image creating apparatus according to claim 1, wherein

the predetermined parallax amount is zero.

9. The stereoscopic panorama image reproducing apparatus according to claim 1, wherein

the parallax adjusting unit selects the image shift amount at a center of a screen of the stereoscopic display or in a portion closest to the center of the screen, and performs a parallax adjustment of the stereoscopic image based on the selected image shift amount.

10. The stereoscopic panorama image reproducing apparatus according to claim 1, wherein

when the reproducing unit reproduces the stereoscopic panorama image by scrolling with a manual instruction input, the parallax adjusting unit selects one image shift amount corresponding to a stereoscopic image displayed in a screen of the stereoscopic display at a time of starting the scrolling with the manual instruction input by a time of stopping scrolling and, when the scrolling is stopped, selects one image shift amount corresponding to a stereoscopic image displayed in the screen of the stereoscopic displayed at the time of stopping scrolling, and performs a parallax adjustment of the stereoscopic images based on the selected image shift amounts.

11. The stereoscopic panorama image reproducing apparatus according to claim 1, wherein

the reproducing unit causes only one of the left image and the right image to be displayed on the stereoscopic display during scrolling by reproduction by scrolling with a manual instruction input, and causes the stereoscopic image to be displayed on the stereoscopic display when the scrolling is stopped, and

at a time of stopping scrolling, the parallax adjusting unit selects one image shift amount corresponding to the stereoscopic image displayed in a screen of the stereoscopic display at the time of stopping scrolling, and performs a parallax adjustment of the stereoscopic image based on the selected image shift amount.

12. The stereoscopic panorama image reproducing apparatus according to claim 1, wherein

the reproducing unit causes a whole of the stereoscopic panorama image to be displayed on the stereoscopic displayed, and then causes the stereoscopic panorama image to be magnified with a predetermined magnification and replayed by scrolling or frame advance.

13. A stereoscopic panorama image reproducing apparatus, comprising:

a stereoscopic panorama image obtaining unit configured to obtain a stereoscopic panorama image formed of left and right panorama images;

corresponding point detecting unit configured to detect, prior to reproduction of the obtained stereoscopic panorama image, a plurality of corresponding points for parallax adjustment in the stereoscopic panorama image, the corresponding points having features matched between the left and right panorama images;

an image shift amount calculating unit configured to calculate image shift amounts for setting parallaxes of the detected plurality of corresponding points each at a predetermined parallax amount;

a storage unit configured to store position information of the plurality of corresponding points for parallax adjustment on the stereoscopic panorama image and the calculated plurality of image shift amounts as being associated with each other;

a reproducing unit configured to cause the obtained stereoscopic panorama image to be magnified with a predetermined magnification, the reproducing, unit configured to cause the magnified stereoscopic panorama image to be reproduced by scrolling on a stereoscopic display or reproduced by frame advance for each of a plurality of frames obtained by dividing the stereoscopic panorama image, automatically or with a manual instruction input; and

a parallax adjusting unit configured to select one image shift amount from among a plurality of image shift amounts stored in the storage unit correspondingly to the stereoscopic panorama image, the image shift amount corresponding to a stereoscopic image in a screen of the stereoscopic display, the parallax adjusting unit configured to perform a parallax adjustment of the stereoscopic image based on the selected image shift amount, when the stereoscopic panorama image is reproduced by scrolling or frame advance by the reproducing unit.

14. The stereoscopic panorama image reproducing apparatus according to claim 13, wherein

the corresponding point detecting unit detects one set of corresponding points for each of slit images each in a strip shape obtained by dividing the panorama image.

15. The stereoscopic panorama image reproducing apparatus according to claim 14, wherein

when necessary corresponding points are not detected by the corresponding point detecting unit in a portion near a center of the stereoscopic image, the image shift amount calculating unit sets image shift amounts set in advance as calculation values or calculates the image shift amounts by interpolation between adjacent image shift amounts.

16. The stereoscopic panorama image reproducing apparatus according to claim 13, further comprising

an object detecting unit configured to detect a necessary object included in the obtained stereoscopic panorama image, wherein

the corresponding point detecting unit detects a corresponding point for each object detected by the object detecting unit.

17. The stereoscopic panorama image reproducing apparatus according to claim 13, further comprising

an interpolation computing unit configured to calculate continuous pixel shift amounts by interpolating pixel shift amounts between the corresponding points based on the plurality of image shift amounts calculated by the image shift amount calculating unit, wherein

the storage unit stores the plurality of image shift amounts calculated by the image shift amount calculating unit and the pixel shift amounts calculated by the interpolation computing unit.

18. The stereoscopic panorama image reproducing apparatus according to claim 13, wherein

the predetermined parallax amount is zero.

19. A stereoscopic panorama image creating method, comprising:

a stereoscopic image obtaining step of obtaining a plurality of stereoscopic images formed of left images and right images shot by a stereoscopic imaging apparatus, the stereoscopic images continuously shot with the stereoscopic imaging apparatus being swung in a constant direction;

a stereoscopic panorama image creating step of creating a stereoscopic panorama image formed of left and right panorama images by synthesizing the left images together and synthesizing the right images together from among the obtained plurality of stereoscopic images;

a corresponding point detecting step of detecting a plurality of corresponding points for parallax adjustment, the corresponding points having features matched between the left images and the right images of the obtained plurality of stereoscopic images or between the left and right panorama images of the created stereoscopic panorama image;

an image shift amount calculating step of calculating image shift amounts for setting parallaxes of the detected plurality of corresponding points each at a predetermined parallax amount; and

a recording step of recording the created stereoscopic panorama image in a recording medium and recording position information of the plurality of corresponding points for parallax adjustment on the stereoscopic panorama image and the calculated plurality of image shift amounts as being associated with each other in the recording medium as auxiliary information of the stereoscopic panorama image;

a reading step of reading a stereoscopic panorama image and auxiliary information of the stereoscopic panorama image from the recording medium;

a reproducing step of causing the read stereoscopic panorama image to be magnified with a predetermined magnification and causing the magnified stereoscopic panorama image to be reproduced by scrolling on a stereoscopic display or reproduced by frame advance for each of a plurality of frames obtained by dividing the stereoscopic panorama image, automatically or with a manual instruction input; and

a parallax adjusting step of selecting one image shift amount from among a plurality of image shift amounts recorded as the auxiliary information of the stereoscopic panorama image, the image shift amount corresponding to a stereoscopic image in a screen of the stereoscopic display, and performing a parallax adjustment of the stereoscopic image based on the selected image shift amount, when the stereoscopic panorama image is reproduced by scrolling or frame advance in the reproducing step.

20. The stereoscopic panorama image creating method according to claim 19, wherein

in the corresponding point detecting step, a set of corresponding points for each set of the left image and the right image of each of the obtained stereoscopic images is detected, and,

in the image shift amount calculating step, one set of corresponding points is specified from among the corresponding points detected for each of the stereoscopic images, and the image shift amounts each for setting a parallax between the corresponding points at the predetermined parallax amount are calculated.

21. The stereoscopic panorama image creating method according to claim 20, wherein

in the corresponding point detecting step, the set of corresponding points for each set of the left image and the right image of each of the stereoscopic images is detected in a portion near a center of the stereoscopic image.

22. The stereoscopic panorama image creating method according to claim 21, wherein

when necessary corresponding points are not detected in the corresponding point detecting step in the portion near the center of the stereoscopic image, image shift amounts set in advance are set as calculation values or the image shift amounts are calculated by interpolation between adjacent image shift amounts in the image shift amount calculating step.

23. The stereoscopic panorama image creating method according to claim 19, wherein

in the stereoscopic panorama image creating step, the stereoscopic panorama image is created by piecing slit images each in a strip shape together, the slit images being in a portion near a center from among the obtained plurality of stereoscopic images.

24. The stereoscopic panorama image creating method according to claim 19, further comprising

an object detecting step of detecting a necessary object included in the obtained plurality of stereoscopic images or the created stereoscopic panorama image, wherein

in the corresponding point detecting step, a corresponding point for each object detected by the object detecting step is detected.

25. The stereoscopic panorama image creating method according to claim 19, further comprising

an interpolation computing step of calculating continuous pixel shift amounts by interpolating pixel shift amounts between the corresponding points based on the plurality of image shift amounts calculated in the image shift amount calculating step, wherein

in the recording step, the plurality of image shift amounts calculated in the image shift amount calculating step and the pixel shift amounts calculated in the interpolation computing step are recorded in the recording medium.

26. The stereoscopic panorama image creating method according to claim 19, wherein

the predetermined parallax amount is zero.

27. The stereoscopic panorama image reproducing method according to claim 19, wherein

in the parallax adjusting step, the image shift amount at a center of a screen of the stereoscopic display or in a portion closest to the center of the screen is selected, and a parallax adjustment of the stereoscopic image is performed based on the selected image shift amount.

28. The stereoscopic panorama image reproducing method according to claim 19, wherein

when the stereoscopic panorama image is reproduced by scrolling with a manual instruction input in the reproducing step, one image shift amount corresponding to a stereoscopic image displayed in a screen of the stereoscopic display at a time of starting the scrolling with the manual instruction input is selected by a time of stopping scrolling and, when the scrolling is stopped, one image shift amount corresponding to a stereoscopic image displayed in the screen of the stereoscopic displayed at the time of stopping scrolling is selected, and a parallax adjustment of the stereoscopic images is performed based on the selected image shift amounts, in the parallax adjusting step.

29. The stereoscopic panorama image reproducing method according to claim 19, wherein

the reproducing step causes only one of the left image and the right image to be displayed on the stereoscopic display during scrolling by reproduction by scrolling with a manual instruction input, and causes the stereoscopic image to be displayed on the stereoscopic display when the scrolling is stopped, and

in the parallax adjusting step, at a time of stopping scrolling, one image shift amount corresponding to the stereoscopic image displayed in a screen of the stereoscopic display at the time of stopping scrolling is selected, and a parallax adjustment of the stereoscopic image is performed based on the selected image shift amount.

30. The stereoscopic panorama image reproducing method according to claim 19, wherein

the reproducing step causes a whole of the stereoscopic panorama image to be displayed on the stereoscopic displayed, and then causes the stereoscopic panorama image to be magnified with a predetermined magnification and replayed by scrolling or frame advance.

31. A stereoscopic panorama image reproducing method, comprising:

a stereoscopic panorama image obtaining step of obtaining a stereoscopic panorama image formed of left and right panorama images;

a corresponding point detecting step of detecting, prior to reproduction of the obtained stereoscopic panorama image, a plurality of corresponding points for parallax adjustment in the stereoscopic panorama image, the corresponding points having features matched between the left and right panorama images;

an image shift amount calculating step of calculating image shift amounts for setting parallaxes of the detected plurality of corresponding points each at a predetermined parallax amount;

a storing step of storing position information of the plurality of corresponding points for parallax adjustment on the stereoscopic panorama image and the calculated plurality of image shift amounts as being associated with each other;

a reproducing step of causing the obtained stereoscopic panorama image to be magnified with a predetermined magnification and causing the magnified stereoscopic panorama image to be reproduced by scrolling on a stereoscopic display or reproduced by frame advance for each of a plurality of frames obtained by dividing the stereoscopic panorama image, automatically or with a manual instruction input; and

a parallax adjusting step of selecting one image shift amount from among a plurality of image shift amounts stored in a storage unit correspondingly to the stereoscopic panorama image, the image shift amount corresponding to a stereoscopic image in a screen of the stereoscopic display, and performing a parallax adjustment of the stereoscopic image based on the selected image shift amount, when the stereoscopic panorama image is reproduced by scrolling or frame advance in the reproducing step.

32. The stereoscopic panorama image reproducing method according to claim 31, wherein

in the corresponding point detecting step, one set of corresponding points for each of slit images each in a strip shape obtained by dividing the panorama image is detected.

33. The stereoscopic panorama image reproducing method according to claim 32, wherein

when necessary corresponding points are not detected in the corresponding point detecting step in a portion near a center of the stereoscopic image, image shift amounts set in advance are set as calculation values, or the image shift amounts are calculated by interpolation between adjacent image shift amounts in the image shift amount calculating step.

34. The stereoscopic panorama image reproducing method according to claim 31, further comprising

an object detecting step of detecting a necessary object included in the obtained stereoscopic panorama image, wherein

in the corresponding point detecting step, a corresponding point for each object detected in the object detecting step is detected.

35. The stereoscopic panorama image reproducing method according to claim 31, further comprising

an interpolation computing step of calculating continuous pixel shift amounts by interpolating pixel shift amounts between the corresponding points based on the plurality of image shift amounts calculated in the image shift amount calculating step, wherein

in the storing step, the plurality of image shift amounts calculated in the image shift amount calculating step and the pixel shift amounts calculated in the interpolation computing step are stored.

36. The stereoscopic panorama image reproducing method according to claim 31, wherein

the predetermined parallax amount is zero.

37. A non-transitory computer-readable recording medium having instruction stored thereon causing, when the instruction is read by a processor and executed, the processor to achieve:

a stereoscopic image obtaining step of obtaining a plurality of stereoscopic images formed of left images and right images shot by a stereoscopic imaging apparatus, the stereoscopic images continuously shot with the stereoscopic imaging apparatus being swung in a constant direction;

a stereoscopic panorama image creating step of creating a stereoscopic panorama image formed of left and right panorama images by synthesizing the left images together and synthesizing the right images together from among the plurality of stereoscopic images obtained in the stereoscopic image obtaining step;

a corresponding point detecting step of detecting a plurality of corresponding points for parallax adjustment, the corresponding points having features matched between the left images and the right images of the plurality of stereoscopic images obtained in the stereoscopic image obtaining step or between the left and right panorama images of the created stereoscopic panorama image;

an image shift amount calculating step of calculating a plurality of image shift amounts for setting parallaxes of the detected plurality of corresponding points each at a predetermined parallax amount; and

a recording step of recording the created stereoscopic panorama image in a recording medium and recording position information of the plurality of corresponding points for parallax adjustment on the stereoscopic panorama image and the calculated plurality of image shift amounts as being associated with each other in the recording medium as auxiliary information of the stereoscopic panorama image;

a reading step of reading a stereoscopic panorama image and auxiliary information of the stereoscopic panorama image from the recording medium;

a reproducing step of causing the read stereoscopic panorama image to be magnified with a predetermined magnification and causing the magnified stereoscopic panorama image to be reproduced by scrolling on a stereoscopic display or reproduced by frame advance for each of a plurality of frames obtained by dividing the stereoscopic panorama image, automatically or with a manual instruction input; and

a parallax adjusting step of selecting one image shift amount from among a plurality of image shift amounts recorded as the auxiliary information of the stereoscopic panorama image, the image shift amount corresponding to a stereoscopic image in a screen of the stereoscopic display, and performing a parallax adjustment of the stereoscopic image based on the selected image shift amount, when the stereoscopic panorama image is reproduced by scrolling or frame advance in the reproducing step.

38. A non-transitory computer-readable recording medium having instruction stored thereon causing, when the instruction is read by a processor and executed, the processor to achieve:

a stereoscopic panorama image obtaining step of obtaining a stereoscopic panorama image formed of left and right panorama images;

a corresponding point detecting step of detecting, prior to reproduction of the obtained stereoscopic panorama image, a plurality of corresponding points for parallax adjustment in the stereoscopic panorama image, the corresponding points having features matched between the left and right panorama images;

an image shift amount calculating step of calculating image shift amounts for setting parallaxes of the detected plurality of corresponding points each at a predetermined parallax amount;

a storing step of storing position information of the plurality of corresponding points for parallax adjustment on the stereoscopic panorama image and the calculated plurality of image shift amounts as being associated with each other;

a reproducing step of causing the obtained stereoscopic panorama image to be magnified with a predetermined magnification and causing the magnified stereoscopic panorama image to be reproduced by scrolling on a stereoscopic display or reproduced by frame advance for each of a plurality of frames obtained by dividing the stereoscopic panorama image, automatically or with a manual instruction input; and

a parallax adjusting step of selecting one image shift amount from among a plurality of image shift amounts stored in the storing step correspondingly to the stereoscopic panorama image, the image shift amount corresponding to a stereoscopic image in a screen of the stereoscopic display, and performing parallax adjustment of the stereoscopic image based on the selected image shift amount, when the stereoscopic panorama image is reproduced by scrolling or frame advance in the reproducing step.