System and method for generating and displaying panoramic images and movies

• US 6,665,003 B1
• Filed: 09/16/1999
• Issued: 12/16/2003
• Est. Priority Date: 09/17/1998
• Status: Expired due to Term

• Current Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Original Assignee: Issum Research Development Company of The Hebrew University Of Jerusalem
• Inventors: Benezra, Moshe, Rosenschein, Robert S., Peleg, Shmuel
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US09/396,248
• Time in Patent Office: 1,552 Days
• Field of Search: 348/42, 348/47, 348/51, 348/53, 348/64, 348/39, 382/154, 382/284, 382/285, 382/294, 349/15, 349/62, 349/74, 349/194
• US Class Current: 348/36
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Systems are disclosed for generating left and right panoramic mosaic images for use in facilitating panoramic stereoscopic viewing of a scene, and for displaying stereoscopic panoramic images to a viewer. The left and right panoramic image generators generate the left and right panoramic mosaic images from a series of images recorded or otherwise generated corresponding to respective angular or other positions, each image having a respective left and right image portion, and mosaics portions of those images together to form the respective left and right panoramic images. The display system displays a stereoscopic panoramic image to a viewer by displaying left and right panoramic images such that each is viewed by a respective one of the viewer'"'"'s eyes.

Claims

1. A system for generating a stereoscopic panoramic mosaic image pair comprising:

A. a strip generator module configured to generate two series of image strips, all of said image strips in each series comprising strips of a series of images of a scene as would be recorded by a camera from a respective series of positions relative to the scene, the image strips of the respective series representing strips of the respective images displaced from one another by at least one selected displacement; and

B. a mosaic image generator module configured to mosaic the respective series of images strips together thereby to construct two panoramic mosaic images, the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair providing a stereoscopic image of the scene as recorded over the path.

2. A system as defined in claim 1 in which the series of positions define a path at least a portion of which corresponds to a curved arc.

3. A system as defined in claim 1 in which the series of positions define a path at least a portion of which corresponds to a straight line.

4. A system as defined in claim 1 in which the series of positions define a translation relative to the scene.

5. A system as defined in claim 1 in which the series of positions define a change in angular orientation relative to the scene.

6. A system as defined in claim 1 in which the image strips are parallel to an axis defined for the respective image.

7. A system as defined in claim 6 which the axis is horizontal.

8. A system as defined in claim 6 in which the axis is vertical.

9. A system as defined in claim 1 in which the strip generator module includes a camera configured to record at least one image strip.

10. A system as defined in claim 9 in which the camera is configured to record a plurality of image strips from both series.

11. A system as defined in claim 10 in which the strip generator module includes a plurality of cameras configured to record at least image strips from both series, the plurality of cameras being disposed at respective positions along the path.

12. A system as defined in claim 9 in which the strip generator module further includes

A. a reflector configured to reflect images of the scene to the camera for recording thereby of at least the image strips; and

B. a reflector position control module configured to position the reflector at least one position to facilitate definition of the path.

13. A system as defined in claim 12 in which the reflector has at least one flat portion.

14. A system as defined in claim 12 in which the reflector has at least one curved portion.

15. A system as defined in claim 12 in which the reflector position control module is configured to position the reflector at a plurality of positions to facilitate definition of the path.

16. A system as defined in claim 1 in which the strip generator module is configured to generate at least one image strip using a selected computer graphics methodology.

17. A system as defined in claim 16 in which the strip generator module is configured to generate a plurality of image strips, at least one of the plurality of image strips being generated using the selected computer graphics methodology.

18. A system as defined in claim 1 in which the strip generator module is configured to generate the two series of image strips such that the displacement varies as between at least two images.

19. A system as defined in claim 18 in which the strip generator module includes

A. a camera configured to record a plurality of images;

and

B. a strip selector module configured to select at least two image strips from respective ones of the images, one of the selected image strips from each of the images being associated with one of the respective series, the strip selector module being configured to control the locations in the respective images from which the image strips are selected thereby to provide the respective displacements.

20. A system as defined in claim 19 in which the strip selector module is configured to control the locations in the respective images from which the image strips are selected by controlling the separations of the image strips in the respective images.

21. A system as defined in claim 18 in which the strip generator module is configured to generate at least some of said image strips using a selected computer graphics methodology.

22. A system as defined in claim 1 in which:

A. the strip generator module is further configured to generate a third series of image strips, all of said image strips in said third series comprising strips of a series of images of a scene as would be recorded by a camera, the image strips of the third series being displaced from the image strips of the other two series by another selected displacement; and

B. the mosaic image generator module is further configured to mosaic the third series of images strips together thereby to construct a third panoramic mosaic image, a selected two of the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair.

23. A system as defined in claim 1 in which:

A. the strip generator module is further configured to generate camera position information representing positions of the camera associated with the respective image strips; and

B. the mosaic image generator module is further configured to use the camera position information in constructing the panoramic mosaic images.

24. A system as defined in claim 1 in which the mosaic image generator module is configured to generate at least one of the panoramic mosaic images by measuring image motion between successive image strips using a selected computer vision technique.

25. A system for generating a stereoscopic panoramic mosaic image pair comprising:

A. a strip generator module configured to generate two series of image strips, all of said image strips in each series comprising strips of a series of images of a scene as would be recorded by a camera from a respective series of positions relative to the scene, the image strips of the respective series representing strips of the respective images displaced from one another by at least two selected displacements;

B. a mosaic image generator module configured to mosaic the respective series of images strips together thereby to construct two panoramic mosaic images, the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair providing a stereoscopic image of the scene.

26. A system as defined in claim 25 in which the strip generator module includes

A. a camera configured to record a plurality of images; and

B. a strip selector module configured to select at least two image strips from respective ones of the images, one of the selected image strips from each of the images being associated with one of the respective series, the strip selector module being configured to control the locations in the respective images from which the image strips are selected thereby to provide the respective displacements.

27. A system as defined in claim 25 in which the strip generator module is configured to generate at least some of said image strips using a selected computer graphics methodology.

28. A system as defined in claim 25 in which:

A. the strip generator module is further configured to generate a third series of image strips, all of said image strips in said third series comprising strips of a series of images of a scene as would be recorded by a camera, the image strips of the third series being displaced from the image strips of the other two series by another selected displacement; and

B. the mosaic image generator module is further configured to mosaic the third series of images strips together thereby to construct a third panoramic mosaic image, a selected two of the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair.

29. A system for displaying a stereo panoramic image to a viewer, said system comprising:

A. a panoramic screen;

B. a plurality of projectors each disposed to project a respective portion of one of a plurality panoramic images in overlapping fashion on the screen, each panoramic image being for viewing by a respective one of a viewer'"'"'s eyes; and

C. a viewing arrangement for facilitating transmission of a respective one of said images to each of the viewer'"'"'s eyes thereby to facilitate stereoscopic viewing of the panoramic image defined by the panoramic images.

30. A system as defined in claim 29 in which:

A. the respective panoramic images are polarized with orthogonal polarizations; and

B. the viewing arrangement comprises a pair of lenses polarized with corresponding orthogonal polarizations.

31. A system as defined in claim 29 in which:

A. the respective panoramic images are of different colors; and

B. the viewing arrangement comprises a pair of lenses having the corresponding colors.

32. A system for displaying a stereo panoramic image to a viewer, said system comprising:

A. a panoramic screen;

B. a plurality of projectors each configured to project a respective portion of a panoramic image on the screen, the panoramic image comprising alternating portions of panoramic images each to be alternatingly viewed by respective ones of a viewers eyes; and

C. a viewing arrangement for facilitating alternating transmission of the respective alternating portions to respective ones of the viewer'"'"'s eyes thereby to facilitate stereoscopic viewing of the panoramic image.

33. A system as defined in claim 32 in which the viewing arrangement comprises a pair of lenses, each lens being associated with a shutter, the shutters being configured to alternatingly open and close in synchrony with the display of the respective alternating portions.

34. A method of generating a stereoscopic panoramic mosaic image pair comprising the steps of:

A. a strip generation step of generating two series of image strips, all of said image strips in each series comprising strips of a series of images of a scene as would be recorded by a camera from a respective series of positions relative to the scene, the image strips of the respective series representing strips of the respective images displaced from one another by at least one selected displacement; and

B. a mosaic image generation step of mosaicing the respective series of images strips together thereby to construct two panoramic mosaic images, the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair providing a stereoscopic image of the scene as recorded over the path.

35. A method as defined in claim 34 in which the series of positions define a path at least a portion of which corresponds to a circular arc.

36. A method as defined in claim 34 in which the series of positions define a path at least a portion of which corresponds to a straight line.

37. A method as defined in claim 34 in which the series of positions define a translation relative to the scene.

38. A method as defined in claim 34 in which the series of positions define a change in angular orientation relative to the scene.

39. A method as defined in claim 34 in which the image strips are parallel to an axis defined for the respective image.

40. A method as defined in claim 39 in which the axis is horizontal.

41. A system as defined in claim 39 in which the axis is vertical.

42. A method as defined in claim 34 in which the strip generation step includes the step of using a camera configured to record at least one image strip.

43. A method as defined in claim 42 in which the strip generation step includes the step of using a camera configured to record a plurality of image strips from both series.

44. A method as defined in claim 43 in which the strip generation step includes the step of using a plurality of cameras configured to record at least image strips from both series, the plurality of cameras being disposed at respective positions along the path.

45. A method as defined in claim 42 in which the strip generation step further includes the steps of

A. using a reflector configured to reflect images of the scene to the camera for recording thereby; and

B. positioning the reflector at least one position to facilitate definition of the path.

46. A method as defined in claim 45 in which the reflector has at least one flat portion.

47. A method as defined in claim 45 in which the reflector has at least one flat portion.

48. A method as defined in claim 45 in which the reflector positioning step includes the step of positioning the reflector at a plurality of positions to facilitate definition of the path.

49. A method as defined in claim 34 in which the strip generation step includes the step of generating at least one image strip using a selected computer graphics methodology.

50. A method as defined in claim 49 in which the strip generation step includes the step of generating a plurality of image strips;

at least one of the plurality of image strips being generated using the selected computer graphics methodology.

51. A method as defined in claim 49 in which the strip generation step includes the step of generating the two series of image strips such that the displacement varies as between at least two images.

52. A method as defined in claim 51 in which the strip generation step includes the steps of

A. using a camera to record a plurality of images;

and

B. a strip selection step of selecting at least two image strips from respective ones of the images, one of the selected image strips from each of the images being associated with one of the respective series, the strip selection step including the step of controlling the locations in the respective images from which the image strips are selected thereby to provide the respective displacements.

53. A method as defined in claim 52 in which the strip selection step includes the step of controlling the locations in the respective images from which the image strips are selected by controlling the separations of the image strips in the respective images.

54. A method as defined in claim 51 in which the strip generation step includes the step of generating at least some of said image strips using a selected computer graphics methodology.

55. A method as defined in claim 34 in which:

A. the strip generation step includes the step of generating a third series of image strips, all of said image strips in said third series comprising strips of a series of images of a scene as would be recorded by a camera, the image strips of the third series being displaced from the image strips of the other two series by another selected displacement; and

B. the mosaic image generation step includes the step of mosaicing the third series of images strips together thereby to construct a third panoramic mosaic image, a selected two of the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair.

56. A method as defined in claim 34 in which:

A. the strip generation step includes the step of generating camera position information representing positions of the camera associated with the respective image strips; and

B. the mosaic image generation step includes the step of using the camera position information in constructing the panoramic mosaic images.

57. A method as defined in claim 34 in which the mosaic image generation step includes the step of generate at least one of the panoramic mosaic images by measuring image motion between successive image strips using a selected computer vision technique.

59. A method as defined in claim 38 in which the strip generation step includes the steps of

A. using a camera to record a plurality of images;

and

B. a strip selection step of selecting at least two image strips from respective ones of the images, one of the selected image strips from each of the images being associated with one of the respective series, the strip selection step including the step of controlling the locations in the respective images from which the image strips are selected thereby to provide the respective displacements.

60. A method as defined in claim 59 in which the strip selection step includes the step of controlling the locations in the respective images from which the image strips are selected by controlling the separations of the image strips in the respective images.

61. A method as defined in claim 38 in which the strip generation step includes the step of generating at least some of said image strips using a selected computer graphics methodology.

62. A method as defined in claim 38 in which:

A. the strip generation step includes the step of generating a third series of image strips, a third series of image strips, all of said image strips in said third series comprising strips of a series of images of a scene as would be recorded by a camera, the image strips of the third series being displaced from the image strips of the other two series by another selected displacement; and

B. the mosaic image generation step includes step of mosaicing the third series of images strips together thereby to construct a third panoramic mosaic image, a selected two of the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair.

58. A method for generating a stereoscopic panoramic mosaic image pair comprising:

A. a strip generation step of generating two series of image strips, all of said image strips in each series comprising strips of a series of images of a scene as would be recorded by a camera from a respective series of positions relative to the scene, the image strips of the respective series representing strips of the respective images displaced from one another by at least two selected displacements; and

B. a mosaic image generation step of mosaicing the respective series of images strips together thereby to construct two panoramic mosaic images, the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair providing a stereoscopic image of the scene.

63. A method as defined in claim 58 in which:

A. the strip generation step includes the step of generating camera position information representing positions of the camera associated with the respective image strips; and

B. the mosaic image generation step includes the step of using the camera position information in constructing the panoramic mosaic images.

64. A method as defined in claim 58 in which the mosaic image generation step includes the step of generate at least one of the panoramic mosaic images by measuring image motion between successive image strips using a selected computer vision technique.

65. A method of displaying a stereo panoramic image to a viewer on a panoramic screen, said method comprising the steps of:

A. using a plurality of projectors each disposed to project a respective portion of one of a plurality panoramic images in overlapping fashion on the screen; and

C. using a viewing arrangement to facilitate transmission of a respective one of said images to each of the viewer'"'"'s eyes thereby to facilitate stereoscopic viewing of the panoramic image defined by the respective panoramic images.

66. A method as defined in claim 65 in which:

A. the respective panoramic images are polarized with orthogonal polarizations; and

B. the viewing arrangement using step includes the step of using a viewing arrangement that comprises a pair of lenses polarized with corresponding orthogonal polarizations.

67. A method as defined in claim 65 in which:

A. the respective panoramic images are of different colors; and

B. the viewing arrangement using step includes the step of using a viewing arrangement that comprises a pair of lenses having the corresponding colors.

68. A method displaying a stereo panoramic image to a viewer on a panoramic screen, said method comprising the steps of:

A. using a plurality of projectors each configured to project a respective portion of a panoramic image on the screen, the panoramic image comprising alternating portions of panoramic images each to be alternatingly viewed by respective ones of a viewers eyes; and

C. using a viewing arrangement for facilitating alternating transmission of the respective alternating portions to respective ones of the viewer'"'"'s eyes thereby to facilitate stereoscopic viewing of the panoramic image.

69. A method as defined in claim 68 in which the viewing arrangement comprises a pair of lenses, each lens being associated with a shutter, the method including the step of controlling the shutters to alternatingly open and close in synchrony with the display of the respective alternating portions.

70. A computer program product for use in connection with a computer to provide a system for generating a stereoscopic panoramic mosaic image pair, the computer program product comprising a computer-readable medium having encoded thereon:

A. a strip generator module configured to enable the computer to generate two series of image strips, all of said image strips in each series comprising strips of a series of images of a scene as would be recorded by a camera from a respective series of positions relative to the scene, the image strips of the respective series representing strips of the respective images displaced from one another by at least one selected displacement; and

B. a mosaic image generator module configured to enable the computer to mosaic the respective series of images strips together thereby to construct two panoramic mosaic images, the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair providing a stereoscopic image of the scene as recorded over the path.

71. A computer program product as defined in claim 70 in which the series of positions define a path at least a portion of which corresponds to a circular arc.

72. A computer program product as defined in claim 70 in which the series of positions define a path at least a portion of which corresponds to a straight line.

73. A computer program product as defined in claim 70 in which the series of positions define a translation relative to the scene.

74. A computer program product as defined in claim 70 in which the series of positions define a change in angular orientation relative to the scene.

75. A computer program product as defined in claim 70 in which the image strips are parallel to an axis defined for the respective image.

76. A computer program product as defined in claim 75 in which the axis is horizontal.

77. A computer program product as defined in claim 75 in which the axis is vertical.

78. A computer program product as defined in claim 70, the system further including a camera configured to record an image including at least one image strip, the strip generator module being further configured to enable the computer to generate the at least one image strip from the image.

79. A computer program product as defined in claim 78 in which the camera is configured to record a plurality of images each including at least one image strip from both series, the strip generator module being further configured to enable the computer to generate the respective image strops from the respective images.

80. A computer program product as defined in claim 79 in which system includes a plurality of cameras configured to record at least image strips from both series, the plurality of cameras being disposed at respective positions along the path.

81. A computer program product as defined in claim 78 in which

A. the system further includes a reflector configured to reflect images of the scene to the camera for recording thereby; and

B. the strip generator module further includes a reflector position control module configured to enable the computer to enable the reflector to be positioned at at least one position to facilitate definition of the path.

82. A computer program product as defined in claim 81 in which the reflector has at least one flat portion.

83. A computer program product as defined in claim 81 in which the reflector has at least one flat portion.

84. A computer program product as defined in claim 81 in which the reflector position control module is configured to enable the computer to enable the reflector to be positioned at a plurality of positions to facilitate definition of the path.

85. A computer program product as defined in claim 70 in which the strip generator module is configured to enable the computer to generate at least one image strip using a selected computer graphics methodology.

86. A computer program product as defined in claim 85 in which the strip generator module is configured to enable the computer to generate a plurality of image strips, at least one of the plurality of image strips being generated using the selected computer graphics methodology.

87. A computer program product as defined in claim 70 in which the strip generator module is configured to enable the computer to generate generate the two series of image strips such that the displacement varies as between at least two images.

88. A computer program product as defined in claim 87 in which:

A. the system includes a camera configured to record a plurality of images; and

B. the strip generator module includes a strip selector module configured to enable the computer to select at least two image strips from respective ones of the images, one of the selected image strips from each of the images being associated with one of the respective series, the strip selector module being configured to enable the computer to control the locations in the respective images from which the image strips are selected thereby to provide the respective displacements.

89. A computer program product as defined in claim 88 in which the strip selector module is configured to enable the computer to control the locations in the respective images from which the image strips are selected by controlling the separations of the image strips in the respective images.

90. A computer program product as defined in claim 87 in which the strip generator module is configured to enable the computer to generate at least some of said image strips using a selected computer graphics methodology.

91. A computer program product as defined in claim 70 in which:

A. the strip generator module is further configured to enable the computer to generate a third series of image strips, all of said image strips in said third series comprising strips of a series of images of a scene as would be recorded by a camera, the image strips of the third series being displaced from the image strips of the other two series by another selected displacement; and

B. the mosaic image generator module is further configured to enable the computer to mosaic the third series of images strips together thereby to construct a third panoramic mosaic image, a selected two of the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair.

92. A computer program product as defined in claim 70 in which:

A. the strip generator module is further configured to enable the computer to generate camera position information representing positions of the camera associated with the respective image strips; and

B. the mosaic image generator module is further configured to enable the computer to use the camera position information in constructing the panoramic mosaic images.

93. A computer program product as defined in claim 70 in which the mosaic image generator module is configured to enable the computer to generate at least one of the panoramic mosaic images by measuring image motion between successive image strips using a selected computer vision technique.

94. A computer program product for use in connection with a computer to provide a system for generating a stereoscopic panoramic mosaic image pair, the computer program product comprising a computer readable medium having encoded thereon:

A. a strip generator module configured to enable the computer to generate two series of image strips, all of said image strips in each series comprising strips of a series of images of a scene as would be recorded by a camera from a respective series of positions relative to the scene, the image strips of the respective series representing strips of the respective images displaced from one another by at least two selected displacements;

B. a mosaic image generator module configured to enable the computer to mosaic the respective series of images strips together thereby to construct two panoramic mosaic images, the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair providing a stereoscopic image of the scene.

95. A computer program product as defined in claim 94 in which

A. the system includes a camera configured to record a plurality of images; and

B. the strip generator module includes a strip selector module configured to enable the computer to select at least two image strips from respective ones of the images, one of the selected image strips from each of the images being associated with one of the respective series, the strip selector module being configured to control the locations in the respective images from which the image strips are selected thereby to provide the respective sets of angles.

96. A computer program product as defined in claim 95 in which the strip selector module is configured to enable the computer to control the locations in the respective images from which the image strips are selected by controlling the separations of the image strips in the respective images.

97. A computer program product as defined in claim 94 in which the strip generator module is configured to enable the computer to generate at least some of said image strips using a selected computer graphics methodology.

98. A computer program product as defined in claim 94 in which:

A. the strip generator module is further configured to enable the computer to generate a third series of image strips, all of said image strips in said third series comprising strips of a series of images of a scene as would be recorded by a camera, the image strips of the third series being displaced from the image strips of the other two series by another selected displacement; and

B. the mosaic image generator module is further configured to enable the computer to mosaic the third series of images strips together thereby to construct a third panoramic mosaic image, a selected two of the panoramic mosaic images comprising the stereoscopic panoramic mosaic image pair.

99. A computer program product as defined in claim 94 in which:

A. the strip generator module is further configured to enable the computer to generate camera position information representing positions of the camera associated with the respective image strips; and

B. the mosaic image generator module is further configured to enable the computer to use the camera position information in constructing the panoramic mosaic images.

100. A computer program product as defined in claim 94 in which the mosaic image generator module is configured to enable the computer to generate at least one of the panoramic mosaic images by measuring image motion between successive image strips using a selected computer vision technique.