Virtual reality camera

• US RE45,785 E1
• Filed: 03/18/2013
• Issued: 10/27/2015
• Est. Priority Date: 09/26/1997
• Status: Expired due to Term

• Current Assignee: Tumbleweed Holdings LLC (Equitable IP Corporation)
• Original Assignee: Intellectual Ventures LLC
• Inventors: Chen, Eric Shenchang
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US13/846,801
• Time in Patent Office: 953 Days
• Field of Search: 348/218.1, 348/207.99, 348/36, 348/239
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 5/23238 Control of image capture or reproduction to achieve a very large field of view, e.g. panorama panoramic or widescreen photography G03B27/00
• CPC Class Codes: H04N 5/23293 Electronic viewfinders
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A method and apparatus for creating and rendering multiple-view images. A camera includes an image sensor to receive images, sampling logic to digitize the images and a processor programmed to combine the images based upon a spatial relationship between the images.

Claims

1. A hand-held camera comprising:

a camera housing;

a camera lens mounted on said housing;

image acquisition circuitry located within said camera housing for acquiring images of fields of view via said camera lens at various orientations of said camera housing;

at least one user input panel for receiving a user request to select a panoramic or non-panoramic image capture mode; and

image processing circuitry located within said camera housing;

responsive to the panoramic image capture mode selection, for at least partially combining each successively acquired image of a field of a view with previously acquired images of fields of view, on an image by image basis in real time, by determining spatial relationships between the images of fields of view, and by mapping the images of fields of view onto regions of a cylindrical surface, based on the spatial relationships.

2. The hand-held camera of claim 1 wherein said image processing circuitry determines spatial relationships between the images based on at least one feature in images that at least partially overlap.

3. The hand-held camera of claim 1 wherein said image processing circuitry determines spatial relationships between the images based on cross-correlations of images that at least partially overlap.

4. The hand-held camera of claim 1 wherein said image processing circuitry determines spatial relationships between the images based on the orientations of said camera housing during image acquisition.

5. The hand-held cameral of claim 4 further comprising a sensor for detecting the orientations of said camera housing.

6. The hand-held camera of claim 5 wherein said image acquisition circuitry uses orientation information from said sensor to automatically determine fields of view for which to acquire images thereof.

7. The hand-held camera of claim 1 wherein the camera is a video camera and wherein sampling logic digitizes the images at a predetermined rate.

8. A hand-held camera comprising:

a camera housing;

a camera lens mounted on said housing;

a display mounted on said camera housing;

image acquisition circuitry located within said cameral housing for acquiring images of fields of view via said camera lens at various orientations of said camera housing;

image processing circuitry located within said camera housing for at least partially combining each successively acquired image of a field of view with previously acquired images of fields of view, on an image by image basis in real time, by determining spatial relationships between the images of fields of view, and by mapping the images of fields of view onto regions of a cylindrical surface, based on spatial relationships;

at least one user input panel to select a panoramic or non-panoramic image view mode, and to receive a user request to display a spatial region of the cylindrical panoramic image on said display; and

view control circuitry, located within said camera housing and responsive to the panoramic view mode, to display a spatial region of the cylindrical panoramic image on said display, wherein said view control circuitry selects the spatial region of the cylindrical panoramic image based upon the user request.

9. The hand-held camera of claim 8 wherein said view control circuitry selects the spatial region of the cylindrical panoramic image to be displayed on said display based upon an orientation of said housing.

10. The hand-held camera of claim 9 further comprising a sensor for detecting the orientation of said camera housing.

11. The hand-held camera of claim 8 further comprising a sensor for detecting the orientation of said camera housing.

12. The hand-held camera of claim 8 wherein said user input panel receives user requests to pan about a panoramic image.

13. The hand-held camera of claim 12 wherein said user input panel comprises left, right, up and down buttons.

14. The hand-held camera of claim 12 further comprising a sensor for detecting the orientation of said camera housing.

15. The hand-held camera of claim 8 wherein said user input panel receives user requests to zoom in and out of a panoramic image.

16. The hand-held camera of claim 15 wherein said user input panel comprises zoom in and zoom out buttons.

17. The hand-held camera of claim 15 further comprising a sensor for detecting the orientation of said camera housing.

18. A method for providing cylindrical panoramic images comprising:

selecting a panoramic or non-panoramic image capture mode;

acquiring images of fields of view at various orientations of a camera; and

when the panoramic image capture mode is selected, at least partially combining each successively acquired image of a field of view with previously acquired images of fields of view, on an image by image basis in real time, comprising;

determining spatial relationships between the image of fields of view; and

mapping the images of fields of view onto regions of a cylindrical surface, based on the spatial relationships.

19. The method of claim 18 wherein said determining is based on at least one feature in images that at least partially overlap.

20. The method of claim 18 wherein said determining is based on cross-correlations of images that at least partially overlap.

21. The method of claim 18 wherein said determining is based on the orientations of the cameral during image acquisitions.

22. The method of claim 21 further comprising detecting the orientation of said camera housing.

23. The method of claim 22 further comprising automatically determining fields of view for which to acquire images thereof, based on detected orientation information.

24. A method for providing cylindrical panoramic images comprising:

acquiring images of fields of view at various orientations of a camera;

at least partially combining each successively acquired image of a fields of view with previously acquired images of fields of view, on an image by image basis in real time, comprising;

determining spatial relationships between the images of fields of view; and

mapping the images of fields of view onto regions of a cylindrical surface, based on the spatial relationships;

selecting a panoramic or non-panoramic image view mode;

when the panoramic image view mode is selected,

receiving a user request to display a spatial region of a cylindrical panoramic image; and

displaying the spatial region of the cylindrical panoramic image.

25. The method of claim 24 further comprising selecting the spatial region of the cylindrical panoramic image to be displayed based upon an orientation of the camera.

26. The method of claim 25 further comprising detecting the orientation of said camera housing.

27. A hand-held camera comprising:

a camera housing;

a camera lens mounted on said housing;

image acquisition circuitry located within said camera housing for acquiring images of fields of view via said camera lens at various orientations of said camera housing;

at least one user input panel for receiving a user request to select a panoramic or non-panoramic image capture mode; and

image processing circuitry located within said camera housing, responsive to the panoramic image capture mode selection, for at least partially combining each successively acquired image of a field of view with previously acquired images of fields of view, on an image by image basis in real time, by determining spatial relationships between the images of fields of view, and by mapping the images of fields of view onto regions of a spherical surface, based on the spatial relationships.

28. The hand-held camera of claim 27 herein the camera is a video camera and wherein sampling logic digitizes the images at a predetermined rate.

29. A hand-held camera comprising:

a careen housing;

a camera lens mounted on said housing;

image acquisition circuitry located within said camera housing for acquiring images of fields of view via said camera lens at various orientations of said camera housing;

at least one user input panel for receiving a user request to select a panoramic or non-panoramic image capture mode; and

image processing circuitry located within said camera housing, responsive to the panoramic image capture mode selection, for at least partially combining each successively acquired images of a field of view with previously acquired images of fields of view, on an image by image basis in real time, by mapping the images of fields of view onto regions of a cylindrical surface, based on spatial relationships between the images of fields of view.

30. The hand-held camera of claim 29 wherein the camera is a video camera and wherein sampling logic digitizes the images at a predetermined rate.

31. A hand-held camera comprising:

a camera housing;

a camera lens mounted on said housing;

image acquisition circuitry located within said camera housing for acquiring images of fields of view via said camera lens at various orientations of said camera housing;

at least one user input panel for receiving a user request to select a panoramic or non-panoramic image capture mode; and

image processing circuitry located within said camera housing, responsive to the panoramic image capture mode selection, for at least partially combining each successively acquired image of a field of view with previously acquired fields of view, on an image by image basis in real time, by napping the images of fields of view onto regions of a spherical surface, based on spatial relationships between the images of fields of view.

32. The hand-held camera of claim 31 wherein the camera is a video camera and wherein sampling logic digitizes the images at a predetermined rate.

33. A method for providing spherical panoramic images comprising:

selecting a panoramic or non-panoramic image capture mode;

acquiring images of fields of view at various orientations of a camera; and

when the panoramic image capture mode is selected, at least partially combining each successively acquired image of a field of view with previously acquired images of fields of view, on an image by image basis in real time, comprising;

determining spatial relationships between the images of fields of view; and

mapping the images of fields of view onto regions of a spherical surface, based on the spatial relationships.

34. A method for providing cylindrical panoramic images comprising:

selecting a panoramic or non-panoramic image capture mode;

acquiring images of fields of view at various orientations of a camera; and

when the panoramic image capture mode is selected, at least partially combining each successively acquired image of a field of view with previously acquired images of fields of view, on an image by image basis in real time, comprising mapping the images of fields of view onto regions of a cylindrical surface, based on spatial relationships between the images of fields of view.

35. A method for providing spherical panoramic images comprising:

selecting a panoramic or non-panoramic image capture mode;

acquiring images of fields of view at various orientations of a camera; and

when the panoramic image capture mode is selected, at least partially combining each successively acquired image of a field of view with previously acquired image of fields of view, on an image by image basis in real time, comprising mapping the images of fields of view onto regions of a spherical surface, based on spatial relationships between the images of fields of view.

36. A non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the steps of:

sensing selection of a panoramic image capture mode;

acquiring images at various orientations of a camera; and

responsive to the selection of the panoramic image capture mode, at least partially combining at least one successively acquired image with one or more previously acquired images, on an image-by-image basis, comprising;

determining spatial relationships between the images; and

mapping the images onto regions of a virtual three-dimensional surface, based on the spatial relationships.

37. A non-transitory computer readable medium of claim 36 further storing instructions that, when executed by the computing device, cause the computing device to perform the steps of sensing an orientation of the camera to determine when a next image in the panoramic image capture mode is to be acquired based at least in part on a camera orientation.

38. The non-transitory computer readable medium of claim 37, wherein the orientation of the camera includes at least one orientation selected from the group consisting of a pitch of the camera, a roll of the camera, and a yaw of the camera.

39. The non-transitory computer readable medium of claim 37 further storing instructions that, when executed by the computing device, further cause the computing device to perform the step of generating a signal to indicate that the next image is to be acquired.

40. The non-transitory computer readable medium of claim 37, wherein the sensing to determine when a next image is to be acquired is further based at least in part on an angle of view of the camera and a distance between the camera and a subject in successive images.

41. The non-transitory computer readable medium of claim 36 further storing instructions that, when executed by the computing device, cause the computing device to perform the steps of collecting image information for each acquired image, and associating the image information for each acquired image with that image, the image information including a spatial location of an acquired image at least relative to spatial locations of other acquired images.

56. The non-transitory computer readable medium of claim 36, wherein the three-dimensional surface is a cylinder.

57. The non-transitory computer readable medium of claim 36, wherein the spatial relationships comprise correlation between portions of the images.

42. A method for providing panoramic images performed by a camera having a processor, the method comprising:

selecting a panoramic or non-panoramic image capture mode;

acquiring images of fields of view at various orientations of the camera; and

if the panoramic image capture mode is selected, at least partially combining by the processor of the camera each successively acquired image of a field of view with previously acquired images of fields of view, on an image-by-image basis, comprising;

determining at least one spatial relationship between the images of fields of view; and

mapping the images of fields of view onto regions of a virtual three-dimensional surface based at least in part on the at least one spatial relationship.

43. The method of claim 42, wherein the virtual three-dimensional surface is a spherical surface.

44. The method of claim 42, wherein the virtual three-dimensional surface is a cylindrical surface.

45. The method of claim 42, wherein determining at least one spatial relationship comprises determining at least one feature in images that at least partially overlap.

46. The method of claim 42, wherein determining at least one spatial relationship comprises determining an orientation of the camera during each image acquisition.

47. The method according to claim 46, wherein the camera has a sensor capable of detecting the orientation of the camera based at least in part on a magnetic field of the earth.

58. The method of claim 42, wherein the at least one spatial relationship comprises correlation between portions of the images of fields of view.

48. A method performed by a camera having a processor, the method for providing panoramic images, comprising:

enabling a selection of a panoramic image capture mode;

acquiring images of fields of view at various orientations of the camera; and

at least partially combining, by the processor of the camera, each successively acquired image of a field of view with a previously acquired image of a field of view on an image-by-image basis in the panoramic image capture mode, comprising;

determining at least one spatial relationship between the images of fields of view; and

mapping the images of fields of view onto regions of a surface based at least in part on the at least one spatial relationship.

49. The method of claim 48, wherein determining at least one spatial relationship comprises determining based on at least one feature that at least partially overlaps in successively acquired images of fields of view.

50. The method of claim 48, wherein determining at least one spatial relationship comprises determining based at least in part on an orientation of the camera during image acquisition.

51. The method of claim 50, further comprising automatically determining fields of view for which to acquire images based at least in part on a determined orientation of the camera.

59. The method of claim 48, wherein the surface is a cylinder.

60. The method of claim 48, wherein the at least one spatial relationship comprises correlation between portions of the images of fields of view.

52. A non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the steps of:

selecting a panoramic or non-panoramic image capture mode;

acquiring images of fields of view at various orientations of a camera; and

in response to selection of the panoramic image capture mode, at least partially combining each successively acquired image of a field of view with previously acquired images of fields of view, on an image by image basis, comprising mapping the images of fields of view onto regions of a smooth surface, based on spatial relationships between the images of fields of view.

53. The non-transitory computer readable medium of claim 52, wherein the smooth surface is a spherical surface.

54. The non-transitory computer readable medium of claim 52, wherein the smooth surface is a cylindrical surface.

55. The non-transitory computer readable medium of claim 52, wherein the camera is a video camera and wherein sampling logic digitizes the images at a predetermined rate.

61. The non-transitory computer readable medium of claim 52, wherein the spatial relationships comprise correlation between portions of the images of fields of view.

62. A camera for providing panoramic images, the camera comprising:

a processor and a memory;

a mode component configured to enable a selection of a panoramic image capture mode;

an image acquisition component configured to acquire images of fields of view at various orientations of the camera; and

an image combining component configured to;

determine the camera is in the panoramic image capture mode; and

at least partially combine, on an image-by-image basis using the processor, two or more successively acquired images from the image acquisition component, the combining performed by;

determining at least one spatial relationship between the successively acquired images; and

mapping the successively acquired images onto regions of a surface based at least in part on the at least one spatial relationship.

63. The camera of claim 62, wherein determining at least one spatial relationship comprises determining based on at least one feature that at least partially overlaps in the successively acquired images.

64. The camera of claim 62, wherein determining at least one spatial relationship comprises determining based at least in part on an orientation of the camera during image acquisition.

65. The camera of claim 64, further comprising automatically determining fields of view for which to acquire images based at least in part on a determined orientation of the camera.

66. The camera of claim 62, wherein the surface is a cylinder.

67. The camera of claim 62, wherein the at least one spatial relationship comprises a correlation between portions of the successively acquired images.