Maintenance of panoramic camera orientation

• US 7,593,042 B2
• Filed: 12/31/2004
• Issued: 09/22/2009
• Est. Priority Date: 07/28/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cutler, Ross G.
• Primary Examiner(s): Chan; Jason
• Assistant Examiner(s): SEGURA, CYNTHIA
• Application Number:
  US11/027,163
• Publication Number:
  US 20060023075A1
• Time in Patent Office: 1,726 Days
• Field of Search: 348/218.1, 348/36, 359/725, 352/69
• US Class Current: 348/218.1
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Systems and methods for maintaining orientation of a non-cylindrical panorama is described wherein an initial magnetometer reading is obtained for a panoramic camera. If the panoramic camera is rotated from an initial position, a new magnetometer reading is obtained and a magnitude of rotation is determined. The magnitude of rotation is factored into an image stitching and warping process to create a panoramic image that maintains the initial orientation.

Claims

1. A 360-degree panoramic video camera system, comprising:

one or more video cameras configured to image a panoramic area;

a processor and memory configured to perform a process in real time, the process comprising;

determining orientations of the panoramic video camera system;

receiving and storing one of the orientations as an initial heading that identifies an orientation of the panoramic video camera system at an initial time;

receiving video images from the one or more cameras, respectively, and mapping the images to panoramic video images comprising 360-degree video images of the panoramic area;

determining a direction and magnitude of a rotation that occurs when the panoramic video camera system is rotated from the initial heading and supplying correction values to the mapping step; and

wherein the mapping incorporates the correction values into the mapping step so that the panoramic video images maintain a similar orientation before and after the rotation such that a point in the panoramic area that is portrayed in the panoramic video images will, if displayed on a display, stay in a substantially same location on the display despite the rotations of the panoramic video camera.

2. The 360-degree panoramic video camera system as recited in claim 1, wherein the determining orientations is performed by a magnetometer that is configured to detected an orientation of the panoramic video camera system with regard to a magnetic pole.

3. The 360-degree panoramic video camera system as recited in claim 1, the process further comprising identifying a shape of a table used to support the panoramic video camera system.

4. The 360-degree panoramic video camera system as recited in claim 3, wherein the identifying the shape of the table further comprises identifying a size of the table used to support the panoramic video camera system.

5. The 360-degree panoramic video camera system as recited in claim 1, wherein the process further comprises applying a non-cylindrical warping function that normalizes sizes of objects appearing in the panoramic video images according to distances of the objects from the panoramic video camera system.

6. The 360-degree panoramic video camera system as recited in claim 1, the process further comprising orienting the panoramic video camera system in a particular way with respect to a table that supports the panoramic video camera system.

7. A method, comprising:

receiving images from a video camera device that includes one or more individual video cameras that image a panoramic area;

mapping the video images to panoramic space to create 360-degree panoramic video images, the mapping comprises applying a same mapping function to each video image in a set of video images that when mapped by the mapping function form a corresponding panoramic image;

detecting a rotation of the video camera device and measuring the size of the rotation; and

adjusting the mapping process to compensate for the detected rotation so that a point in the panoramic area that is portrayed in the panoramic video images is rendered in the panoramic video images to approximately a same display location of the panoramic space before and after the rotation, wherein in the adjusting comprises changing the mapping function based on the size of the rotation.

8. The method as recited in claim 7, wherein the detecting step further comprises:

storing an initial heading that identifies a first orientation of the video camera device at a first time;

receiving a detected heading that identifies a second orientation of the video camera device at a second time, the second orientation being different from the first orientation; and

deriving a direction and magnitude of the rotation from a difference between the initial heading and the detected heading.

9. The method as recited in claim 8, further comprising receiving the initial heading and the detected heading from an orientation device that detects an orientation of the video camera device with respect to a magnetic pole.

10. The method as recited in claim 7, further comprising receiving data regarding a shape of a table on which the video camera device is situated.

11. The method as recited in claim 10, further comprising applying a non-cylindrical warping function to the panoramic video images to normalize objects according to the shape of the table on which the video camera device is situated.

12. The method as recited in claim 10, further comprising receiving data regarding a size of the table on which the video camera device is situated.

13. The method as recited in claim 7, further comprising identifying a camera device orientation that indicates a way in which the video camera device should be physically oriented.

14. One or more computer-readable media storing executable instructions that, when executed, by a processor would implement steps comprising:

receiving an initial heading that indicates an orientation of a video camera device;

receiving from the video camera device video images portraying a 360-degree panoramic scene;

mapping the video images received from the video camera device to form 360-degree panoramic video images of the 360-degree panoramic scene;

receiving a detected heading that indicates an orientation of the video camera device that is different from the orientation indicated by the initial heading; and

computing mapping correction values that are used in the mapping process that alter the mapping process so that the panoramic video images maintain a similar orientation before and after the occurrence of the rotation, wherein a point in the 360-degree panoramic scene that is portrayed in the panoramic video images will, if displayed on a display, stay in a substantially same location on the display despite the rotations of the panoramic video camera.

15. The one or more computer-readable media as recited in claim 14, further comprising additional executable instructions that, when executed, implement an additional step of applying a non-cylindrical warping function to the panoramic video images to normalize sizes of objects appearing in the panoramic video images according to a location of each object with respect to the video camera device.

16. The one or more computer-readable media as recited in claim 15, further comprising additional executable instructions that, when executed, implement an additional step of receiving table data regarding a shape of a table on which the video camera device is situated and using the table data in the non-cylindrical warping function.

17. The one or more computer-readable media as recited in claim 16, wherein the table data further comprises data regarding a size of the table on which the video camera device is situated.

18. The one or more computer-readable media as recited in claim 14, further comprising additional executable instructions that, when executed, implement an additional step of determining a direction and magnitude of the rotation of the video camera device.

19. The one or more computer-readable media as recited in claim 14, wherein the indication of the orientation of the video camera device further comprises an indication of the orientation of the video camera device with respect to a magnetic pole of the earth.

20. A panoramic video camera comprising:

one or more video sensors arranged to capture video images of a panoramic scene wherein their respective captured video images are processed in real time to form a stream of 360-degree panoramic video images, wherein the panoramic video camera is configured to detect rotations of the panoramic video camera, wherein the video camera uses the detected rotation changes to maintain an image orientation of 360-degree panoramic video images, wherein the 360-degree panoramic video images are rotated to compensate for the detected rotation changes such that a point in the panoramic scene that is portrayed in the panoramic video images will, if displayed on a display, stay in a substantially same location on the display despite the rotations of the panoramic video camera.

21. A panoramic video camera system according to claim 20, further comprising one or more physical sensors detecting the rotation changes.

22. A panoramic video camera system according to claim 20, wherein the panoramic video images comprise cylindrical panoramic video images.

23. A panoramic video camera system according to claim 20, wherein the video camera system has a rotation threshold value that is used to determine whether to maintain the image orientation for different detected rotations, such that rotations of a magnitude below the threshold do not have a corresponding maintenance of the image orientation and rotations of a magnitude above the threshold have a corresponding maintenance of the image orientation.