Augmenting images for panoramic display

• US 8,009,178 B2
• Filed: 06/29/2007
• Issued: 08/30/2011
• Est. Priority Date: 06/29/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Chen, Billy P.
• Primary Examiner(s): Wang; Andrew
• Assistant Examiner(s): Chu; David H
• Application Number:
  US11/771,434
• Publication Number:
  US 20090002394A1
• Time in Patent Office: 1,523 Days
• Field of Search: None
• US Class Current: 345/629
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

Methods and systems are provided methods and systems for augmenting image data (e.g., still image data or video image data) utilizing image context data to generate panoramic images. In accordance with embodiments hereof, a position and orientation of received image data is utilized to identify image context data (e.g., three-dimensional model data, two-dimensional image data, and/or 360° image data from another source) rendered based upon the same position or a nearby position relative to the image data and the image data is augmented utilizing the identified context data to create a panoramic image. The panoramic image may then be displayed (e.g., shown on a LCD/CRT screen or projected) to create a user experience that is more immersive than the original image data could create.

Claims

1. One or more computer-storage media having computer-executable instructions embodied thereon for performing a method for generating panoramic images, the method comprising:

receiving video image data having a known position and orientation from a first source, wherein the first source includes a user'"'"'s image capture device;

receiving image context data from a second source;

extrapolating image data in at least one direction utilizing received image context data to create a panoramic image by rendering a plurality of three-dimensional synthetic views of an area surrounding the capture point of the image data;

alpha-compositing the image data onto at least one of the plurality of synthetic views to create a composite image;

warping the composite image into spherical coordinates;

augmenting the video image data with the image context data; and

storing the generated panoramic image.

2. The one or more computer-storage media of claim 1, wherein receiving video image data having a known position and orientation comprises receiving video image data having coordinates associated therewith.

3. The one or more computer-storage media of claim 2, wherein receiving image context data comprises receiving three-dimensional model data having coordinates associated therewith, the coordinates associated with the received video image data and the coordinates associated with the three-dimensional model data sharing a common coordinate system.

4. The one or more computer-storage media of claim 1, wherein receiving image context data comprises receiving three-dimensional model data.

5. The one or more computer-storage media of claim 1, wherein the received video image data has at least one defined boundary, and wherein augmenting the video image data with the image context data comprises extrapolating the video image data beyond the at least one defined boundary in at least one direction.

6. The one or more computer-storage media of claim 1, wherein the method further comprises displaying the generated panoramic image.

7. The one or more computer-storage media of claim 6, wherein displaying the generated panoramic image comprises projecting the generated panoramic image.

8. A computer-implemented method for augmenting image data, the method comprising:

receiving, at a user'"'"'s computing device, image data having a known position and orientation, wherein the image data is received from an image capture device;

determining, using the user'"'"'s computing device, if image context data is available that includes the known position;

if it is determined that image context data is not available that includes the known position, receiving, at the user'"'"'s computing device, image context data having a position near the known position;

augmenting, using the user'"'"'s computing device, the image data with the image context data to generate augmented image data, wherein transitions between the image data and the image context data are feathered;

alpha-compositing the image data and warping the augmented image data into spherical coordinates; and

storing the augmented image data on the user'"'"'s computing device.

9. The method of claim 8, wherein receiving image data having a known position and orientation comprises receiving at least one of still image data and video image data having coordinates associated therewith.

10. The method of claim 9, wherein receiving image context data having a position near the known position comprises receiving three-dimensional model data having coordinates associated therewith, and wherein the coordinates associated with the received image data and the coordinates associated with the three-dimensional model data share a common coordinate system.

11. The method of claim 8, wherein receiving image context data having a position near the known position comprises receiving three-dimensional model data.

12. The method of claim 11, wherein receiving image context data having a position near the known position comprise receiving image context data having coordinates within a pre-determined proximity of the coordinates associated with the image data.

13. The method of claim 8, wherein the received image data includes at least one defined boundary, and wherein augmenting the image data with the image context data comprises extrapolating the image data beyond the at least one defined boundary in at least one direction.

14. The method of claim 8, further comprising displaying the augmented image data.

15. A computerized system for generating panoramic images, the system comprising:

an image receiving component configured for receiving image data having a known position and orientation, wherein the image data is received from a camera;

an image context receiving component that receives image context data having a known position and orientation;

a panoramic extrapolation component that extrapolates received image data in at least one direction utilizing received image context data to create a panoramic image, wherein the panoramic extrapolation component accomplishes the extrapolation by rendering a plurality of three-dimensional synthetic views of an area surrounding the capture point of the image data, alpha-compositing the image data onto at least one of the plurality of synthetic views to create a composite image, and warping the composite image into spherical coordinates; and

a storage component that stores the image context data and the panoramic image.

16. The computerized system of claim 15, further comprising a panoramic display component configured for transmitting the panoramic image to at least one display device.

17. The computerized system of claim 15, wherein the image receiving component is configured for receiving image data having image coordinates associated therewith, and wherein the image context receiving component is configured for receiving image context data having context coordinates associated therewith.

18. The computerized system of claim 17, further comprising a determining component configured for determining if image context data is available that includes context coordinates that are the same as the image coordinates.

19. The computerized system of claim 18, wherein the image context receiving component is further configured to receive image context data having context coordinates within a pre-determined proximity of the image coordinates.

20. The computerized system of claim 15, wherein the camera is configured for capturing video image data and wherein the digital image data includes video image data.