Method and apparatus for displaying panoramas with streaming video

• US 6,356,297 B1
• Filed: 01/15/1998
• Issued: 03/12/2002
• Est. Priority Date: 01/15/1998
• Status: Expired due to Term

• Current Assignee: Activision Publishing Incorporated (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Cheng, Keh-Shin Fu, Kumar, Keeranoor G., Lipscomb, James Sargent, Menon, Jai Prakash, Willebeek-LeMair, Marc Hubert
• Primary Examiner(s): Britton, Howard
• Assistant Examiner(s): DIEP, NHON THANH
• Application Number:
  US09/007,729
• Time in Patent Office: 1,517 Days
• Field of Search: H04/N.700, H04/N 1.300, H04/N 1.500, 348/36, 348/39, 348/42, 348/47, 348/48, 345/423, 345/425, 345/302, 345/419
• US Class Current: 348/36
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic

Abstract

In this invention streaming video data is embedded within a panoramic image for display. The invention teaches a method and system for displaying an image of a scene of interest from a stored environment map of the scene and from video data of the scene. The method includes the steps of: (A) receiving and storing the video data of the scene; (B) selecting an orientation of the scene; (C) retrieving video data according to the selected orientation of the scene; (D) rendering the environment map according to the selected orientation of the scene to generate a first image for the selected orientation; (E) synchronously combining the retrieved video data and the first image to form a composite image for the selected orientation of the scene of interest; and (F) displaying the composite image. In one embodiment the method for displaying the image of the scene of interest further includes a step of continuously receiving video data packets that include image data and sound data. In this embodiment the sound data is continuously played while selected portions of the image data are displayed. In another embodiment, the method for displaying the image of the scene of interest further includes a step of layering the video data and the first image generated by rendering the environment map. In the present invention layering techniques include, for example, drawing order layering, z-buffer-hiding layering, and chroma-key layering.

Claims

1. A method for displaying an image of a scene of interest from a stored environment map of the scene and from video data of the scene, the method comprising the steps of:

receiving and storing the video data of the scene;

selecting an orientation of the scene;

retrieving video data according to the selected orientation of the scene;

rendering the environment map according to the selected orientation of the scene to generate a first image for the selected orientation;

synchronously combining the retrieved video data and the first image to form a composite image for the selected orientation of the scene of interest; and

displaying the composite image.

2. A method for displaying an image of a scene of interest as set forth in claim 1, wherein the step of receiving the video data further comprises a step of continuously receiving video data packets that include image data and sound data.

3. A method for displaying an image of a scene of interest as set forth in claim 1, wherein the received video data is stored in a video memory buffer and the step of synchronously combining further comprises the steps of:

retrieving the received video data from the video memory buffer; and

embedding the retrieved video data within the first image to form the composite image.

4. A method for displaying an image of a scene of interest as set forth in claim 3, further comprising a step of transforming the video data to a desired position and size by at least one of scaling, shearing, and stretching the video data.

5. A method for displaying an image of a scene of interest as set forth in claim 3, wherein the step of retrieving further comprises a step of evaluating a value of a pointer to a memory location and retrieving the contents of the memory location.

6. A method for displaying an image of a scene of interest as set forth in claim 3, wherein the step of embedding further comprises a step of layering the retrieved video data and the first image to form the composite image.

7. A method for displaying an image of a scene of interest as set forth in claim 6, wherein the step of layering is one of a step of drawing order layering, z-buffer-hiding layering, and chroma-key layering of one of the retrieved video data onto the first image and the first image onto the video data.

8. An image processing system for displaying an image of a scene of interest from a stored environment map of said scene and from video data of said scene comprising:

means for receiving and storing said video data of said scene;

means for selecting an orientation of said scene;

means for retrieving video data according to said selected orientation of said scene;

means for rendering said environment map according to said selected orientation of said scene to generate a first image for said selected orientation;

means for synchronously combining said retrieved video data and said first image to form a composite image for said selected orientation of said scene of interest; and

means for displaying said composite image.

9. An image processing system for displaying an image of a scene of interest as set forth in claim 8, wherein said means for receiving said video data further comprises means for continuously receiving video data packets that include image data and sound data.

10. An image processing system for displaying an image of a scene of interest as set forth in claim 8, wherein said means for receiving said video data further comprises a video memory buffer.

11. An image processing system for displaying an image of a scene of interest as set forth in claim 8, wherein said means for synchronously combining said retrieved video data and said first image further comprises:

means for retrieving said received video data from said video memory buffer; and

means for embedding said retrieved video data within said first image to form said composite image.

12. An image processing system for displaying an image of a scene of interest as set forth in claim 11, further comprising means for transforming said video data retrieved from said video memory buffer to a desired position and size by at least one of scaling, shearing, and stretching the video data prior to embedding said video data within said first image.

13. An image processing system for displaying an image of a scene of interest as set forth in claim 11, wherein said means for retrieving said video data from said video memory buffer further comprises means for evaluating a value of a pointer to a memory location and for retrieving said contents of said memory location.

14. An image processing system for displaying an image of a scene of interest as set forth in claim 11, wherein said means for embedding further comprises means for layering said retrieved video data and said first image to form said composite image.

15. An image processing system for displaying an image of a scene of interest as set forth in claim 14, wherein said means for layering further comprises means for chroma-key layering of one of said retrieved video data onto said first image and said first image onto said video data.

16. A method for displaying an image of a scene of interest from a stored environment map of the scene and from image data of the scene, the method comprising the steps of:

continuously receiving and storing video data packets that include the image data and sound data of the scene;

selecting an orientation of the scene;

retrieving image data according to the selected orientation of the scene;

rendering the environment map according to the selected orientation of the scene to generate a first image for the selected orientation;

synchronously combining the retrieved image data and the first image to form a composite image for the selected orientation of the scene of interest;

continuously playing sound data of the scene; and

displaying the composite image.

17. A method for displaying an image of a scene of interest as set forth in claim 16, wherein the step of continuously playing sound data further comprises a step of controlling the volume of the sound data being played.

18. A method for displaying an image of a scene of interest from a stored environment map of the scene and from video data of the scene, the method comprising the steps of:

receiving the video data of the scene and storing the video data in a video buffer;

selecting an orientation of the scene;

retrieving video data from the video buffer according to the selected orientation of the scene, wherein the video data remains separate from the environment map;

rendering the environment map according to the selected orientation of the scene to generate a first image for the selected orientation;

synchronously combining the retrieved video data and the first image to form a composite image for the selected orientation of the scene of interest; and

displaying the composite image.