Process for displaying and navigating panoramic video, and method and user interface for streaming panoramic video and images between a server and browser-based client application

• US 8,074,241 B2
• Filed: 03/12/2008
• Issued: 12/06/2011
• Est. Priority Date: 03/30/2007
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Thrun, Sebastian, Arfidsson, Joakim, Dahlkamp, Hendrik, Lookingbill, Andrew
• Primary Examiner(s): Koenig, Andrew
• Assistant Examiner(s): Castro, Alfonso
• Application Number:
  US12/047,160
• Publication Number:
  US 20080244648A1
• Time in Patent Office: 1,364 Days
• Field of Search: 725/93, 725/116, 725/141, 715/757
• US Class Current: 725/38
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: H04L 65/612 for unicast

Abstract

A method for providing panoramic videos and images to a user using a server-client architecture while minimizing the wait time necessary before still images are available for viewing or videos begin playing. A series of location-referenced panoramic images are separated into one-dimensional tracks. Intuitive user controls are provided which allow the user to start and stop video playback, step through the panoramas in a track one at a time, and change the viewing orientation within the panorama. A video will start playing as soon as the video files for the preferred projected cube faces have been downloaded. This delay is reduced by storing the videos as keyframe distance files for opposing directions for each cube face and further reduced by encoding videos with different starting points so that they are staggered by a portion of the keyframe distance.

Claims

1. A method for organizing and delivering compressed video for streaming panoramic video to a user, the panoramic video obtained by projecting two dimensional video onto a three dimensional proxy and developing a two dimensional video of a desired view, the method comprising the steps of:

for a given location, storing a series of sequential, two dimensional, compressed video segments along a movement track in first and second directions for each of a plurality of viewable surfaces of the three dimensional proxy as separate video segment files, each of said video segment files having a length of approximately one keyframe distance, each keyframe distance including a plurality of frames, with every keyframe starting a different video segment file; and

retrieving the video segment files appropriate for providing the user a streamed video in a desired direction for projection.

2. The method of claim 1, wherein each stored video segment file in each direction overlaps at least one other stored video segment file and starts a portion of a keyframe distance from the overlapping stored video segment files.

3. The method of claim 2, wherein a user may select a second location and direction after being at a first location, the method further comprising the step of:

retrieving the video segment files closest to the second location and of the correct direction, said closest video segment file being selected based on both the keyframe distance and the portion of the keyframe distance of the second location.

4. The method of claim 2, wherein the three dimensional proxy is a cube, the viewable surfaces are the cube faces and the directions are forward and backward.

5. The method of claim 2, wherein the portion of the keyframe distance is one-third.

6. The method of claim 2, wherein each video segment is stored in a plurality of combinations of resolutions and bit rates.

7. The method of claim 2, further comprising the steps of:

storing compressed still images having a resolution higher than the retrieved video segment file corresponding to selected locations; and

retrieving said compressed still images related to a particular location if movement has stopped at the selected location.

8. The method of claim 2, further comprising the steps of:

projecting the retrieved video segments onto the three dimensional proxy and developing a two dimensional video segment of a desired view;

compressing the developed two dimensional video segment; and

providing the compressed video segment to the user.

9. The method of claim 1, wherein a user may select a second location and direction after being at a first location, the method further comprising the step of:

retrieving the video segment files closest to the second location and of the correct direction, said closest video segment file being selected based on the keyframe distance.

10. The method of claim 1, wherein the three dimensional proxy is a cube, the viewable surfaces are the cube faces and the directions are forward and backward.

11. The method of claim 1, wherein each video segment is stored in a plurality of combinations of resolutions and bit rates.

12. The method of claim 1, further comprising the steps of:

storing compressed still images having a resolution higher than the retrieved video segment file corresponding to selected locations; and

retrieving said compressed still images related to a particular location if movement has stopped at the selected location.

13. The method of claim 1, further comprising the steps of:

projecting the retrieved video segments onto the three dimensional proxy and developing a two dimensional video segment of a desired view;

compressing the developed two dimensional video segment; and

providing the compressed video segment to the user.

14. A non-transitory machine-readable medium having machine-executable instructions for performing a method for streaming panoramic video to a user, the method comprising the steps of:

retrieving video segments appropriate for providing the user a streamed video in a desired direction, the video segments stored as a series of sequential, two dimensional, compressed video segments along a movement track in first and second directions for each of a plurality of viewable surfaces of a three dimensional proxy as separate video segment files, each of said video segment files having a length of approximately one keyframe distance, each keyframe distance including a plurality of frames, with every keyframe starting a different video segment file;

projecting the retrieved video segments onto the three dimensional proxy and developing a two dimensional video segment of a desired view;

compressing the developed two dimensional video segment; and

providing the compressed video segment to the user.

15. The non-transitory machine-readable medium of claim 14, wherein each stored video segment file in each direction overlaps at least one other stored video segment file and starts a portion of a keyframe distance from the overlapping stored video segment file.

16. The non-transitory machine-readable medium of claim 15, wherein a user may select a second location and direction after being at a first location, the method further comprising the step of:

retrieving the video segment files closest to the second location and of the correct direction, said closest video segment file being selected based on both the keyframe distance and the portion of the keyframe distance of the second location.

17. The non-transitory machine-readable medium of claim 15, wherein the three dimensional proxy is a cube, the viewable surfaces are the cube faces and the directions are forward and backward.

18. The non-transitory machine-readable medium of claim 15, wherein the portion of the keyframe distance is one-third.

19. The non-transitory machine-readable medium of claim 15, wherein each video segment is stored in a plurality of combinations of resolutions and bit rates and the retrieved video segments are those that provide the best trade-off between download wait time and viewing quality based on connection bandwidth.

20. The non-transitory machine-readable medium of claim 15, the method further comprising the steps of:

retrieving compressed still images related to a particular location if movement has stopped at the selected location, the compressed still images having a resolution higher than the retrieved video segment file corresponding to selected location; and

providing the compressed still images to the user.

21. The non-transitory machine-readable medium of claim 14, wherein a user may select a second location and direction after being at a first location, the method further comprising the step of:

retrieving the video segment files closest to the second location and of the correct direction, said closest video segment file being selected based on the keyframe distance.

22. The non-transitory machine-readable medium of claim 14, wherein the three dimensional proxy is a cube, the viewable surfaces are the cube faces and the directions are forward and backward.

23. The non-transitory machine-readable medium of claim 14, wherein each video segment is stored in a plurality of combinations of resolutions and bit rates and the retrieved video segments are those that provide the best trade-off between download wait time and viewing quality based on connection bandwidth.

24. The non-transitory machine-readable medium of claim 14, further comprising the steps of:

retrieving compressed still images related to a particular location if movement has stopped at the selected location, the compressed still images having a resolution higher than the retrieved video segment file corresponding to selected location; and

providing the compressed still images to the user.