System and process for viewing and navigating through an interactive video tour

• US 6,968,973 B2
• Filed: 05/31/2003
• Issued: 11/29/2005
• Est. Priority Date: 05/31/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kang, Sing Bing, Criminisi, Antonio, Uyttendaele, Matthew T., Szeliski, Richard
• Primary Examiner(s): Noland, Kenneth
• Application Number:
  US10/449,700
• Publication Number:
  US 20040239699A1
• Time in Patent Office: 913 Days
• Field of Search: 221/1, 221/2, 221/9, 715/855, 715/854, 715/764, 345/619
• US Class Current: 221/2
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858

Abstract

A system and process for providing an interactive video tour of a tour site to a user is presented. In general, the system and process provides an image-based rendering system that enables users to explore remote real world locations, such as a house or a garden. The present approach is based directly on filming an environment, and then using image-based rendering techniques to replay the tour in an interactive manner. As such, the resulting experience is referred to as Interactive Video Tours. The experience is interactive in that the user can move freely along a path, choose between different directions of motion at branch points in the path, and look around in any direction. The user experience is additionally enhanced with multimedia elements such as overview maps, video textures, and sound.

Claims

1. In a computer system having a user interface comprising a display and user interface selection device, a process for presenting an interactive video tour of a tour site, comprising process actions for:

inputting data associated with the video tour, said inputting comprising,

inputting video data comprising a sequence of video frames each of which represents a panoramic view of the tour site from a particular location along said path, and

inputting scene data, which given a viewing direction and optionally a user-specified direction of motion along the tour path, is used to determine which video frame and what part of that video frame is needed to render and display the next image of the video tour;

displaying a viewing and navigation window on the display, wherein the window comprises a plurality of sectors for viewing and controlling the video tour; and

implementing commands entered by a user directly via said selection device and via the user selecting displayed items within the viewing and navigation window sectors using said selection device, so as to give the user continuous control over both the viewing position along a path through the tour site and viewing direction at points on the path.

2. The process of claim 1, wherein the path through the tour site has at least one branch point from each of which the path either turns an abrupt corner or diverges in more than one direction thereby allowing the user to follow any part of the path extending away from the branch, and wherein the process action of inputting scene data comprises an action of inputting a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current user-specified viewing direction.

3. The process of claim 2, wherein the action of inputting a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current user-specified viewing direction, comprises an action of defining a range of viewing directions for each part of the path diverging from each branch point such that when a user specifies a viewing direction within the defined range for a part of the path extending away from a viewing location that is a branch point, the frame associated with the first specifiable location along that part of the path from the branch point is identified as the next video frame that is to be played.

4. The process of claim 2, wherein the branch points comprise at least one (i) a point where the path turns a corner, (ii) a point where the path crosses itself to form a four or more-way intersection, (iii) a point where the path forms an approximately T-shaped three-way intersection, and (iv) a point where the path forms an approximately Y-shaped three-way intersection.

5. The process of claim 1, wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a portion of the frame associated with a current viewing position which corresponds to a prescribed field of view around a current viewing direction, and wherein the process action of inputting scene data comprises an action of inputting a frame rotation table which for each video frame identifies a global orientation direction corresponding to a direction designated when the frame was captured as a zero degree line from which the user-specified viewing direction is measured thereby facilitating finding the portion of the frame to be displayed.

6. The process of claim 5, wherein the plurality of viewing and navigation window sectors comprises a sector for displaying an icon representing an object of interest residing in the tour site which when selected by the user causes additional information about the object to be displayed, and wherein the process action of inputting scene data comprises an action of inputting an object information table that for a prescribed set of frame ranges and viewing directions associated with the frames of those frame ranges in which the object of interest appears, identifies the particular icon that is to be displayed and link details needed to access said information about the object.

7. The process of claim 5, wherein the process action of displaying a portion of the frame associated with the current viewing position which corresponds to a prescribed field of view around the current viewing direction plurality comprises displaying a replaceable object within the displayed frame portion, and wherein the process action of inputting scene data comprises an action of inputting an object replacement information table that for a prescribed set of frame ranges and viewing directions associated with the frames of those frame ranges in which the replaceable object appears, identifies a menu that is to be displayed in the viewing and navigation window comprising a list of replacement objects and link details for each replacement object listed that are needed to access said replacement objects which are to be displayed in place of the replaceable object upon selection of that replacement object from the menu by the user.

8. The process of claim 5, wherein the process action of displaying a portion of the frame associated with the current viewing position which corresponds to a prescribed field of view around the current viewing direction plurality comprises displaying a replaceable object within the displayed frame portion, and wherein the process action of inputting scene data comprises an action of inputting an object replacement information table that for a prescribed set of frame ranges and viewing directions associated with the frames of those frame ranges in which the replaceable object appears, identifies link details for a group of replacement objects that are needed to access each of said objects, which are displayed one at a time in place of the replaceable object in a prescribed cyclical order upon entry of a change command by the user.

9. The process of claim 1, wherein the method employed to capture the video frames introduced jitter, and wherein the process action of inputting scene data comprises an action of inputting a frame rotation table which for each video frame identifies a jitter correction factor thereby allowing the jitter to be removed from each frame prior to a portion thereof being rendered and displayed.

10. The process of claim 1, wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a map graphic of the tour site, and wherein the process action of inputting scene data comprises an action of inputting a map region table which for each of a prescribed set of frame ranges identifies the particular map graphic that is to be displayed whenever a portion of a video frame within that frame range is rendered and displayed.

11. The process of claim 1, wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a name associated with a region of the tour site that the user is currently viewing, and wherein the process action of inputting scene data comprises an action of inputting a map region table that for each of a prescribed set of frame ranges, each of which is associated with a particular region of the tour site, identifies the name associated with that region.

12. The process of claim 1, wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a map graphic of the tour site that comprises a line representing the tour path and an indicator which identifies the point on the tour path from which the user is currently viewing the tour site, and wherein the process action of inputting scene data comprises an action of inputting a path table which for each video frame identifies a location on the map graphic representing the point on the tour path that the frame was captured thereby facilitating the placement of said indicator on the map graphic.

13. The process of claim 1, wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a map graphic of the tour site that comprises a line representing the tour path, and wherein the process action of inputting scene data comprises an action of inputting a path table which for each video frame identifies a location on the map graphic representing the point on the tour path that the frame was captured, and wherein whenever the user selects a point on the tour path line of the map graphic, the next frame from which a portion is rendered and displayed is the frame identified in the path table as corresponding to the user-selected path point, thereby allowing the user to view another part of the tour site without having to follow the tour path to that part.

14. A computer-implemented process for providing an interactive video tour of a tour site to a user, comprising using a computer to perform the following process actions:

(a) inputting video data comprising a sequence of video frames each of which represents a panoramic view of the tour site from a particular location along a path through the tour site;

(b) inputting scene data which comprises information to identify the frame associated with each specifiable location along the path, and that given a viewing direction and a user-specified direction of motion along the tour path, is used to determine which video frame and what part of that video frame is needed to render and display the next image of the video tour;

(c) for a given viewing position within the site along the path, identifying the frame associated with that position using the scene data and designating it as the current frame;

(d) inputting a current user-specified viewing direction, and a current user-specified direction of motion along the tour path, if one;

(e) obtaining from the video data that portion of the current frame which corresponds to a prescribed field of view around the current viewing direction;

(f) rendering the obtained portion of the current frame and displaying it to the user, while simultaneously identifying the frame needed to display the next image of the video tour in view of the user-specified direction of motion along the tour path using the scene data;

(g) designating the newly identified frame as the current frame in lieu of the previously identified frame;

(h) repeating process action (d) through (g) for as long as the user wants to view the video tour.

15. The process of claim 14, wherein the video data is compressed using a compression method which allows the decompression of any frame, not just the next sequential frame, and which allows any portion of a frame to be decompressed without having to decompress the rest of the frame, and wherein the process action of obtaining from the video data that portion of the current frame which corresponds to the prescribed field of view around the current viewing direction, comprises an action of exclusively decompressing just said portion of the current frame.

16. The process of claim 14, wherein the scene data comprises a global orientation direction for each video frame which corresponds to a direction designated when the frame was captured as a zero degree line from which the viewing direction is measured, and wherein the process action of obtaining from the video data that portion of the current frame which corresponds to the prescribed field of view around the current viewing direction, comprises an action of identifying said portion of the current frame which corresponds to the prescribed field of view around the current viewing direction by measuring the viewing direction from the global orientation direction.

17. The process of claim 16, wherein the method employed to capture the video frames introduced jitter, and wherein the scene data comprises a jitter correction factor for each video frame, and wherein the process action of obtaining from the video data that portion of the current frame which corresponds to the prescribed field of view around the current viewing direction, comprises the actions of:

obtaining the jitter correction factor associated with the current frame from the scene data; and

employing said jitter correction factor to remove the jitter from said portion of the current frame.

18. The process of claim 14, wherein the process action of identifying the frame needed to display the next image of the video tour in view of the user-specified direction of motion, comprises an action of designating the previously designated frame as the newly designated frame whenever no direction of motion is specified.

19. The process of claim 14, wherein the path through the tour site has at least one branch point from each of which the path either turns an abrupt corner or diverges in more than one direction such that the user can follow any part of the path extending away from the branch, and wherein the scene data comprises a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current viewing direction, and wherein the process action of identifying the frame needed to display the next image of the video tour in view of the user-specified direction of motion, comprises the actions of:

determining from the scene data whether the current frame is associated with a branch point; and

whenever the current frame is not associated with a branch point,

identifying the next sequential frame of the video data as the newly identified frame if the user has specified a forward motion direction as related to what was the forward direction when the video frames were captured, and

identifying the previous sequential frame of the video data as the newly identified frame if the user has specified a backwards motion direction as related to what was the forward direction when the video frames were captured.

20. The process of claim 19, further comprising an action of determining if the current viewing direction is outside a prescribed forward range, and whenever it is determine the current viewing direction is outside the forward range and a forward direction of motion is specified by the user, the motion command is ignored and rather than identifying the next sequential frame of the video data as the newly identified frame, the previously identified frame is designated as the newly identified frame such that the user continues to view the tour site from the same location.

21. The process of claim 19, further comprising an action of determining if the current viewing direction is outside a prescribed backward range, and whenever it is determine the current viewing direction is outside the backward range and a backward direction of motion is specified by the user, the motion command is ignored and rather than identifying the previous sequential frame of the video data as the newly identified frame, the previously identified frame is designated as the newly identified frame such that the user continues to view the tour site from the same location.

22. The process of claim 16, wherein the path through the tour site has at least one branch point from each of which the path either turn an abrupt corner or diverges in more than one direction such that the user can follow any part of the path extending away from the branch, and wherein the scene data comprises a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current viewing direction, and wherein the process action of identifying the frame needed to display the next image of the video tour in view of the user-specified direction of motion, comprises the actions of:

determining from the scene data whether the current frame is associated with a branch point; and

whenever the current frame is associated with a branch point, identifying the frame of the video data specified in the scene data as the next frame for the current viewing direction as the newly identified frame.

23. The process of claim 22, wherein a substantial divergence in the global orientation direction may exist between a frame associated with a branch point and the frame designated as the newly identified frame, and wherein process action of obtaining from the video data that portion of the current frame which corresponds to the prescribed field of view around the current viewing direction comprises the actions of:

estimating the relative rotation between the frames using the global orientation information for those frames from the scene data; and

correcting for any mismatch in order to minimize a visual jump that the user can see when the tour image associated with the frame portion is displayed.

24. The process of claim 22, wherein the video data is pre- rotated such that all panoramic frames have the same orientation, and wherein process action of obtaining from the video data that portion of the current frame which corresponds to the prescribed field of view around the current viewing direction does not require any rotation adjustment.

25. The process of claim 14, wherein the user is allowed to input a new path location at any time during the progress of the interactive video tour, and wherein the process further comprises an action of restarting at process action (c) whenever the user inputs a new path location.

26. A system for providing an interactive video tour of a tour site to a user, comprising:

a general purpose computing device comprising a display and a user selection device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input video data comprising a sequence of video frames each of which represents a panoramic view of the tour site from a particular location along a path through the tour site,

input scene data comprising information to identify the frame associated with each specifiable location along the path, and that given a viewing direction and a user-specified direction of motion along the tour path, is used to determine which video frame and what part of that video frame is needed to render and display the next image of the video tour,

display a viewing and navigation window on the display, wherein the window comprises a plurality of sectors for viewing and controlling the video tour, and

implementing commands comprising a current user-specified viewing direction and a current user-specified direction of motion along the tour path, if one, which are entered by a user directly via said selection device and via the user selecting displayed items within the viewing and navigation window sectors using said selection device, so as to give the user continuous control over both the viewing position along a path through the tour site and viewing direction at points on the path.

27. The system of claim 26, wherein the program module for displaying the viewing and navigation window comprises a sub-module for displaying a frame display sector that presents a portion of the frame associated with a current viewing position in the tour site, wherein said frame portion corresponds to a prescribed field of view around the current viewing direction.

28. The system of claim 27, wherein the program module for displaying the viewing and navigation window comprises a sub-module for displaying direction indicators on the portion of the frame associated with a current viewing position in the tour site displayed in the display sector, wherein the direction indicators point to the direction to regions of the tour site which are adjacent to the region currently being viewed by the user.

29. The system of claim 27, wherein the program module for displaying the viewing and navigation window comprises sub-modules for:

displaying a replaceable object within the portion of the frame displayed in the display sector;

upon selection of the replaceable object by the user, displaying a menu in the viewing and navigation window comprising a list of replacement objects; and

upon selection of a replacement object from said list by the user, displaying that replacement object within the portion of the frame associated with the replaceable object.

30. The system of claim 29, wherein the sub-modules for displaying the replaceable object within the portion of the frame displayed in the display sector and displaying a replacement object within the portion of the frame associated with the replaceable object, each comprises sub-modules for:

obtaining an occlusion mask associated with the frame, wherein the occlusion mask specifies the degree of visibility for each pixel of the replaceable or replacement object in a range from completely occluded through partially visible to totally visible;

displaying each pixel of the replaceable object or replacement object with the visibility indicated for that pixel in the occlusion map.

31. The system of claim 30, wherein the sub-module for displaying each pixel of the replaceable object or replacement object with the visibility indicated for that pixel in the occlusion map, comprises a sub-module for obtaining alignment information associated with the occlusion map which specifies the position of the map relative to the frame so that the pixels of the frame subject to the visibility indicators of the occlusion map can be identified.

32. The system of claim 27, wherein the program module for displaying the viewing and navigation window comprises sub-modules for:

displaying a replaceable object within the portion of the frame displayed in the display sector; and

displaying a replacement object within the portion of the frame associated with the replaceable object in lieu thereof, each time the user selects the replaceable or replacement object, wherein the replacement objects are displayed in a prescribed order and are displayed cyclically.

33. The system of claim 32, wherein the sub-modules for displaying the replaceable object within the portion of the frame displayed in the display sector and displaying a replacement object within the portion of the frame associated with the replaceable object, each comprises sub-modules for:

obtaining an occlusion mask associated with the frame, wherein the occlusion mask specifies the degree of visibility for each pixel of the replaceable or replacement object in a range from completely occluded through partially visible to totally visible;

displaying each pixel of the replaceable object or replacement object with the visibility indicated for that pixel in the occlusion map.

34. The system of claim 33, wherein the sub-module for displaying each pixel of the replaceable object or replacement object with the visibility indicated for that pixel in the occlusion map, comprises a sub-module for obtaining alignment information associated with the occlusion map which specifies the position of the map relative to the frame so that the pixels of the frame subject to the visibility indicators of the occlusion map can be identified.

35. The system of claim 26, wherein the program module for displaying the viewing and navigation window comprises a sub-module for displaying a navigation sector that indicates the name and direction to regions of the tour site which are adjacent to the region currently being viewed by the user in the frame display sector.

36. The system of claim 26, wherein the program module for displaying the viewing and navigation window comprises a sub-module for displaying a information sector that comprises one or more icons representing an object of interest which is visible in the portion of the frame associated with a current viewing position that is displayed in the display sector, and wherein, whenever the user selects one of said icons, additional information about the associated object is displayed in a separate window on the display.

37. The system of claim 26, wherein the program module for displaying the viewing and navigation window comprises a sub-module for displaying a map sector which comprises a map graphic of the tour site.

38. The system of claim 37, wherein the sub-module for displaying the map sector further comprises displaying a line representing the tour path on the map graphic.

39. The system of claim 38, wherein the sub-module for displaying the map sector further comprises displaying an indicator which identifies the point on the tour path from which the user is currently viewing the tour site.

40. The system of claim 38, wherein the program module for displaying the viewing and navigation window comprises a sub-module for displaying a frame display sector that presents a portion of the frame associated with a current viewing position in the tour site which corresponds to a prescribed field of view around the current viewing direction, and wherein whenever the user selects a point on the tour path line of the map graphic, the next frame from which a portion is displayed in the frame display sector is the frame corresponding to the user-selected path point, thereby allowing the user to view another part of the tour site without having to follow the tour path to that part.

41. The system of claim 37, wherein the sub-module for displaying the map sector further comprises displaying a name given to the region currently being viewed by the user on the map graphic.

42. The system of claim 26, wherein said user selection device allows the user to pan the viewing direction left/right/up/down and select the direction of motion to move the user'"'"'s viewing position forwards and backwards along the tour path.

43. The system of claim 42, wherein the user selection device is a gamepad.

44. A computer-readable medium having computer-executable instructions for providing an interactive video tour of a tour site to a user, said computer-executable instructions comprising:

(a) inputting video data comprising a sequence of video frames each of which represents a panoramic view of the tour site from a particular location along a path through the tour site;

(b) inputting audio data which represents sounds associated with the tour site;

(c) inputting scene data which comprises information to identify the frame associated with each specifiable location along the path, and that given a viewing direction and a user-specified direction of motion along the tour path, is used to determine which video frame and what part of that video frame is needed to render and display the next image of the video tour;

(d) for a given viewing position within the site along the path, identifying the frame associated with that position using the scene data and designating it as the current frame;

(e) inputting a current user-specified viewing direction, and a current user-specified direction of motion along the tour path, if one;

(f) obtaining from the video data that portion of the current frame which corresponds to a prescribed field of view around the current viewing direction;

(g) rendering the obtained portion of the current frame and displaying it to the user, while simultaneously identifying the frame needed to display the next image of the video tour in view of the user-specified direction of motion along the tour path using the scene data;

(h) designating the newly identified frame as the current frame in lieu of the previously identified frame;

(i) repeating process action (e) through (h) for as long as the user wants to view the video tour.

45. The computer-readable medium of claim 44, wherein the instruction for inputting audio data, comprises sub-instructions for:

inputting a plurality of audio tracks, each of which was captured at a different known location within the tour site;

attenuating each track based on the inverse square distance rule in view of the user'"'"'s current viewing location in the tour site;

mixing the attenuated tracks together; and

playing the mixed audio tracks to the user for as long as the user remains at the current viewing location.