Method and apparatus for creating interactive virtual tours

• US 20040169724A1
• Filed: 12/03/2003
• Published: 09/02/2004
• Est. Priority Date: 12/09/2002
• Status: Active Grant

• Current Assignee: Frank Edughom Ekpar
• Original Assignee: Frank Edughom Ekpar
• Inventors: Ekpar, Frank Edughom
• Granted Patent: US 7,567,274 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

The present invention discloses a method and apparatus for creating interactive virtual tours. In one embodiment, panoramic imaging systems characterized by a 360-degree lateral field of view and a vertical field of view that is usually less than 180 degrees are utilized and polynomial-based interpolation algorithms are used to correct distortions in the input panoramic images, video or rendered scenes, permitting the use of an arbitrary panoramic imaging system characterized by the ability to generate a seamless 360-degree panoramic view of the imaged scene in each single image frame. The input panoramic images, video or rendered scenes are used to create spherical environment maps that are packaged into completely immersive (giving the impression of being there) virtual tours that are rendered on a display device by a viewing engine that corrects perspective distortions in the spherical environment maps contained in the virtual tour packages and permits a plurality of viewers to view any portion of any of the panoramic scenes in the virtual tour package in a manner that is free from distortions. The present invention also discloses a control engine for representing the individual panoramic scenes contained in the virtual tour package in multi-dimensional space and/or time and for transmitting control signals to the viewing engine that cause the viewing engine to render any particular panoramic scene selected by the viewer and any view window on the selected panoramic scene desired by the viewer. Furthermore, the control engine communicates bi-directionally with the viewing engine and provides a means of indicating which particular panoramic scene is currently viewed by the viewing engine as well as a means of indicating what portion of the selected panoramic scene is currently displayed by the viewing engine.

Claims

1. An apparatus for creating, managing and publishing interactive virtual tours, the apparatus comprising:

a panorama data acquisition unit implementing means of capturing panoramic data and optionally preparing said panoramic data for further processing;

a transform engine responsive to said panoramic data and implementing means of correcting distortions in said panoramic data and/or performing automatic, manual or interactive calibration of said panoramic data and/or transforming said panoramic data into a desired format or set of formats when necessary, wherein said distortion correction could optionally be accomplished through steps of;

a) loading data including reference data that could be used to derive panoramic imaging system distortion profiles;

b) specifying a linear or other predetermined distortion function or set of distortion functions and optionally displaying representations of said distortion function or set of distortion functions;

c) using distortion function or set of distortion functions to build a distortion profile for the panoramic imaging system that was used to acquire the reference data;

d) performing transformation on panoramic data using distortion profile specified in c) and optionally displaying results of said transformation;

e) determining whether distortion is satisfactorily corrected;

f) continuing with steps g) and h) if distortion is deemed not to have been satisfactorily corrected and continuing with step i) if distortion is deemed to have been satisfactorily corrected;

g) automatically or interactively modifying the distortion function or set of distortion functions using feedback from optionally displayed distortion function representation or transformation results obtained in step d) or by using data loaded in step a);

h) repeating steps c) to g) until distortion is judged to have been satisfactorily corrected;

i) storing distortion profile obtained in step c) for use in performing transformations on panoramic data acquired using the panoramic imaging system for which data was loaded in step a);

an optional package generator adapted to generate virtual tour packages containing said panoramic data, commands and/or optional virtual tour data;

a viewing engine responsive to said panoramic data and/or virtual tour packages and implementing means for perspective correction, and user interaction with, said panoramic data and/or optional virtual tour data when necessary;

an optional control engine adapted to facilitate a higher level of interaction with said panoramic data and/or optional virtual tour data, wherein said control engine is optionally connected operatively to and communicates bi-directionally with said viewing engine, renders representative information about all or parts of said virtual tour, permits a particular portion to be selected from said virtual tour and optionally sends signals to said viewing engine that cause said viewing engine to permit the interactive navigation of said selected portion of said virtual tour, wherein said control engine also indicates or causes to be indicated what portion of said virtual tour is currently selected and what sub-part of said selected portion of said virtual tour is currently rendered, wherein said control engine is responsive to user input and/or commands from said viewing engine and is in turn capable of modifying said representative information about all or parts of said virtual tour in response to said user input and/or said commands from said viewing engine and is further optionally capable of communicating information indicative of such externally induced modifications to said user and/or said viewing engine;

a display means for rendering output of said viewing engine, control engine, package generator, transform engine, and/or panoramic data acquisition unit.

2. The apparatus of claim 1, wherein said panoramic data acquisition unit is adapted to capture data representing 2D panoramic images or video, 3D or stereoscopic panoramic images or video, holographic data, infrared data, ultrasonic data or ultraviolet data.

3. The apparatus of claim 1, wherein said panoramic data acquisition unit comprises at least one versatile device for creating representations of stimuli covering substantially all directions around a given reference or view point, said versatile device for creating representations of stimuli comprising at least one grid of one or more focusing elements disposed on an N-dimensional and arbitrarily shaped surface, at least one grid of one or more sensor elements disposed on an N-dimensional and arbitrarily shaped surface, and optionally, at least one grid of one or more stimulus guide elements disposed on an N-dimensional and arbitrarily shaped surface, wherein said focusing element grid is adapted to focus stimuli covering substantially all directions around a given reference or view point onto the sensor element grid and may optionally do so via the stimulus guide unit if necessary, wherein each focusing element or group of focusing elements is associated with and focuses a subset (typically the subset impinging on it) of the entire stimulus space onto a sensor element or group of sensor elements responsive to the stimuli, wherein when an optional stimulus guide element grid is provided, the focusing element grid can be adapted to focus the stimuli onto the stimulus guide element grid for suitable formatting and onward transmission to the sensor element grid, wherein when used, each stimulus guide element or group of elements is associated with and receives stimuli from a focusing element or group of focusing elements and is in turn associated with and transmits stimuli to a sensor element or group of sensor elements.

4. The apparatus of claim 3, wherein said versatile device for creating representations of stimuli covering substantially all directions around a given reference or view point comprises at least one grid of one or more sensor elements disposed on an N-dimensional and arbitrarily shaped surface, said at least one grid of one or more sensor elements responsive to electromagnetic radiation.

5. The apparatus of claim 1, wherein said panoramic data acquisition unit comprises a spherical image/video acquisition unit comprising at least one grid of one or more photosensitive elements on a surface with a spherical geometry or an approximation thereto and at least one enclosing concentric grid of one or more focusing elements on a surface with a spherical geometry or an approximation thereto.

6. The apparatus of claim 1, wherein said panoramic data acquisition unit comprises a panoramic imaging system characterized by a 360-degree lateral field of view and a vertical field of view that is usually less than 180 degrees.

7. The apparatus of claim 1, wherein said panoramic data acquisition unit is adapted to capture each complete panoramic data block in a single image/video frame.

8. The apparatus of claim 1, wherein said panoramic data acquisition unit is adapted to capture each complete panoramic data block in a plurality of image/video frames that are combined to form complete panoramic data blocks.

9. The apparatus of claim 1, wherein said associated optional virtual tour data includes audio, video, text, graphics, pointers to resources on local and distributed networks or virtual tour packages.

10. The apparatus of claim 1, wherein said panoramic data acquisition unit further comprises means of preparing and transferring all or portions of acquired panoramic data to said transform engine, said optional package generator, said viewing engine, said optional control engine or said display;

wherein said preparation and/or transfer optionally involves compression and/or decompression of all or portions of acquired panoramic data;

wherein said preparation and/or transfer is optionally in response to user commands or information inferred from the state of said viewing engine or said control engine.

11. The apparatus of claim 1, wherein said transform engine implements means of converting panoramic data from a first format to one or more formats and/or vice versa.

12. The apparatus of claim 11, wherein said panoramic data conversion is between polar and rectilinear coordinates, polar and spherical coordinates, spherical and rectilinear coordinates, cubic and spherical coordinates, cubic and polar coordinates or cubic and rectilinear coordinates.

13. The apparatus of claim 1, wherein said distortion function or set of distortion functions is based on a polynomial or a set of polynomials of suitable degree.

14. The apparatus of claim 1, wherein said package generator implements means of specifying active regions and/or volumes on said panoramic data and/or virtual tour data for programming reference.

15. The apparatus of claim 1, wherein said package generator implements means of specifying navigable paths or walk-through sequences on said panoramic data and/or virtual tour data.

16. The apparatus of claim 15, wherein said navigable paths or walk-through sequences are navigated in automatic or guided mode.

17. The apparatus of claim 1, wherein said panoramic data, virtual tour data, commands, navigable paths or walk-through sequences, virtual tour packages and/or distortion profiles are managed using a universal file format, said universal file format specifying a header identifying the file type and containing information as to the number, types, locations and sizes of elements it contains, wherein each element in the file is in turn described by a header specifying the type of element, its size and any relevant data, commands or attributes and the types, locations and sizes of any additional elements it contains, thus making it possible for arbitrary types and numbers of elements to be managed by said universal file format.

18. The apparatus of claim 1, wherein said viewing engine is responsive to user input and implements means for perspective correction of said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages.

19. The apparatus of claim 1, wherein said viewing engine further implements means of navigating said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages.

20. The apparatus of claim 1, wherein said viewing engine implements means for a single user or a plurality of users to independently and simultaneously interact with and navigate said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages.

21. The apparatus of claim 1, wherein said viewing engine implements means for a plurality of users to interact with and navigate said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages in a collaborative, competitive, coordinated, and/or synchronized manner.

22. The apparatus of claim 1, wherein said representative information rendered by said control engine about all or parts of said virtual tour is rendered in 2D, 3D or higher dimensional space and/or time.

23. The apparatus of claim 1, wherein said viewing engine and control engine comprise a unitary structure.

24. A method of creating, managing and publishing interactive virtual tours, said method comprising:

a panorama data acquisition step for capturing panoramic data and optionally preparing said panoramic data for further processing;

a transform step for correcting distortions in said panoramic data and/or performing automatic, manual or interactive calibration of said panoramic data and/or transforming said panoramic data into a desired format or set of formats when necessary, wherein said distortion correction could optionally be accomplished through steps of;

a) loading data including reference data that could be used to derive panoramic imaging system distortion profiles;

b) specifying a linear or other predetermined distortion function or set of distortion functions and optionally displaying representations of said distortion function or set of distortion functions;

c) using distortion function or set of distortion functions to build a distortion profile for the panoramic imaging system that was used to acquire the reference data;

d) performing transformation on panoramic data using distortion profile specified in c) and optionally displaying results of said transformation;

e) determining whether distortion is satisfactorily corrected;

f) continuing with steps g) and h) if distortion is deemed not to have been satisfactorily corrected and continuing with step i) if distortion is deemed to have been satisfactorily corrected;

g) automatically or interactively modifying the distortion function or set of distortion functions using feedback from optionally displayed distortion function representation or transformation results obtained in step d) or by using data loaded in step a);

h) repeating steps c) to g) until distortion is judged to have been satisfactorily corrected;

i) storing distortion profile obtained in step c) for use in performing transformations on panoramic data acquired using the panoramic imaging system for which data was loaded in step a);

an optional package generation step adapted to generate virtual tour packages containing said panoramic data, commands and/or optional virtual tour data;

a viewing step responsive to said panoramic data and/or virtual tour packages and providing means for perspective correction of, and user interaction with, said panoramic data and/or optional virtual tour data when necessary;

an optional control step adapted to facilitate a higher level of interaction with said panoramic data and/or optional virtual tour data, wherein said control step is optionally connected operatively to and communicates bi-directionally with said viewing step, renders representative information about all or parts of said virtual tour, permits a particular portion to be selected from said virtual tour and optionally sends signals to said viewing step that cause said viewing step to permit the interactive navigation of said selected portion of said virtual tour, wherein said control step also indicates or causes to be indicated what portion of said virtual tour is currently selected and what sub-part of said selected portion of said virtual tour is currently rendered, wherein said control step is responsive to user input and/or commands from said viewing step and is in turn capable of modifying said representative information about all or parts of said virtual tour in response to said user input and/or said commands from said viewing step and is further optionally capable of communicating information indicative of such externally induced modifications to said user and/or said viewing step;

a display step providing means for rendering output of said viewing step, control step, package generation step, transform step, and/or panoramic data acquisition step.

25. The method of claim 24, wherein said panoramic data acquisition step is adapted to capture data representing 2D panoramic images or video, 3D or stereoscopic panoramic images or video, holographic data, infrared data, ultrasonic data or ultraviolet data.

26. The method of claim 24, wherein said panoramic data acquisition step utilizes at least one versatile device for creating representations of stimuli covering substantially all directions around a given reference or view point, said versatile device for creating representations of stimuli comprising at least one grid of one or more focusing elements disposed on an N-dimensional and arbitrarily shaped surface, at least one grid of one or more sensor elements disposed on an N-dimensional and arbitrarily shaped surface, and optionally, at least one grid of one or more stimulus guide elements disposed on an N-dimensional and arbitrarily shaped surface, wherein said focusing element grid is adapted to focus stimuli covering substantially all directions around a given reference or view point onto the sensor element grid and may optionally do so via the stimulus guide unit if necessary, wherein each focusing element or group of focusing elements is associated with and focuses a subset (typically the subset impinging on it) of the entire stimulus space onto a sensor element or group of sensor elements responsive to the stimuli, wherein when an optional stimulus guide element grid is provided, the focusing element grid can be adapted to focus the stimuli onto the stimulus guide element grid for suitable formatting and onward transmission to the sensor element grid, wherein when used, each stimulus guide element or group of elements is associated with and receives stimuli from a focusing element or group of focusing elements and is in turn associated with and transmits stimuli to a sensor element or group of sensor elements.

27. The method of claim 26, wherein said versatile device for creating representations of stimuli covering substantially all directions around a given reference or view point comprises at least one grid of one or more sensor elements disposed on an N-dimensional and arbitrarily shaped surface, said at least one grid of one or more sensor elements responsive to electromagnetic radiation.

28. The method of claim 24, wherein said panoramic data acquisition step utilizes a spherical image/video acquisition unit comprising at least one grid of one or more photosensitive elements on a surface with a spherical geometry or an approximation thereto and at least one enclosing concentric grid of one or more focusing elements on a surface with a spherical geometry or an approximation thereto.

29. The method of claim 24, wherein said panoramic data acquisition step utilizes a panoramic imaging system characterized by a 360-degree lateral field of view and a vertical field of view that is usually less than 180 degrees.

30. The method of claim 24, wherein said panoramic data acquisition step is adapted to capture each complete panoramic data block in a single image/video flame.

31. The method of claim 24, wherein said panoramic data acquisition step is adapted to capture each complete panoramic data block in a plurality of image/video frames that are combined to form complete panoramic data blocks.

32. The method of claim 24, wherein said optional virtual tour data includes audio, video, text, graphics, pointers to resources on local and distributed networks or virtual tour packages.

33. The method of claim 24, wherein said panoramic data acquisition step further comprises providing means of preparing and transferring all or portions of acquired panoramic data to said transform engine, said optional package generator, said viewing engine, said optional control engine or said display;

wherein said preparation and/or transfer optionally involves compression and/or decompression of all or portions of acquired panoramic data;

wherein said preparation and/or transfer is optionally in response to user commands or information inferred from the state of said viewing engine or said control engine.

34. The method of claim 24, wherein said transform step implements means of converting panoramic data from a first format to one or more formats and/or vice versa.

35. The method of claim 34, wherein said panoramic data conversion is between polar and rectilinear coordinates, polar and spherical coordinates, spherical and rectilinear coordinates, cubic and spherical coordinates, cubic and polar coordinates or cubic and rectilinear coordinates.

36. The method of claim 24, wherein said distortion function or set of distortion functions is based on a polynomial or a set of polynomials of suitable degree.

37. The method of claim 24, wherein said package generation step implements means of specifying active regions and/or volumes on said panoramic data and/or virtual tour data for programming reference.

38. The method of claim 24, wherein said package generation step implements means of specifying navigable paths or walk-through sequences on said panoramic data and/or virtual tour data.

39. The method of claim 38, wherein said navigable paths or walk-through sequences are navigated in automatic or guided mode.

40. The method of claim 24, wherein said panoramic data, virtual tour data, commands, navigable paths or walk-through sequences, virtual tour packages and/or distortion profiles are managed using a universal file format, said universal file format specifying a header identifying the file type and containing information as to the number, types, locations and sizes of elements it contains, wherein each element in the file is in turn described by a header specifying the type of element, its size and any relevant data, commands or attributes and the types, locations and sizes of any additional elements it contains, thus making it possible for arbitrary types and numbers of elements to be managed by said universal file format.

41. The method of claim 26, wherein said viewing step is responsive to user input and implements means for perspective correction of said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages.

42. The method of claim 24, wherein said viewing step further implements means of navigating said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages.

43. The method of claim 24, wherein said viewing step implements means for a single user or a plurality of users to independently and simultaneously interact with and navigate said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages.

44. The method of claim 24, wherein said viewing step implements means for a plurality of users to interact with and navigate said panoramic data, navigable paths or walk-through sequences and/or virtual tour packages in a collaborative, competitive, coordinated, and/or synchronized manner.

45. The method of claim 24, wherein said representative information rendered by said control step about all or parts of said virtual tour is rendered in 2D, 3D or higher dimensional space and/or time.

46. The method of claim 26, wherein said viewing step and control step are implemented in a unitary structure.