Method and apparatus for sampling images to simulate movement within a multidimensional space

• US 4,807,158 A
• Filed: 09/30/1986
• Issued: 02/21/1989
• Est. Priority Date: 09/30/1986
• Status: Expired due to Fees

• Current Assignee: Daleco/Ivex Partners Limited A CA Limited Partnership
• Original Assignee: Daleco/Ivex Partners Limited
• Inventors: Blanton, Keith A., Tumblin, John E.
• Primary Examiner(s): Harkcom, Gary V.
• Assistant Examiner(s): HERNDON, HEATHER R
• Application Number:
  US06/913,507
• Time in Patent Office: 875 Days
• Field of Search: 364/518, 364/521, 364/424, 364/431, 434/43, 358/104
• US Class Current: 345/581
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G09B 9/301 by computer-processed or -generated image
• CPC Class Codes: Y10S 345/96 Iterative display of preconfigured images

Abstract

A method and apparatus for simulating movement within a multidimensional space by sampling a prestored panoramic image. The image formed by sampling is displayed on a CRT display screen. Each of the stored images represents the view derived from data stored as a panoramic projection of a flat world in an imaginary space, selected as a function of an observer'"'"'s position within the imaginary space and orientation with respect to that space. The preferred embodiment selects prestored panoramic image data which has been mapped to a geometric projection surface called a "keypoint". The apparatus includes a random access video disc system for storing the keypoint data. The method involves mapping each pixel on the display screen to the imaginary space and thence to the keypoint projection surface in order to retrieve pixel information for display. Each keypoint projection surface represents an entire 360° panoramic view of the imaginary world for all possible orientations of the observer with respect to the closest keypoint.

Claims

1. A method of generating an image to simulate a view seen by an observer within an imaginary space, comprising the steps of:

(1) mapping at least a portion of a panoramic view of a first surface to be simulated in the imaginary space onto a second mapping surface by providing as picture elements the picture values of areas of said mapping surface corresponding to areas of said first surface projected onto said mapping surface;

(2) storing said picture elements on a storage medium as a panoramic image;

(3) retrieving from said storage medium a portion of said picture elements as a function of the position and orientation of an observer with respect to the imaginary space; and

(4) displaying said plurality of picture element on a display.

2. The method of claim 1, further comprising the steps of retrieving from said storage medium a plurality of sky picture elements associated with regions of the panoramic image corresponding to the sky when the observer'"'"'s position and attitude indicate that the field of view of the observer includes regions above the horizon of the panoramic image, and displaying said sky picture elements.

3. The method of claim 1, wherein said each of said panoramic images comprises a plurality of picture elements mapped in a predetermined geometric projection.

4. The method of claim 3, wherein said predetermined geometric projection comprises a conic projection.

5. The method of claim 4, wherein said conic projection is an inverted conic surface, and wherein the uppermost limit of the conic surface is associated with the boundary between the sky and the horizon in said panoramic image.

6. The method of claim 1, wherein said plurality of picture elements retrieved in said retrieving step comprises a predetermined number N of picture elements, and further comprising the step of sampling said predetermined number N of picture elements to obtain a second predetermined number n of picture elements for display in the displaying step, as a function of observer position and attitude within the imaginary space.

7. The method of claim 1, wherein said panoramic image comprises all data required to reconstruct all possible views of an imaginary landscape within the imaginary space.

8. The method of claim 1, wherein the step of storing on said storage medium comprises storing a plurality of panoramic images, each of said panoramic images being associated with a different predetermined region in said imaginary space, and further comprising the step of selecting one of said plurality of said panoramic images as a function of observer position and attitude within the imaginary space.

9. The method of claim 8, wherein the step of selecting a particular one of said panoramic images comprises selecting one of the image which is closer to the region of the imaginary space in the field of view of the observer.

10. The method of claim 1, wherein said plurality of displayed picture elements are displayed on a CRT.

11. The method of claim 1, wherein the field of view of the observer is a field of view, and further comprising the step of retrieving from said storage medium a second plurality of said picture elements associated with a second field of view of the observer within the imaginary space to stimulate movement from a first point of view of the observer to a second point of view.

12. The method of claim 1, wherein said picture elements of said panoramic image are stored on said storage medium in addressable locations at addresses corresponding to the coordinates of said mapping surface.

13. The method of claim 1, wherein said panoramic image comprises a 360 degree panoramic view.

14. The method of claim 1, wherein said panoramic image comprises a less than 360 degree view.

15. The method of claim 1, wherein said second mapping surface comprises an inverted conic surface.

16. The method of claim 15, wherein the inverted conic surface comprises a plurality of circular horizontal sections, each of said sections being associated with a predetermined number of picture element locations, and wherein the uppermost one of said sections comprises picture elements associated with the horizon of the panoramic image.

17. A system for displaying video images for simulating movement within an imaginary space, comprising:

mass data storage means for storing a plurality of sets of keypoint image data corresponding to a plurality of panoramic views within said imaginary space, said keypoint image data comprising a plurality of picture elements obtained from mapping at least a portion of a panoramic view of a first surface to be simulated in said imaginary space onto a second mapping surface, said picture elements comprising the picture values of areas of said mapping surface corresponding to areas of said first surface projected onto said mapping surface;

means for selecting a particular one of said keypoint data sets store in said mass storage means as a function of input observer position signals;

addressable memory means for storing data corresponding to a selected one of said keypoint data sets selected by said selecting means;

means for generating addresses for said addressable memory means corresponding to said observer position signals and input observer spatial orientation signals;

means for retrieving a selected plurality of picture elements from said addressable memory means from addresses provided by said address generating means; and

means for displaying said selected plurality of picture to provide a video image.

18. Image reproducing apparatus for generating images to simulate movement within an imaginary space, comprising:

image storage means for retrievably storing panoramic image data, said panoramic image data comprising a plurality of picture elements obtained from mapping at least a portion of panoramic view of a first surface to be simulated in said imaginary space onto a second mapping surface, said picture elements comprising the picture values of area;

of said mapping surface corresponding to areas of said first surface projected onto said mapping surface;

means for sampling a particular portion of said stored panoramic image data corresponding to an observer field of view with respect to said imaginary space to select a plurality of picture elements; and

means for displaying said selected plurality of picture elements.

19. The apparatus of claim 18, further comprising means for providing position signals corresponding to the position and attitude of said observer field of view with respect to said imaginary space, and wherein said sampling means is responsive to said position signals to select said plurality of picture elements.

20. The apparatus of claim 19, further comprising means for retrieving from said storage means a plurality of sky picture elements associated with regions of said panoramic view corresponding to the sky in response to indication by said position signals that said observer field of view includes regions above the horizon of said panoramic view, and means for displaying said sky picture elements.

21. The apparatus of claim 18, wherein said panoramic view comprises a less than 360 degree view.

22. The apparatus of claim 18, wherein said second mapping surface comprises an inverted conic surface.

23. The apparatus of claim 22, wherein said inverted conic surface comprises a plurality of circular horizontal sections, each of said sections being associated with a predetermined number of picture element locations, and wherein the upper one of said sections comprises picture elements associated with the horizon of said panoramic view.

24. The apparatus of claim 18, wherein said image storage means comprises a video disc.

25. The apparatus of claim 18, wherein said image storage means comprises:

a video disc for storing a keypoint data set corresponding said panoramic image data associated with a particular keypoint position in said imaginary space, said keypoint data set comprising a predetermined number N of picture elements, and an addressable memory for storing said N picture elements for accessing by said sample means.

26. The apparatus of claim 25, wherein said sampling means selects from said predetermined number N of picture elements stored in said addressable memory to obtain a second predetermined number n of picture elements for display as a function of observer position and attitude with respect to said imaginary space.

27. The apparatus of claim 25, wherein said video disc stores a plurality of said keypoint data sets, and said addressable memory means stores a particular one of said plurality of keypoint data sets corresponding to said observer field of view.

28. The apparatus of claim 18, wherein said plurality of picture elements stored by said image storage means comprises a predetermined nu N of picture elements, and further comprising addressable memory means for storing said number N of picture elements for accessing by said sampling means, and wherein said sampling means samples said predetermined number N of picture elements to obtain a second predetermined number n of picture elements for display as a function of observer position and attitude with respect to said imaginary space.

29. The apparatus of claim 18, wherein said image storage means stores a plurality of keypoint data sets, each of said keypoint data sets corresponding to a different one of said panoramic views, each of said different panoramic views being associated with a different predetermined region in said imaginary space, and further comprising keypoint selecting means for selecting one of said keyppoint data sets as a function of observer position and attitude with respect to said imaginary space.

30. The apparatus of claim 29, wherein said keypoint selecting means selects one of said keypoint data sets which is closest to the region of the imaginary space in the field of view of the observer.

31. The apparatus of claim 18, wherein said displayed selected plurality of picture elements are displayed on a CRT.

32. The apparatus of claim 18, wherein the field of view of the observer is a first field of view, and wherein said sampling means samples a second particular portion of said stored image data corresponding to a second observer field of view to select a second plurality of picture elements to simulate movement from a first point of view of the observer associated with said first field of view to a second point of view associated with said second field of view.

33. The apparatus of claim 18, wherein said panoramic image data are stored in said storage means in addressable locations at addresses corresponding to the coordinates of said mapping surface.

34. The apparatus of claim 18, wherein said panoramic view comprises a 360 degree panoramic view.

35. A method of reproducing a video image to simulate movement within an imaginary space, comprising the steps of:

(1) mapping a panoramic view of the imaginary space onto a first projection surface associated with a keypoint, the keypoint comprising a point of view of a mapping observer of the imaginary space;

(2) storing picture elements at addresses in an addressable memory corresponding to the coordinates of the first projection surface;

(3) determining the intersection of a first ray emanating from the point of view of a viewing observer within the imaginary space through a predetermined picture element location on a viewing screen associated with the viewing observer with a second projection surface corresponding to the ground of the imaginary space;

(4) tracing a second ray from the intersection of the first ray with the second projection surface to the keypoint;

(5) determining the address in the addressable memory of a picture elements on the first projection surface as a function of coordinates of the intersection of the second ray with the first projection surface;

(6) retrieving data from the addressable memory corresponding to the determined picture element address;

(7) displaying a picture element derived from said retrieved data as the picture element for the predetermined picture element location on a display corresponding to the viewing screen; and

(8) repeating the above steps (3) through (7) to create an entire viewing screen image for all picture element locations of the viewing screen.

36. The method of claim 35, further comprising the steps of:

(9) mapping a plurality of panoramic views of the imaginary space onto a plurality of first projection surfaces associated with a plurality of keypoints, each of the keypoints comprising a different point of view of the mapping observer of the imaginary space;

(10) storing the plurality of panoramic views in a mass storage device;

(11) providing position signals corresponding to the location and orientation of the point of view of the viewing observer in the imaginary space;

(12) selecting a particular one of the stored plurality of panoramic views as a function of the viewing observer position and orientation signals;

(13) storing picture elements of the selected particular one of the panoramic views at addresses in the addressable memory corresponding to the coordinates of the first projection surface; and

(14) carrying out the above steps (3) through (8) to create an entire viewing screen image for all picture element locations of the viewing screen.

37. The method of claim 35, wherein the first projection surface is a conic projection surface.

38. The method of claim 35, wherein the second projection surface corresponds to the ground in the imaginary space.

39. The method of claim 35 further comprising the step of reducing the occurrence of sampling artifacts in the image displayed in the displaying step.

40. The method of claim 39, wherein the data retrieved in the retrieving step comprises a selected subset of a predetermined number m of picture elements, and wherein the step of reducing the occurrence of sampling artifacts in the image displayed comprises filtering said predetermined number m of picture elements to be obtain each one of the picture elements displayed in the displaying step.

41. The method of claim 40, where m is at least four.

42. The method of claim 40, wherein the step of filtering comprises the step of computing a resultant picture element R as a function of the values of the m picture elements.

43. A method of reproducing a series of video images to simulate movement within an imaginary space, comprising the steps of:

(1) creating a database comprising a plurality of panoramic images, each of the images comprising a plurality of picture elements, each of the images being associated with a predefined keypoint region and set of orientations of an observer within said imaginary space, each of the images including picture elements corresponding to all visible picture elements within a field of view of the observer positioned within a predefined subspace and set of observer orientations within said imaginary space associated with said keypoint region;

(2) storing the panoramic images on a storage medium;

(3) selecting one of the stored panoramic images as a function of the spatial location and spatial orientation of the observer within the predefined subspace;

(4) retrieving from the storage medium image data corresponding to the selected one of said stored panoramic images;

(5) storing the retrieved image data in an addressable buffer memory;

(6) selecting a plurality of picture elements stored in the buffer memory corresponding to the location and orientation of the observer within the predefined subspace; and

(7) displaying the selected plurality of picture elements on a display.

44. The method of claim 43, further comprising the steps of:

(8) selecting a second ones of the stored panoramic images in the storage medium when the observer changes from the predefined subspace and set of observer orientations to a second subspace and set of observer orientations associated with a second keypoint region in the imaginary space;

(9) retrieving image data corresponding to the second one of the stored panoramic images;

(10) storing the retrieved image data for the second one of the stored panoramic images in the addressable buffer memory;

(11) selecting a plurality of picture elements stored in the buffer memory corresponding to the location and orientation of the observer within the second subspace; and

(12) displaying the selected plurality of picture elements on a display.

63. The system of claim 44, wherein each one of said addresses for said addressable memory means generated by said keypoint address generator means comprises a first address portion corresponding to an azimuthal spatial coordinate in said imaginary space and a second address portion corresponding to an elevational spatial coordinate in said imaginary space.

64. The system of claim 63, wherein said second address portion includes a terrain address portion and a sky address portion, and wherein said sky address portion corresponds to data stored in said addressable memory means for generating a sky region of said imaginary space in said video image displayed to the observer.

65. The system of claim 64, further comprising sky address generator means responsive to said position signals and said attitude signals for generating said sky address portion when the picture elements visible to said observer in said simulator correspond to regions of sky in said imaginary space.

66. The system of claim 63, wherein said first address portion corresponds to a horizontal spatial coordinate of a geometric projection of a predefined region of said imaginary space, and said second address portion corresponds to a vertical spatial coordinate of said geometric projection of said predefined region of said imaginary space.

45. In an aircraft flight simulator system, a system for displaying images to simulate movement through an imaginary space, comprising:

data storage means for storing a plurality of sets of keypoint data, each of said keypoint data sets comprising a plurality of picture elements corresponding to a panoramic view from a predetermined location within said imaginary space;

means for receiving position signals and attitude signals form a host controller associated with said simulator system corresponding to an imaginary position and attitude within said imaginary space;

control means responsive to said position signals and said attitude signals for computing frame rate constant signals;

second control means responsive to said position signals and said attitude signals for selecting for display one of said plurality of sets of keypoint data associated with said imaginary position and attitude of said simulator within said imaginary space;

addressable memory means for storing data corresponding to said selected one of said plurality of sets of keypoint data;

keypoint address generator means responsive to said frame rate constant signals for generating addresses for said addressable memory means corresponding to a selected subset of said picture elements of said selected one of said plurality of sets of keypoint data stored in said addressable memory, said selected subset of said picture elements corresponding to particular ones of said picture elements visible to an observer in said simulator system positioned at said imaginary position and attitude within said imaginary space;

means for retrieving said selected subset of said picture elements form said addressable memory means from said addresses provided by said keypoint address generator means; and

means for displaying said selected subset of said picture elements to provide an image corresponding to the view visible to the observer in said simulator positioned at said imaginary position and attitude within said imaginary space.

46. The system of claim 45, wherein said data storage means memory comprises a video disc.

47. The system of claim 46, further comprising means for storing said keypoint data on said videodisc to maximize the signal-to-noise ratio.

48. The system of claim 47, wherein said storing means comprises a multiplexed analog component coder/decoder.

49. The system of claim 45 wherein said keypoint data comprises signals corresponding to color component video signals, and further comprising means for storing said signals in a predetermined storage format on said data storage means so as to maximize the signal-to-noise ratio.

50. The system of claim 49, wherein said predetermined storage format comprises a multiplexed analog component format.

51. The system of claim 45, further comprising antialiasing means for reducing the occurrence of sampling artifacts in said image displayed by said displaying means.

52. The system of claim 51, wherein said selected subset of said picture elements displayed by said displaying means comprises a predetermined number n of displayed picture elements, and wherein said antialiasing means comprises:

means for retrieving a predetermined number m of said picture elements from said addressable memory means for each one of said n displayed picture elements; and

means for filtering said predetermined number m of said picture elements to obtain each one of said n displayed picture elements.

53. The system of claim 52, where m is at least four.

54. The system of claim 52, wherein said filtering means is responsive to compute a resultant picture element R as a function of the values of said m picture elements.

55. The system of claim 51, wherein said addresses generated by said keypoint address generator means comprises an integer portion and a fractional portion, and wherein said antialiasing means comprises:

means responsive to said integer portion for retrieving a predetermined number m of pictures elements;

filter means responsive to said fractional portion for computing a resultant picture element R as a function of the values of said m picture elements.

56. The system of claim 55, wherein said filter means comprises a finite impulse response filter.

57. The system of claim 55, wherein m is at least four.

58. The system of claim 45 further comprising means for generating a haze overlay on a video image displayed by said display means to simulate haze conditions.

59. The system of claim 58, wherein said haze overlay generating means is selectable as to haze color.

60. The system of claim 59, wherein said haze overlay generating means is selectable as to haze intensity.

61. The system of claim 58, wherein said haze overlay generating means is responsive to an angle-to-horizon signal corresponding to the angle to the imaginary horizon of a point on said image displayed by said displaying means.

62. The system of claim 61, wherein said angle-to-horizon signal is F and has the form:

space="preserve" listing-type="equation">F=F1 * u+f2 * v+f3

where f1, f2, and f3 are frame rate constants, u is a horizontal pixel rate variable, and v is a vertical line rate variable.

67. The system of claim 45, wherein said frame rate constant signals have the form:

space="preserve" listing-type="equation">A=a1 * u+a2 * v+a3

space="preserve" listing-type="equation">B=b1 * u+b2 * v+b3

space="preserve" listing-type="equation">C=c1 * u+C2 * v+c3

where a1, a2, a3, b1, b2, b3, c1, c2, and c3 are frame rate constants, u is a horizontal pixel rate variable, and v is a vertical line rate variable.

68. The system of claim 67, further comprising row/column multiplier means responsive to said frame rate constants for calculating the equations for A, B, and C at pixel rates.

69. The system of claim 45, wherein said addressable memory means comprises a double buffered random access memory.

70. In a visual system for a vehicle simulator, said visual system generating a plurality of output images to simulate an imaginary space, a system for generating a haze overlay for said images to simulate variable visibility conditions, comprising:

means for providing distance signals corresponding to the distance from an observer in the simulator to a point in the imaginary space;

means responsive to said distance signals for computing a haze factor corresponding to the degree of haze to be imposed upon said images; and

output means responsive to said haze factor and input signals representing said images for providing output signals corresponding to said images overlaid with haze.

71. The system of claim 70, further comprising means for providing a haze color signal corresponding to a desired color of haze for said images, and wherein said output means is responsive to provide said output signals as a weighted average between said haze color signal and the brightness value of each picture element of said images as a function of said haze factor.

72. The system of claim 70, wherein said haze factor computing means comprises:

means for providing an angle-to-horizon signal corresponding to the angle to the imaginary horizon of a point on the output image, and means responsive to said angle-to-horizon signal for computing said haze factor.

73. The system of claim 72, wherein said angle-to-horizon signal is F and has the form:

space="preserve" listing-type="equation">F=f1 * u+f2 * v+f3

where f1, f2, and f3 are frame rate constants, u is a horizontal pixel rate variable, and v is a vertical line rate variable.

74. The system of claim 70, wherein said haze factor computing means is operative to provide said haze factor at a constant value for all portions of said image which are parallel to a horizon line in said imaginary space.

75. A method of generating a displayed image to simulate a view seen by an observer moving within an imaginary space, comprising the steps of:

(1) obtaining a panoramic image by mapping at least a portion of a panoramic view of a first surface to be simulated in the imaginary space onto a second mapping surface by providing as picture elements the picture values of areas of said mapping surface corresponding to areas of said first surface projected onto said mapping surface;

(2) storing said panoramic image on a storage medium;

(3) retrieving from said storage medium portions of said panoramic image associated with the point of view of an observer within the imaginary space; and

(4) displaying on a display a plurality of displayed picture elements formed from spatial modification of said retrieved portions of said panoramic image.

76. The method of claim 75, further comprising the steps of retrieving from said storage medium portions of said panoramic image associated with regions of the panoramic view corresponding to the sky when the observer'"'"'s position and attitude indicate that the field of view of the observer includes regions of the sky of the panoramic view, and displaying a plurality of picture element formed from said retrieved sky portions of said panoramic image.

77. The method of claim 75, wherein said mapping surface comprises a cone.

78. The method of claim 77, wherein said cone is a inverted conic surface, and wherein the uppermost limit of the conic surface is associated with the boundary between the sky and the horizon in said panoramic image.

79. The method of claim 75, wherein said panoramic image comprises a predetermined number N of stored picture elements, and wherein the step of retrieving portions of said panoramic image comprises the step of sampling said predetermined number N of stored picture elements to obtain a second predetermined number n of sampled picture elements for use in the displaying step, as a function of observer position and attitude with respect to the imaginary space.

80. The method of claim 75, wherein said panoramic image comprises all data required to reconstruct all possible views of an imaginary landscape within the imaginary space.

81. The method of claim 75, wherein the step of storing on said storage medium comprises storing a plurality of panoramic images, each of said panoramic images being associated with a different predetermined keypoint position in said imaginary space, and further comprising the step of selecting one of said plurality of said panoramic images as a function of observer position and attitude with respect to the imaginary space.

82. The method of claim 81, wherein the step of selecting one of said panoramic images comprises selecting said image as a function of observer position and attitude relative to said predetermined keypoint position.

83. The method of claim 81, wherein the step of selecting one of said panoramic images comprises selecting the image associated with the keypoint which is closest to the region of the imaginary space in the field of view of the observer.

84. The method of claim 75, wherein the displayed picture elements are displayed on a CRT.

85. The method of claim 75, wherein the step of displaying provides a first displayed image, and further comprising the step of retrieving from said storage medium other portions of said panoramic image associated with a second point of view of the observer within the imaginary space to stimulate movement from the first point of view of the observer to the second point of view.

86. A method of generating a video image to simulate movement within an imaginary space, comprising the steps of:

(1) mapping at least a portion of a panoramic view of a first surface to be simulated in the imaginary space onto a second mapping surface by providing as picture elements the picture values of areas of said mapping surface corresponding to areas of said first surface projected onto said mapping surface, a plurality of picture elements associated with a predetermined keypoint of view in the imaginary space corresponding to a panoramic image;

(2) storing on a storage medium a plurality of said panoramic images, each of said panoramic images being associated with a different one of a plurality of said predetermined keypoints of view within the imaginary space;

(3) selecting a particular one of said panoramic images stored on said storage medium as a first function of the spatial position and orientation of the field of view of an observer with respect to the imaginary space;

(4) retrieving said particular one of said panoramic images from said storage medium;

(5) selecting a particular subset of said plurality of picture elements of said particular retrieved one of said panoramic images as a second function of the spatial position and orientation of the field of view of the observer with respect to the imaginary space; and

(6) displaying said selected subset of said picture elements on a display.

87. The method of claim 86, wherein each of said plurality of panoramic images is a 360°

panoramic image.

88. The method of claim 86, wherein each of said plurality of panoramic images corresponds to a predetermined region in the imaginary space related to the spatial position and orientation of the field of views of the observer.

89. The method of claim 86, wherein said plurality of predetermined keypoints of view within said imaginary space are dispersed in a predetermined pattern throughout the imaginary space.

90. The method of claim 86, wherein said first function comprises a function related to a point on the ground in the imaginary space associated with the bottom of the field of view of the observer.

91. The method of claim 90, wherein said first function comprises the steps of mapping the imaginary ground of the imaginary space to the field of view of the observer;

determining the closet predetermined keypoint of view to a ground point in the imaginary space at the bottom of the field of view of the observer; and

selecting the predetermined keypoint of view for use in generating the image which is either (1) the closet predetermined keypoint of view to said ground point in the imaginary space or (2) no more than one keypoint of view away from said ground point in the imaginary space.