Method and apparatus for head tracked display of precomputed stereo images

• US 5,287,437 A
• Filed: 06/02/1992
• Issued: 02/15/1994
• Est. Priority Date: 06/02/1992
• Status: Expired due to Term

• Current Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Deering, Michael
• Primary Examiner(s): Harkcom, Gary V.
• Assistant Examiner(s): SMITH, MICHAEL
• Application Number:
  US07/893,196
• Time in Patent Office: 623 Days
• Field of Search: 395/127, 358/222, 358/88, 358/92, 358/103, 358/104
• US Class Current: 345/427
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/32 using arrays of controllable light sources; using moving apertures or moving light sources
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 13/368 for two or more viewers
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/378 for tracking rotational head movements around an axis perpendicular to the screen
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A method and apparatus for generating complex 3D stereo images in response to real time viewer head movement by dynamically accessing pairs of stereo images selected from among a precomputed set of images. The precomputed images correspond to a predetermined set of possible viewpoints, and are accessed based upon a prediction of viewer head movement. The arrangement of predetermined viewpoints may be along one, two, or three dimensions.

Claims

1. A method for generating a head tracked stereo image, comprising the steps of:

(a) generating precomputed images corresponding to a set of predetermined viewpoints of a virtual object, such that the predetermined viewpoints and the virtual object are defined in a display plate coordinate space registered to a display screen of a display device;

(b) determining a time index for displaying the precomputed images in stereo on the display device;

(c) predicting an eye first nodal point for each eye of a viewer corresponding to the time index, such that the eye first nodal points are located in the display plate coordinate space;

(d) determining the predetermined viewpoints having the closest proximity in the display plate coordinate space to the eye first nodal points;

(e) accessing the precomputed images corresponding to the predetermined viewpoints having the closest proximity in the display plate coordinate space to the eye first nodal points, and transferring the precomputed images to a frame buffer;

(f) displaying the precomputed images in the frame buffer in stereo on the display device.

2. The method of claim 1, wherein step (a) comprises the steps of:

(g) selecting an arrangement in the display plate coordinate space for the predetermined viewpoints;

(h) rendering the precomputed images corresponding to the predetermined viewpoints of the virtual object using a graphics rendering system;

(i) storing the precomputed images, such that the precomputed images are accessed by specifying the corresponding predetermined viewpoint.

3. The method of claim 2, wherein step (b) comprises the step of generating the precomputed images corresponding to the predetermined viewpoints of the virtual object by image sampling a real object having the appearance of the virtual object.

4. The method of claim 2, further comprising the step of performing a data compression on the precomputed images after step (b) and before step (c).

5. The method of claim 1, wherein step (c) comprises the steps of:

(g) sensing dynamic locations and orientations of the viewer in the display plate coordinate space;

(h) determining a location and orientation of the viewer for a next frame of the display device by performing an interpolation of the dynamic locations and orientations of the viewer;

(i) determining locations of the eyeballs of the viewer in the display plate coordinate space for the next frame, based upon the location and orientation of the viewer for the next frame of the display device, and a predetermined location of the eyeballs of the viewer relative to the location of the head of the viewer;

(j) determining locations of the eye first nodal points of the viewer in the display plate coordinate space based upon the locations of the eyeballs.

6. The method of claim 5, wherein the interpolation in step (b) is reduced by a filter function at small velocities of the head of the viewer in order to reduce jitter of a stereo image when the viewer'"'"'s head is still.

7. The method of claim 1, wherein step (e) comprises the steps of:

(g) retrieving the precomputed images from a storage means, such that the precomputed images are in a compressed format;

(h) decompressing the precomputed images retrieved from the storage means;

(i) storing the decompressed precomputed images into the frame buffer, such that the frame buffer is double buffered and the decompressed precomputed images are transferred to a first buffer in the frame buffer while a second buffer in the frame buffer is accessed by the display device.

8. A method for generating a head tracked stereo image, comprising the steps of:

(a) predicting an eye first nodal point for each eye of a viewer, such that the eye first nodal points correspond to points in a display plate coordinate space registered to a display screen of a display device;

(b) comparing the eye first nodal points to a set of predetermined viewpoints defined in the display plate coordinate space, each of the predetermined viewpoints corresponding to a viewpoint of a video camera generating a live video signal;

(c) determining the predetermined viewpoints having the closest proximity in the display plate coordinate space to the eye first nodal points;

(d) selectively coupling to the display device the video signals from the video cameras corresponding to the predetermined viewpoints having the closest proximity in the display plate coordinate space to the eye first nodal points;

(e) displaying the video signals in stero on the display device.

9. An apparatus for generating a head tracked stereo image, comprising:

means for generating precomputed images corresponding to a set of predetermined viewpoints of a virtual object, such that the predetermined viewpoints and the virtual object are defined in a display plate coordinate space registered to a display screen of a display device;

means for determining a time index for displaying the precomputed images in stereo on the display device;

means for predicting an eye first nodal point for each eye of a viewer corresponding to the time index, such that the eye first nodal points are located in the display plate coordinate space;

means for determining the predetermined viewpoints having the closest proximity in the display plate coordinate space to the eye first nodal points;

means for accessing the precomputed images corresponding to the predetermined viewpoints having the closest proximity in the display plate coordinate space to the eye first nodal points, and transferring the precomputed images to a frame buffer;

means for displaying the precomputed images in the frame buffer in stereo on the display device.

10. The apparatus of claim 9, wherein the means for generating precomputed images corresponding to a set of predetermined viewpoints of a virtual object comprises:

means for selecting an arrangement in the display plate coordinate space for the predetermined viewpoints;

graphics rendering means, the graphics rendering means generating the precomputed images corresponding to the predetermined viewpoints of the virtual object;

means for storing the precomputed images, such that the precomputed images are accessed by specifying the corresponding predetermined viewpoint.

11. The apparatus of claim 10, wherein the graphics rendering means comprises means for image sampling a real object having the appearance of the virtual object.

12. The apparatus of claim 10, further comprising means for performing a data compression on the precomputed images.

13. The apparatus of claim 9, wherein the means for predicting an eye first nodal point for each eye of a viewer comprises:

means for sensing dynamic locations and orientations of the viewer in the display plate coordinate space;

means for determining a location and orientation of the viewer for a next frame of the display device by performing an interpolation of the dynamic locations and orientations of the viewer;

means for determining locations of the eyeballs of the viewer in the display plate coordinate space for the next frame, based upon the location and orientation of the head of the viewer for the next frame of the display device, and a predetermined location of the eyeballs of the viewer relative to the location of the head of the viewer;

means for determining locations of the eye first nodal points of the viewer in the display plate coordinate space based upon the locations of the eyeballs.

14. The apparatus of claim 13, wherein the interpolation is reduced by a filter function at small velocities of the head of the viewer in order to reduce jitter of a stereo image when the viewer'"'"'s head is still.

15. The apparatus of claim 13, wherein interocular distance for the viewer is used in addition to the location and orientation of the head of the viewer for the next frame of the display device to accurately determine the eyeball locations.

16. The apparatus of claim 9, wherein the means for accessing the precomputed images comprises:

means for retrieving the precomputed images from a storage means, such that the precomputed images are in a compressed format;

means for decompressing the precomputed images retrieved from the storage means;

means for storing the decompressed precomputed images into the frame buffer, such that the frame buffer is double buffered and the decompressed precomputed images are transferred to a first buffer in the frame buffer while a second buffer in the frame buffer is accessed by the display device.