System and methods for performing electronic display stabilization via retained lightfield rendering

• US 10,419,747 B2
• Filed: 09/21/2016
• Issued: 09/17/2019
• Est. Priority Date: 12/22/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Parker, Evan Hardesty, Pharr, Matthew Milton, Lee, Johnny Chung, Donner, Craig, Erickson, Daniel, Kaeser, Dominik Philemon
• Primary Examiner(s): Vazquez Colon, Maria E
• Application Number:
  US15/271,813
• Publication Number:
  US 20170180721A1
• Time in Patent Office: 1,091 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

In a system having a user-portable display device, a method includes maintaining a lightfield data structure representing at least a portion of a four-dimensional (4D) lightfield for a three-dimensional (3D) world in association with a first pose of the user-portable display device relative to the 3D world. The method further includes determining a second pose of the user-portable display device relative to the 3D world, the second pose comprising an updated pose of the user-portable display device. The method additionally includes generating a display frame from the lightfield data structure based on the second pose, the display frame representing a field of view of the 3D world from the second pose.

Claims

1. A method comprising:

maintaining a lightfield data structure representing at least a portion of a four-dimensional (4D) lightfield for a three-dimensional (3D) world in association with a first pose of a user-portable display device relative to the 3D world, by;

determining a set of potential poses based on the first pose, the set of potential poses representing at least some of the poses that the user-portable display device is capable of taking within a specified time span by;

determining a probability distribution of potential poses for the specified time span; and

determining the set of potential poses from the probability distribution; and

for each potential pose of the set of potential poses, rendering a lightfield frame representing a field of view of the 3D world from the potential pose and incorporating the lightfield frame into the lightfield data structure;

determining a second pose of the user-portable display device relative to the 3D world based on the set of potential poses, the second pose comprising an updated pose of the user-portable display device;

generating a display frame from the lightfield data structure based on the second pose, the display frame representing a field of view of the 3D world from the second pose; and

displaying the generated display frame at the user-portable display device.

2. The method of claim 1, wherein:

rendering wherein rendering the lightfield frame comprises;

rendering the lightfield frame at a first rate; and

rendering the display frame comprises rendering the display frame at a second rate greater than the first rate.

3. The method of claim 1, wherein:

the user-portable display device comprises a head mounted display; and

determining the probability distribution of potential poses comprises determining the probability distribution of potential poses further based on potential poses of a user'"'"'s eyes for the specified time span.

4. The method of claim 1, wherein:

generating the display frame from the lightfield data structure based on the second pose comprises generating the display frame based at least in part on ray-tracing-based interpolation from at least one rendered lightfield frame represented in the lightfield data structure.

5. The method of claim 1, wherein:

the user-portable display device comprises a head mounted display having a left-eye display and a right-eye display; and

generating the display frame from the lightfield data structure based on the second pose comprises;

generating a first display frame for display on the left-eye display based on the second pose and an orientation of a left eye of the user relative to the second pose; and

generating a second display frame for display on the right-eye display based on the second pose and an orientation of a right eye of the user relative to the second pose.

6. A user-portable display device comprising:

a display;

a storage component;

an inertial sensor;

an application processor coupled to the inertial sensor and the storage component, the application processor configured to;

maintain, in the storage component, a lightfield data structure representing at least a portion of a four-dimensional (4D) lightfield of a three-dimensional (3D) world in association with one or more poses of the user-portable display device relative to the 3D world;

determine a set of potential poses, the set of potential poses representing at least some of the poses that the user-portable display device is capable of taking within a specified time span by;

determining a probability distribution of potential poses for the specified time span; and

determining the set of potential poses from the probability distribution; and

for each potential pose of the set of poses, generate a lightfield frame representing a field of view of the 3D world from the respective potential pose and incorporating the lightfield frame into the lightfield data structure; and

a compositor coupled to the storage component and the inertial sensor, the compositor configured to;

determine a current pose of the user-portable display device relative to the 3D world based on output from the inertial sensor, the current pose comprising an updated pose of the user-portable display device; and

generate a display frame from the lightfield data structure based on the current pose, the display frame representing a field of view of the 3D world from the current pose, wherein the user-portable display device is configured to display the generated display frame.

7. The user-portable display device of claim 6, wherein the application processor is further configured to:

generate lightfield frames at a first rate; and

render display frames at a second rate different than the first rate.

8. The user-portable display device of claim 6, wherein:

the user-portable display device comprises a head mounted display; and

the application processor is further configured to determine the probability distribution of potential poses further based on potential poses of a user'"'"'s eyes for the specified time span.

9. The user-portable display device of claim 6, wherein:

the application processor is further configured to generate the display frame based at least in part on ray-tracing-based interpolation from at least one rendered lightfield frame represented in the lightfield data structure.

10. The user-portable display device of claim 6, wherein:

the user-portable display device comprises a head mounted display having a left-eye display and a right-eye display; and

the application processor is further configured to;

generate a first display frame for display on the left-eye display based on the current pose and an orientation of a left eye of the user relative to the current pose; and

generate a second display frame for display on the right-eye display based on the current pose and an orientation of a right eye of the user relative to the current pose.

11. A head mounted display (HMD) system comprising:

a left-eye display and a right-eye display;

an inertial sensor; and

a processing system coupled to the left-eye display, right-eye display, and the inertial sensor, the processing system to;

determine a probability distribution of potential future poses of the HMD system based on a current pose of the HMD system;

within a specified time span, select a set of potential poses from the probability distribution;

render a set of lightfield frames based on the set of potential poses, each lightfield frame representing a view of a 3D world from a corresponding potential pose of the set;

determine an updated pose of the HMD system;

generate a first display frame for display at the left-eye display based on the set of lightfield frames and the updated pose; and

generate a second display frame for display at the right-eye display based on the set of lightfield frames and the updated pose.

12. The HMD system of claim 11, wherein:

the processing system is to generate the first display frame further based on an orientation of a left eye of a user; and

the processing system is to generate the second display frame further based on an orientation of a right eye of the user.