Predictive virtual reality display system with post rendering correction

• US 10,089,790 B2
• Filed: 09/28/2016
• Issued: 10/02/2018
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Ariadne's Thread (Usa), Inc.
• Inventors: Lawson, David, Li, Adam
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/279,079
• Publication Number:
  US 20170018121A1
• Time in Patent Office: 734 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 9/31 Projection devices for colour picture display , e.g. using electronic spatial light modulators [ESLM]

Abstract

A virtual reality display system that generates display images in two phases: the first phase renders images based on a predicted pose at the time the display will be updated; the second phase re-predicts the pose using recent sensor data, and corrects the images based on changes since the initial prediction. The second phase may be delayed so that it occurs just in time for a display update cycle, to ensure that sensor data is as accurate as possible for the revised pose prediction. Pose prediction may extrapolate sensor data by integrating differential equations of motion. It may incorporate biomechanical models of the user, which may be learned by prompting the user to perform specific movements. Pose prediction may take into account a user'"'"'s tendency to look towards regions of interest. Multiple parallel pose predictions may be made to reflect uncertainty in the user'"'"'s movement.

Claims

1. A predictive virtual reality display system with post rendering correction, comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to receive a region of interest; and

,

calculate a predicted pose of said one or more body parts of said user at a future point in time, based on

said sensor data generated by said at least one sensor,

said future point in time; and

,

said region of interest;

a 3D model of a scene;

a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to

obtain an initial predicted pose at a future display update point in time from said pose predictor; and

,

calculate one or more 2D projections of said 3D model, based on said initial predicted pose; and

,

an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to

receive said initial predicted pose at said future display update point in time from said scene renderer;

receive said one or more 2D projections from said scene renderer;

obtain a revised predicted pose at said future display update point in time from said pose predictor;

calculate a change in pose between said initial predicted pose and said revised predicted pose;

generate a rerendering approximation of said one or more 2D projections based on said change in pose;

modify said one or more 2D projections based on said rerendering approximation; and

,

transmit said one or more 2D projections to said at least one display.

2. The system of claim 1 wherein said one or more body parts of said user comprise a head of said user.

3. The system of claim 1 wherein said modify said one or more 2D projections is further based on a lens distortion correction.

4. The system of claim 1 wherein said future display update point in time is a halfway point in a pixel update cycle of said at least one display.

5. The system of claim 1 wherein said image warper is further configured to delay said obtain said revised predicted pose until a just in time pose revision time, wherein said just in time pose revision time is a latest time that allows said transmit said one or more 2D projections to said at least one display to occur by said future display update point in time.

6. The system of claim 5 wherein said occur by said future display update point in time comprises update half of pixels of said at least one display by said future display update point in time.

7. The system of claim 1 wherein said calculate a predicted pose of said one or more body parts of said user at a future point in time comprises

calculate a current pose from said sensor data;

calculate one or more derivatives of pose from said sensor data; and

,

extrapolate said current pose and said one or more derivatives of pose to said future point in time to form said predicted pose.

8. The system of claim 7 wherein

said at least one sensor comprises a rate gyroscope;

and,

said calculate said one or more derivatives of pose comprises

obtain an angular velocity from said rate gyroscope; and

,

set a first derivative of pose of said one or more derivatives to said angular velocity.

9. The system of claim 7 wherein said extrapolate said current pose and said one or more derivatives of pose comprises

apply one or both of an Euler'"'"'s method or a Runge-Kutta method to numerically integrate said one or more derivatives of pose.

10. The system of claim 1 wherein

said pose predictor is further coupled to a body model of said one or more body parts of said user;

and,

said calculate a predicted pose of said one or more body parts of said user at a future point in time is further based on said body model.

11. The system of claim 10 wherein said body model comprises one or more of a limit on a range of motion of said one or more body parts;

and,

a model of forces applied by or applied to said one or more body parts.

12. The system of claim 10 further comprising a body model generator that is configured to analyze said sensor data to create said body model.

13. The system of claim 12 wherein said body model generator is further configured to

prompt said user to execute one or more movements;

and,

analyze said sensor data during said one or more movements to create said body model.

14. The system of claim 1 wherein said region of interest is based on one or more of

a location in said 3D model with action;

a location in said 3D model that emits a sound; and

,

a location of a menu item on said at least one display or in said 3D model.

15. The system of claim 1 wherein said pose predictor is further configured to calculate a plurality of predicted poses of said one or more body parts of said user at said future point in time.

16. The system of claim 15 wherein

said scene renderer is further configured to

receive a plurality of initial predicted poses at said future display update point in time from said pose predictor;

and,

calculate one or more 2D projections of said 3D model for each initial predicted pose of said plurality of initial predicted poses; and

,

said image warper is further configured to

receive said plurality of initial predicted poses at said future display update point in time from said pose predictor;

receive said one or more 2D projections of said 3D model for each initial predicted pose of said plurality of initial predicted poses from said scene renderer;

obtain said revised predicted pose at said future display update point in time from said pose predictor;

select a closest initial predicted pose from said plurality of initial predicted poses that best matches said revised predicted pose;

select one or more closest 2D projections that were rendered based on said closest initial predicted pose;

calculate a change in pose between said closest initial predicted pose and said revised predicted pose;

generate a rerendering approximation of said one or more closest 2D projections based on said change in pose;

modify said one or more closest 2D projections based on said rerendering approximation; and

,

transmit said one or more closest 2D projections to said at least one display.

17. The system of claim 1 wherein said image warper is further configured to

partition at least one of said one or more 2D projections into a plurality of tiles;

and,

for each tile of said plurality of tiles

determine a tile display update point in time as a halfway point in a pixel update cycle of said tile on at least one display;

obtain a tile revised predicted pose at said tile display update point in time from said pose predictor;

calculate a tile change in pose between said initial predicted pose and said tile revised predicted pose;

generate a rerendering approximation of said tile based on said tile change in pose;

modify said tile based on said rerendering approximation of said tile; and

,

transmit said tile to said at least one display.

18. A predictive virtual reality display system with post rendering correction, comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose predictor coupled to said at least one sensor and to a body model of said one or more body parts of said user, wherein said pose predictor is configured to calculate a predicted pose of said one or more body parts of said user at a future point in time, based on

said sensor data generated by said at least one sensor,

said future point in time, and,

said body model;

a 3D model of a scene;

a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to

obtain an initial predicted pose at a future display update point in time from said pose predictor; and

,

calculate one or more 2D projections of said 3D model, based on said initial predicted pose;

an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to

receive said initial predicted pose at said future display update point in time from said scene renderer;

receive said one or more 2D projections from said scene renderer;

obtain a revised predicted pose at said future display update point in time from said pose predictor;

calculate a change in pose between said initial predicted pose and said revised predicted pose;

generate a rerendering approximation of said one or more 2D projections based on said change in pose;

modify said one or more 2D projections based on said rerendering approximation; and

,

transmit said one or more 2D projections to said at least one display; and

,

a body model generator that is configured to

analyze said sensor data to create said body model;

prompt said user to execute one or more movements; and

,

analyze said sensor data during said one or more movements to create said body model.

19. A predictive virtual reality display system with post rendering correction, comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to

calculate a predicted pose of said one or more body parts of said user at a future point in time, based on said sensor data generated by said at least one sensor and based on said future point in time, and,

calculate a plurality of predicted poses of said one or more body parts of said user at said future point in time;

a 3D model of a scene;

a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to

obtain an initial predicted pose at a future display update point in time from said pose predictor;

calculate one or more 2D projections of said 3D model, based on said initial predicted pose;

an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to

receive said initial predicted pose at said future display update point in time from said scene renderer;

receive said one or more 2D projections from said scene renderer;

obtain a revised predicted pose at said future display update point in time from said pose predictor;

calculate a change in pose between said initial predicted pose and said revised predicted pose;

generate a rerendering approximation of said one or more 2D projections based on said change in pose;

modify said one or more 2D projections based on said rerendering approximation; and

,

transmit said one or more 2D projections to said at least one display;

wherein said scene renderer is further configured to

receive a plurality of initial predicted poses at said future display update point in time from said pose predictor; and

,

calculate one or more 2D projections of said 3D model for each initial predicted pose of said plurality of initial predicted poses; and

,

wherein said image warper is further configured to

receive said plurality of initial predicted poses at said future display update point in time from said pose predictor;

receive said one or more 2D projections of said 3D model for each initial predicted pose of said plurality of initial predicted poses from said scene renderer;

obtain said revised predicted pose at said future display update point in time from said pose predictor;

select a closest initial predicted pose from said plurality of initial predicted poses that best matches said revised predicted pose;

select one or more closest 2D projections that were rendered based on said closest initial predicted pose;

calculate a change in pose between said closest initial predicted pose and said revised predicted pose;

generate a rerendering approximation of said one or more closest 2D projections based on said change in pose;

modify said one or more closest 2D projections based on said rerendering approximation; and

,

transmit said one or more closest 2D projections to said at least one display.

20. A predictive virtual reality display system with post rendering correction, comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to calculate a predicted pose of said one or more body parts of said user at a future point in time, based on said sensor data generated by said at least one sensor and based on said future point in time;

a 3D model of a scene;

a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to

obtain an initial predicted pose at a future display update point in time from said pose predictor;

calculate one or more 2D projections of said 3D model, based on said initial predicted pose;

an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to

receive said initial predicted pose at said future display update point in time from said scene renderer;

receive said one or more 2D projections from said scene renderer;

obtain a revised predicted pose at said future display update point in time from said pose predictor;

calculate a change in pose between said initial predicted pose and said revised predicted pose;

generate a rerendering approximation of said one or more 2D projections based on said change in pose;

modify said one or more 2D projections based on said rerendering approximation;

transmit said one or more 2D projections to said at least one display;

partition at least one of said one or more 2D projections into a plurality of tiles; and

,

for each tile of said plurality of tiles

determine a tile display update point in time as a halfway point in a pixel update cycle of said tile on at least one display;

obtain a tile revised predicted pose at said tile display update point in time from said pose predictor;

calculate a tile change in pose between said initial predicted pose and said tile revised predicted pose;

generate a rerendering approximation of said tile based on said tile change in pose;

modify said tile based on said rerendering approximation of said tile; and

,

transmit said tile to said at least one display.