Predictive virtual reality display system with post rendering correction
First Claim
1. A predictive virtual reality display system with post rendering correction, comprising:
- at least one display viewable by a user;
at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;
a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to receive a region of interest; and
,calculate a predicted pose of said one or more body parts of said user at a future point in time, based onsaid sensor data generated by said at least one sensor,said future point in time; and
,said region of interest;
a 3D model of a scene;
a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured toobtain an initial predicted pose at a future display update point in time from said pose predictor; and
,calculate one or more 2D projections of said 3D model, based on said initial predicted pose; and
,an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured toreceive said initial predicted pose at said future display update point in time from said scene renderer;
receive said one or more 2D projections from said scene renderer;
obtain a revised predicted pose at said future display update point in time from said pose predictor;
calculate a change in pose between said initial predicted pose and said revised predicted pose;
generate a rerendering approximation of said one or more 2D projections based on said change in pose;
modify said one or more 2D projections based on said rerendering approximation; and
,transmit said one or more 2D projections to said at least one display.
3 Assignments
0 Petitions
Accused Products
Abstract
A virtual reality display system that generates display images in two phases: the first phase renders images based on a predicted pose at the time the display will be updated; the second phase re-predicts the pose using recent sensor data, and corrects the images based on changes since the initial prediction. The second phase may be delayed so that it occurs just in time for a display update cycle, to ensure that sensor data is as accurate as possible for the revised pose prediction. Pose prediction may extrapolate sensor data by integrating differential equations of motion. It may incorporate biomechanical models of the user, which may be learned by prompting the user to perform specific movements. Pose prediction may take into account a user'"'"'s tendency to look towards regions of interest. Multiple parallel pose predictions may be made to reflect uncertainty in the user'"'"'s movement.
-
Citations
20 Claims
-
1. A predictive virtual reality display system with post rendering correction, comprising:
-
at least one display viewable by a user; at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user; a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to receive a region of interest; and
,calculate a predicted pose of said one or more body parts of said user at a future point in time, based on said sensor data generated by said at least one sensor, said future point in time; and
,said region of interest; a 3D model of a scene; a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to obtain an initial predicted pose at a future display update point in time from said pose predictor; and
,calculate one or more 2D projections of said 3D model, based on said initial predicted pose; and
,an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to receive said initial predicted pose at said future display update point in time from said scene renderer; receive said one or more 2D projections from said scene renderer; obtain a revised predicted pose at said future display update point in time from said pose predictor; calculate a change in pose between said initial predicted pose and said revised predicted pose; generate a rerendering approximation of said one or more 2D projections based on said change in pose; modify said one or more 2D projections based on said rerendering approximation; and
,transmit said one or more 2D projections to said at least one display. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
-
-
18. A predictive virtual reality display system with post rendering correction, comprising:
-
at least one display viewable by a user; at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user; a pose predictor coupled to said at least one sensor and to a body model of said one or more body parts of said user, wherein said pose predictor is configured to calculate a predicted pose of said one or more body parts of said user at a future point in time, based on said sensor data generated by said at least one sensor, said future point in time, and, said body model; a 3D model of a scene; a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to obtain an initial predicted pose at a future display update point in time from said pose predictor; and
,calculate one or more 2D projections of said 3D model, based on said initial predicted pose; an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to receive said initial predicted pose at said future display update point in time from said scene renderer; receive said one or more 2D projections from said scene renderer; obtain a revised predicted pose at said future display update point in time from said pose predictor; calculate a change in pose between said initial predicted pose and said revised predicted pose; generate a rerendering approximation of said one or more 2D projections based on said change in pose; modify said one or more 2D projections based on said rerendering approximation; and
,transmit said one or more 2D projections to said at least one display; and
,a body model generator that is configured to analyze said sensor data to create said body model; prompt said user to execute one or more movements; and
,analyze said sensor data during said one or more movements to create said body model.
-
-
19. A predictive virtual reality display system with post rendering correction, comprising:
-
at least one display viewable by a user; at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user; a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to calculate a predicted pose of said one or more body parts of said user at a future point in time, based on said sensor data generated by said at least one sensor and based on said future point in time, and, calculate a plurality of predicted poses of said one or more body parts of said user at said future point in time; a 3D model of a scene; a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to obtain an initial predicted pose at a future display update point in time from said pose predictor; calculate one or more 2D projections of said 3D model, based on said initial predicted pose; an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to receive said initial predicted pose at said future display update point in time from said scene renderer; receive said one or more 2D projections from said scene renderer; obtain a revised predicted pose at said future display update point in time from said pose predictor; calculate a change in pose between said initial predicted pose and said revised predicted pose; generate a rerendering approximation of said one or more 2D projections based on said change in pose; modify said one or more 2D projections based on said rerendering approximation; and
,transmit said one or more 2D projections to said at least one display; wherein said scene renderer is further configured to receive a plurality of initial predicted poses at said future display update point in time from said pose predictor; and
,calculate one or more 2D projections of said 3D model for each initial predicted pose of said plurality of initial predicted poses; and
,wherein said image warper is further configured to receive said plurality of initial predicted poses at said future display update point in time from said pose predictor; receive said one or more 2D projections of said 3D model for each initial predicted pose of said plurality of initial predicted poses from said scene renderer; obtain said revised predicted pose at said future display update point in time from said pose predictor; select a closest initial predicted pose from said plurality of initial predicted poses that best matches said revised predicted pose; select one or more closest 2D projections that were rendered based on said closest initial predicted pose; calculate a change in pose between said closest initial predicted pose and said revised predicted pose; generate a rerendering approximation of said one or more closest 2D projections based on said change in pose; modify said one or more closest 2D projections based on said rerendering approximation; and
,transmit said one or more closest 2D projections to said at least one display.
-
-
20. A predictive virtual reality display system with post rendering correction, comprising:
-
at least one display viewable by a user; at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user; a pose predictor coupled to said at least one sensor, wherein said pose predictor is configured to calculate a predicted pose of said one or more body parts of said user at a future point in time, based on said sensor data generated by said at least one sensor and based on said future point in time; a 3D model of a scene; a scene renderer coupled to said at least one display, said pose predictor, and said 3D model, wherein said scene renderer is configured to obtain an initial predicted pose at a future display update point in time from said pose predictor; calculate one or more 2D projections of said 3D model, based on said initial predicted pose; an image warper coupled to said at least one display, said scene renderer, and said pose predictor, wherein said image warper is configured to receive said initial predicted pose at said future display update point in time from said scene renderer; receive said one or more 2D projections from said scene renderer; obtain a revised predicted pose at said future display update point in time from said pose predictor; calculate a change in pose between said initial predicted pose and said revised predicted pose; generate a rerendering approximation of said one or more 2D projections based on said change in pose; modify said one or more 2D projections based on said rerendering approximation; transmit said one or more 2D projections to said at least one display; partition at least one of said one or more 2D projections into a plurality of tiles; and
,for each tile of said plurality of tiles determine a tile display update point in time as a halfway point in a pixel update cycle of said tile on at least one display; obtain a tile revised predicted pose at said tile display update point in time from said pose predictor; calculate a tile change in pose between said initial predicted pose and said tile revised predicted pose; generate a rerendering approximation of said tile based on said tile change in pose; modify said tile based on said rerendering approximation of said tile; and
,transmit said tile to said at least one display.
-
Specification