Variable resolution virtual reality display system

• US 9,607,428 B2
• Filed: 10/01/2015
• Issued: 03/28/2017
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Ariadne's Thread (Usa), Inc.
• Inventors: Li, Adam
• Primary Examiner(s): Du, Haixia
• Application Number:
  US14/872,488
• Publication Number:
  US 20170004648A1
• Time in Patent Office: 544 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/20 : Perspective computation
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2016 : Rotation, translation, scaling
• CPC Class Codes: G06T 3/18 : Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting

Abstract

A virtual reality display system that renders images at different resolutions in different parts of a display. Reduces rendering latency by rendering at a lower resolution in selected regions, for example on the sides of a display where human vision has lower resolution than in the center. Pixels in low resolution regions are combined into grid elements, and rendering may generate grid element values rather than individual pixel values. Rendering may use ray casting, rasterization, or both. Variable resolution rendering may be combined with variable level of detail geometry models to further reduce rendering time. Selected objects may be designed as high resolution objects that are rendered at a high resolution even in low resolution display regions.

Claims

1. A variable resolution virtual reality display system comprising

at least one display viewable by a user and comprising a plurality of display pixels, wherein said plurality of display pixels is partitioned into at least two display regions;

wherein each display region of said at least two display regions is partitioned into a grid of grid elements, each grid element of said grid of grid elements comprising one or more display pixels; and

,

at least two of said at least two display regions have different ratios of display pixels per grid element;

a 3D model of a scene;

a scene renderer coupled to said at least one display, to said at least two display regions, and to said 3D model, wherein said scene renderer

calculates one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions, to form a grid element value for each grid element; and

,

for each grid element, assigns a pixel value to said one or more display pixels associated with said grid element based on said grid element value of said grid element;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

wherein said scene renderer is coupled to said pose analyzer, and said scene renderer receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data; and

,

calculates said one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions based on said one or more rendering virtual camera poses; and

,

an image warper coupled to said at least one display, said scene renderer, and said pose analyzer, wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said pose data from said pose analyzer;

calculates a change in pose between said one or more virtual camera poses and said pose data;

generates a rerendering approximation of said one or more 2D projections on said at least one display based on said change in pose;

modifies one or more pixels of said at least one display based on said rerendering approximation;

compares said change in pose to a threshold value; and

,

generates said rerendering approximation if said change in pose is below said threshold value.

2. The system of claim 1, wherein

said grid element value comprises one or more of a color and an intensity;

said pixel value comprises one or more of a color and an intensity; and

,

said assigns a pixel value to said one or more display pixels associated with said grid element comprises

copy said grid element value to said pixel value of each of said one or more display pixels associated with said grid element.

3. The system of claim 1, wherein

said at least two display regions comprise

a high resolution center region located in a center of a field of view of said user;

a low resolution left region located in a left portion of said field of view; and

,

a low resolution right region located in a right portion of said field of view;

a ratio of display pixels per grid element for said low resolution left region is greater than said ratio of display pixels per grid element for said high resolution center region; and

,

a ratio of display pixels per grid element for said low resolution right region is greater than said ratio of display pixels per grid element for said high resolution center region.

4. The system of claim 3, wherein

said ratio of display pixels per grid element for said high resolution center region is 1:

1.

5. The system of claim 3, wherein

said ratio of display pixels per grid element for said low resolution left region is at least 4:

1; and

,

said ratio of display pixels per grid element for said low resolution right region equals said ratio of display pixels per grid element for said low resolution left region.

6. The system of claim 1, wherein

at least one object in said 3D model comprises at least two geometry models at different levels of detail;

for each display region of said at least two display regions, said scene renderer selects a level of detail for said at least one object; and

,

calculates a 2D projection of said at least one object onto said grid of grid elements using a geometry model for said level of detail associated with said at least one object; and

,

said scene renderer selects at least two different levels of detail for said at least one object for at least two different display regions.

7. The system of claim 1, wherein said calculates said one or more 2D projections of said 3D model onto said grid of grid elements for at least one of said at least two display regions comprises

uses ray casting to project a ray from a point within each grid element to a first intersected object in said 3D model;

and,

sets said grid element value for each grid element based on one or more characteristics of said first intersected object in said 3D model at an intersection between said ray and said first intersected object.

8. The system of claim 1, wherein said calculates said one or more 2D projections of said 3D model onto said grid of grid elements comprises projects one or more 3D geometric primitives from said 3D model onto an image plane corresponding to a display region, to form one or more projected primitives;

rasterizes said one or more projected primitives to form one or more grid element fragments, each associated with

a grid element; and

,

one or more display pixels within said grid element; and

,

blends said one or more grid element fragments associated with each grid element to form said grid element value for said grid element.

9. The system of claim 8, wherein

said 3D model comprises at least one high resolution object and at least one low resolution object;

each fragment formed from projection and rasterization of said at least one high resolution object is associated with a single display pixel; and

,

each fragment formed from projection and rasterization of said at least one low resolution object is associated with all display pixels within said grid element.

10. The system of claim 1, wherein said rerendering approximation comprises

calculating a pixel translation vector;

and,

translating one or more pixels of said one or more 2D projections by said pixel translation vector.

11. The system of claim 10, wherein calculating said pixel translation vector comprises

approximating said change in pose as a rotation around a unit vector {circumflex over (ω

)} comprising {circumflex over (ω

)}_y and {circumflex over (ω

)}_x by an angle Δ

θ

;

calculating a spatial translation vector ({circumflex over (ω

)}_yΔ

θ

, −

{circumflex over (ω

)}_xΔ

θ

);

calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections; and

,

calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.

12. A variable resolution virtual reality display system comprising

at least one display viewable by a user and comprising a plurality of display pixels, wherein said plurality of display pixels is partitioned into at least two display regions;

wherein each display region of said at least two display regions is partitioned into a grid of grid elements, each grid element of said grid of grid elements comprising one or more display pixels; and

,

at least two of said at least two display regions have different ratios of display pixels per grid element;

a 3D model of a scene;

a scene renderer coupled to said at least one display, to said at least two display regions, and to said 3D model, wherein said scene renderer

calculates one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions, to form a grid element value for each grid element; and

,

for each grid element, assigns a pixel value to said one or more display pixels associated with said grid element based on said grid element value of said grid element;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

wherein said scene renderer is coupled to said pose analyzer, and said scene renderer receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data; and

,

calculates said one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions based on said one or more rendering virtual camera poses; and

,

an image warper coupled to said at least one display, said scene renderer, and said pose analyzer, wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said pose data from said pose analyzer;

calculates a change in pose between said one or more virtual camera poses and said pose data;

generates a rerendering approximation of said one or more 2D projections on said at least one display based on said change in pose;

modifies one or more pixels of said at least one display based on said rerendering approximation;

wherein said rerendering approximation comprises

calculating a pixel translation vector; and

,

translating one or more pixels of said one or more 2D projections by said pixel translation vector; and

,

wherein calculating said pixel translation vector comprises

approximating said change in pose as a rotation around a unit vector) comprising {circumflex over (ω

)}_y and {circumflex over (ω

)}_x by an angle Δ

θ

;

calculating a spatial translation vector ({circumflex over (ω

)}_yΔ

θ

, −

{circumflex over (ω

)}_xΔ

θ

);

calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections; and

,

calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.

13. The system of claim 12, wherein

said grid element value comprises one or more of a color and an intensity;

said pixel value comprises one or more of a color and an intensity; and

,

said assigns a pixel value to said one or more display pixels associated with said grid element comprises

copy said grid element value to said pixel value of each of said one or more display pixels associated with said grid element.

14. The system of claim 12, wherein

said at least two display regions comprise

a high resolution center region located in a center of a field of view of said user;

a low resolution left region located in a left portion of said field of view; and

,

a low resolution right region located in a right portion of said field of view;

a ratio of display pixels per grid element for said low resolution left region is greater than said ratio of display pixels per grid element for said high resolution center region; and

,

a ratio of display pixels per grid element for said low resolution right region is greater than said ratio of display pixels per grid element for said high resolution center region.

15. The system of claim 14, wherein

said ratio of display pixels per grid element for said high resolution center region is 1:

1.

16. The system of claim 14, wherein

said ratio of display pixels per grid element for said low resolution left region is at least 4:

1; and

,

said ratio of display pixels per grid element for said low resolution right region equals said ratio of display pixels per grid element for said low resolution left region.

17. The system of claim 12, wherein

at least one object in said 3D model comprises at least two geometry models at different levels of detail;

for each display region of said at least two display regions, said scene renderer

selects a level of detail for said at least one object; and

,

calculates a 2D projection of said at least one object onto said grid of grid elements using a geometry model for said level of detail associated with said at least one object; and

,

said scene renderer selects at least two different levels of detail for said at least one object for at least two different display regions.

18. The system of claim 12, wherein said calculates one or more 2D projections of said 3D model onto said grid of grid elements for at least one of said at least two display regions comprises

uses ray casting to project a ray from a point within each grid element to a first intersected object in said 3D model;

and,

sets said grid element value for each grid element based on one or more characteristics of said first intersected object in said 3D model at an intersection between said ray and said first intersected object.

19. The system of claim 12, wherein said calculates one or more 2D projections of said 3D model onto said grid of grid elements comprises

projects one or more 3D geometric primitives from said 3D model onto an image plane corresponding to a display region, to form one or more projected primitives;

rasterizes said one or more projected primitives to form one or more grid element fragments, each associated with

a grid element; and

,

one or more display pixels within said grid element; and

,

blends said one or more grid element fragments associated with each grid element to form said grid element value for said grid element.

20. The system of claim 19, wherein

said 3D model comprises at least one high resolution object and at least one low resolution object;

each fragment formed from projection and rasterization of said at least one high resolution object is associated with a single display pixel; and

,

each fragment formed from projection and rasterization of said at least one low resolution object is associated with all display pixels within said grid element.

21. The system of claim 12, wherein said image warper

compares said change in pose to a threshold value;

and,

generates said rerendering approximation if said change in pose is below said threshold value.