Time-warping adjustment based on depth information in a virtual/augmented reality system

• US 10,453,272 B2
• Filed: 05/26/2017
• Issued: 10/22/2019
• Est. Priority Date: 05/29/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bates, John Paul
• Primary Examiner(s): Liu, Gordon G
• Application Number:
  US15/606,569
• Publication Number:
  US 20170345220A1
• Time in Patent Office: 879 Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A technique includes determining a depth value for each of a plurality of pixels of a frame, down-sampling the depth values of a tile of the frame to obtain a plurality of down-sampled depth values, the frame including one or more tiles, determining a change in a head pose, determining, from the plurality of down-sampled depth values, a down-sampled depth value for a vertex, determining an adjusted position for the vertex based on the change in head pose and the down-sampled depth value for the vertex, performing, based on at least the adjusted position for the vertex, a depth-adjusted time-warping of the frame to obtain a depth-adjusted time-warped frame, and triggering display of the depth-adjusted time-warped frame.

Claims

1. A method of adjusting time-warping of a frame for a virtual reality or augmented reality system based on depth information, the method including:

rendering a frame having a plurality of pixels, the rendering including determining a depth value for each of the plurality of pixels of the frame, the frame being associated with a frame vertex grid that divides the frame into a plurality of polygons;

down-sampling the depth values of the frame to obtain a plurality of down-sampled depth values;

determining a change in a head pose;

determining, from the plurality of down-sampled depth values, a down-sampled depth value for a vertex of the frame vertex grid;

determining an adjusted position for the vertex of the frame vertex grid based on the change in head pose and the down-sampled depth value for the vertex of the frame vertex grid;

performing, based on at least the adjusted position for the vertex of the frame vertex grid, a depth-adjusted time-warping of the frame to obtain a depth-adjusted time-warped frame; and

triggering display of the depth-adjusted time-warped frame.

2. The method of claim 1,

wherein the determining, from the plurality of down-sampled depth values, a down-sampled depth value for a vertex comprises:

determining a coordinate location of the vertex; and

determining, from the plurality of down-sampled depth values, a down-sampled depth value for the coordinate location as a down-sampled depth value for the vertex.

3. The method of claim 1,

wherein the determining, from the plurality of down-sampled depth values, a down-sampled depth value for a vertex comprises:

determining a first coordinate location of a first vertex;

determining, from the plurality of down-sampled depth values, a first down-sampled depth value for the first coordinate location as a down-sampled depth value for the first vertex;

determining a second coordinate location of a second vertex; and

determining, from the plurality of down-sampled depth values, a second down-sampled depth value for the second coordinate location as a down-sampled depth value for the second vertex;

wherein the determining an adjusted position for the vertex comprises;

determining an adjusted position for the first vertex based on the change in head pose and the first down-sampled depth value; and

determining an adjusted position for the second vertex based on the change in head pose and the second down-sampled depth value.

4. The method of claim 1,

wherein the determining, from the plurality of depth values of the frame, a down-sampled depth value for a vertex comprises determining a first down-sampled depth value for a first vertex and a second down-sampled depth value for a second vertex;

wherein the determining an adjusted position for the vertex based on the change in head pose and the down-sampled depth value for the vertex comprises;

determining an adjusted position for the first vertex and an adjusted position of the second vertex based on the first down-sampled depth value and the second down-sampled depth value, respectively; and

wherein the performing comprises performing, based on at least the adjusted position for the first vertex and the adjusted position for the second vertex, a depth-adjusted time-warping of the frame to obtain a depth-adjusted time-warped frame.

5. The method of claim 1, wherein the determining a change in a head pose comprises:

determining an initial head pose information;

rendering the frame based on color values of the plurality of pixels and the initial head pose information; and

determining, during or after the rendering, an updated head pose information, where a change in head pose is based on a difference between the updated head pose information and the initial head pose information.

6. The method of claim 1, wherein down-sampling comprises:

determining a tile for the frame; and

down-sampling the depth values of the tile to obtain one or more down-sampled depth values.

7. The method of claim 1, wherein the time-warping comprises rotational time-warping due to rotation of a user'"'"'s head, and positional time-warping due to translation or change in location of a user'"'"'s eyes, wherein the positional time-warping is scaled or adjusted based on a depth value for one or more of the vertexes.

8. The method of claim 1, wherein the adjusted position for the vertex is based on a position offset for the vertex, the position offset for the vertex including a positional time-warp component that is scaled or adjusted based on the down-sampled depth value for the vertex.

9. The method of claim 1, wherein the adjusted position for the vertex comprises an adjusted x, y position based on an x, y offset from an initial x, y position of the vertex.

10. The method of claim 1, wherein the determining an adjusted position for the vertex comprises:

determining, based on a transform matrix, the adjusted position for the vertex based on a change in the head pose and an initial position of the vertex, wherein the transform matrix and the adjusted position for the vertex is adjusted based on the down-sampled depth value for the vertex.

11. The method of claim 1, wherein the vertex comprises a first vertex, wherein the first vertex and a second vertex are provided for a polygon within the frame, wherein an adjusted position is determined for each of the first vertex and the second vertex, and wherein the performing a depth-adjusted time-warping on the frame comprises performing the following for the polygon:

determining adjusted positions for pixels within the polygon by interpolating between an adjusted position of the first vertex and an adjusted position of the second vertex.

12. The method of claim 1,

wherein the vertex is provided at an x, y frame coordinate, and wherein the adjusted position for the vertex comprises an adjusted x, y frame coordinate of the vertex based on an x, y offset, wherein the x, y offset is determined based on a change in head pose and a depth value at the x, y coordinate at the vertex.

13. The method of claim 1, wherein the vertex comprises a first vertex, wherein the first vertex and a second vertex are provided for a polygon within the frame,

wherein the first vertex is provided at a first x, y frame coordinate, and wherein an adjusted position for the first vertex comprises a first adjusted x, y frame coordinate of the first vertex based on a first x, y offset, wherein the first x, y offset is determined based on a change in head pose and a depth value of the first x, y coordinate;

wherein the second vertex is provided at a second x, y frame coordinate, and wherein an adjusted position for the second vertex comprises a second adjusted x, y frame coordinate of the second vertex based on a second x, y offset, wherein the second x, y offset is determined based on a change in head pose and a depth value of the second x, y coordinate;

and wherein the performing a depth-adjusted time-warping on the frame comprises determining an x, y offset for one or more pixels within the polygon by interpolating between the first x, y offset of the first vertex and the second x, y offset of the second vertex.

14. The method of claim 1,

wherein the vertex comprises a first vertex provided at a first x, y frame coordinate, wherein a second vertex is provided at a second x, y frame coordinate, the first vertex and the second vertex provided for a polygon;

wherein an adjusted position for the first vertex comprises a first adjusted x, y frame coordinate of the first vertex based on a first x, y offset that is determined based on a change in head pose and a depth value at the first x, y coordinate;

wherein an adjusted position for the second vertex comprises a second adjusted x, y frame coordinate of the second vertex based on a second x, y offset that is determined based on the change in head pose and a depth value at the second x, y coordinate;

and wherein the performing a depth-adjusted time-warping on the frame comprises;

determining an x, y offset for a pixel within the polygon by interpolating between the first x, y offset of the first vertex and the second x, y offset of the second vertex; and

adjusting a position of the pixel within the polygon based on the x, y offset for the pixel.

15. The method of claim 1 wherein the method further comprises:

generating, by a head-mounted audio visual device, a virtual world immersive experience within a virtual space; and

wherein the displaying comprises;

displaying, by the head-mounted audio visual device on a display device, the depth-adjusted time-warped frame.

16. An apparatus comprising at least one processor and at least one memory including computer instructions that, when executed by the at least one processor, cause the apparatus to:

render a frame having a plurality of pixels, the frame being associated with a frame vertex grid that divides the frame into a plurality of polygons;

determine a depth value for each of the plurality of pixels of the frame;

down-sample the depth values of the frame to obtain a plurality of down-sampled depth values;

determine a change in a head pose;

determine, from the plurality of down-sampled depth values, a down-sampled depth value for a vertex of the frame vertex grid;

determine an adjusted position for the vertex of the frame vertex grid based on the change in head pose and the down-sampled depth value for the vertex of the frame vertex grid;

perform, based on at least the adjusted position for the vertex of the frame vertex grid, a depth-adjusted time-warping of the frame to obtain a depth-adjusted time-warped frame; and

trigger display of the depth-adjusted time-warped frame.

17. The apparatus of claim 16, wherein the instructions that cause the apparatus to determine a change in a head pose include instructions that cause the computing device to:

determine an initial head pose information;

render the frame based on color values of the plurality of pixels and the initial head pose information; and

determine, during or after the rendering, an updated head pose information, where a change in head pose is based on a difference between the updated head pose information and the initial head pose information.

18. A computer program product, the computer program product comprising a computer-readable storage medium and storing executable code that, when executed by at least one data processing apparatus, is configured to cause the at least one data processing apparatus to perform a method comprising:

rendering a frame having a plurality of pixels, the rendering including determining a depth value for each of the plurality of pixels of the frame, the frame being associated with a frame vertex grid that divides the frame into a plurality of polygons;

down-sampling the depth values of the frame to obtain a plurality of down-sampled depth values;

determining a change in a head pose;

determining, from the plurality of down-sampled depth values, a down-sampled depth value for a vertex of the frame vertex grid;

determining an adjusted position for the vertex of the frame vertex grid based on the change in head pose and the down-sampled depth value for the vertex of the frame vertex grid;

performing, based on at least the adjusted position for the vertex of the frame vertex grid, a depth-adjusted time-warping of the frame to obtain a depth-adjusted time-warped frame; and

triggering display of the depth-adjusted time-warped frame.

19. The computer program product of claim 18, wherein the determining a change in a head pose comprises:

determining an initial head pose information;

rendering the frame based on color values of the plurality of pixels and the initial head pose information; and

determining, during or after the rendering, an updated head pose information, where a change in head pose is based on a difference between the updated head pose information and the initial head pose information.

20. The computer program product of claim 18, wherein the frame vertex grid includes a plurality of vertices defining the plurality of polygons and the performing the depth-adjusted time-warping of the frame to obtain a depth-adjusted time-warped frame includes performing the depth-adjusted time-warping based on depth-adjusted positions of the plurality of vertices of the frame vertex grid.