Generating a new frame using rendered content and non-rendered content from a previous perspective

• US 10,489,966 B2
• Filed: 08/31/2018
• Issued: 11/26/2019
• Est. Priority Date: 09/01/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Nourai, Reza, Taylor, Robert Blake, Liebenow, Michael Harold
• Primary Examiner(s): Nguyen, Vu
• Application Number:
  US16/120,020
• Publication Number:
  US 20190073821A1
• Time in Patent Office: 452 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually

Abstract

Disclosed is an approach for constructing a new frame using rendered content and non-rendered content from a previous perspective. Points of visible surfaces of a first set of objects from a first perspective are rendered. Both rendered content and non-rendered content from the first perspective are stored. The new frame from the second perspective is generated using the rendered content and the non-rendered content from the first perspective.

Claims

1. A method for constructing a new frame using rendered content and non-rendered content from a previous perspective, the method comprising:

rendering points of visible surfaces of a first set of objects from a first perspective;

storing both rendered content and non-rendered content from the first perspective, the rendered content corresponding to the points of the visible surfaces of the first set of objects from the first perspective, and the non-rendered content corresponding to non-visible portions of the first set of objects from the first perspective;

receiving a request to display a new frame from a second perspective;

generating the new frame from the second perspective using the rendered content and the non-rendered content from the first perspective; and

displaying the new frame from the second perspective.

2. The method of claim 1, further comprising identifying the first perspective by capturing a first pose of a user.

3. The method of claim 1, wherein storing both the rendered content and the non-rendered content from the first perspective comprises storing both the rendered content and the non-rendered content in at least one of linked lists, array structures, true volumetric representations, voxels, surface definitions, N-dimensional data structures, and N-dimensional graph representations.

4. The method of claim 1, further comprising determining different granularities for both rendered content and non-rendered content for the one or more objects from the first perspective.

5. The method of claim 1, further comprising:

rendering points of visible surfaces of a second set of objects from the second perspective;

warping at least some of the stored rendered content from the first perspective to the second perspective;

wherein generating the second from the second perspective comprises combining new rendered content and warped rendered content, the new rendered content corresponding to the points of the visible surfaces of the second set of objects from the second perspective, and the warped rendered content corresponding to the stored rendered content warped from the first perspective to the second perspective.

6. The method of claim 5, wherein rendering the points of the visible surfaces of the second set of objects from the second perspective comprises:

determining a third set of objects visible from the second perspective;

comparing the third set of objects from the second perspective to the stored rendered content from the first perspective to identify the second set of objects, wherein the second set of objects includes objects from the third set of objects that are not in the stored rendered content; and

rendering the points of the visible surfaces of the second set of objects.

7. The method of claim 5, wherein warping the at least some of the stored content from the first perspective to the second perspective comprises:

determining a third set of objects visible from the second perspective;

comparing the third set of objects from the second perspective to the stored rendered content from the first perspective to identify objects that are in both the third set of objects and the stored rendered content;

warping the objects that are in both the third set of objects and the stored rendered content.

8. The method of claim 1, wherein storing both the rendered content and the non-rendered content from the first perspective comprises:

identifying the rendered content and the non-rendered content from the first perspective;

analyzing the rendered content and the non-rendered content to determine if content corresponds to one of a surface point and a non-surface point;

discarding the content corresponding to a non-surface point; and

storing the content corresponding to the surface point.

9. The method of claim 1, wherein storing both the rendered content and the non-rendered content from the first perspective comprises:

predicting a head movement of a user;

calculating bounds for the predicted head movement of the user based at least in part on one or more of a predicted direction, a current speed of movement of the user, and a predicted speed of movement for the user, the bounds corresponding to a maximum predicted head movement for the second perspective;

determine content within the calculated bounds for the predicted head movement; and

storing the content within the calculated bounds.

10. The method of claim 1, wherein storing both rendered content and non-rendered content for the one or more objects from the first perspective further comprises:

determining opaque points of one or more opaque objects;

storing image data corresponding to the opaque points;

determining semi-transparent points of one or more semi-transparent objects; and

storing image data of the semi-transparent points, wherein the first set of objects comprises the one or more opaque objects and the one or more semi-transparent objects.

11. The system of claim 1, wherein storing both rendered content and non-rendered content for the one or more objects from the first perspective further comprises:

determining opaque points of one or more opaque objects;

storing image data corresponding to the opaque points;

determining semi-transparent points of one or more semi-transparent objects; and

storing image data of the semi-transparent points, wherein the first set of objects comprises the one or more opaque objects and the one or more semi-transparent objects.

12. A system for constructing a new frame using rendered content and non-rendered content from a previous perspective, the system comprising:

a non-transitory computer-readable storage medium storing executable computer instructions that, when executed, perform steps comprising;

rendering points of visible surfaces of a first set of objects from a first perspective;

storing both rendered content and non-rendered content from the first perspective, the rendered content corresponding to the points of the visible surfaces of the first set of objects from the first perspective, and the non-rendered content corresponding to non-visible portions of the first set of objects from the first perspective;

receiving a request to display a new frame from a second perspective;

generating the new frame from the second perspective using the rendered content and the non-rendered content from the first perspective; and

displaying the new frame from the second perspective; and

a processor configured to execute the computer instructions.

13. The system of claim 12, wherein the instructions, when executed, perform further steps comprising identifying the first perspective by capturing a first pose of a user.

14. The system of claim 12, wherein storing both the rendered content and the non-rendered content from the first perspective comprises storing both the rendered content and the non-rendered content in at least one of linked lists, array structures, true volumetric representations, voxels, surface definitions, N-dimensional data structures, and N-dimensional graph representations.

15. The system of claim 12, wherein the instructions, when executed, perform further steps comprising determining different granularities for both rendered content and non-rendered content for the one or more objects from the first perspective.

16. The system of claim 12, wherein the instructions, when executed, perform further steps comprising:

rendering points of visible surfaces of a second set of objects from the second perspective;

warping at least some of the stored rendered content from the first perspective to the second perspective;

wherein generating the second from the second perspective comprises combining new rendered content and warped rendered content, the new rendered content corresponding to the points of the visible surfaces of the second set of objects from the second perspective, and the warped rendered content corresponding to the stored rendered content warped from the first perspective to the second perspective.

17. The system of claim 16, rendering the points of the visible surfaces of the second set of objects from the second perspective comprises:

determining a third set of objects visible from the second perspective;

comparing the third set of objects from the second perspective to the stored rendered content from the first perspective to identify the second set of objects, wherein the second set of objects includes objects from the third set of objects that are not in the stored rendered content; and

rendering the points of the visible surfaces of the second set of objects.

18. The system of claim 16, wherein warping the at least some of the stored content from the first perspective to the second perspective comprises:

determining a third set of objects visible from the second perspective;

comparing the third set of objects from the second perspective to the stored rendered content from the first perspective to identify objects that are in both the third set of objects and the stored rendered content;

warping the objects that are in both the third set of objects and the stored rendered content.

19. The system of claim 12, wherein storing both the rendered content and the non-rendered content from the first perspective comprises:

identifying the rendered content and the non-rendered content from the first perspective;

analyzing the rendered content and the non-rendered content to determine if content corresponds to one of a surface point and a non-surface point;

discarding the content corresponding to a non-surface point; and

storing the content corresponding to the surface point.

20. The system of claim 12, wherein storing both the rendered content and the non-rendered content from the first perspective comprises:

predicting a head movement of a user;

calculating bounds for the predicted head movement of the user based at least in part on one or more of a predicted direction, a current speed of movement of the user, and a predicted speed of movement for the user, the bounds corresponding to a maximum predicted head movement for the second perspective;

determine content within the calculated bounds for the predicted head movement; and

storing the content within the calculated bounds.

21. A computer program product embodied in a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for constructing a new frame using rendered content and non-rendered content from a previous perspective, the method comprising:

rendering points of visible surfaces of a first set of objects from a first perspective;

storing both rendered content and non-rendered content from the first perspective, the rendered content corresponding to the points of the visible surfaces of the first set of objects from the first perspective, and the non-rendered content corresponding to non-visible portions of the first set of objects from the first perspective;

receiving a request to display a new frame from a second perspective;

generating the new frame from the second perspective using the rendered content and the non-rendered content from the first perspective; and

displaying the new frame from the second perspective.

22. The computer program product of claim 21, further comprising:

rendering points of visible surfaces of a second set of objects from the second perspective;

warping at least some of the stored rendered content from the first perspective to the second perspective;

wherein generating the second from the second perspective comprises combining new rendered content and warped rendered content, the new rendered content corresponding to the points of the visible surfaces of the second set of objects from the second perspective, and the warped rendered content corresponding to the stored rendered content warped from the first perspective to the second perspective.

23. The computer program product of claim 21, wherein storing both the rendered content and the non-rendered content from the first perspective comprises:

identifying the rendered content and the non-rendered content from the first perspective;

analyzing the rendered content and the non-rendered content to determine if content corresponds to one of a surface point and a non-surface point;

discarding the content corresponding to a non-surface point; and

storing the content corresponding to the surface point.

24. The computer program product of claim 21, wherein storing both the rendered content and the non-rendered content from the first perspective comprises:

predicting a head movement of a user;

calculating bounds for the predicted head movement of the user based at least in part on one or more of a predicted direction, a current speed of movement of the user, and a predicted speed of movement for the user, the bounds corresponding to a maximum predicted head movement for the second perspective;

determine content within the calculated bounds for the predicted head movement; and

storing the content within the calculated bounds.

25. The computer program product of claim 21, wherein storing both rendered content and non-rendered content for the one or more objects from the first perspective further comprises:

determining opaque points of one or more opaque objects;

storing image data corresponding to the opaque points;

determining semi-transparent points of one or more semi-transparent objects; and

storing image data of the semi-transparent points, wherein the first set of objects comprises the one or more opaque objects and the one or more semi-transparent objects.

26. A method for constructing a new frame using rendered content and non-rendered content from a previous perspective, the method comprising:

rendering one or more objects from a first perspective;

storing both rendered content and non-rendered content for the one or more objects from the first perspective, the non-rendered content corresponding to non-visible portions of the object from the first perspective;

receiving a request to display the one or more object from a second perspective;

warping at least some of the rendered content for the one or more objects to create warped content from the second perspective;

rendering at least some of the non-rendered content for the one or more objects to create second rendered content from the second perspective; and

constructing the one or more objects from the second perspective by combining both the warped content and the second rendered content from the second perspective; and

displaying the one or more objects from the second perspective.