Generating and Modifying Representations of Objects in an Augmented-Reality or Virtual-Reality Scene

• US 20200134923A1
• Filed: 09/27/2019
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Seiler, Larry, Nankervis, Alex
• Granted Patent: US 11,170,577 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/08 Bandwidth reduction
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

In one embodiment, a method for generating subframes may include accessing a rendered frame depicting a virtual scene from a first viewpoint of a user. The rendered frame may be generated based on at least one 3D model of an object in the virtual scene. A computing system may generate plurality of surfaces based on the rendered frame. Each surface may be positioned within a 3D space and comprises visual information. The system may determine a second viewpoint of the user in the 3D space and determine a visibility of the plurality of surfaces from the second viewpoint. The system may then render a subframe depicting the virtual scene from the second viewpoint based on the visual information and determined visibility of the plurality of surfaces.

Claims

1. A method comprising, by one or more computing systems:

accessing a rendered frame depicting a virtual scene from a first viewpoint of a user, wherein the rendered frame is generated based on at least one 3D model of an object in the virtual scene;

generating a plurality of surfaces based on the rendered frame, wherein each surface is positioned within a 3D space and comprises visual information;

determining a second viewpoint of the user in the 3D space;

determining a visibility of the plurality of surfaces from the second viewpoint; and

rendering a subframe depicting the virtual scene from the second viewpoint based on the visual information and determined visibility of the plurality of surfaces.

2. The method of claim 1, wherein:

the visual information of each of the plurality of surfaces is a texture;

the determined visibility of the plurality of surfaces identifies points on the plurality of surfaces visible from the second view point; and

the rendering of the subframe comprises sampling the texture of each of the plurality of surfaces according to the points on the plurality of surfaces.

3. The method of claim 1, wherein:

the 3D model of the object comprises polygons; and

the rendered frame is generated based on computed intersections between one or more of the polygons and rays cast from the first viewpoint.

4. The method of claim 1, wherein:

the rendered frame is one of a plurality of frames generated based on at least the 3D model of the object;

the rendered subframe is one of a plurality of subframes generated based on the plurality of surfaces; and

the plurality of subframes are generated after the rendered frame and before a subsequent one of the plurality of frames.

5. The method of claim 4, wherein:

the plurality of frames are generated at a first frame rate; and

the plurality of subframes are generated at a second frame rate that is higher than the first frame rate.

6. The method of claim 1, wherein the visibility of the plurality of surfaces is determined using ray casting.

7. The method of claim 6, wherein the rendered frame is generated using ray tracing.

8. The method of claim 1, wherein the first viewpoint and the second viewpoint are determined at different times.

9. The method of claim 1, wherein:

the rendered frame is generated based on the 3D model of the object and a second 3D model of a second object in the virtual scene; and

at least one of the plurality of surfaces comprises visual information that depict both the object and the second object.

10. The method of claim 1, wherein each of the plurality of surfaces is flat.

11. The method of claim 1, wherein the rendered frame and the subframe are different.

12. The method of claim 1, wherein the rendered frame and the rendered subframe are rendered by different computing systems.

13. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a rendered frame depicting a virtual scene from a first viewpoint of a user, wherein the rendered frame is generated based on at least one 3D model of an object in the virtual scene;

generate a plurality of surfaces based on the rendered frame, wherein each surface is positioned within a 3D space and comprises visual information;

determine a second viewpoint of the user in the 3D space;

determine a visibility of the plurality of surfaces from the second viewpoint; and

render a subframe depicting the virtual scene from the second viewpoint based on the visual information and determined visibility of the plurality of surfaces.

14. The media of claim 13, wherein:

the visual information of each of the plurality of surfaces is a texture;

the determined visibility of the plurality of surfaces identifies points on the plurality of surfaces visible from the second view point; and

the rendering of the subframe comprises sampling the texture of each of the plurality of surfaces according to the points on the plurality of surfaces.

15. The media of claim 13, wherein:

the 3D model of the object comprises polygons; and

the rendered frame is generated based on computed intersections between one or more of the polygons and rays cast from the first viewpoint.

16. The media of claim 13, wherein:

the rendered frame is one of a plurality of frames generated based on at least the 3D model of the object;

the rendered subframe is one of a plurality of subframes generated based on the plurality of surfaces; and

the plurality of subframes are generated after the rendered frame and before a subsequent one of the plurality of frames.

17. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to;

access a rendered frame depicting a virtual scene from a first viewpoint of a user, wherein the rendered frame is generated based on at least one 3D model of an object in the virtual scene;

generate a plurality of surfaces based on the rendered frame, wherein each surface is positioned within a 3D space and comprises visual information;

determine a second viewpoint of the user in the 3D space;

determine a visibility of the plurality of surfaces from the second viewpoint; and

render a subframe depicting the virtual scene from the second viewpoint based on the visual information and determined visibility of the plurality of surfaces.

18. The system of claim 17, wherein:

the visual information of each of the plurality of surfaces is a texture;

the determined visibility of the plurality of surfaces identifies points on the plurality of surfaces visible from the second view point; and

the rendering of the subframe comprises sampling the texture of each of the plurality of surfaces according to the points on the plurality of surfaces.

19. The system of claim 17, wherein:

the 3D model of the object comprises polygons; and

the rendered frame is generated based on computed intersections between one or more of the polygons and rays cast from the first viewpoint.

20. The system of claim 17, wherein:

the rendered frame is one of a plurality of frames generated based on at least the 3D model of the object;

the rendered subframe is one of a plurality of subframes generated based on the plurality of surfaces; and

the plurality of subframes are generated after the rendered frame and before a subsequent one of the plurality of frames.