SYSTEM AND METHOD FOR DISPLAYING COMPUTER-BASED CONTENT IN A VIRTUAL OR AUGMENTED ENVIRONMENT

• US 20180033204A1
• Filed: 11/02/2016
• Published: 02/01/2018
• Est. Priority Date: 07/26/2016
• Status: Active Grant

• Current Assignee: Allison Phuong Huynh, Rouslan Lyubomirov Dimitrov
• Original Assignee: Allison Phuong Huynh, Rouslan Lyubomirov Dimitrov
• Inventors: DIMITROV, Rouslan Lyubomirov, HUYNH, Allison Phuong
• Granted Patent: US 10,489,978 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method for displaying computer-based content in a virtual or augmented environment includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen independently orientable from other virtual screens corresponding to other windows of the computing system, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing properties of the virtual screen based on manipulation of the virtual controller by the user. In some embodiments, the system and method further include one or more of rendering a pointing ray and changing the properties based on movement of the virtual controller as if the virtual controller and the virtual screen are connected by a rigid link.

Claims

1. A method comprising:

receiving, by one or more processors, a selection of content for display on a window of a computing system;

rendering, by the one or more processors, the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in a virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;

compositing, by the one or more processors, the virtual screen into the virtual or augmented reality environment;

detecting, by the one or more processors, a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both; and

changing, by the one or more processors, an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user.

2. The method of claim 1, wherein the virtual controller corresponds to a hand-held input device or a virtual hand corresponding to a hand of the user.

3. The method of claim 2, further comprising changing one or more properties of the virtual screen, the one or more properties including one or more of a curvature, a position, and the orientation of the virtual screen, the changing of the position, the orientation, or both of the virtual screen being based on movement of the hand-held input device as if the virtual controller and the virtual screen are connected by a rigid link.

4. The method of claim 1, further comprising using the virtual screen to obtain a virtual object corresponding to an element displayed on the virtual screen.

5. The method of claim 1, further comprising saving a state of the virtual or the augmented reality environment.

6. The method of claim 1, further comprising adding a sound source to the virtual or augmented environment when the content includes an audio component.

7. The method of claim 1, further comprising using the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented environment.

8. The method of claim 1, further comprising aligning the virtual screen with an up direction in the virtual or augmented reality environment.

9. The method of claim 1, further comprising aligning the virtual screen so it is directed toward the user in the virtual or augmented environment.

10. The method of claim 1, wherein changing the orientation of the virtual screen comprises temporarily changing a transparency of the virtual screen so that it is semi-transparent and other virtual objects are trackable while the orientation is being changed.

11. The method of claim 1, wherein the user may navigate past or move behind the virtual screen within the virtual or augmented environment.

12. The method of claim 1, further comprising rendering a respective menu for each of the virtual screen and the other virtual screens, each respective menu being usable by the user to manipulate a corresponding virtual screen.

13. A system comprising:

a head-mounted display;

a hand-held controller; and

one or more processors coupled to the head-mounted display and the hand-held controller, the one or more processors being configured to;

receive a selection of content for display on a window of a computing system;

render the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in a virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;

render the hand-held controller as a virtual controller, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls;

composite the virtual screen and the virtual controller into a virtual or augmented reality environment displayed on the head-mounted display;

detect a selection of the virtual screen or other virtual object using the virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both; and

change an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user.

14. The system of claim 13, wherein the one or more processors are further configured to render a pointing ray between the virtual controller and a selected point on the virtual screen or the other virtual object.

15. The system of claim 13, wherein the one or more processors are further configured to change one or more of a curvature, a position, and the orientation of the virtual screen, the change in the position, the orientation, or both of the virtual screen being based on movement of the hand-held controller as if the virtual controller and the virtual screen are connected by a rigid link.

16. The system of claim 13, wherein the one or more processors are further configured to add a sound source to the virtual or augmented reality environment when the content includes an audio component, use the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented reality environment, or both.

17. The system of claim 13, wherein the one or more processors are further configured to align the virtual screen with an up direction in the virtual or augmented reality environment, so it is directed toward the user in the virtual or augmented environment, or both.

18. A non-transitory computer-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors associated with a virtual or augmented reality system are adapted to cause the one or more processors to perform a method comprising:

receiving a selection of content for display on a window of a computing system;

rendering the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in the virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;

rendering a hand-held controller as a virtual controller, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls;

compositing the virtual screen into a virtual or augmented reality environment;

detecting a selection of the virtual screen using the virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both; and

change an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user.

19. The non-transitory computer-readable medium of claim 18, wherein the method further comprises:

rendering a pointing ray between the virtual controller and a selected point on the virtual screen; and

changing one or more of a curvature, a position, and an orientation of the virtual screen, the changing of the position, the orientation, or both of the virtual screen being based on movement of the hand-held controller as if the virtual controller and the virtual screen are connected by a rigid link.

20. The non-transitory computer-readable medium of claim 18, wherein the method further comprises adding a sound source to the virtual or augmented reality environment when the content includes an audio component, using the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented reality environment, or both.