System and method for displaying computer-based content in a virtual or augmented environment

• US 10,489,978 B2
• Filed: 11/02/2016
• Issued: 11/26/2019
• Est. Priority Date: 07/26/2016
• Status: Active Grant

• Current Assignee: Allison Phuong Huynh, Rouslan Lyubomirov Dimitrov
• Original Assignee: Allison Phuong Huynh, Rouslan Lyubomirov Dimitrov
• Inventors: Dimitrov, Rouslan Lyubomirov, Huynh, Allison Phuong
• Primary Examiner(s): Flores, Roberto W
• Application Number:
  US15/341,812
• Publication Number:
  US 20180033204A1
• Time in Patent Office: 1,119 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method for displaying computer-based content in a virtual or augmented environment includes receiving a selection of content for display on a window of a computing system, rendering the content as a virtual screen independently orientable from other virtual screens corresponding to other windows of the computing system, compositing the virtual screen into a virtual or augmented reality environment, detecting a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and changing properties of the virtual screen based on manipulation of the virtual controller by the user. In some embodiments, the system and method further include one or more of rendering a pointing ray and changing the properties based on movement of the virtual controller as if the virtual controller and the virtual screen are connected by a rigid link.

Claims

1. A method comprising:

receiving, by one or more processors, a selection of content for display on a window of a computing system;

rendering, by the one or more processors, the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in a virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;

compositing, by the one or more processors, the virtual screen into the virtual or augmented reality environment;

rendering a hand-held input device as a virtual controller spaced from the virtual screens, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls;

rendering a pointing ray extending from the virtual controller;

detecting, by the one or more processors, a selection of the virtual screen using a virtual controller, based on a gaze of a user within the virtual or augmented reality environment, or both, and representing said selection in the virtual or augmented reality environment as selection by the virtual controller aiming at the virtual screen from a distance as aided by the pointing ray;

changing, by the one or more processors, an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user so that rotations of the hand-held input device result in corresponding rotations of the virtual screen as if the virtual controller and the virtual screen are connected via a rigid link between a representative point on the virtual controller and a point on the virtual screen, the rigid link extending along the pointing ray, the pointing ray reaching said point on the virtual screen, wherein the manipulation of the virtual controller corresponds to a filtered version of a manipulation of the hand-held input device by a user;

automatically reorienting the virtual screen, after the virtual screen is deselected, to align a screen up direction of the virtual screen with a view up direction of virtual or augmented reality environment when the screen up direction of the virtual screen is within a first angular separation from the view up direction of the virtual or augmented reality environment;

automatically reorienting the virtual screen, after the virtual screen is deselected, to align an orientation of the virtual screen with a direction of view of the user when the orientation of the virtual screen is within a second angular separation from the direction of view of the user;

relocating, after the virtual screen is deselected, a sound source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller; and

relocating, after the virtual screen is deselected, an illumination source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller, the illumination source contributing to illumination of the virtual or augmented reality environment as a computer screen displaying a same video content as being displayed on the virtual screen would illuminate a real environment.

2. The method of claim 1, wherein each of the one or more rendered input controls being selected from a group consisting of a button, a touchpad, a slider bar, a scroll wheel, and a rocker switch.

3. The method of claim 2, further comprising:

changing one or more properties of the virtual screen, the one or more properties including one or more of a position of the virtual screen, a size of the virtual screen, or the orientation of the virtual screen;

wherein;

the changing of the position of the virtual screen is based on changes in a position of the hand-held input device as if the virtual controller and the virtual screen are connected by rigid link; and

the changing of the size of the virtual screen is based on up and down swipes on a touchpad, movement of a slider bar, rotation of a scroll wheel, or activation of a rocker switch.

4. The method of claim 1, further comprising using the virtual screen to obtain a virtual object corresponding to an element displayed on the virtual screen.

5. The method of claim 1, further comprising saving a state of the virtual or the augmented reality environment.

6. The method of claim 1, further comprising adding a sound source to the virtual or augmented environment when the content includes an audio component.

7. The method of claim 1, further comprising using the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented environment.

8. The method of claim 1, further comprising aligning the virtual screen with an up direction in the virtual or augmented reality environment.

9. The method of claim 1, further comprising aligning the virtual screen so it is directed toward the user in the virtual or augmented environment.

10. The method of claim 1, wherein changing the orientation of the virtual screen comprises temporarily changing a transparency of the virtual screen so that it becomes semi-transparent when the virtual screen is selected and an opacity of the virtual screen is restored after the orientation is changed.

11. The method of claim 1, wherein the user may navigate past or move behind the virtual screen within the virtual or augmented environment.

12. The method of claim 1, further comprising rendering a respective menu for each of the virtual screen and the other virtual screens, each respective menu being usable by the user to manipulate a corresponding virtual screen.

13. The method of claim 1, wherein a level of the filtering is adjusted based on a translational or a rotational velocity of the hand-held input device.

14. A system comprising:

a head-mounted display;

a hand-held controller; and

one or more processors coupled to the head-mounted display and the hand-held controller, the one or more processors being configured to;

receive a selection of content for display on a window of a computing system;

render the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in a virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;

render the hand-held controller as a virtual controller, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls;

composite the virtual screen and the virtual controller into the virtual or augmented reality environment displayed on the head-mounted display;

detect a selection of the virtual screen by a user;

represent a selection of the virtual screen as selected by the virtual controller in the virtual or augmented reality environment when the virtual controller is at a distance from the virtual screen, wherein representing the selection of the virtual screen comprises;

rendering the virtual controller to aim at the virtual screen from a distance;

rendering a pointing ray extending from the virtual controller to the virtual screen; and

rendering a manipulation of the one or more rendered input controls of the virtual controller to select the virtual screen by the virtual controller aiming at the virtual screen from a distance;

change an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by the user so that rotations of the hand-held controller result in corresponding rotations of the virtual screen as if the pointing ray is at least part of a rigid link connecting a representative point on the virtual controller to a point where the pointing ray meets the virtual screen, wherein the manipulation of the virtual controller corresponds to a filtered version of a manipulation of the hand-held controller by a user;

automatically reorienting the virtual screen, after the virtual screen is deselected, to align a screen up direction of the virtual screen with a view up direction of virtual or augmented reality environment when the screen up direction of the virtual screen is within a first angular separation from the view up direction of the virtual or augmented reality environment;

automatically reorienting the virtual screen, after the virtual screen is deselected, to align an orientation of the virtual screen with a direction of view of the user when the orientation of the virtual screen is within a second angular separation from the direction of view of the user;

relocating, after the virtual screen is deselected, a sound source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller; and

relocating, after the virtual screen is deselected, an illumination source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller, the illumination source contributing to illumination of the virtual or augmented reality environment as a computer screen displaying a same video content as being displayed on the virtual screen would illuminate a real environment.

15. The system of claim 14, wherein:

the one or more processors are further configured to change one or more of a position of the virtual screen or, a size of the virtual screen, or the orientation of the virtual screen;

the change in the position of the virtual screen is based on changes in position of the hand-held controller as if the virtual controller and the virtual screen are connected by the rigid link; and

the change of the size of the virtual screen is based on up and down swipes on a touchpad, movement of a slider bar, rotation of a scroll wheel, or activation of a rocker switch.

16. The system of claim 14, wherein the one or more processors are further configured to add a sound source to the virtual or augmented reality environment when the content includes an audio component, use the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented reality environment, or both.

17. The system of claim 14, wherein the one or more processors are further configured to align the virtual screen with an up direction in the virtual or augmented reality environment, so it is directed toward the user in the virtual or augmented environment, or both.

18. A non-transitory computer-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors associated with a virtual or augmented reality system are adapted to cause the one or more processors to perform a method comprising:

receiving a selection of content for display on a window of a computing system;

rendering the content as a virtual screen, the virtual screen being independently rotationally orientable about any combination of x, y, or z axes in the virtual or augmented reality environment from one or more other virtual screens corresponding to other windows of the computing system;

rendering a hand-held controller as a virtual controller, the virtual controller being manipulatable as a virtual object within the virtual or augmented reality environment, the virtual controller having one or more rendered input controls, each of the one or more rendered input controls being selected from a group consisting of a button, a touchpad, a slider bar, a scroll wheel, and a rocker switch;

compositing the virtual screen into the virtual or augmented reality environment;

rendering a pointing ray extending from the virtual controller;

detecting a selection of the virtual screen by detecting that the virtual controller is aiming toward the virtual screen from a distance as indicated by the pointing ray;

changing an orientation of the virtual screen about any combination of the x, y, or z axes in the virtual or augmented reality environment independent of an orientation of the one or more other virtual screens based on manipulation of the virtual controller by a user so that rotations of the hand-held controller result in corresponding rotations of the virtual screen as if the virtual controller and the virtual screen are connected via a rigid link between a representative point on the virtual controller and a point on the virtual screen, the rigid link extending along the pointing ray, the pointing ray reaching said point on the virtual screen, wherein the manipulation of the virtual controller corresponds to a filtered version of a manipulation of the hand-held controller by a user;

automatically reorienting the virtual screen, after the virtual screen is deselected, to align a screen up direction of the virtual screen with a view up direction of virtual or augmented reality environment when the screen up direction of the virtual screen is within a first angular separation from the view up direction of the virtual or augmented reality environment;

automatically reorienting the virtual screen, after the virtual screen is deselected, to align an orientation of the virtual screen with a direction of view of the user when the orientation of the virtual screen is within a second angular separation from the direction of view of the user;

relocating, after the virtual screen is deselected, a sound source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller; and

relocating, after the virtual screen is deselected, an illumination source associated with the virtual screen based on a change in location or the orientation of the virtual screen due to the manipulation of the virtual controller, the illumination source contributing to illumination of the virtual or augmented reality environment as a computer screen displaying a same video content as being displayed on the virtual screen would illuminate a real environment.

19. The non-transitory computer-readable medium of claim 18, wherein the method further comprises:

changing one or more of a curvature, a position, and an orientation of the virtual screen, the changing of the position of the virtual screen being based on movement of the hand-held controller as if the virtual controller and the virtual screen are connected by the rigid link.

20. The non-transitory computer-readable medium of claim 18, wherein the method further comprises adding a sound source to the virtual or augmented reality environment when the content includes an audio component, using the virtual screen as an illumination source for one or more other virtual objects within the virtual or augmented reality environment, or both.