Controller visualization in virtual and augmented reality environments

• US 10,286,308 B2
• Filed: 02/27/2016
• Issued: 05/14/2019
• Est. Priority Date: 11/10/2014
• Status: Active Grant

• Current Assignee: Valve Corporation
• Original Assignee: Valve Corporation
• Inventors: Sawyer, David, Reidford, Tristan, Malaika, Yasser, Yates, Alan, Selan, Jeremy
• Primary Examiner(s): Balaoing, Ariel A
• Application Number:
  US15/055,592
• Publication Number:
  US 20170249019A1
• Time in Patent Office: 1,172 Days
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Controller visualization systems and methods for use in virtual/augmented reality environments such as walk-around virtual reality environments are described. The virtual representation of a physical control device may be altered based on the context of the virtual environment. Certain embodiments combine the virtual rendering of the tracked control device with a real-time video representation of part of the operating space. In certain embodiments, the display of additional information relating to the function of interactive elements may be displayed based on context, such as when a specific action is required from a user in the virtual space.

Claims

1. An optical tracking system, comprising;

a movable tracked object positionable within an operating space, the movable tracked object comprising one or more rigidly attached optical sensors;

one or more optical transmitters positionable within said operating space, wherein each optical transmitter comprises two rotors that sweep two optical beams across a tracking volume on orthogonal axes during repeating sweep cycles, wherein each optical transmitter emits an omni-directional synchronization optical pulse detectable by said optical sensors at the beginning of each said sweep cycle; and

control circuitry operatively coupled to the one or more optical sensors, in operation, the control circuitry;

for each of the one or more optical sensors, computes an angular location within said tracking volume by calculating the elapsed time between detecting said synchronization pulse and detecting said optical beams;

determines the location and orientation of said movable tracked object based on the determined angular location of the one or more optical sensors;

causes a display to present a virtual representation of said movable tracked object in a virtual reality environment with a location and orientation within the virtual reality environment that corresponds to the determined location and orientation of said movable tracked object in said operating space; and

alters the virtual representation of said movable tracked object based on the context of said virtual reality environment.

3. The system of claim 1, wherein said movable tracked object is located at a first position within said operating space relative to a user'"'"'s eyes, and wherein said virtual representation of said movable tracked object is located at a second position within said virtual reality environment that corresponds to said first position, so that the placement and orientation of said movable tracked object within said virtual reality environment appears in substantially the same position relative to said user'"'"'s eyes as said movable tracked object appears relative to said user'"'"'s eyes within said operating space.

5. The system of claim 1, wherein said virtual representation of said movable tracked object is altered based on detecting that a user'"'"'s gaze has been directed toward said virtual representation of said movable tracked object.

2. An optical tracking system, comprising;

a movable tracked object positionable within an operating space, the movable tracked object comprising one or more rigidly attached optical sensors;

one or more optical transmitters positionable within said operating space; and

control circuitry operatively coupled to the one or more optical sensors, in operation, the control circuitry;

for each of the one or more optical sensors, computes an angular location within a tracking volume with respect to said one or more optical transmitters;

determines the location and orientation of said movable tracked object based on the determined angular location of the one or more optical sensors;

causes a display to present a virtual representation of said movable tracked object in a virtual reality environment with a location and orientation within the virtual reality environment that corresponds to the determined location and orientation of said movable tracked object in said operating space; and

alters the virtual representation of said movable tracked object based on the context of said virtual reality environment.

4. The system of claim 2, wherein said movable tracked object is located at a first position within said operating space relative to a user'"'"'s eyes, and wherein said virtual representation of said movable tracked object is located at a second position within said virtual reality environment that corresponds to said first position, so that the placement and orientation of said movable tracked object within said virtual reality environment appears in substantially the same position relative to said user'"'"'s eyes as said movable tracked object appears relative to said user'"'"'s eyes within said operating space.

6. The system of claim 2, wherein said virtual representation of said movable tracked object is altered based on detecting that a user'"'"'s gaze has been directed toward said virtual representation of said movable tracked object.

7. The system of claim 2 wherein the control circuitry provides an augmented virtual representation of said movable tracked object based on the context of said virtual reality environment.

8. The system of claim 7 wherein the augmented virtual representation of said movable tracked object comprises a labeling of at least one interactive element of the movable tracked object that indicates a context-specific function of the at least one interactive element.

9. The system of claim 8 wherein the labeling comprises at least one of icon-based labeling or animation-based labeling.

10. The system of claim 8 wherein the at least one interactive element comprises at least one of a button, joystick, directional pad, software button, touch sensitive input device, dial, trigger, knob, or motion sensor.

11. The system of claim 2 wherein the movable tracked object comprises a video game controller, an input device for a computer, or a remote control device.

12. The system of claim 2 wherein the control circuitry alters the virtual representation of said movable tracked object based on the context of at least one of said operating space or an event that takes place outside of said virtual reality environment.

13. The system of claim 2 wherein the control circuitry alters the virtual representation of said movable tracked object responsive to a user of the movable tracked object taking a pre-defined action.

14. The system of claim 2 wherein the control circuitry alters the virtual representation of said moveable tracked object such that the virtual representation of the movable tracked object comprises an item in the virtual reality environment.

15. An optical tracking system, comprising;

a movable tracked object positionable within an operating space, the movable tracked object comprising one or more rigidly attached optical sensors;

one or more optical transmitters positionable within said operating space; and

control circuitry operatively coupled to the one or more optical sensors, in operation, the control circuitry;

for each of the one or more optical sensors, computes an angular location within a tracking volume with respect to said one or more optical transmitters;

determines the location and orientation of said movable tracked object based on the determined angular location of the one or more optical sensors;

causes a display to present a virtual representation of said movable tracked object in a virtual reality environment with a location and orientation within the virtual reality environment that corresponds to the determined location and orientation of said movable tracked object in said operating space; and

alters the virtual representation of said movable tracked object based on the context of at least one of said operating space or an event that takes place outside of said virtual reality environment.

16. The optical tracking system of claim 15 wherein the control circuitry receives a notification from an external device, and alters the virtual representation of said movable tracked object based on the received notification.

17. The optical tracking system of claim 16 wherein the notification comprises a notification that the external device is receiving a phone call.

18. The optical tracking system of claim 17 wherein the control circuitry alters the virtual representation of said movable object to indicate that an interactive element of the movable tracked object can be selected to answer the incoming phone call.

19. The optical tracking system of claim 16 wherein the notification comprises a notification from the external device that a person is at a door of a user'"'"'s home.