Method and system for receiving gesture input via virtual control objects

• US 9,852,546 B2
• Filed: 01/28/2015
• Issued: 12/26/2017
• Est. Priority Date: 01/28/2015
• Status: Expired due to Fees

• Current Assignee: CCP hf (PearlAbyss Corp.)
• Original Assignee: CCP hf (PearlAbyss Corp.)
• Inventors: Kraver, Adam, McGinnis, Joel Patrick, Rittenhouse, John Warren
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US14/608,047
• Publication Number:
  US 20160217614A1
• Time in Patent Office: 1,063 Days
• Field of Search: None
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Aspects of the present invention provide computer systems, apparatuses, computer-executable methods and one or more non-transitory computer-readable media for receiving gesture input via virtual controls. Examples include a computer-implemented method that includes receiving data indicating a physical environment state, processing the data to determine a physical position of at least one user, determining at least one physical anchor position within the physical environment state, mapping the physical anchor position to a virtual anchor position within a virtual environment state, wherein the virtual environment state includes a plurality of virtual coordinate positions that map to at least a portion of the physical environment state, determining a particular virtual coordinate position for at least one virtual control from the plurality of virtual coordinate positions, and instructing a display device configured to display the virtual environment state to display the virtual control at the particular virtual coordinate position.

Claims

1. A computer-implemented method for implementing a virtual control in a virtual reality environment, the method comprising:

receiving data indicating a physical environment state;

processing, by a processor, the data indicating the physical environment state to determine a physical position of at least one user;

determining, by the processor and based on the position of the at least one user, at least one physical anchor position within the physical environment state;

mapping, by virtual environment state management circuitry, the physical anchor position to a virtual anchor position within a virtual environment state, wherein the virtual environment state includes a plurality of virtual coordinate positions that map to at least a portion of the physical environment state;

predefining, by the virtual environment state management circuitry, a fixed virtual coordinate position for at least one virtual control from the plurality of virtual coordinate positions by;

identifying a virtual control positioning offset indicating a relative position with respect to the virtual anchor position; and

applying the virtual control positioning offset to the virtual anchor position to determine the fixed virtual coordinate position, wherein a first relative position between the fixed virtual coordinate position and the virtual anchor position corresponds to a second relative position between a particular physical coordinate position mapped to the fixed virtual coordinate position and the physical anchor position;

determining a view angle of the at least one user; and

only in response to determining that the view angle of the at least one user includes an area of a physical environment encompassing the fixed virtual coordinate position predefined by the virtual environment state management circuitry, instructing a display device configured to display the virtual environment state to display the virtual control at the fixed virtual coordinate position.

2. The method of claim 1, further comprising:

determining that the physical position of at least a portion of the user is within a defined area of a physical environment corresponding to the physical environment state that maps to the fixed virtual coordinate position; and

in response to mapping the physical position of the at least one the user to the fixed virtual coordinate position, executing functionality associated with the virtual control.

3. The method of claim 2, further comprising executing the functionality associated with the virtual control only in response to determining that the physical position of the at least one user has been proximate to the physical position that maps to the fixed virtual coordinate position for at least a threshold period of time.

4. The method of claim 1, wherein the physical position of the at least one user is the user'"'"'s head or shoulders.

5. The method of claim 1, wherein the virtual control is associated with one or more particular gestures, and wherein functionality of the virtual control is only executed in response to performing at least one of the one or more gestures at a physical position that maps to a defined area around the fixed virtual coordinate position.

6. The method of claim 1, wherein the data indicating the physical environment state is received from at least one sensor and wherein the at least one sensor is at least one of a camera or an infrared receiver.

7. The method of claim 1, wherein the display device is a head-mounted display.

8. The method of claim 7, wherein the physical anchor position is determined at least in part by detecting an infrared emitter mounted on the head-mounted display.

9. The method of claim 1, further comprising:

determining a view angle of the at least one user; and

enabling interaction with the virtual control in response to determining that the view angle corresponds to an area of a physical environment corresponding to the fixed virtual coordinate position of the virtual control object.

10. The method of claim 9, wherein the view angle of the user is determined based at least in part on a sensor reading received from a head-mounted device.

11. The method of claim 1, wherein determining the physical anchor position is performed in response to at least one of turning on the display device or placing the display device on the user'"'"'s head.

12. The method of claim 1, wherein a component value of the virtual control positioning offset is determined based at least in part on a number of initialized virtual controls, or a detected height of the user.

13. A computer-implemented method for implementing a virtual control in a virtual reality environment, the method comprising:

receiving data indicating a physical environment state;

processing, by a processor, the data indicating the physical environment state to determine a physical position of a physical object;

determining, by the processor, based on the physical position of the physical object, at least one virtual anchor position within a virtual environment state;

predefining, by virtual environment state management circuitry, a fixed virtual coordinate position of a virtual control in the virtual environment state relative to the virtual anchor position;

mapping, by the virtual environment state management circuitry, the fixed virtual coordinate position to a physical position of the control in a physical environment corresponding to the physical environment state;

determining, by the virtual environment state management circuitry, a view angle of at least one user; and

only in response to determining that the view angle includes an area of a physical environment encompassing the fixed virtual coordinate position predefined by the virtual environment state management circuitry, instructing a display device configured to display the virtual environment state to display the virtual environment state such that the virtual control is displayed at the fixed virtual coordinate position.

14. The method of claim 13, further comprising determining that a view angle of the display device is angled toward the physical position prior to displaying the virtual control via the display device.

15. The method of claim 13, further comprising:

determining that the physical position of the physical object is within a defined area of a physical environment corresponding to the physical environment state that maps to the fixed virtual coordinate position; and

in response to determining that the physical position of the user is proximate to the physical position of the control, executing functionality associated with the virtual control.

16. The method of claim 15, further comprising executing the functionality associated with the virtual control only in response to determining that the physical position of physical object has been proximate to the physical position of the control for at least a threshold period of time.

17. The method of claim 13, wherein the virtual anchor position is associated with a physical position of a user'"'"'s head.

18. The method of claim 13, wherein the data indicating the physical environment state is received from at least one sensor and wherein the at least one sensor is at least one of a camera or an infrared receiver.

19. The method of claim 13, wherein the display device is a head-mounted display.

20. The method of claim 13, wherein determining the virtual anchor position is performed in response to at least one of turning on the display device or placing the display device on the user'"'"'s head.

21. A non-transitory computer-readable storage medium comprising instructions, that, when executed by a processor, cause the processor to implement a virtual control in a virtual reality environment, by at least:

receiving data indicating a physical environment state;

processing, by a processor, the data indicating the physical environment state to determine a physical position of a physical object;

determining, by the processor, based on the physical position of the physical object, at least one virtual anchor position within a virtual environment state;

predefining, by virtual environment state management circuitry, a fixed virtual coordinate position of a virtual control in the virtual environment state relative to the virtual anchor position;

mapping, by the virtual environment state management circuitry, the fixed virtual coordinate position to a physical position of the control in a physical environment corresponding to the physical environment state;

determining, by the virtual environment state management circuitry, a view angle of at least one user; and

only in response to determining that the view angle includes an area of a physical environment encompassing the fixed virtual coordinate position predefined by the virtual environment state management circuitry, instructing a display device configured to display the virtual environment state to display the virtual environment state such that the virtual control is displayed at the fixed virtual coordinate position.

22. A system for providing a virtual reality environment, the system comprising:

circuitry configured to output a virtual environment state provided by virtual environment state management circuitry to at least one display device, the virtual environment state including one or more virtual controls, wherein the virtual environment state comprises a virtual coordinate system for indicating locations of virtual objects within the virtual environment state;

circuitry configured to receive data indicating a physical environment state from at least one sensor, wherein the physical environment state comprises a physical coordinate system for indicating locations of physical objects within the physical environment state;

physical environment state management circuitry configured to determine a physical anchor position at a particular physical coordinate position within the physical coordinate system using the data indicating the physical environment state; and

the virtual environment state management circuitry configured to;

generate a mapping between a set of virtual coordinate positions and a set of physical coordinate positions;

determine, based on the mapping, a particular virtual coordinate position corresponding to the particular physical coordinate position;

place a virtual control anchor at the particular virtual coordinate position;

determine fixed virtual coordinate positions for the one or more virtual controls according to respective virtual control positioning offsets by applying the virtual control positioning offset to the particular virtual coordinate position of the virtual control anchor;

determine a view angle of at least one user; and

only in response to determining that the view angle corresponds to an area of a physical environment corresponding to a subset of the fixed virtual coordinate positions of the one or more virtual controls, place the respective one or more virtual controls within the virtual environment state at the corresponding fixed virtual coordinate positions.

23. The system of claim 22, wherein the display device is a head-mounted display.

24. The system of claim 22, wherein the virtual environment state management circuitry is further configured to display only a portion of the virtual environment state via the display device, and wherein the portion of the virtual environment state is determined based on a view angle of the display device.

25. The system of claim 22, wherein the physical environment state management circuitry is further configured to determine a user position, and wherein the virtual environment state management circuitry is further configured to:

determine that the user position is proximate to at least one physical coordinate position that maps to a particular fixed virtual coordinate position; and

perform a function associated with the corresponding virtual control in response to determining that the user position is proximate to at least one physical coordinate position that maps to the particular fixed virtual coordinate position.