Dynamic switching and merging of head, gesture and touch input in virtual reality

• US 10,101,803 B2
• Filed: 08/26/2015
• Issued: 10/16/2018
• Est. Priority Date: 08/26/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Faaborg, Alexander James, Clement, Manuel Christian, McKenzie, Chris
• Primary Examiner(s): Cobb, Michael J
• Application Number:
  US14/836,311
• Publication Number:
  US 20170060230A1
• Time in Patent Office: 1,147 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/306 User profiles

Abstract

In a system for dynamic switching and merging of head, gesture and touch input in virtual reality, a virtual object may be selected by a user in response to a first input implementing one of a number of different input modes. Once selected, with focus established on the first object by the first input, the first object may be manipulated in the virtual world in response to a second input implementing another of the different input modes. In response to a third input, another object may be selected, and focus may be shifted from the first object to the second object in response to a third input if, for example, a priority value of the third input is higher than a priority value of the first input that established focus on the first object. If the priority value of the third input is less than the priority value of the first input that established focus on the first object, focus may remain on the first object. In response to certain trigger inputs, a display of virtual objects may be shifted between a far field display and a near field display to accommodate a particular mode of interaction with and manipulation of the virtual objects.

Claims

1. A method, comprising:

displaying an immersive virtual environment including a plurality of virtual objects, each of the plurality of virtual objects being selectable and manipulatable in response to a plurality of input modes;

detecting a first input implementing a first input mode of the plurality of input modes;

selecting a first virtual object of the plurality of virtual objects, in response to the first input, and setting an active focus on the selected first virtual object for manipulation in the virtual environment;

detecting a second input implementing a second input mode of the plurality of input modes;

manipulating at least one virtual element of the first virtual object in response to the second input while the active focus is maintained on the first virtual object;

detecting a third input implementing a third input mode of the plurality of input modes;

comparing a priority value of the third input mode and a priority value of the first input mode; and

based on the comparison, releasing the active focus on the first virtual object and shifting the active focus to a second virtual object of the plurality of virtual objects, or maintaining the active focus on the first virtual object, in response to the third input.

2. The method of claim 1, wherein releasing the active focus on the first virtual object and shifting the active focus to the second virtual object in response to the third input includes releasing the active focus on the first virtual object and shifting the active focus to the second virtual object in response to the third input when the priority value of the third input is greater than the priority value of the first input.

3. The method of claim 2, further comprising maintaining the active focus on the first virtual object when the priority value of the third input is less than the priority value of the first input.

4. The method of claim 1, each of the plurality of input modes having an associated priority value, the respective priority value associated with each of the plurality of input modes being preset or being set by a user in accordance with a user profile.

5. The method of claim 1, wherein the plurality of input modes includes a head gaze input mode, an eye gaze input mode, a point input mode, a reach input mode, and a gesture input mode.

6. The method of claim 1, wherein detecting the first input and selecting the first virtual object includes detecting a gaze input directed to the first virtual object to select the first virtual object for manipulation.

7. The method of claim 6, wherein detecting the second input includes:

detecting a touch input on a touch sensitive surface of a handheld electronic device; and

moving at least one virtual element of the selected first virtual object in response to the touch input.

8. The method of claim 7, wherein the touch input on the touch sensitive surface of the handheld electronic device is a touch and drag input, and wherein the movement of the at least one virtual element of the selected first virtual object corresponds to movement of the touch and drag input on the touch sensitive surface.

9. The method of claim 6, wherein detecting the gaze input directed to the first virtual object to select the first virtual object includes:

receiving the gaze input at a far field display of the plurality of virtual objects including the first virtual object; and

wherein receiving the second input includes;

detecting a gesture input;

recognizing the gesture input as a trigger for dynamic switching of the plurality of virtual objects from the far field display to a near field display;

morphing an appearance of the plurality of virtual objects for near field display of the plurality of virtual objects; and

displaying the morphed plurality of virtual objects in the near field display.

10. The method of claim 9, wherein morphing the appearance of the plurality of virtual objects for near field display of the plurality of virtual objects includes decreasing a virtual distance from a user to the plurality of virtual objects and at least one of changing a size of the plurality of virtual objects, changing an image associated with the plurality of virtual objects, changing text associated with the plurality of virtual objects, or changing menu items associated with the plurality of virtual objects as the plurality of virtual objects move from the far field display to the near field display.

11. The method of claim 10, further comprising detecting a fourth input, the fourth input including a reach gesture to select and manipulate a virtual object of the plurality of virtual objects displayed in the near field display.

12. The method of claim 1, wherein detecting the first input and selecting the first virtual object includes detecting a virtual beam directed from a handheld electronic device to the first virtual object to select the first virtual object, for manipulation of at least one virtual element of the first virtual object.

13. The method of claim 12, wherein detecting the second input includes:

detecting a touch input on a touch sensitive surface of a handheld electronic device;

detecting a drag of the touch input along the touch sensitive surface; and

moving at least one virtual element of the selected first virtual object in response to the touch and drag input, movement of the at least one virtual element of the selected first virtual object corresponding to movement of the drag input along the touch sensitive surface of the handheld electronic device.

14. The method of claim 12, wherein detecting the second input includes:

detecting a touch input on a touch sensitive surface of a handheld electronic device, and connecting the virtual beam between the handheld electronic device and the selected first virtual object in response to the detected touch input;

detecting a physical movement of the handheld electronic device;

moving at least one virtual element of the selected first virtual object in response to the detected physical movement of the handheld electronic device, movement of the at least one virtual element of the selected first virtual object corresponding to the detected physical movement of the handheld electronic device;

detecting release of the touch input from the touch sensitive surface of the handheld electronic device; and

releasing the active focus on the selected first virtual object in response to the detected release of the touch input to reposition the at least one virtual element of the selected first virtual object at a release point of the touch input.

15. An apparatus, comprising:

a head mounted electronic device configured to be operably coupled with a handheld electronic device, the head mounted electronic device including a control system controlling operation of the head mounted electronic device to;

display an immersive virtual environment including a plurality of virtual objects, each of the plurality of virtual objects being selectable and manipulatable in response to a plurality of input modes;

detect a first input implementing a first input mode of the plurality of input modes;

select a first virtual object, of the plurality of virtual objects, in response to the first input, and set an active focus on the selected first virtual object for manipulation in the virtual environment;

detect a second input implementing a second input mode of the plurality of input modes;

manipulate at least one virtual element of the first virtual object in response to the second input while the active focus is maintained on the first virtual object;

detect a third input implementing a third input mode of the plurality of input modes;

compare a priority value of the third input mode and a priority value of the first input mode; and

based on the comparison, release the active focus on the first virtual object and shift the active focus to a second virtual object of the plurality of virtual objects, or maintain the active focus on the first virtual object, in response to the third input.

16. The apparatus of claim 15, each of the plurality of input modes having an associated priority value, the respective priority value associated with each of the plurality of input modes being preset or being set by a user in accordance with a user profile, the control system being further configured to:

release the active focus on the first virtual object and shift the active focus to the second virtual object when the priority value of the third input is greater than the priority value of the first input; and

maintain the active focus on the first virtual object when the priority value of the third input is less than the priority value of the first input.

17. The apparatus of claim 15, wherein the first input includes a gaze input directed to a far field display of the plurality of virtual objects including the first virtual object, and the second input includes a gesture triggering dynamic switching of the plurality of virtual objects from the far field display to a near field display, and wherein, in response to the second input, the control system is further configured to:

morph an appearance of the plurality of virtual objects for near field display of the plurality of virtual objects; and

display the morphed plurality of virtual objects in the near field display.

18. A computer program product embodied on a non-transitory computer-readable storage medium storing instructions that, when executed, cause a computing device to perform a process, the instructions comprising instructions to:

display an immersive virtual environment including a plurality of virtual objects, each of the plurality of virtual objects being selectable and manipulatable in response to a plurality of input modes;

detect a first input implementing a first input mode of the plurality of input modes;

select a first virtual object, of the plurality of virtual objects, in response to the first input, and set an active focus on the selected first virtual object for manipulation of the first virtual object in the virtual environment;

detect a second input implementing a second input mode of the plurality of input modes;

manipulate at least one virtual element of the first virtual object in response to the second input while the active focus is maintained on the first virtual object;

detect a third input implementing a third input mode of the plurality of input modes;

compare a priority value of the third input mode and a priority value of the first input mode; and

based on the comparison, release the active focus on the first virtual object and shift the active focus to a second virtual object of the plurality of virtual objects, or maintain the active focus on the first virtual object, in response to the third input.

19. The computer program product embodied on the non-transitory computer-readable storage medium of claim 18, each of the plurality of input modes having an associated priority value, the respective priority value associated with each of the plurality of input modes being preset or being set by a user in accordance with a user profile.

20. The computer program product embodied on the non-transitory computer-readable storage medium of claim 18, the active focus on the first virtual object being released and shifted to the second virtual object when the priority value of the third input is greater than the priority value of the first input, and the active focus on the first virtual object being maintained when the priority value of the third input is less than the priority value of the first input.

21. The computer program product embodied on the non-transitory computer-readable storage medium of claim 18, wherein the first input is directed to a far field display of the plurality of virtual objects, and the second input triggers a dynamic switching of the plurality of virtual objects from the far field display to a near field display, in response to the second input, the instructions also including instructions to:

morph an appearance of the plurality of virtual objects for near field display of the plurality of virtual objects; and

display the morphed plurality of virtual objects in the near field display.

22. The computer program product embodied on the non-transitory computer-readable storage medium of claim 21, wherein the morphed appearance of the plurality of virtual objects for near field display of the plurality of virtual objects includes a decrease in a virtual distance from a user to the plurality of virtual objects and at least one of a change in a size of the plurality of virtual objects, a change in an image associated with the plurality of virtual objects, a change in text associated with the plurality of virtual objects, or a change menu items associated with the plurality of virtual objects as the plurality of virtual objects move from the far field display to the near field display.