USER INTERFACES FOR BI-MANUAL CONTROL

• US 20180088792A1
• Filed: 09/29/2016
• Published: 03/29/2018
• Est. Priority Date: 09/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Klein, Christian, Glass, Andrew S.
• Granted Patent: US 11,073,980 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Techniques for bi-manual control of an operation may allow gestural input for the operation to be inputted blindly with one hand while a user'"'"'s attention remains on the other hand. An operation invoked and controlled by the gestural input may automatically associate with an operation controlled by the other hand. The gestural input may facilitate blind control of the operation by being able to be initially recognized without regard for the scale, orientation, and/or location of the invoking gestural input. Moreover, scale and/or orientation of the operation may be determined by the gestural input. Changes in scale and/or orientation of the gestural input during the blind operation may change the scale and/or orientation of the operation. The operation may manifest by displaying corresponding graphics at a location independent of the gestural input or by associating functionality of the blind operation with an operation performed by the other hand.

Claims

1. A method performed by a computing device comprising storage hardware, processing hardware, a display, and an input device, the method comprising:

while a graphical user operation displayed at a first region of the display is being interactively performed by a first hand that is providing input via the input device at the first region of the display, recognizing a gesture inputted by a second hand and responding to the recognized gesture by associating inputs inputted by the second hand with the graphical user operation being interactively performed by the first hand, wherein the gesture is initially recognized as such based on invocation features thereof that are independent of the orientation and scale of the gesture, and wherein the graphical user operation is augmented, controlled, or modified based on the inputs being associated with the graphical user operation.

2. A method according to claim 1, wherein the recognizing the gesture is performed regardless of which arbitrary applications are executing when the gesture is recognized.

3. A method according to claim 2, wherein the user inputs are able to be associated with any of the arbitrary applications.

4. A method according to claim 1, wherein the inputs associated with the graphical user operation are interpreted based on an orientation and/or scale of the gesture and not based on a location of the gesture.

5. A method according to claim 1, further comprising applying the inputs associated with the graphical user operation at a location based on a location of the graphical user operation.

6. A method according to claim 1, wherein an application displaying the graphical user operation receives events or messages corresponding to the user inputs associated with the graphical user operation.

8. A method according to claim 1, further comprising recognizing the summoning gesture while third user input is being inputted to control an operation displayed on the display, the operation being controlled and displayed before the first user input is inputted.

9. A method according to claim 8, wherein the characteristic comprises orientation and/or scale and/or a number of distinct input pointers and the value comprises an orientation value and/or scale value and/or the number of distinct input pointers.

10. A method according to claim 9, wherein the orientation and/or scale of the user interface control dynamically adapts to the orientation and/or scale of the second user input.

7. A method performed by a computing device comprised of input hardware, a display, storage hardware, and processing hardware, the method comprising:

responsive to receiving first user input, recognizing the first user input as a summoning gesture; and

determining a value of a characteristic of the first user input;

based on the recognizing of the summoning gesture, invoking a corresponding user interface control and associating second user input with the user interface control, wherein the user interface control is configured to be operated by the second user input according to the determined value of the first user input, and wherein the user interface control, while being operated by the second user input, either, (i) lacks any graphic representation displayed on the display or, (ii) comprises a graphic representation displayed at a location of the display that is not dependent on a location of the first user input.

11. A method according to claim 7, the method further comprising providing an application programming interface (API) through which applications are able to request notifications corresponding to activations or instantiations of the user interface control.

12. A method according to claim 11, wherein the API allows the applications to receive events and/or messages generated by the user interface control responsive operation thereof by the second user input.

13. A method according to claim 7, wherein an operating system performs the method, wherein the graphic representation and/or a function of the user interface control is provided by the operating system, the method further comprising the operating system, not the application, associating the graphic representation and/or the function with the location.

14. A computing device comprising:

input hardware;

a display;

processing hardware; and

storage hardware;

the storage hardware storing instructions executable by the processing hardware, the instructions configured to perform a process, the process comprising;

allowing summoning gestures to be recognized for user inputs entered above arbitrary applications and based on recognition features of the user inputs that do not depend on orientation and/or scale of the user inputs;

when a summoning gesture is recognized for a user input, automatically causing a corresponding user interface control to be summoned, by the user input, for user-manipulation (i) at a display location of the user input location of the display and (ii) at a display scale and/or orientation of the user input; and

automatically associating a functional and/or graphical manifestation of the user interface control with an object displayed on the display or at a location that does not depend on the display location of the user input.

15. A computing device according to claim 14, wherein the user interface control does not comprise a graphical manifestation on the display.

16. A computing device according to claim 14, wherein the configured display scale and/or orientation of the user interface control is configured to dynamically change according to changes to the display scale and/or orientation of the user input.

17. A computing device according to claim 14, wherein the instructions comprise an operating system executable by the processing hardware, the operating system configured to determine whether the user inputs correspond to the summoning gesture regardless of the display locations where the user inputs are inputted.

18. A computing device according to claim 14, the process further comprising receiving the user input inputted by a first hand of a person while another user input is being inputted by a second hand of the person, and automatically manifesting the functional and/or graphical manifestation based on the user input inputted by the second hand.

19. A computing device according to claim 18, the process further comprising causing the functional and/or graphical manifestation to automatically track the second user input.

20. A computing device according to claim 14, the process further comprising mapping user inputs that manipulate the user interface control to corresponding control events received by an application comprised of the object.