Graphical user interface for large-scale, multi-user, multi-touch systems

• US 8,181,122 B2
• Filed: 07/30/2008
• Issued: 05/15/2012
• Est. Priority Date: 07/30/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: Davidson, Philip L.
• Primary Examiner(s): BAUTISTA, XIOMARA L
• Application Number:
  US12/182,385
• Publication Number:
  US 20090193366A1
• Time in Patent Office: 1,385 Days
• Field of Search: 715/702, 715/764, 715/765, 715/810, 715/861, 715/863, 715/864, 345/473, 345/169, 345/173
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method implemented on the graphical user interface device to invoke an independent, user-localized menu in an application environment, by making a predetermined gesture with a pointing device on an arbitrary part of a display screen or surface, especially when applied in a multi-touch, multi-user environment, and in environments where multiple concurrent pointing devices are present. As an example, the user may trace out a closed loop of a specific size that invokes a default system menu at any location on the surface, even when a second user may be operating a different portion of the system elsewhere on the same surface. As an additional aspect of the invention, the method allows the user to smoothly transition between the menu-invocation and menu control.

Claims

1. A method of invoking a control element on a display device, comprising the steps of:

generating by a user a stroke on a display device using a moving pointer element;

ascertaining, during generation of the stroke, a shape of the stroke;

determining whether the ascertained shape corresponds to a predetermined shape; and

invoking a control element based on whether the stroke is ascertained to have a shape that corresponds to the predetermined shape,

wherein invoking the control element includes;

determining a display location for the control element as a function of the shape of the stroke; and

displaying the control element at the determined display location on the display device.

2. The method of claim 1, wherein the display device is a touch sensitive display device and the generating step comprises generating by the user the stroke on the display device by actuating the touch sensitive display device.

3. The method of claim 1, wherein the predetermined shape is a closed loop.

4. The method of claim 1, wherein the predetermined shape is a signature shape associated with the user.

5. The method of claim 1, wherein the stroke generated by the user is a subsection of a continuous sequence of input by the user using the moving pointer element.

6. The method of claim 1, wherein the generating step comprises generating by at least one user a plurality of strokes on the display device, each of the strokes using a respective moving pointing element.

7. The method of claim 1,

wherein, after ascertaining that the stroke corresponds to the predetermined shape, the method comprises the step of conducting a secondary analysis of a characteristic of the stroke;

and

wherein the invoking step comprises invoking a first control element or a second control element based on results of the conducted secondary analysis.

8. The method of claim 1, wherein the ascertaining step comprises ascertaining if the stroke corresponds to either a first predetermined shape or a second predetermined shape;

and the invoking step comprises invoking a first control element based on whether the stroke is ascertained to correspond to the first predetermined shape and invoking a second control element based on whether the stroke is ascertained to correspond to the second predetermined shape.

9. The method of claim 1, wherein the ascertaining step also ascertains a direction of travel of the stroke;

and the invoking step comprises invoking a first control element based on whether the stroke is ascertained to correspond to a predetermined shape with a measurable clockwise or counterclockwise winding and that the ascertained direction of travel corresponds to a clockwise direction, and invoking a second control element based on whether the stroke is ascertained to correspond to the predetermined shape and the ascertained direction of travel corresponds to a counter-clockwise direction.

10. The method of claim 9, wherein the invoking step comprises invoking a first control element based on whether the stroke is ascertained to correspond to a closed loop with a measurable clockwise or counterclockwise winding and that the ascertained direction of travel corresponds to a clockwise direction, and invoking a second control element based on whether the stroke is ascertained to correspond to the closed loop and the ascertained direction of travel corresponds to a counter-clockwise direction.

11. The method of claim 9, wherein the invoking step comprises invoking a first control element based on whether the stroke is ascertained to correspond to a closed polygon with a measurable clockwise or counterclockwise winding and that the ascertained direction of travel corresponds to a clockwise direction, and invoking a second control element based on whether the stroke is ascertained to correspond to the closed polygon and the ascertained direction of travel corresponds to a counter-clockwise direction.

12. The method of claim 1, further comprising recognizing, during generation of the stroke, if the generated stroke corresponds to a substantial portion of the predetermined shape;

and providing visual feedback on the display corresponding to at least a portion of the generated stroke if the generated stroke is recognized to correspond to said substantial portion of the predetermined shape, the visual feedback identifying to the user that the stroke being generated corresponds to a partial amount of the predetermined shape.

13. The method of claim 1, wherein the ascertaining step comprises ascertaining if the stroke corresponds to a predetermined shape and is within a predetermined size threshold;

and the invoking step comprises invoking the control element based on whether the stroke is ascertained to correspond to the predetermined shape and the stroke is within the predetermined size threshold.

14. The method of claim 1, wherein the generating step comprises generating by a plurality of users respective strokes on said display device;

the ascertaining step comprises ascertaining, for each of the strokes being generated, if the respective stroke corresponds to the predetermined shape; and

the invoking step comprises invoking the control element for each ascertained stroke that corresponds to the predetermined shape.

15. The method of claim 1, wherein displaying the control element at the determined display location comprises displaying the control element at a center of the stroke that corresponds to the predetermined shape.

16. The method of claim 1, wherein displaying the control element at the determined display location comprises displaying the control element at a display location that is a function of the shape of the generated stroke and also is a function of boundaries of the display device so that the displayed control element is fully displayed within the display device.

17. The method of claim 1, wherein displaying the control element at the determined display location comprises displaying the control element at a position and orientation that is additionally a function of the position and orientation of the generated stroke and also is a function of positions and orientations of other control elements on the display device, of application objects present on the display device, or of the activity of other users of the device.

18. The method of claim 1,

wherein the shape of the stroke comprises a closed loop,

wherein determining the display location for the control element comprises determining a display location of a point on the loop where closure of the loop occurred, and

wherein displaying the control element comprises displaying the control element at the determined display location of the point on the loop where closure of the loop occurred.

19. The method of claim 1,

wherein the shape of the stroke comprises an enclosed area,

wherein determining the display location for the control element comprises determining a display location of a centroid of the enclosed area, and

wherein displaying the control element comprises displaying the control element at the determined display location of the centroid of the enclosed area.

20. The method of claim 19,

wherein displaying the control element comprises determining a display location of a centroid of the control element, and

wherein displaying the control element comprises displaying the control element such that the display location of the centroid of the control element coincides with the display location of the centroid of the enclosed area.

21. The method of claim 20, wherein the control element is a menu element having a plurality of selectable menu items.

22. A method of activating a control element on a display device, comprising the steps of:

invoking a control element on a display device in response to a generated stroke made using a moving pointer element, wherein invoking the control element includes;

ascertaining a speed of movement of the moving pointer element during generation of the stroke,

determining a location where the control element will be displayed on the display device as a function of the ascertained speed, and

displaying the control element on the display device at the determined display location.

23. The method of claim 22, wherein the ascertaining step comprises ascertaining whether the speed of movement of the moving pointer element corresponds to a relatively slow speed, a relatively moderate speed, or a relatively high speed.

24. The method of claim 23, further comprising establishing a manner of activation of the control element as being immediately activatable by subsequent movement of the moving pointer element based on whether the speed of movement of the moving pointer element is ascertained to correspond to the relatively slow speed.

25. The method of claim 23, further comprising establishing a manner of activation of the control element as being activatable by subsequent movement of the moving pointer element after discontinued movement of the moving pointer element based on whether the speed of movement of the moving pointer element is ascertained to correspond to the relatively moderate speed.

26. The method of claim 23, wherein determining a location where the control element will be displayed comprises determining a display location disposed away from the moving pointer element and disposed behind a direction of travel of the moving pointer element based on whether the speed of the movement of the moving pointer element is ascertained to correspond to the relatively high speed.

27. The method of claim 23, wherein determining a location where the control element will be displayed comprises determining a display location disposed away from the moving pointer element and disposed behind a direction of travel of the moving pointer element based on whether the speed of the movement of the moving pointer element is ascertained to correspond to the relatively high speed, and the control element being immediately activatable based on whether the pointer element returns to a predetermined active area of the control element.

28. The method of claim 22, wherein the ascertaining step comprises selecting the speed of movement of the moving pointer element from among a relatively slow speed, a relatively moderate speed, and a relatively high speed.

29. The method of claim 22,

wherein ascertaining the speed of movement comprises determining that the speed corresponds to a relatively high speed or to a relative low speed, and

wherein determining a location where the control element will be displayed comprises:

determining a first location where the control element will be displayed if the determined speed corresponds to a relatively low speed, and

determining a second and different location where the control element will be displayed if the determined speed corresponds to a relatively high speed.

30. The method of claim 29, wherein the control element comprises a menu element having a plurality of selectable menu items.

31. A method of activating a menu element, comprising the steps of:

invoking a menu element on a display device in response to a generated first stroke made by contacting a first moving pointer element with a display surface of the display device, the display device being a multi-input display device and the menu element having a plurality of selectable menu options, wherein invoking the menu element comprises displaying the menu element and the plurality of selectable menu options on the display surface; and

activating one of the selectable menu options as a function of a generated second stroke made by contacting a second moving pointer element with the display surface, the second moving pointer element contacting the display surface to begin generating the second stroke before the first moving pointer element discontinues contact with the display surface to complete the first strike,

wherein the menu element is invoked at a position of the first moving pointer element when the first stroke was ascertained to correspond to a predetermined shape, and

further comprising translating the menu element on the display surface after invocation of the menu element based on movement of the first moving pointer element as the first moving pointer element completes the first stroke.

32. The method of claim 31, wherein the first moving pointer element generates the first stroke by providing input to an input device, and the second moving pointer element generates the second stroke by providing input to said input device.

33. The method of claim 31, wherein the invoking step includes displaying the menu element on the display device at a position that corresponds to a position of the generated first stroke.

34. The method of claim 33, wherein the invoking step includes moving the displayed menu element to positions corresponding to positions of the generated first stroke during movement of the first moving pointer element.

35. A system including

a display device;

and

a processor configured to;

detect generation by a user of a stroke on the display device in response to user movement of a pointer element;

ascertain, during generation of the stroke, a shape of the stroke;

determine whether the ascertained shape corresponds to a predetermined shape; and

invoke a control element based on whether the stroke is ascertained to have a shape that corresponds to the predetermined shape,

wherein the processor being configured to invoke the control element includes the processor being configured to;

determine a display location for the control element as a function of the shape of the stroke; and

display the control element at the determined display location on the display device.

36. A method of invoking a menu element on a display device, comprising the steps of:

detecting generation by a user of a stroke made by the user contacting a moving pointer element with a display surface of the display device;

ascertaining, during generation of the stroke, a shape of the stroke;

determining whether the ascertained shape corresponds to a predetermined shape; and

invoking a menu element based on whether the stroke is ascertained to have a shape that corresponds to the predetermined shape,

wherein invoking the control element includes;

determining a location on the display surface where the moving pointer element discontinues contact with the display surface of the display device, and

triggering display of the menu element at a display location corresponding to the location of discontinued contact.

37. The method of claim 36, wherein the triggering of the display of the menu element at the display location corresponding to the location of discontinued contact comprises triggering the display of at least a portion of the menu element at the location of discontinued contact in response to the discontinued contact.