Graphical user interface for large-scale, multi-user, multi-touch systems

• US 8,429,568 B2
• Filed: 05/11/2012
• Issued: 04/23/2013
• Est. Priority Date: 07/30/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Perceptive Pixel Incorporated (Microsoft Corporation)
• Inventors: Davidson, Philip L.
• Primary Examiner(s): BAUTISTA, XIOMARA L
• Application Number:
  US13/469,850
• Publication Number:
  US 20120227012A1
• Time in Patent Office: 347 Days
• Field of Search: 715/702, 715/764, 715/765, 715/810, 715/861, 715/863, 715/864, 345/473, 345/169, 345/173
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method implemented on the graphical user interface device to invoke an independent, user-localized menu in an application environment, by making a predetermined gesture with a pointing device on an arbitrary part of a display screen or surface, especially when applied in a multi-touch, multi-user environment, and in environments where multiple concurrent pointing devices are present. As an example, the user may trace out a closed loop of a specific size that invokes a default system menu at any location on the surface, even when a second user may be operating a different portion of the system elsewhere on the same surface. As an additional aspect of the invention, the method allows the user to smoothly transition between the menu-invocation and menu control.

Claims

1. A method of activating a menu element, the method comprising:

invoking a menu element on a display device in response to a generated first stroke made by positioning a first moving pointer element on a display surface of the display device, the display device being a multi-input display device and the menu element having a plurality of selectable menu options, wherein invoking the menu element comprises displaying the menu element and the plurality of selectable menu options on the display surface; and

selecting one of the selectable menu options as a function of a generated second stroke made by positioning a second moving pointer element on the display surface, the second moving pointer element being positioned on the display surface to select the one of the selectable menu options before the first moving pointer element is removed from the display surface, wherein, once the menu element has been invoked on the display device, the first moving pointer element is unable to select any of the selectable menu options.

2. The method of claim 1, wherein the first moving pointer element generates the first stroke by providing input to an input device, and the second moving pointer element generates the second stroke by providing input to the input device.

3. The method of claim 1, wherein the first stroke is generated by contacting the first moving pointer element on the display surface, and the second stroke is generated by contacting the second moving pointer element on the display surface.

4. The method of claim 3, wherein activating one of the selectable menu options comprises activating one of the selectable menu options as a function of the generated second stroke conditioned on the second moving pointer element begin positioned on the display surface to begin generating the second stroke before the first moving pointer element discontinues contact with the display surface to complete the first stroke.

5. The method of claim 1,

further comprising determining a position of the first moving pointer element when the first stroke is ascertained to correspond to a predetermined shape, and

wherein invoking the menu element comprises displaying the menu element at a translated position, the translated position corresponding to the position of the first moving pointer element when the first stroke was ascertained to correspond to the predetermined shape translated as a function of continued movement of the first moving pointer element prior to completion of the first stroke.

6. The method of claim 5,

wherein determining the position of the first moving pointer element comprises determining a position of the first moving pointer element when the first stroke crosses over itself to form a loop, and

wherein displaying the menu element at the translated position comprises displaying the menu element at a position corresponding to the position of the first moving pointer element when the first stroke was ascertained to cross over itself to form a loop translated as a function of continued movement of the first moving pointer element prior to completion of the first stroke.

7. The method of claim 1,

further comprising determining a first position of the first moving pointer element when the first stroke is ascertained to correspond to a predetermined shape, and

wherein invoking the menu element includes:

initially displaying the menu element at the first position, and

subsequently displaying the menu element at multiple different and sequential translated positions to visually translate the menu element across the display surface from the first position as a function of continued movement of the first moving pointer element as the first moving pointer element completes the first stroke.

8. The method of claim 7,

wherein determining the first position comprises determining a cross-over position of the first moving pointer element, the cross-over position being a position of the first moving pointer element when the first stroke crosses over itself to form a loop;

wherein initially displaying the menu element comprises initially displaying the menu element at the cross-over position, and

wherein subsequently displaying the menu element at multiple different and sequential translated positions comprises subsequently displaying the menu element at multiple different and sequential translated positions to visually translate the menu element across the display surface from the cross-over position as a function of continued movement of the first moving pointer element as the first moving pointer element completes the first stroke.

9. The method of claim 1,

wherein a shape of the first stroke comprises an enclosed area, and

wherein displaying the menu element comprises:

determining a display location of a centroid of the enclosed area, and

displaying the menu element at the determined display location of the centroid of the enclosed area.

10. The method of claim 9, wherein displaying the menu element at the determined display location of the centroid of the enclosed area comprises:

determining a display location of a centroid of the menu element, and

displaying the menu element such that the display location of the centroid of the menu element coincides with the display location of the centroid of the enclosed area.

11. The method of claim 1 wherein the displayed selectable menu options remain separated from the first moving pointer element regardless of movement of first moving pointer element.

12. The method of claim 1 wherein continued movement of the first moving pointer element causes the displayed menu element to be repositioned on the display surface as a function of the continued movement of the first moving pointer element.

13. A system including:

a multi-input display device; and

a processor configured to perform operation comprising;

detect generation by a user of a first stroke made by positioning a first moving pointer element on a display surface of the display device;

invoke a menu element on the display device in response to detecting the generation of the first stroke, the menu element having a plurality of selectable menu options, wherein invoking the menu element comprises enabling display of the menu element and the plurality of selectable menu options on the display surface;

detect generation by the user of a second stroke made by positioning a second moving pointer element on a display surface of the display device; and

select one of the selectable menu options as a function of the detected generated second stroke, the second moving pointer element being positioned on the display surface to select the one of the selectable menu options before the first moving pointer element is removed from the display surface, wherein, once the menu element has been invoked on the display device, the first moving pointer element is unable to select any of the selectable menu options.

14. The system of claim 13,

further comprising an input device, and

wherein the first moving pointer element generates the first stroke by providing input to the input device, and the second moving pointer element generates the second stroke by providing input to the input device.

15. The system of claim 14,

wherein the processor is configured to detect generation by the user of the first stroke by detecting the first moving pointer element contacting the display surface, and

wherein the processor is configured to detect generation by the user of the second stroke by detecting the second moving pointer element contacting the display surface.

16. The system of claim 15, wherein the processor being configured to activate one of the selectable menu options comprises the processor being configured to activate one of the selectable menu options as a function of the generated second stroke conditioned on the second moving pointer element begin positioned on the display surface to begin generating the second stroke before the first moving pointer element discontinues contact with the display surface to complete the first stroke.

17. The system of claim 13,

wherein the processor is further configured to determine a position of the first moving pointer element when the first stroke is ascertained to correspond to a predetermined shape, and

wherein the processor being configured to invoke the menu element comprises the processor being configured to enable display of the menu element at a translated position, the translated position corresponding to the position of the first moving pointer element when the first stroke was ascertained to correspond to the predetermined shape translated as a function of continued movement of the first moving pointer element prior to completion of the first stroke.

18. The system of claim 17,

wherein the processor being configured to determine the position of the first moving pointer element comprises the processor being configured to determine a position of the first moving pointer element when the first stroke crosses over itself to form a loop, and

wherein the processor being configured to enable display of the menu element at the translated position comprises the processor being configured to enable display of the menu element at a position corresponding to the position of the first moving pointer element when the first stroke was ascertained to cross over itself to form a loop translated as a function of continued movement of the first moving pointer element prior to completion of the first stroke.

19. The system of claim 13,

wherein the processor is further configured to determine a first position of the first moving pointer element when the first stroke is ascertained to correspond to a predetermined shape, and

wherein the processor being configured to invoke the menu element comprises the processor being configured to:

enable an initial display of the menu element at the first position, and

enable subsequent display of the menu element at multiple different and sequential translated positions to visually translate the menu element across the display surface from the first position as a function of continued movement of the first moving pointer element as the first moving pointer element completes the first stroke.

20. The system of claim 19,

wherein the processor being configured to determine the first position comprises the processor being configured to determine a cross-over position of the first moving pointer element, the cross-over position being a position of the first moving pointer element when the first stroke crosses over itself to form a loop;

wherein the processor being configured to enable an initial display of the menu element comprises the processor being configured to enable an initial display of the menu element at the cross-over position, and

wherein the processor being configured to enable subsequent display of the menu element at multiple different and sequential translated positions comprises the processor being configured to enable subsequent display of the menu element at multiple different and sequential translated positions to visually translate the menu element across the display surface from the cross-over position as a function of continued movement of the first moving pointer element as the first moving pointer element completes the first stroke.

21. The system of claim 13,

wherein a shape of the first stroke comprises an enclosed area, and

wherein the processor being configured to enable display of the menu element comprises the processor being configured to:

determine a display location of a centroid of the enclosed area, and

enable display of the menu element at the determined display location of the centroid of the enclosed area.

22. The system of claim 21, wherein the processor being configured to enable display of the menu element at the determined display location of the centroid of the enclosed area comprises the processor being configured to:

determine a display location of a centroid of the menu element, and

enable display of the menu element such that the display location of the centroid of the menu element coincides with the display location of the centroid of the enclosed area.

23. The system of claim 13 wherein the displayed selectable menu options remain separated from the first moving pointer element regardless of movement of first moving pointer element.

24. The method of claim 13 wherein continued movement of the first moving pointer element causes the displayed menu element to be repositioned on the display surface as a function of the continued movement of the first moving pointer element.