Computer with graphical user interface for interaction

• US 9,740,364 B2
• Filed: 05/03/2010
• Issued: 08/22/2017
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D.
• Primary Examiner(s): Xiao, Di
• Application Number:
  US12/772,864
• Publication Number:
  US 20110271216A1
• Time in Patent Office: 2,668 Days
• Field of Search: 715863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Different techniques of processing user interactions with a computing system are described. In one implementation, an interactive display is configured to depict a graphical user interface which includes a plurality of different types of user interface elements (e.g., button-type element, scroll bar-type element). A user may use one or more user input object (e.g., finger, hand, stylus) to simultaneously interact with the interactive display. A plurality of different user input processing methods are used to process user inputs received by the graphical user interface differently and in accordance with the types of the user interface elements which are displayed. The processing of the user inputs is implemented to determine whether the user inputs control the respective user interface elements. The processing may determine whether the user inputs activate and/or manipulate the displayed user interface elements in but one example.

Claims

1. A user interaction method implemented by a system comprising:

receiving a user input image via a graphical user interface displayed on a display screen, wherein the graphical user interface includes a plurality of user interface elements, the user input image defining a plurality of pixel locations of the display screen, the plurality of pixel locations activated by one or more input objects interacting with the graphical user interface, the plurality of activated pixel locations defining one or more regions of activated pixel locations of the display screen; and

processing the user input image to determine whether the user input image controls at least one user interface element displayed in the graphical user interface, the processing including;

identifying a number of first pixels activated by the user input image and which are within an area defined by a geometric boundary of the user interface element,

weighting the number of first pixels to provide a first value,

identifying a number of second pixels which are proximate to the user interface element but outside the geometric boundary of the user interface element,

combining the number of first pixels without the weighting and the number of second pixels to provide a second value,

subtracting the second value from the first value to provide a third value, and

comparing the third value with a threshold to determine whether the user input image controls the user interface element.

2. The method of claim 1, wherein the receiving comprises receiving the user input image resulting from a plurality of user input objects interacting with the graphical user interface at substantially the same moment in time.

3. The method of claim 1 further comprising processing the user input image to identify at least one of a plurality of different types of user input objects which interacted with the graphical user interface to create the user input image.

4. The method of claim 1, wherein the processing comprises determining a first region comprising a plurality of pixels of the graphical user interface selected by the user input and comparing the first region with the at least one user interface element displayed in the graphical user interface.

5. The method of claim 1, wherein the at least one user interface element comprises a first user interface element and a second user interface element in the graphical user interface, wherein the second user interface element is different from the first user interface element, and wherein a first input processing method corresponds to the first user interface element and a second input processing method corresponds to the second user interface element.

6. The method of claim 1 further comprising changing the at least one user interface element as a result of the processing determining that the user input image controls the at least one user interface elements which is displayed in the graphical user interface.

7. The method of claim 1, wherein the at least one user interface element is a button-type element, and wherein the processing comprises:

identifying a number of first pixels activated by the user input image and which are within an area defined by a geometric boundary of the button-type element;

weighting the number of first pixels to provide a first value;

identifying a number of second pixels which are proximate to the button-type element but outside the geometric boundary of the button-type element;

combining the number of first pixels without the weighting and the number of second pixels to provide a second value;

subtracting the second value from the first value to provide a third value; and

comparing the third value with a threshold to determine whether the user input image controls the button-type element.

8. The method of claim 5, wherein a second user interface element is a scroll bar-type element, and wherein the processing comprises identifying a motion vector of the user input image and comparing the motion vector with respect to the scroll bar-type element to determine whether the user input image controls the scroll bar-type element.

9. The method of claim 5, wherein the first user interface element is a button-type element, and wherein the processing comprises identifying a number of pixels activated by the user input image and which are within an area corresponding to the button-type element and comparing the number of pixels with respect to a threshold to determine whether the user input image controls the button-type element.

10. The method of claim 1, wherein each of the plurality of user interface elements corresponds to a type of a user interface element, and the method further comprising accessing information regarding the type of the at least one user interface element displayed in the graphical user interface.

11. A system comprising:

processing circuitry;

at least one computer-readable storage medium comprising programming stored thereon that is configured to cause the processing circuitry to perform processing comprising;

first processing a user input image received by a graphical user interface using a first user input processing method which corresponds to a first type of user interface element, the first processing including;

identifying a number of first pixels activated by the user input image and which are within an area defined by a geometric boundary of the user interface element,

weighting the number of first pixels to provide a first value,

identifying a number of second pixels which are proximate to the user interface element but outside the geometric boundary of the user interface element,

combining the number of first pixels without the weighting and the number of second pixels to provide a second value,

subtracting the second value from the first value to provide a third value, and

comparing the third value with a threshold to determine whether the user input image controls the user interface element;

second processing the user input image using a second user input processing method which is different than the first user input processing method and which corresponds to a second type of user interface element, the second type different from the first type;

using the first processing, determining whether the user input image controls one of the user interface elements that is displayed on the display screen, which is the first type of user interface element; and

using the second processing, determining whether the user input image controls another user interface element that is displayed on the display screen which is the second type of user interface element.

12. The system of claim 11, wherein the programming is configured to cause the processing circuitry to perform processing that further comprises receiving the user input image resulting from a plurality of user input objects interacting with the graphical user interface at substantially the same moment in time.

13. The system of claim 11, wherein the programming is configured to cause the processing circuitry to perform processing that further comprises processing the user input image to identify at least one of a plurality of different types of user input objects interacting with the graphical user interface.

14. The system of claim 11, wherein the first processing comprises determining a region of a plurality of pixels of the graphical user interface selected by the user input image and comparing the region with the one of the user interface elements.

15. The system of claim 11, wherein the programming is configured to cause the processing circuitry to perform processing that further comprises accessing information regarding any user interface elements displayed in the graphical user interface, and further comprising selecting the first and second user input processing methods using the information.

16. A system comprising:

processing circuitry coupled with an interactive display, wherein the processing circuitry is configured to access information regarding one of a plurality of types of user interface elements which is displayed in the graphical user interface, to access the user input image, to select one of a plurality of different user input processing methods which corresponds to the one of the plurality of types of user interface elements using the information regarding the one of the plurality of types of user interface elements, the user input processing methods being different for each different type of user interface element, and to process the user input image using the selected one of the user input processing methods to determine if the user input image controls the one of the user interface elements which is displayed in the graphical user interface, the processing including;

identifying a number of first pixels activated by the user input image and which are within an area defined by a geometric boundary of the user interface element,

weighting the number of first pixels to provide a first value,

identifying a number of second pixels which are proximate to the user interface element but outside the geometric boundary of the user interface element,

combining the number of first pixels without the weighting and the number of second pixels to provide a second value,

subtracting the second value from the first value to provide a third value, and

comparing the third value with a threshold to determine whether the user input image controls the user interface element.

17. The system of claim 16, wherein the interactive display comprises a multi-touch interface which is configured to receive the user input image resulting from a plurality of user input objects interacting with the multi-touch interface at substantially the same moment in time.

18. The system of claim 16 wherein the user interface element of the graphical user interface comprises a first user interface element, wherein the graphical user interface includes a second user interface element which is another of the plurality of types of user interface elements.

19. The system of claim 16, wherein the user interface element is a button-type element and wherein the interactive display is a multi-touch interface configured to receive the user input image resulting from a plurality of user input objects interacting with the multi-touch interface at substantially the same moment in time, and wherein the processing circuitry is configured to select one of a plurality of different user input processing methods which corresponds to the button-type element, to process the user input image using the one of the user input processing methods including identifying a region of pixels activated by one of the user input objects interacting with the multi-touch interface and which region overlaps with an area of the button-type element and comparing the region of pixels with a threshold to determine if the user input controls the button-type element, and to change the button-type element if it is determined that the user input image controls the button-type element.

20. The system of claim 16, wherein the processing circuitry is configured to access information regarding a type of one of the plurality of types of user interface elements which is displayed in the graphical user interface.