Multi-modal interaction for a screen magnifier

• US 8,176,438 B2
• Filed: 09/26/2008
• Issued: 05/08/2012
• Est. Priority Date: 09/26/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zaman, Nazia, Reid, Paul J.
• Primary Examiner(s): Hailu, Tadeese
• Application Number:
  US12/238,464
• Publication Number:
  US 20100079498A1
• Time in Patent Office: 1,320 Days
• Field of Search: 715/862, 715/863, 715/815, 715/800, 715/801, 715/786, 715/857, 345/661
• US Class Current: 715/815
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04805 Virtual magnifying lens, i.e. window or frame movable on top of displayed information to enlarge it for better reading or selection
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A multi-model magnification system is described that can be used with multiple forms of input, including natural input such as touch-based devices. Users can zoom in/out and pan with their preferred mode of interaction, including mouse, keyboard, pen, and touch interactions. The system provides dedicated features for some modes, such as pinch/scroll gestures for touch input, that allow users to easily zoom in/out and pan the magnified area (e.g., on a touch laptop). The system provides an interactive panning mode in which the user the user can interact with the desktop or applications (e.g., clicking on icons) and pan using special gestures. When the system is in the panning mode, the system consumes received gestures or other input, otherwise the system passes received input to a magnified application.

Claims

1. A computer-implemented method for responding to multiple modes of user input in a screen magnifier application, the method comprising:

receiving input from one or more input devices, wherein the input devices comprise multiple modes of user input;

determining whether the received input is a touch-based gesture, wherein a touch-based gesture is a gesture produced by a user touching at least two fingers to an input device;

if the received input is a touch-based gesture,

detecting a type of the gesture, and

modifying a magnified area based on the detected type of the gesture; and

if the received input is not a touch-based gesture, modifying the magnified area based on the received input, wherein the screen magnifier application and a magnified application for which the screen magnifier application provides a magnified view are separate applications that run on top of an operation system.

2. The method of claim 1 wherein, when the detected type of the gesture is a pinch, modifying the magnified area comprises zooming in or out of the magnified area based on whether the pinch gesture was an expanding or contracting pinch gesture.

3. The method of claim 1 wherein, when the detected type of the gesture is a multi-finger slide, modifying the magnified area comprises panning contents of the magnified area based on a direction of the slide.

4. The method of claim 1 wherein the multiple modes comprise at least mouse input, keyboard input, and touch input.

5. A computer system for handling input associated with a magnifier application, the system comprising:

an input device configured to receive input from a user and communicate the input to an operating system;

an input detection component configured to convert the received input into coordinates of a displayed cursor, wherein when a user moves the input device, the input detection component moves the displayed cursor;

a location identification component configured to identify one or more user interface elements present at a current location of the displayed cursor;

a mode selection component configured to select between a panning mode in which the magnifier application consumes received input or an interaction mode in which the magnifier application forwards the received input to a magnified application;

a display device configured to display a graphical representation of one or more applications and a magnified view of at least a portion of the graphical representation; and

a magnification component configured to generate the magnified view from a selected area of the graphical representation.

6. The system of claim 5 wherein the input device component comprises a touch screen that receives a location of the screen that the user touches and gestures that describe a type of touch at the received location.

7. The system of claim 5 wherein the input detection component invokes an operating system API to determine a type of device that generated the received input.

8. The system of claim 5 wherein the mode selection component further comprises:

a panning component configured to consume received input and modify behavior of the magnifier application, and a forwarding component configured to pass received user input to the magnified application.

9. The system of claim 8 wherein the forwarding component forwards messages to the magnified application in a way that the magnified application is not aware that the system received and forwarded the user input.

10. The system of claim 5 wherein the magnification component receives a bitmap of the selected area and performs a stretch blit to display a larger than usual view of the selected area in the magnified view.

11. The system of claim 5 further comprising a configuration component configured to receive configuration information from the user, and wherein the mode selection component is further configured to access the configuration information to determine which mode to select.

12. A computer-readable storage device encoded with instructions for controlling a computer system to process input when a screen magnifier that displays magnified contents of a screen of the computer system is running, by a method comprising:

registering to receive gesture-based touch input;

receiving input from a user input device, wherein the input indicates a location selected by the user;

determining whether a user has indicated for the screen magnifier to consume received input or whether the user has indicated for the screen magnifier to forward received input to a magnified application, wherein the screen magnifier and the magnified application are separate applications running on an operating system of the computer system;

if the user has indicated for the screen magnifier to consume received input, processing the received user input within the screen magnifier to modify the behavior of the screen magnifier; and

if the user has indicated for the screen magnifier to forward received input to the magnified application, forwarding the received input to the magnified application.

13. The device of claim 12 wherein registering to receive gesture-based touch input comprises calling an operating system API to indicate that the screen magnifier handles gesture-based touch input.

14. The device of claim 12 wherein the magnified application comprises an operating system desktop that is in the magnified contents.

15. The device of claim 12 wherein determining whether a user has indicated for the screen magnifier to consume received input comprises detecting a state of a toggled user interface setting.

16. The device of claim 12 wherein determining whether a user has indicated for the screen magnifier to consume received input comprises automatically determining a mode to use based on the received user input.

17. The device of claim 12 wherein processing the received user input within the screen magnifier comprises intercepting the received user input and not passing the received user input to the magnified application.

18. The device of claim 12 wherein when the received user input is a pinch gesture and the user has indicated for the screen magnifier to consume received input, processing the received user input within the screen magnifier comprises zooming the magnified contents in or out based on the received input.

19. The device of claim 12 wherein when the received user input is a slide gesture and the user has indicated for the screen magnifier to consume received input, processing the received user input within the screen magnifier comprises panning the magnified contents based on the received input.

20. The device of claim 12 further comprising, if the user has indicated for the screen magnifier to consume received input, modifying a displayed cursor to indicate that the screen magnifier will consume received input.