Device, method, and graphical user interface for integrating recognition of handwriting gestures with a screen reader

• US 9,633,191 B2
• Filed: 10/17/2014
• Issued: 04/25/2017
• Est. Priority Date: 03/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Pedersen, II, Michael M., Fleizach, Christopher B., Minifie, Darren C., Hughes, Gregory F., Dour, Ryan N., Fisch, Ian M., Lopes Da Silva, Joel M., Seymour, Eric T., Naik, Devang K., Dixon, Ryan S.
• Primary Examiner(s): Korsak, Oleg
• Application Number:
  US14/517,691
• Publication Number:
  US 20150040213A1
• Time in Patent Office: 921 Days
• Field of Search: 726 13, 345175
• CPC Class Codes: G06F 21/36 by graphic or iconic representation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

While an electronic device with a display and a touch-sensitive surface is in a screen reader accessibility mode, the device displays an application launcher screen including a plurality of application icons. A respective application icon corresponds to a respective application stored in the device. The device detects a sequence of one or more gestures on the touch-sensitive surface that correspond to one or more characters. A respective gesture that corresponds to a respective character is a single finger gesture that moves across the touch-sensitive surface along a respective path that corresponds to the respective character. The device determines whether the detected sequence of one or more gestures corresponds to a respective application icon of the plurality of application icons, and, in response to determining that the detected sequence of one or more gestures corresponds to the respective application icon, performs a predefined operation associated with the respective application icon.

Claims

1. A method, comprising:

while an electronic device with a display and a touch-sensitive surface is in a screen reader accessibility mode;

displaying at least a portion of a web page on the display, the web page including a plurality of user interface elements;

detecting a first navigation gesture on the touch-sensitive surface;

in response to detecting the first navigation gesture on the touch-sensitive surface, navigating to a first set of one or more user interface elements of the plurality of user interface elements that corresponds to a current navigable unit type, wherein the current navigable unit type is set to a first navigable unit type selected from a plurality of predefined navigable unit types, wherein the plurality of predefined navigable unit types includes at least four different navigable unit types including the first navigable unit type, a second navigable unit type, a third navigable unit type, and a fourth navigable unit type;

detecting a navigation setting gesture on the touch-sensitive surface that corresponds to a respective character, wherein;

the navigation setting gesture that corresponds to the respective character is a single finger gesture that moves across the touch-sensitive surface along a respective path that corresponds to the respective character;

determining whether the navigation setting gesture corresponds to one of the second navigable unit type, the third navigable unit type, and the fourth navigable unit type;

in accordance with a determination that the navigation setting gesture corresponds to the second navigable unit type of the plurality of predefined navigable unit types;

changing the current navigable unit type from the first navigable unit type to the second navigable unit type; and

outputting audible accessibility information indicating that the second navigable unit type has been selected;

in accordance with a determination that the navigation setting gesture corresponds to the third navigable unit type of the plurality of predefined navigable unit types;

changing the current navigable unit type from the first navigable unit type to the third navigable unit type; and

outputting audible accessibility information indicating that the third navigable unit type has been selected;

in accordance with a determination that the navigation setting gesture corresponds to the fourth navigable unit type of the plurality of predefined navigable unit types;

changing the current navigable unit type from the first navigable unit type to the fourth navigable unit type; and

outputting audible accessibility information indicating that the fourth navigable unit type has been selected;

after changing the current navigable unit type, detecting a second navigation gesture on the touch-sensitive surface; and

,

in response to detecting the second navigation gesture on the touch-sensitive surface, navigating to a second set of one or more user interface elements of the plurality of user interface elements that corresponds to the changed current navigable unit type.

2. The method of claim 1, wherein:

the respective path traverses one or more locations on the touch-sensitive surface that corresponds to one or more user interface elements of the plurality of user interface elements in the web page without activating the one or more user interface elements.

3. The method of claim 1, wherein the respective path is in the shape of the respective character.

4. The method of claim 1, wherein the respective gesture is independent of contacting a location on the touch-sensitive surface that corresponds to any of the plurality of user interface elements.

5. The method of claim 1, wherein:

the plurality of predefined navigable unit types includes four or more navigable unit types in a circular sequence so that a navigable unit type that has a second position in the circular sequence is deemed to be adjacent to a navigable unit type that has a first position in the circular sequence, the navigable unit type that has a third position in the circular sequence is deemed to be adjacent to the navigable unit type that has the second position in the circular sequence, and a navigable unit type that has a last position in the circular sequence is deemed to be adjacent to the navigable unit type that has the first position in the circular sequence; and

the first navigable unit type is not adjacent to the second navigable unit type in the circular sequence.

6. The method of claim 5, including:

detecting a second navigation setting gesture on the touch-sensitive surface, wherein the second navigation setting gesture is a multi-finger gesture, and the second navigation setting gesture is independent of contacting a location on the touch-sensitive surface that corresponds to any of the plurality of user interface elements; and

,

in response to detecting the second navigation setting gesture on the touch-sensitive surface;

changing the current navigable unit type from the second navigable unit type to a third navigable unit type that is adjacent to the second navigable unit type in the sequence of the plurality of navigable unit types; and

outputting accessibility information about the third navigable unit type.

7. The method of claim 1, including:

while the electronic device is in the screen reader accessibility mode, prior to detecting the navigation setting gesture on the touch-sensitive surface;

detecting a predefined input; and

,

in response to detecting the predefined input, entering a handwriting gesture recognition mode within the screen reader accessibility mode.

8. The method of claim 1, wherein a label for the second navigable unit type starts with the respective character.

9. An electronic device, comprising:

a display;

a touch-sensitive surface;

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

while the electronic device is in a screen reader accessibility mode;

displaying at least a portion of a web page on the display, the web page including a plurality of user interface elements;

detecting a first navigation gesture on the touch-sensitive surface;

in response to detecting the first navigation gesture on the touch-sensitive surface, navigating to a first set of one or more user interface elements of the plurality of user interface elements that corresponds to a current navigable unit type, wherein the current navigable unit type is set to a first navigable unit type selected from a plurality of predefined navigable unit types, wherein the plurality of predefined navigable unit types includes at least four different navigable unit types including the first navigable unit type, a second navigable unit type, a third navigable unit type, and a fourth navigable unit type;

detecting a navigation setting gesture on the touch-sensitive surface that corresponds to a respective character, wherein;

the navigation setting gesture that corresponds to the respective character is a single finger gesture that moves across the touch-sensitive surface along a respective path that corresponds to the respective character;

determining whether the navigation setting gesture corresponds to one of the second navigable unit type, the third navigable unit type, and the fourth navigable unit type;

in accordance with a determination that the navigation setting gesture corresponds to the second navigable unit type of the plurality of predefined navigable unit types;

changing the current navigable unit type from the first navigable unit type to the second navigable unit type; and

outputting audible accessibility information indicating that the second navigable unit type has been selected;

in accordance with a determination that the navigation setting gesture corresponds to the third navigable unit type of the plurality of predefined navigable unit types;

changing the current navigable unit type from the first navigable unit type to the third navigable unit type; and

outputting audible accessibility information indicating that the third navigable unit type has been selected;

in accordance with a determination that the navigation setting gesture corresponds to the fourth navigable unit type of the plurality of predefined navigable unit types;

changing the current navigable unit type from the first navigable unit type to the fourth navigable unit type; and

outputting audible accessibility information indicating that the fourth navigable unit type has been selected;

after changing the current navigable unit type, detecting a second navigation gesture on the touch-sensitive surface; and

,

in response to detecting the second navigation gesture on the touch-sensitive surface, navigating to a second set of one or more user interface elements of the plurality of user interface elements that corresponds to the changed current navigable unit type.

10. The electronic device of claim 9, wherein:

the respective path traverses one or more locations on the touch-sensitive surface that corresponds to one or more user interface elements of the plurality of user interface elements in the web page without activating the one or more user interface elements.

11. The electronic device of claim 9, wherein the respective path is in the shape of the respective character.

12. The electronic device of claim 9, wherein the respective gesture is independent of contacting a location on the touch-sensitive surface that corresponds to any of the plurality of user interface elements.

13. The electronic device of claim 9, wherein:

the plurality of predefined navigable unit types includes four or more navigable unit types in a circular sequence so that a navigable unit type that has a second position in the circular sequence is deemed to be adjacent to a navigable unit type that has a first position in the circular sequence,

the navigable unit type that has a third position in the circular sequence is deemed to be adjacent to the navigable unit type that has the second position in the circular sequence, and a navigable unit type that has a last position in the circular sequence is deemed to be adjacent to the navigable unit type that has the first position in the circular sequence; and

the first navigable unit type is not adjacent to the second navigable unit type in the circular sequence.

14. The electronic device of claim 13, wherein the one or more programs further include instructions for:

detecting a second navigation setting gesture on the touch-sensitive surface, wherein the second navigation setting gesture is a multi-finger gesture, and the second navigation setting gesture is independent of contacting a location on the touch-sensitive surface that corresponds to any of the plurality of user interface elements; and

,

in response to detecting the second navigation setting gesture on the touch-sensitive surface;

changing the current navigable unit type from the second navigable unit type to a third navigable unit type that is adjacent to the second navigable unit type in the sequence of the plurality of navigable unit types; and

outputting accessibility information about the third navigable unit type.

15. The electronic device of claim 9, wherein the one or more programs further include instructions for:

while the electronic device is in the screen reader accessibility mode, prior to detecting the navigation setting gesture on the touch-sensitive surface;

detecting a predefined input; and

,

in response to detecting the predefined input, entering a handwriting gesture recognition mode within the screen reader accessibility mode.

16. The electronic device of claim 9, wherein a label for the second navigable unit type starts with the respective character.

17. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a display and a touch-sensitive surface, cause the device to:

while the electronic device is in a screen reader accessibility mode;

display at least a portion of a web page on the display, the web page including a plurality of user interface elements;

detect a first navigation gesture on the touch-sensitive surface;

in response to detecting the first navigation gesture on the touch-sensitive surface, navigate to a first set of one or more user interface elements of the plurality of user interface elements that corresponds to a current navigable unit type, wherein the current navigable unit type is set to a first navigable unit type selected from a plurality of predefined navigable unit types, wherein the plurality of predefined navigable unit types includes at least four different navigable unit types including the first navigable unit type, a second navigable unit type, a third navigable unit type, and a fourth navigable unit type;

detect a navigation setting gesture on the touch-sensitive surface that corresponds to a respective character, wherein;

the navigation setting gesture that corresponds to the respective character is a single finger gesture that moves across the touch-sensitive surface along a respective path that corresponds to the respective character;

determine whether the navigation setting gesture corresponds to one of the second navigable unit type, the third navigable unit type, and the fourth navigable unit type;

in accordance with a determination that the navigation setting gesture corresponds to the second navigable unit type of the plurality of predefined navigable unit types;

change the current navigable unit type from the first navigable unit type to the second navigable unit type; and

output audible accessibility information indicating that the second navigable unit type has been selected;

in accordance with a determination that the navigation setting gesture corresponds to the third navigable unit type of the plurality of predefined navigable unit types;

change the current navigable unit type from the first navigable unit type to the third navigable unit type; and

output audible accessibility information indicating that the third navigable unit type has been selected;

in accordance with a determination that the navigation setting gesture corresponds to the fourth navigable unit type of the plurality of predefined navigable unit types;

change the current navigable unit type from the first navigable unit type to the fourth navigable unit type; and

output audible accessibility information indicating that the fourth navigable unit type has been selected

after changing the current navigable unit type, detect a second navigation gesture on the touch-sensitive surface; and

,

in response to detecting the second navigation gesture on the touch-sensitive surface, navigate to a second set of one or more user interface elements of the plurality of user interface elements that corresponds to the changed current navigable unit type.

18. The non-transitory computer readable storage medium of claim 17, wherein:

the respective path traverses one or more locations on the touch-sensitive surface that corresponds to one or more user interface elements of the plurality of user interface elements in the web page without activating the one or more user interface elements.

19. The non-transitory computer readable storage medium of claim 17, wherein the respective path is in the shape of the respective character.

20. The non-transitory computer readable storage medium of claim 17, wherein the respective gesture is independent of contacting a location on the touch-sensitive surface that corresponds to any of the plurality of user interface elements.

21. The non-transitory computer readable storage medium of claim 17, wherein:

the plurality of predefined navigable unit types includes four or more navigable unit types in a circular sequence so that a navigable unit type that has a second position in the circular sequence is deemed to be adjacent to a navigable unit type that has a first position in the circular sequence, the navigable unit type that has a third position in the circular sequence is deemed to be adjacent to the navigable unit type that has the second position in the circular sequence, and a navigable unit type that has a last position in the circular sequence is deemed to be adjacent to the navigable unit type that has the first position in the circular sequence; and

the first navigable unit type is not adjacent to the second navigable unit type in the circular sequence.

22. The non-transitory computer readable storage medium of claim 21, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

detect a second navigation setting gesture on the touch-sensitive surface, wherein the second navigation setting gesture is a multi-finger gesture, and the second navigation setting gesture is independent of contacting a location on the touch-sensitive surface that corresponds to any of the plurality of user interface elements; and

,

in response to detecting the second navigation setting gesture on the touch-sensitive surface;

change the current navigable unit type from the second navigable unit type to a third navigable unit type that is adjacent to the second navigable unit type in the sequence of the plurality of navigable unit types; and

output accessibility information about the third navigable unit type.

23. The non-transitory computer readable storage medium of claim 17, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

while the electronic device is in the screen reader accessibility mode, prior to detecting the navigation setting gesture on the touch-sensitive surface;

detect a predefined input; and

,

in response to detecting the predefined input, enter a handwriting gesture recognition mode within the screen reader accessibility mode.

24. The non-transitory computer readable storage medium of claim 17, wherein a label for the second navigable unit type starts with the respective character.