Instantaneous speaking of content on touch devices

• US 10,592,095 B2
• Filed: 09/18/2014
• Issued: 03/17/2020
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kasemset, Clare T., Fleizach, Christopher B.
• Primary Examiner(s): McIntosh, Andrew T
• Application Number:
  US14/490,397
• Publication Number:
  US 20150339049A1
• Time in Patent Office: 2,007 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers

Abstract

Systems and processes are disclosed for initiating and controlling content speaking on touch-sensitive devices. A gesture can be detected on a touchscreen for causing text to be spoken. Displayed content can be analyzed, and a determination can be made based on size, position, and other attributes as to which portion of displayed text should be spoken. In response to detecting the gesture, the identified portion of text can be spoken using a text-to-speech process. A menu of controls can be displayed for controlling the speaking. The menu can automatically be hidden and a persistent virtual button can be displayed that can remain available on the touchscreen despite the user navigating to another view. Selecting the persistent virtual button can restore the full menu of controls, thereby allowing the user to continue to control the speaking even after navigating away from the content being spoken.

Claims

1. A method for speaking touchscreen content, the method comprising:

at an electronic device having a processor, memory, and a touchscreen;

displaying, on the touchscreen, a content container containing a plurality of portions of text,

wherein the content container is one of a plurality of content containers displayed on the touchscreen, and

wherein each portion of text of the plurality of portions of text occupies a respective area and a respective position in the content container;

detecting a gesture on the touchscreen, wherein the gesture does not identify a first portion of text of the plurality of portions of text;

in response to detecting the gesture;

determining the first portion of text to be spoken based on;

comparing an attribute of the content container relative to an attribute of at least one other content container of the plurality of content containers;

comparing a size of the respective area occupied by the first portion of text relative to a size of the respective area occupied by a second portion of text of the plurality of portions of text; and

the respective position occupied by the first portion of text in the content container; and

speaking the determined first portion of text using a text-to-speech process.

2. The method of claim 1, wherein the gesture comprises a swipe of two input points on the touchscreen from an edge of the touchscreen toward a center of the touchscreen.

3. The method of claim 1, wherein the gesture comprises a tapping of three input points on the touchscreen.

4. The method of claim 1, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

in an email application, selecting one of a sender field, a subject field, and a message body field as the determined first portion.

5. The method of claim 1, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

in a messaging application, selecting one or more text messages in reverse chronological order as the determined first portion.

6. The method of claim 1, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

where two or more applications are displayed on the touchscreen, selecting text in an application with focus as the determined first portion.

7. The method of claim 1, further comprising:

in response to reaching an end of the first portion of text displayed on the touchscreen, speaking additional text associated with the first portion of text displayed on the touchscreen.

8. The method of claim 7, wherein the additional text comprises a next paragraph, a next page, a next message, or a next email.

9. The method of claim 1, further comprising:

while text is being spoken, displaying a menu of controls on the touchscreen for controlling the speaking, wherein the menu of controls comprises a pause control and a speed control;

in response to a predetermined time elapsing without selection of a menu control, ceasing to display the menu of controls on the touchscreen and displaying a persistent virtual button;

in response to selection of the persistent virtual button, ceasing to display the persistent virtual button and displaying the menu of controls on the touchscreen;

receiving a user command to display new content on the touchscreen; and

in response to receiving the user command, displaying the new content, wherein the persistent virtual button is overlaid over the new content.

10. The method of claim 9, wherein the menu of controls further comprises a skip forward control and a skip backward control.

11. The method of claim 10, further comprising:

determining a text skipping distance based on a length of the portion of text being spoken;

in response to selection of the skip forward control, causing the speaking to skip forward in the portion of text by the text skipping distance; and

in response to selection of the skip backward control, causing the speaking to skip backward in the portion of text by the text skipping distance.

12. The method of claim 1, wherein determining the first portion of text to be spoken comprises:

determining that the size of the respective area occupied by the first portion of text is larger or smaller than the size of the respective area occupied by the second portion of text.

13. The method of claim 1, wherein the attribute of the content container is a size of a respective area occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a size of a respective area occupied by the at least one other content container on the touchscreen.

14. The method of claim 1, wherein the attribute of the content container is a respective position occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a respective position occupied by the at least one other content container on the touchscreen.

15. The method of claim 1, wherein determining the first portion of text to be spoken is further based on metadata associated with the first portion of text, and wherein the metadata indicates a type of content corresponding to the first portion of text.

16. A method for speaking displayed content, the method comprising:

at an electronic device having a processor, memory, and a touchscreen;

displaying, on the touchscreen, a content container containing a plurality of portions of text,

wherein the content container is one of a plurality of content containers displayed on the touchscreen, and

wherein each portion of text of the plurality of portions of text occupies a respective area and a respective position in the content container;

receiving a voice command, wherein the voice command does not identify a first portion of text of the plurality of portions of text;

in response to receiving the voice command;

determining the first portion of text to be spoken based on;

comparing an attribute of the content container relative to an attribute of at least one other content container of the plurality of content containers;

comparing a size of the respective area occupied by the first portion of text relative to a size of the respective area occupied by a second portion of text of the plurality of portions of text; and

the respective position occupied by the first portion of text in the content container; and

speaking the determined portion of text using a text-to-speech process.

17. The method of claim 16, wherein determining the first portion of displayed text to be spoken comprises:

in a messaging application, selecting one or more text messages in reverse chronological order as the determined first portion.

18. The method of claim 16, wherein determining the first portion of displayed text to be spoken comprises:

where two or more applications are displayed on the display, selecting text in an application with focus as the determined first portion.

19. The method of claim 16, further comprising:

in response to reaching an end of the first portion of text displayed on the touchscreen, speaking additional text associated with the first portion of text displayed on the touchscreen.

20. The method of claim 16, further comprising:

while text is being spoken, displaying a menu of controls on the touchscreen for controlling the speaking, wherein the menu of controls comprises a pause control and a speed control;

in response to a predetermined time elapsing without selection of a menu control, ceasing to display the menu of controls on the touchscreen and displaying a persistent virtual button;

in response to selection of the persistent virtual button, ceasing to display the persistent virtual button and displaying the menu of controls on the touchscreen;

receiving a user command to display new content on the touchscreen; and

in response to receiving the user command, displaying the new content, wherein the persistent virtual button is overlaid over the new content.

21. The method of claim 16, wherein determining the first portion of text to be spoken comprises:

determining that the size of the respective area occupied by the first portion of text is larger or smaller than the size of the respective area occupied by the second portion of text.

22. The method of claim 16, wherein the attribute of the content container is a size of a respective area occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a size of a respective area occupied by the at least one other content container on the touchscreen.

23. The method of claim 16, wherein the attribute of the content container is a respective position occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a respective position occupied by the at least one other content container on the touchscreen.

24. The method of claim 16, wherein determining the first portion of text to be spoken is further based on metadata associated with the first portion of text, and wherein the metadata indicates a type of content corresponding to the first portion of text.

25. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by an electronic device having a processor, memory, and a touchscreen, the one or more programs including instructions for:

displaying, on the touchscreen, a content container containing a plurality of portions of text,

wherein the content container is one of a plurality of content containers displayed on the touchscreen, and

wherein each portion of text of the plurality of portions of text occupies a respective area and a respective position in the content container;

detecting a gesture on the touchscreen, wherein the gesture does not identify a first portion of text of the plurality of portions of text;

in response to detecting the gesture;

determining the first portion of text to be spoken based on;

comparing an attribute of the content container relative to an attribute of at least one other content container of the plurality of content containers;

comparing a size of the respective area occupied by the first portion of text relative to a size of the respective area occupied by a second portion of text of the plurality of portions of text; and

the respective position occupied by the first portion of text in the content container; and

speaking the determined first portion of text using a text-to-speech process.

26. The non-transitory computer-readable storage medium of claim 25, wherein the gesture comprises a swipe of two input points on the touchscreen from an edge of the touchscreen toward a center of the touchscreen.

27. The non-transitory computer-readable storage medium of claim 25, wherein the gesture comprises a tapping of three input points on the touchscreen.

28. The non-transitory computer-readable storage medium of claim 25, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

in an email application, selecting one of a sender field, a subject field, and a message body field as the determined first portion.

29. The non-transitory computer-readable storage medium of claim 25, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

in a messaging application, selecting one or more text messages in reverse chronological order as the determined first portion.

30. The non-transitory computer-readable storage medium of claim 25, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

where two or more applications are displayed on the touchscreen, selecting text in an application with focus as the determined first portion.

31. The non-transitory computer-readable storage medium of claim 25, wherein the one or more programs further include instructions for:

in response to reaching an end of the first portion of text displayed on the touchscreen, speaking additional text associated with the first portion of text displayed on the touchscreen.

32. The non-transitory computer-readable storage medium of claim 25, wherein the one or more programs further include instructions for:

while text is being spoken, displaying a menu of controls on the touchscreen for controlling the speaking, wherein the menu of controls comprises a pause control and a speed control;

in response to a predetermined time elapsing without selection of a menu control, ceasing to display the menu of controls on the touchscreen and displaying a persistent virtual button;

in response to selection of the persistent virtual button, ceasing to display the persistent virtual button and displaying the menu of controls on the touchscreen;

receiving a user command to display new content on the touchscreen; and

in response to receiving the user command, displaying the new content, wherein the persistent virtual button is overlaid over the new content.

33. The non-transitory computer-readable storage medium of claim 25, wherein determining the first portion of text to be spoken comprises:

determining that the size of the respective area occupied by the first portion of text is larger or smaller than the size of the respective area occupied by the second portion of text.

34. The non-transitory computer-readable storage medium of claim 25, wherein the attribute of the content container is a size of a respective area occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a size of a respective area occupied by the at least one other content container on the touchscreen.

35. The non-transitory computer-readable storage medium of claim 25, wherein the attribute of the content container is a respective position occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a respective position occupied by the at least one other content container on the touchscreen.

36. The non-transitory computer-readable storage medium of claim 25, wherein determining the first portion of text to be spoken is further based on metadata associated with the first portion of text, and wherein the metadata indicates a type of content corresponding to the first portion of text.

37. A system comprising:

one or more processors;

memory;

a touchscreen; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying, on the touchscreen, a content container containing a plurality of portions of text,

wherein the content container is one of a plurality of content containers displayed on the touchscreen, and

wherein each portion of text of the plurality of portions of text occupies a respective area and a respective position in the content container;

detecting a gesture on the touchscreen, wherein the gesture does not identify a first portion of text of the plurality of portions of text;

in response to detecting the gesture;

determining the first portion of text to be spoken based on;

comparing an attribute of the content container relative to an attribute of at least one other content container of the plurality of content containers;

comparing a size of the respective area occupied by the first portion of text relative to a size of the respective area occupied by a second portion of text of the plurality of portions of text; and

the respective position occupied by the first portion of text in the content container; and

speaking the determined first portion of text using a text-to-speech process.

38. The system of claim 37, wherein the gesture comprises a swipe of two input points on the touchscreen from an edge of the touchscreen toward a center of the touchscreen.

39. The system of claim 37, wherein the gesture comprises a tapping of three input points on the touchscreen.

40. The system of claim 37, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

in an email application, selecting one of a sender field, a subject field, and a message body field as the determined first portion.

41. The system of claim 37, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

in a messaging application, selecting one or more text messages in reverse chronological order as the determined first portion.

42. The system of claim 37, wherein determining the first portion of text displayed on the touchscreen to be spoken comprises:

where two or more applications are displayed on the touchscreen, selecting text in an application with focus as the determined first portion.

43. The system of claim 37, wherein the one or more programs further include instructions for:

in response to reaching an end of the first portion of text displayed on the touchscreen, speaking additional text associated with the first portion of text displayed on the touchscreen.

44. The system of claim 37, wherein the one or more programs further include instructions for:

while text is being spoken, displaying a menu of controls on the touchscreen for controlling the speaking, wherein the menu of controls comprises a pause control and a speed control;

in response to a predetermined time elapsing without selection of a menu control, ceasing to display the menu of controls on the touchscreen and displaying a persistent virtual button;

in response to selection of the persistent virtual button, ceasing to display the persistent virtual button and displaying the menu of controls on the touchscreen;

receiving a user command to display new content on the touchscreen; and

in response to receiving the user command, displaying the new content, wherein the persistent virtual button is overlaid over the new content.

45. The system of claim 37, wherein determining the first portion of text to be spoken comprises:

determining that the size of the respective area occupied by the first portion of text is larger or smaller than the size of the respective area occupied by the second portion of text.

46. The system of claim 37, wherein the attribute of the content container is a size of a respective area occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a size of a respective area occupied by the at least one other content container on the touchscreen.

47. The system of claim 37, wherein the attribute of the content container is a respective position occupied by the content container on the touchscreen, and wherein the attribute of the at least one other content container of the plurality of content containers is a respective position occupied by the at least one other content container on the touchscreen.

48. The system of claim 37, wherein determining the first portion of text to be spoken is further based on metadata associated with the first portion of text, and wherein the metadata indicates a type of content corresponding to the first portion of text.