PROVIDING TEXT INPUT USING SPEECH DATA AND NON-SPEECH DATA

• US 20120166192A1
• Filed: 11/18/2011
• Published: 06/28/2012
• Est. Priority Date: 02/22/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Yanagihara, Kazuhisa
• Granted Patent: US 8,688,446 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Systems, methods, and computer readable media providing a speech input interface. The interface can receive speech input and non-speech input from a user through a user interface. The speech input can be converted to text data and the text data can be combined with the non-speech input for presentation to a user.

Claims

1. A system comprising:

a mobile device configured to perform operations comprising;

receiving a speech input, the speech input including a speech data stream;

while receiving the speech input, receiving a non-speech input and time information associated with the non-speech input, the time information operable to specify a sequence in which the speech input and non-speech input are assembled, wherein the time information includes a start streaming node and a stop streaming node;

retrieving text data from the speech input using speech recognition;

assembling the text data with the non-speech input based on the time information to produce output data; and

providing the output data to a user through a user interface.

2. The system of claim 1, wherein the time information includes a timestamp of the non-speech input.

3. The system of claim 2, wherein the timestamp indicates a relative time of the non-speech input to the speech input.

4. The system of claim 3, wherein assembling the text data with the non-speech input comprises multiplexing a stream of the speech input and a stream of the non-speech input based on the timestamp.

5. The system of claim 1, wherein:

the start streaming node is operable to indicate a time when the user selects to enable a speech input mode of a computing device, and

the stop streaming node is operable to indicate a time when the user selects to disable a speech input mode of the computing device.

6. The system of claim 1, wherein retrieving text data from the speech input includes one of:

communicating with a speech recognition module located remotely from the one or more;

or

retrieving the text data using with a speech recognition module of the one or more computing devices.

7. The system of claim 1, comprising a communication interface, wherein the communication interface is operable to multiplex the speech input and the non-speech input into a single communication stream.

8. A computer-implemented method comprising:

receiving, by a mobile device, speech data and non-speech data, at least one of the speech data or the non-speech data being associated with time information, the time information operable to specify a sequence in which the speech data and non-speech data are assembled;

retrieving, by the mobile device, text data from the speech data using speech recognition;

assembling, by the mobile device, the text data with the non-speech data based on the time information to produce output data;

performing, by the mobile device, a function using the output data as a parameter; and

providing, by the mobile device, a result of the function for presentation in a user interface.

9. The method of claim 8, wherein assembling the text data with the non-speech data comprises synchronizing the speech data and the non-speech data.

10. The method of claim 9, wherein synchronizing the speech data and the non-speech data comprises combining a speech data stream and a non-speech data using a real-time transport protocol (RTP) or real-time streaming protocol (RTSP).

11. The method of claim 9, wherein synchronizing the speech data and the non-speech data comprises comparing a sequence indicator of the speech data and a sequence indication of the non-speech data.

12. The method of claim 8, wherein receiving the speech data includes receiving the speech data using a microphone.

13. One or more non-transitory computer readable media storing software program code operable to cause a mobile device to perform operations comprising:

receiving speech data and non-speech data, at least one of the speech data or the non-speech data being associated with time information, the time information operable to specify a sequence in which the speech data and non-speech data are assembled;

retrieving text data from the speech data using speech recognition;

assembling the text data with the non-speech data based on the time information to produce output data;

performing a function using the output data as a parameter; and

providing a result of the function for presentation in a user interface.

14. The computer readable media of claim 13, wherein the time information includes a timestamp of the non-speech data, the timestamp indicating a relative time of the non-speech data to the speech data.

15. A system comprising:

a mobile device configured to perform operations comprising;

receiving speech data and non-speech data, at least one of the speech data or the non-speech data being associated with time information, the time information operable to specify a sequence in which the speech data and non-speech data are assembled;

retrieving text data from the speech data using speech recognition;

assembling the text data with the non-speech data based on the time information to produce output data;

performing a function using the output data as a parameter; and

providing a result of the function for presentation in a user interface.

16. A computer-implemented method, comprising:

receiving a speech input, the speech input including a speech data stream;

while receiving the speech input, receiving a non-speech input and time information associated with the non-speech input, the time information operable to specify a sequence in which the speech input and non-speech input are assembled, wherein the time information includes a start streaming node and a stop streaming node;

retrieving text data from the speech input using speech recognition;

assembling the text data with the non-speech input based on the time information to produce output data; and

providing the output data to a user through a user interface.

17. The method of claim 16, further comprising multiplexing the speech input and the non-speech input into a single communication stream.

18. One or more non-transitory computer readable media storing software program code operable cause a mobile device to perform operations comprising:

receiving a speech input, the speech input including a speech data stream;

while receiving the speech input, receiving a non-speech input and time information associated with the non-speech input, the time information operable to specify a sequence in which the speech input and non-speech input are assembled, wherein the time information includes a start streaming node and a stop streaming node;

retrieving text data from the speech input using speech recognition;

assembling the text data with the non-speech input based on the time information to produce output data; and

providing the output data to a user through a user interface.

19. The media of claim 18, wherein the time information includes a timestamp of the non-speech input, the timestamp indicating a relative time of the non-speech input to the speech input.

20. The media of claim 19, wherein assembling the text data with the non-speech input comprises multiplexing a stream of the speech input and a stream of the non-speech input based on the timestamp.

The media of claim 18, wherein assembling the text data with the non-speech input comprises multiplexing a stream of the speech input and a stream of the non-speech input based on the timestamp.