PERFORMING AN OPERATION RELATIVE TO TABULAR DATA BASED UPON VOICE INPUT

• US 20150019216A1
• Filed: 05/21/2014
• Published: 01/15/2015
• Est. Priority Date: 07/15/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ganjam, Kris, Gulwani, Sumit, Ju, Yun-Cheng, Chakrabarti, Kaushik, Singh, Prabhdeep, Marron, Mark
• Granted Patent: US 10,956,433 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 40/18 of spreadsheets form-filling G06F40/174
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Described herein are various technologies pertaining to performing an operation relative to tabular data based upon voice input. An ASR system includes a language model that is customized based upon content of the tabular data. The ASR system receives a voice signal that is representative of speech of a user. The ASR system creates a transcription of the voice signal based upon the ASR being customized with the content of the tabular data. The operation relative to the tabular data is performed based upon the transcription of the voice signal.

Claims

1. A computing device comprising:

a processor; and

a memory that comprises an application that is executed by the processor, the application comprises;

tabular data loaded into the application, the tabular data comprises a text string;

an executor component that executes a computing operation relative to the tabular data, the executor component comprises;

an automatic speech recognition (ASR) system that is customized based upon the text string being included in the tabular data, the ASR system receives a voice signal that is representative of voice input to the application, the voice input includes a reference to the text string, the ASR system generates a transcription of the voice signal; and

a table manipulation system that is in communication with the ASR system, the table manipulation system receives the transcription of the voice signal from the ASR system and performs the computing operation relative to the tabular data based upon the transcription of the voice signal.

2. The computing device of claim 1 being a client computing device, the client computing device being one of a tablet computing device or a mobile telephone.

3. The computing device of claim 1 being a server computing device, the ASR system of the application receives the voice signal from a client computing device in network communication with the server computing device.

4. The computing device of claim 1, the application being a spreadsheet application, the tabular data loaded into a spreadsheet of the spreadsheet application.

5. The computing device of claim 1, the ASR system comprises an acoustic model that models phones in a language of the spoken utterance, a lexicon model that defines probabilities over elements that include respective sequences of phones, and a language model that defines probabilities over sequences of elements, the language model updated to include at least one of the text string or a synonym of the text string responsive to the text string being loaded into the application.

6. The computing device of claim 1, wherein the ASR system is customized to constrain potential interpretations of the voice input.

7. The computing device of claim 1, wherein the voice input is a natural language query, and wherein the table manipulation system performs the computing operation based upon the natural language query.

8. The computing device of claim 1, the computing operation being one of a sort of the tabular data, a filter of the tabular data, a mathematical operation performed over entries in the tabular data, a visualization of the tabular data, or an augmentation of the tabular data.

9. The computing device of claim 8, the computing operation is the augmentation of the tabular data, and wherein the table manipulation system generates a query based upon the transcription, the search system executes a search over a network accessible index of tables based upon query augments the tabular data loaded into the application with additional tabular data included in the index of tables.

10. The computing device of claim 1, the table manipulation system generates a program based upon the transcription, the table manipulation system executes the program over at least a portion of the tabular data loaded into the application to perform the computing operation.

11. A method comprising:

receiving tabular data that has been loaded into a computer-executable application;

responsive to receiving the tabular data, updating a language model of an automatic speech recognition (ASR) system based upon the tabular data;

receiving a voice signal that is indicative of an operation to be performed with respect to the tabular data;

decoding the voice signal based upon the updating of the language model of the ASR system; and

performing the operation based upon the decoding of the voice signal.

12. The method of claim 11 executed by a client computing device, wherein the voice signal represents a query, and wherein the operation comprises:

augmenting the tabular data with additional data retrieved from a data source that is accessible to the client computing device by way of a network, the augmenting based upon the query; and

performing a subsequent operation over the tabular data and the additional data based upon the query.

13. The method of claim 11, the computer-executable application being a web browser.

14. The method of claim 11, the tabular data comprises an entry, the entry comprises a character sequence, and wherein updating the language model comprises including the character sequence in the language model.

15. The method of claim 14, the entry being one of a column header or a row header.

16. The method of claim 14, the language model comprises a partially completed phrase that is representative of historically observed commands, wherein updating the language model comprises including the character sequence in the language model to complete the partially completed phrase.

17. The method of claim 16, wherein updating the language model further comprises:

assigning a type to the character sequence based upon the tabular data, the type indicating that the character sequence is representative of one of people, places, or things in the tabular data; and

assigning a probability value to the completed phrase based upon the type assigned to the character sequence.

18. The method of claim 11, wherein receiving the voice signal comprises:

receiving an indication that a graphical button has been selected; and

monitoring output of a microphone responsive to receiving the indication.

19. The method of claim 11, wherein the voice signal represents a query, the performing of the operation causes updated tabular data to be generated, the method further comprising:

subsequent to performing the operation based upon the query, receiving a second voice signal that is representative of a second query;

decoding the second voice signal based upon the updating of the language model of the ASR system; and

performing a second operation relative to the updated tabular data, the second operation performed based upon the query and the second query.

20. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving tabular data loaded into a spreadsheet application, the tabular data comprises an entry, the entry comprises a character sequence;

responsive to receiving the tabular data, updating a language model of an automatic speech recognition (ASR) system to include the character sequence and a known synonym of the character sequence;

receiving a voice signal that is representative of a spoken utterance, the spoken utterance includes the character sequence or the known synonym of the character sequence;

decoding the voice signal, wherein decoding the voice signal comprises identifying that the spoken utterance includes the character sequence or the known synonym of the character sequence; and

performing an operation relative to the tabular data based upon the decoding of the voice signal.