Multiple parallel dialogs in smart phone applications

• US 9,431,008 B2
• Filed: 05/29/2013
• Issued: 08/30/2016
• Est. Priority Date: 05/29/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Robichaud, Jean-Philippe, Hebert, Matthieu
• Primary Examiner(s): Sirjani, Fariba
• Application Number:
  US13/904,269
• Publication Number:
  US 20140358545A1
• Time in Patent Office: 1,189 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/228 of application context

Abstract

An arrangement is described for conducting natural language dialogs with a user on a mobile device using automatic speech recognition (ASR) and multiple different dialog applications. A user interface provides for user interaction with the dialogue applications in natural language dialogs. An ASR engine processes unknown speech inputs from the user to produce corresponding speech recognition results. A dialog concept module develops dialog concept items from the speech recognition results and stores the dialog concept items and additional dialog information in a dialog concept database. A dialog processor accesses dialog concept database information and coordinates operation of the ASR engine and the dialog applications to conduct with the user a plurality of separate parallel natural language dialogs in the dialog applications.

Claims

1. A method comprising:

receiving first recognized text corresponding to first speech input from a user;

displaying a first tab of a user interface, the first tab corresponding to a first application, wherein the first tab comprises the first recognized text with an icon associated with the first application;

receiving second recognized text corresponding to second speech input from the user;

displaying, responsive to the first speech input and the second speech input, a second tab of the user interface, the second tab corresponding to a second application, wherein the second application is selected based on the second speech input;

receiving third recognized text corresponding to third speech input from the user; and

displaying the first tab of the user interface, wherein the first tab further comprises the second recognized text with an icon associated with the second application and the third recognized text with the icon associated with the first application.

2. The method of claim 1, wherein the first application comprises a text messaging application and the second application comprises an email application, a social media application, a weather application, or a map application.

3. The method of claim 1, further comprising receiving the first speech input from the user by conducting a natural language dialog with the user by a domain expert agent coordinating with a dialog processor.

4. The method of claim 3, wherein the dialog processor pushes information stored in a dialog concept database to the first application and the second application.

5. The method of claim 1, wherein the first tab of the user interface and the second tab of the user interface are configured to allow the user to enter input.

6. A method comprising:

receiving first speech input from a user;

determining a first recognized text corresponding to the first speech input;

storing, in a dialog concept database and based on the first speech input, a first dialog concept item and an associated first indicator of a first application;

displaying a user interface for the first application, the user interface for the first application comprising the first recognized text with the first indicator;

receiving second speech input from the user;

determining a second recognized text corresponding to the second speech input;

storing, in the dialog concept database and based on the second speech input, a second dialog concept item and a second indicator of a second application corresponding to the second dialog concept item;

receiving third speech input from the user;

determining a third recognized text corresponding to the third speech input;

storing, in the dialog concept database and based on the third speech input, a third dialog concept item and a third indicator of a third application corresponding to the third dialog concept item; and

modifying the user interface for the first application to further comprise the second recognized text with the second indicator and the third recognized text with the third indicator.

7. The method of claim 6, wherein the first application is a text messaging application, the second application is an email application, a social media application, a weather application, or a map application, and the third application is an email application, a social media application, a weather application, or a map application.

8. The method of claim 6, wherein one or more dialog concept items are pushed from the dialog concept database to the first application, the second application, and the third application.

9. The method of claim 6, wherein one or more dialog concept items are pulled from the dialog concept database by the first application, the second application, and the third application.

10. The method of claim 6, further comprising:

receiving a selection to delete the first recognized text, the second recognized text, or the third recognized text; and

modifying the user interface based on the selection.

11. The method of claim 10, wherein receiving the selection to delete the first recognized text, the second recognized text, or the third recognized text comprises receiving an indication that the user swiped the first recognized text, the second recognized text, or the third recognized text.

12. The method of claim 6, further comprising displaying a second user interface for the second application.

13. The method of claim 6, further comprising displaying a third user interface for the third application.

14. A method comprising:

receiving first speech input from a user;

determining a first recognized text corresponding to the first speech input;

storing, in a dialog concept database and based on the first speech input, a first dialog concept item with an associated first indicator of a first application in which the first speech input originated;

displaying a plurality of tabs, a first tab of the plurality of tabs having a user interface corresponding to the first application, the user interface comprising the first recognized text with the first indicator;

receiving second speech input;

determining a second recognized text corresponding to the second speech input;

storing, in the dialog concept database and based on the second speech input, a second dialog concept item with an associated second indicator of a second application; and

modifying the user interface to further display the second recognized text with the second indicator.

15. The method of claim 14, wherein receiving the first speech input from the user comprises conducting, by a domain expert, a natural language dialog with the user.

16. The method of claim 14, wherein one or more dialog concept items are pushed from the dialog concept database to the first application and the second application.

17. The method of claim 14, wherein one or more dialog concept items are pulled from the dialog concept database by the first application and the second application.