Device for extracting information from a dialog

• US 10,318,623 B2
• Filed: 10/31/2016
• Issued: 06/11/2019
• Est. Priority Date: 03/08/2012
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Waibel, Alexander
• Primary Examiner(s): Pham, Thierry L
• Application Number:
  US15/338,750
• Publication Number:
  US 20170046326A1
• Time in Patent Office: 953 Days
• Field of Search: 704 2, 704 4, 704 9, 704211, 704246, 704277
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/027 Concept to speech synthesisers; Generation of natural phrases from machine-based concepts generation of parameters for speech synthesis out of text G10L13/08
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G16H 10/20 for electronic clinical trials or questionnaires
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof
• CPC Class Codes: G16H 80/00 ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring

Abstract

Computer-implemented systems and methods for extracting information during a human-to-human mono-lingual or multi-lingual dialog between two speakers are disclosed. Information from either the recognized speech (or the translation thereof) by the second speaker and/or the recognized speech by the first speaker (or the translation thereof) is extracted. The extracted information is then entered into an electronic form stored in a data store.

Claims

1. A method comprising:

receiving a first speech input in a first language from a first speaker and a second speech input in a second language from a second speaker, the first speech input and the second speech input being part of a dialogue between the first speaker and the second speaker;

determining, by a speech recognition system, a first recognized speech result based on the first speech input;

determining, by the speech recognition system, a second recognized speech result based on the second speech input;

extracting information from the second recognized speech result;

automatically populating a form having a plurality of fields based on the dialogue, the automatic population comprising;

automatically identifying one of the plurality of fields based on the first recognized speech result in the first language,

translating at least a portion of the information extracted from the second recognized speech result from the second language to the first language, and

inserting the translated information in the first language into the identified field; and

displaying the populated form in the first language, wherein the population form comprises a toggle switch that is configured to cause the populated form to change from the first language to the second language.

2. The method of claim 1, further comprising:

receiving a correction to the populated form by detecting a touch input from a user dragging information from the second recognized speech result to an empty field of the populated form; and

entering the dragged information into the empty field.

3. The method of claim 1, further comprising:

receiving a correction to the populated form, wherein receiving the correction comprises;

receiving a selection of an empty field of the populated form,

receiving a correction speech input, and

determining a recognized correction speech result based on the correction speech input; and

entering the recognized correction speech result into the empty field.

4. The method of claim 1, further comprising:

receiving a correction to the populated form, wherein receiving the correction comprises;

receiving selection of a filled field of the populated form, and

receiving selection of a portion of the second recognized speech result; and

replacing contents of the filled field with the portion of the recognized speech result.

5. The method of claim 1, further comprising:

receiving a correction to the populated form, wherein receiving the correction comprises;

receiving a selection of a filled field of the populated form;

receiving a correction speech input, and

determining a recognized correction speech result based on the correction speech input; and

replacing contents of the filled field with the recognized correction speech result.

6. The method of claim 5, wherein the correction speech input is a dictated spelling.

7. The method of claim 5, wherein the correction speech input is a restatement of at least a portion of the second speech input.

8. The method of claim 1, further comprising:

receiving a gesture from a user, the gesture indicating that the populated form needs to be corrected.

9. The method of claim 1, wherein the information is extracted based on one or more prompts associated with the plurality of fields of the form.

10. The method of claim 1, wherein the information is extracted using a semantic parser.

11. The method of claim 1, wherein the information is extracted using keyword spotting.

12. The method of claim 1, wherein the information is extracted via spoken term detection.

13. The method of claim 1, wherein the information is extracted using an entity tagger.

15. The method of claim 1, further comprising displaying the second recognized speech result, wherein the second recognized speech result is displayed separately from, simultaneously with and not overlapping the populated form.

14. A non-transitory computer-readable medium comprising instructions that when executed by a processor cause the processor to perform steps comprising:

receiving a first speech input in a first language from a first speaker and a second speech input in a second language from a second speaker, the first speech input and the second speech input being part of a dialogue between the first speaker and the second speaker;

determining, by a speech recognition system, a first recognized speech result based on the first speech input;

determining, by the speech recognition system, a second recognized speech result based on the second speech input;

extracting information from the second recognized speech result;

automatically populating a form having a plurality of fields based on the dialogue, the automatic population comprising;

in the first language,

translating at least a portion of the information extracted from the second recognized speech result from the second language to the first language, and

inserting the translated information in the first language into the identified field; and

displaying the populated form and the recognized speech result in the first language, wherein the population form comprises a toggle switch that is configured to cause the populated form to change from the first language to the second language.

16. The non-transitory computer-readable medium of claim 14, wherein the information is extracted using a semantic parser.

17. The non-transitory computer-readable medium of claim 14, wherein the steps further comprises:

receiving a correction to the populated form; and

revising the populated form based on the received correction.

18. The non-transitory computer-readable medium of claim 17, wherein receiving the correction comprises:

detecting a touch input from a user dragging information from the second recognized speech result to an empty field of the populated form; and

wherein revising the populated form based on the received correction comprises;

entering the dragged information into the empty field.

19. The non-transitory computer-readable medium of claim 17, wherein receiving the correction comprises:

receiving selection of a filled field of the populated form, and

receiving selection of a portion of the second recognized speech result; and

wherein revising the populated form based on the received correction comprises;

replacing contents of the filled field with the portion of the second recognized speech result.

20. The non-transitory computer-readable medium of claim 17, wherein receiving the correction comprises:

receiving a selection of a filled field of the populated form,

receiving a correction speech input, and

determining a recognized correction speech result based on the correction speech input; and

wherein revising the populated form based on the received correction comprises;

replacing contents of the filled field with the recognized correction speech result.