Speech recognition candidate selection based on non-acoustic input

• US 9,632,589 B2
• Filed: 07/01/2015
• Issued: 04/25/2017
• Est. Priority Date: 11/13/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Connell, II, Jonathan H., Marcheret, Etienne
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US14/789,501
• Publication Number:
  US 20160140963A1
• Time in Patent Office: 664 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 2021/065 Aids for the handicapped in understanding
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

A method includes the following steps. A speech input is received. At least two speech recognition candidates are generated from the speech input. A scene related to the speech input is observed using one or more non-acoustic sensors. The observed scene is segmented into one or more regions. One or more properties for the one or more regions are computed. One of the speech recognition candidates is selected based on the one or more computed properties of the one or more regions.

Claims

1. A method, comprising:

receiving a speech input;

generating at least two speech recognition candidates from the speech input;

observing a scene related to the speech input using one or more non-acoustic sensors;

segmenting the observed scene into one or more regions;

computing one or more properties for the one or more regions, wherein computing the one or more properties comprises determining a textual label using optical character recognition; and

selecting one of the speech recognition candidates based on the one or more computed properties of the one or more regions.

2. The method of claim 1, wherein each speech recognition candidate comprises one of a word and a phrase.

3. The method of claim 1, wherein the speech recognition candidate is selected based on a comparison of the one or more computed properties of the one or more regions with the two or more speech recognition candidates generated based on the speech input.

4. The method of claim 3, further comprising transcribing the selected speech recognition candidate as text for display on a device.

5. The method of claim 3, further comprising interpreting the selected speech recognition candidate as a portion of an action command.

6. The method of claim 1, wherein the one or more non-acoustic sensors comprise at least one camera.

7. The method of claim 1, wherein each of the one or more segmented regions corresponds to an object or a surface in the observed scene.

8. The method of claim 1, wherein the one or more properties further comprise a color and a shape.

9. The method of claim 1, wherein computing one or more properties of the one or more regions comprises determining one or more spatial relations between the one or more regions.

10. The method of claim 1, wherein computing one or more properties of the one or more regions comprises detecting one or more gestures.

11. The method of claim 10, further comprising associating the one or more detected gestures with at least a portion of the one or more regions.