Method and apparatus for enabling multimodal tags in a communication device by discarding redundant information in the tags training signals

• US 9,031,843 B2
• Filed: 09/28/2007
• Issued: 05/12/2015
• Est. Priority Date: 09/28/2007
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Ma, Changxue, Bliss, Harry M.
• Primary Examiner(s): Kazeminezhad, Farzad
• Application Number:
  US11/863,763
• Publication Number:
  US 20090089059A1
• Time in Patent Office: 2,783 Days
• Field of Search: 704/254, 704/250, 726/5
• US Class Current: 704/254
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: H04M 1/72421 with automatic activation of emergency service functions, e.g. upon sensing an alarm
• CPC Class Codes: H04M 1/7243 with interactive means for internal management of messages
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A method and apparatus for enabling multimodal tags in a communication device is disclosed. The method comprises receiving a first training signal and receiving a second training signal in conjunction with the first training signal. A multimodal tag is created by discarding redundant or non-discriminative information associated with each of the first and second training signals to represent a combination of the first training signal and the second training signal and a function is associated with the created multimodal tag.

Claims

1. A method comprising:

receiving, by an electronic device, one or more first user inputs indicative of a first selection of a first input modality for a multimodal tag and a second input modality for the multimodal tag, wherein the first and second input modalities are selected from a set of input modalities comprising at least three different types of input modalities;

detecting, with the first input modality of the electronic device, a first training signal;

detecting, with the second input modality of the electronic device, a second training signal;

creating a multimodal tag by at least discarding redundant or non-discriminative information associated with each of the first and second training signals and producing a training signal feature sequence that defines one or more salient features of both the first and the second training signals as a function of time;

receiving at least one second user input indicative of a second selection of a function from a plurality of selectable functions; and

associating the function with the multimodal tag.

2. The method of claim 1, further comprising:

receiving a plurality of input signals; and

in response to determining that the plurality of input signals correspond to the multimodal tag performing the function associated with the multimodal tag.

3. The method of claim 2, wherein at least one of the plurality of input signals includes a tactile sensor based signal.

4. The method of claim 2, wherein determining whether the received plurality of input signals correspond to the multimodal tag is done using a pattern matching technique.

5. The method of claim 4, wherein the pattern matching technique is performed on a first feature sequence and a second feature sequence, wherein the first feature sequence characterizes the first input signal as a function of time and the second feature sequence characterizes the second input signal as a function of time.

6. The method of claim 1, wherein the second training signal is a motion signal detected by a motion and the first training signal is an audio signal detected by an audio sensor.

7. The method of claim 6, further comprising:

detecting the first training signal by at least analyzing the audio signal using speech analysis techniques.

8. The method of claim 1, wherein the second training signal is tactile signal detected by a tactile sensor and the first training signal is an audio signal detected by an audio sensor.

9. The method of claim 8, wherein the tactile signal is one of a touch based input, a key press input, or a stylus input.

10. The method of claim 1, further comprising detecting at least a third training signal, wherein creating the multimodal tag comprises combining the first, second, and third training signals into the training signal feature sequence.

11. The method of claim 1, wherein registering the training signal feature sequence comprises storing the training signal feature sequence as the multimodal tag.

12. The method of claim 1, wherein the second training signal is a motion based signal detected by a motion sensor and the first training signal is a tactile signal detected by a tactile sensor.

13. The method of claim 1, wherein at least one of the first training signal and the second training signal is based on at least one of velocity, acceleration, direction, trajectory, gravity, or rotation detected by a motion sensor.

14. The method of claim 1, wherein the second training signal is a motion based signal and the first training signal is a gesture based signal.

15. The method of claim 1, wherein the second training signal is a gesture based signal and the first training signal is a tactile sensor based signal.

16. The method of claim 1, wherein creating the multimodal tag further comprises combining the first and second training signals into the training signal feature sequence based on a temporal order at which the first training signal and the second training signal are detected.

17. The method of claim 1, wherein the first input signal is a motion based signal indicative of a shaking motion, the second input signal is a tactile based signal indicative of a key press, and the function is dialing a an emergency response service.

18. An electronic device comprising:

a first sensor configured as a first input modality for detecting a first training signal associated with a multimodal tag;

a second sensor configured as a second input modality for detecting a second training signal associated with a multimodal tag;

a third sensor configured as a third input modality for detecting a third training signal associated with a multimodal tag;

a memory for storing registered multimodal tags and associated functions; and

a processor operably coupled to the first sensor, the second sensor, and the third sensor, wherein the processor is configured to;

receive one or more first user inputs indicative of a first selection of the first input modality, the second input modality, and the third input modality;

receive the first training signal, the second training signal, and the third training signal;

receive at least one second user input indicative of a second selection of a function from a plurality of selectable functions;

discard redundant or non-discriminative information associated with each of the first, second, and third training signals;

produce a training signal feature sequence that defines one or more salient features of a combination of the first, second, and third training signals as a function of time; and

register the training signal feature sequence as a multimodal tag by at least storing the user selected function in association with the multimodal tag.

19. The electronic device of claim 18, wherein producing the training signal feature sequence comprises combining the first, and second training signals into the training signal feature sequence based on a temporal order at which the first, and second training signals are received.

20. The electronic device of claim 18, wherein one of the first and second sensors are motion based sensors for detecting motion of the electronic device and the third sensor is a tactile based sensor for detecting tactile input associated with the electronic device.

21. A method comprising:

receiving, by an electronic device, one or more first user inputs indicative of;

a first selection of at least a first input modality for a multimodal tag,

a second input modality for the multimodal tag, and

a third input modality for the multimodal tag;

detecting, with the first input modality of the electronic device, a first training signal;

detecting, with the second input modality of the electronic device, a second training signal;

detecting, with the third input modality of the electronic device, a third training signal;

discarding redundant or non-discriminative information associated with each of the first, second, and third training signals;

creating a multimodal tag by at least combining the first, second, and third training signals into one or more training signal feature sequences that define one or more salient features of a combination of the first, second, and third training signals as a function of time;

receiving at least one second user input indicative of a second selection of a function from a plurality of selectable functions;

and associating the function with the multimodal tag.