System-resource-based multi-modal input fusion

• US 20050216254A1
• Filed: 03/24/2004
• Published: 09/29/2005
• Est. Priority Date: 03/24/2004
• Status: Abandoned Application

• Current Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Anastasakos, Tasos, Gupta, Anurag K.
• Application Number:
  US10/808,126
• Publication Number:
  US 20050216254A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

A multi-modal input fusion (MMIF) (200) is made scalable based on the resources available. When system resources are low, the MMIF module will limit the number of elements in each set of related interpretations. Additionally, the number of sets generated can be increased or reduced based on an amount of system resources available. In order to accommodate the scalable MMIF module, a resource profile (205) is provided to the MMIF describing the amount of resources (memory, processing power, etc.) available, and/or an amount of resources the MMIF module can utilize. Based on the amount of resources the MMIF module calculates threshold values that are used to adjust the number of sets produced and the number of elements included within each set.

Claims

1. A method for operating a system-resource-based multi-modal input fusion, the method comprising the steps of:

receiving a plurality of user inputs;

determining an amount of system resources available; and

creating sets of similar user inputs, wherein a number of similar user inputs within a set is based on the amount of system resources available.

2. The method of claim 1 further comprising the steps of:

converting the plurality of user inputs into Typed Feature Structures (TFSs); and

wherein the step of creating sets of similar user inputs comprises the step of creating sets of similar TFSs, wherein the number of TFSs within a set is based on the amount of system resources available.

3. The method of claim 2 wherein the step of converting the plurality of user inputs into Typed Feature Structures comprises the step of converting the plurality of user inputs into a plurality of attribute value pairs and confidence scores.

4. The method of claim 2 wherein the step of creating sets of similar TFSs comprises the step of creating sets of similar TFSs, wherein a TFS is included in a set if it has a content score greater than a threshold, wherein

ContentScore(TFS)=f(N, N_A, N_R, N_M, CS(i)|_i=1^N),

where

N=number of attributes in TFS,

N_A=number of attributes in TFS having a value,

N_R=number of attributes in TFS with redundant values,

N_M=number of attributes in TFS with missing explicit reference, and

CS(i)=confidence score of the i^th attribute of TFS.

5. The method of claim 2 wherein the step of creating sets of similar TFSs comprises the step of creating sets of similar TFSs, wherein a TFS is included in a set if it has a context score greater than a threshold.

6. The method of claim 5 wherein the step of creating sets of similar TFSs comprises the step of creating sets of similar TFSs, wherein a TFS is included in a set if it has a context score greater than a threshold wherein

ContextScore(TFS)=h(D_m, RS(TFS,TFS_m))

where

D_m=number of turns elapsed since receiving TFS_m from a modality

RS=Relationship Score between TFS (current input) and TFS_m

TFS_m=a TFS received D_m turns ago.

7. The method of claim 1 wherein a number of sets created is based on the amount of system resources available.

8. The method of claim 1 wherein the step of receiving the plurality of user inputs comprises the step of receiving a plurality of multi-modal user inputs.

9. The method of claim 1 wherein the step of determining the amount of system resources available comprises the step of determining an amount of memory or processing power available.

10. The method of claim 1 wherein the step of creating sets of similar user inputs comprises the step of creating sets of similar user inputs, wherein a user input is included in a set if it has a content score greater than a threshold.

11. A method for operating a system-resource-based multi-modal input fusion, the method comprising the steps of:

receiving a plurality of user inputs;

determining an amount of system resources available; and

creating sets of similar user inputs, wherein a number of similar user inputs within a set is based on the amount of system resources available, and wherein a number of sets created is limited based on the amount of system resources available.

12. The method of claim 11 further comprising the steps of:

converting the plurality of user inputs into Typed Feature Structures (TFSs); and

wherein the step of creating sets of similar user inputs comprises the step of creating sets of similar TFSs, wherein the number of TFSs within a set is based on the amount of system resources available.

13. The method of claim 12 wherein the step of converting the plurality of user inputs into Typed Feature Structures comprises the step of converting the plurality of user inputs into a plurality of attribute value pairs and confidence scores.

14. The method of claim 11 wherein the step of receiving the plurality of user inputs comprises the step of receiving a plurality of multi-modal user inputs.

15. The method of claim 11 wherein the step of determining the amount of system resources available comprises the step of determining an amount of memory or processing power available.

16. An apparatus comprising:

a plurality of modality recognizers receiving a plurality of user inputs; and

a semantic classifier determining an amount of system resources available and creating sets of similar user inputs, wherein a number of user inputs within a set is based on the amount of system resources available.

17. The apparatus of claim 16 further comprising:

segmentation circuitry converting the plurality of user inputs into a plurality of Typed Feature Structures (TFSs); and

wherein the semantic classifier creates sets of similar TFSs, wherein the number of TFSs within a set is based on the amount of system resources available.

18. The apparatus of claim 17 wherein the number of sets created is limited based on the amount of system resources available.

19. The apparatus of claim 16 wherein the number of sets created is limited based on the amount of system resources available.