Method, system and module for mult-modal data fusion

• US 20040093215A1
• Filed: 11/12/2002
• Published: 05/13/2004
• Est. Priority Date: 11/12/2002
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Cheng, Ying Catherine, Gupta, Anurag Kumar
• Granted Patent: US 7,152,033 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

A method for multi-modal data fusion (100), a multi-modal system data fusion system (10) and module (24) that in use operates by receiving segments (125) of multi-modal data associated respectively with a modalitiy. Initiating (130) a dynamically variable wait period after one of the segments is received is then performed. The dynamically variable wait period has a duration determined from data fusion timing statistics of the system (10). A waiting (140) for reception of any further segments during the dynamically variable wait period is then effected and thereafter a fusing (145) of the segments received provides fused data that is sent (160) to a dialog manager (25).

Claims

1. A method for multi-modal data fusion in a multi-modal system comprising a plurality of input modalities, the method comprising the steps of:

(i) Receiving one or more segments of multi-modal data, each of said segments being associated respectively with said modalities;

(ii) Initiating a dynamically variable wait period after one of said segments is received, said dynamically variable wait period having a duration determined from data fusion timing statistics of the system;

(iii) Waiting for reception of any further said segments during said dynamically variable wait period; and

(v) Fusing said segments received during said steps (i) to (iii) to provide fused data.

2. A method for multi-modal data fusion, as claimed in claim 1, further characterized by repeating steps (ii) to (iv) if one or more said further said segments are received during said dynamically variable wait period.

3. A method for multi-modal data fusion, as claimed in claim 1, including the further step of sending said fused data to a dialog manager.

4. A method for multi-modal data fusion, as claimed in claim 1, wherein the duration of the timing statistics are determined from historical statistical data of the system.

5. A method for multi-modal data fusion, as claimed in claim 4, wherein the historical statistical data includes average segment creation time for each modality of said system.

6. A method for multi-modal data fusion, as claimed in claim 5, the duration of the dynamically variable wait period is further determined from analysis of a period starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments.

7. A method for multi-modal data fusion, as claimed in claim 1, wherein the duration of the dynamically variable wait period is determined from analysis of the following:

min(max(AvgDur_m) or (AvgTimeDiff+AvgDur))

where AvgDur_m is a value indicative of the average duration associated with processing a user request into a one of said segments for a modality m, AvgTimeDiff is a value indicative of an average period for periods starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments, and AvgDur is a value indicative of the average duration associated with processing a user request into a one of said segments for every modality in the system.

8. A method for multi-modal data fusion, as claimed in claim 1, wherein said segments are frames.

9. A method for multi-modal data fusion, as claimed in claim 8, wherein said frames include temporal characteristics including at least part of said historical statistical data. The frames may also include semantic representations an associated user request. This is correct.

10. A multi-modal data fusion system comprising:

a plurality of modality processing modules;

a plurality of parsers coupled to a respective one of said modality processing modules;

a multi-modal fusion module having inputs coupled to outputs of said parsers, wherein in use said fusion module receives one or more segments of multi-modal data from at least one of said parsers, and initiates a dynamically variable wait period after one of said segments is received, said dynamically variable wait period having a duration determined from data fusion timing statistics of the system, the fusion module then waits for reception of any further said segments during said dynamically variable wait period and fuses said segments received to provide fused data.

11. A multi-modal data fusion system as claimed in claim 10, wherein there is a dialogue manager coupled to an output of said fusion module.

12. A multi-modal data fusion system as claimed in claim 10, there are user input devices coupled respectively to said modality processing modules.

13. A multi-modal data fusion system as claimed in claim 10, wherein the duration of the timing statistics are determined from historical statistical data of the system.

14. A multi-modal data fusion system as claimed in claim 13, wherein the historical statistical data includes average segment creation time for each modality of said system.

15. A multi-modal data fusion system as claimed in claim 14, wherein the duration of the dynamically variable wait period is further determined from analysis of a period starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments.

16. A multi-modal data fusion system as claimed in claim 10, wherein the duration of the dynamically variable wait period is determined from analysis of the following:

min(max(AvgDur_m) or (AvgTimeDiff+AvgDur))

where AvgDur_m is a value indicative of the average duration associated with processing a user request into a one of said segments for a modality m, AvgTimeDiff is a value indicative of an average period for periods starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments, and AvgDur is a value indicative of the average duration associated with processing a user request into a one of said segments for every modality in the system.

17. A multi-modal fusion module having inputs for coupling to outputs of parsers, wherein in use said fusion module receives one or more segments of multi-modal data from at least one of said parsers, and initiates a dynamically variable wait period after one of said segments is received, said dynamically variable wait period having a duration determined from data fusion timing statistics of the system, the fusion module then waits for reception of any further said segments during said dynamically variable wait period and fuses said segments received to provide fused data.

18. A multi-modal fusion module as claimed in claim 17, wherein the duration of the timing statistics are determined from historical statistical data of the system.

19. A multi-modal fusion module as claimed in claim 18, wherein the historical statistical data includes average segment creation time for each modality of said system.

20. A multi-modal fusion module as claimed in claim 19, wherein the duration of the dynamically variable wait period is further determined from analysis of a period starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments.

21. A multi-modal fusion module as claimed in claim 17, wherein the duration of the dynamically variable wait period is determined from analysis of the following:

min(max(AvgDur₁) or (AvgTimeDiff+AvgDur))

where AvgDur_m is a value indicative of the average duration associated with processing a user request into a one of said segments for a modality m, AvgTimeDiff is a value indicative of an average period for periods starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments, and AvgDur is a value indicative of the average duration associated with processing a user request into a one of said segments for every modality in the system.