Multimodal input processing

• US 9,530,102 B2
• Filed: 02/17/2015
• Issued: 12/27/2016
• Est. Priority Date: 02/17/2015
• Status: Active Grant

• Current Assignee: The MathWorks, Inc.
• Original Assignee: The MathWorks, Inc.
• Inventors: Desai, Ankit A., Quinn, Michael J., Mosterman, Pieter J., Parupati, Srikanth, Uthama, Ashish
• Primary Examiner(s): CHEN, ALAN S
• Application Number:
  US14/623,985
• Publication Number:
  US 20160239751A1
• Time in Patent Office: 679 Days
• US Class Current: 1/1
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

Exemplary embodiments described herein provide methods, mediums, and systems for recognizing a formalism type present in an input and generating an output that is consistent with the formalism type. In order to generate an output that is consistent with the formalism type, exemplary embodiments analyze elements of the input together. Such a holistic analysis determines or uses a likelihood that the different elements coexist together in a given formalism type. Based on this holistic analysis, an appropriate formalism type that is consistent with the coexistence of the elements may be selected. After the appropriate formalism type is selected, the input may be processed to generate an output consistent with the formalism type. The output may be computer-based representations of the input defined according to a program associated with the identified formalism type, and/or the input may be translated or transformed into another representation.

Claims

1. A non-transitory computer readable medium storing instructions, the instructions comprising:

one or more instructions that, when executed, cause a processor to;

receive an input, the input comprising a plurality of input elements;

parse the input to identify;

a first element from among the plurality of input elements, the first element represented according to a formalism type that identifies having a syntax for organizing elements of the formalism type, and

a second element from among the plurality of input elements, the second element represented according to the formalism type;

access a library comprising entries for different formalism types, the formalism type included in the different formalism types;

determine a likelihood that the first element and the second element coexist in a selected formalism type from among the different types of formalisms in the library, and

select, based on the determining, the selected formalism type that is consistent with a coexistence of the first element and the second element;

wherein the selected formalism type is one of the following;

an equation, a state diagram, a plot, a chart, a table, a Unified Modeling Language (UML) diagram, a chemical formula, a block diagram model, a type of diagram, a multi-body modeling formalism, a computer-aided design formalism, a physical modeling formalism or source code,

generate an output, the output;

comprising the first element and the second element, and

at least partly represented in a format corresponding to the selected formalism type; and

wherein if the selected formalism type is an equation, the format is a solution to the equation or a source code function representing the equation;

wherein if the selected formalism type is a state diagram, the format is a state chart model representing the state diagram;

wherein if the selected formalism type is a plot, the format is a source code command that when executed generates the plot or a table of data represented in the plot;

wherein if the selected formalism type is a table, the format is a plot or a chart representing the table;

wherein if the selected formalism type is a chart, the format is a table of data represented in the chart;

wherein if the selected formalism type is a UML diagram, the format is a source code class representing the UML diagram;

wherein if the selected formalism type is a chemical formula, the format is a name of a chemical represented by the chemical formula;

wherein if the selected formalism type is a block diagram model, the format is a source code representation of the block diagram model; and

wherein if the selected formalism type represents source code, the format is a block diagram model.

2. The medium of claim 1, wherein the plurality of input elements of the input include a first set of elements represented according to a first formalism type and a second set of elements represented according to a second formalism type distinct from the first formalism type.

3. The medium of claim 1, wherein the output comprises output elements corresponding to a plurality of distinct formalisms such that the output is partly represented in the format corresponding to the selected formalism type, and is further represented in a second format corresponding to a second selected formalism type that is distinct from the selected formalism type.

4. The medium of claim 1, wherein different formalism types are processed by different types of processing engines or environments.

5. The medium of claim 1, wherein different formalism types have different alphabets, syntaxes, or semantics.

6. The medium of claim 1, wherein the input comprises one or more of:

an image;

video;

audio; and

gesture data.

7. The medium of claim 1, wherein the library comprises a table of probabilities that a type of element is found in each of the formalism types in the library.

8. The medium of claim 7, wherein the probabilities are dynamically updated based on user-feedback about a classification of the input into a formalism type.

9. The medium of claim 7, wherein the probabilities are dynamically updated based on historical classifications of formalism types.

10. The medium of claim 7, wherein analyzing the first element and the second element comprises:

determining a first probability, the first probability representing a probability that the first element belongs to the selected formalism type;

determining a second probability, the second probability representing a probability that the second element belongs to the selected formalism type; and

adding the first probability to the second probability.

11. The medium of claim 1, wherein the second element is identified, at least in part, based on a predicted formalism type that is determined based on an analysis of the first element.

12. The medium of claim 1, wherein generating the output comprises applying a formalism-specific rule to the first element or the second element, the formalism-specific rule being particular to the selected formalism type.

13. The medium of claim 1, wherein generating the output comprises processing the first element and the second element using a formalism-specific procedure.

14. The medium of claim 1, further comprising displaying help information specific to the selected formalism type, the first element, or the second element.

15. A method comprising:

receiving, using a processor of a computing device, an input, the input comprising a plurality of input elements;

parsing the input, the parsing;

performed using the processor,

identifying a first element from among the plurality of input elements, and

identifying a second element from among the plurality of input elements,

wherein the first element and the second element are represented according to a formalism type defining a syntax for organizing elements of the formalism type;

access a library comprising entries for different formalism types;

evaluating a likelihood that the first element and the second element coexist together in a selected formalism type from among the different formalism types in the library, and

selecting, based on the evaluating, the selected formalism type that is consistent with a coexistence of the first element and the second element;

wherein the selected formalism type is one of the following;

an equation, a state diagram, a plot, a chart, a table, a Unified Modeling Language (UML) diagram, a chemical formula, a block diagram model, a type of diagram, a multi-body modeling formalism, a computer aided design formalism, a physical modeling-formalism or source code,

generating, using the processor, an output, the output;

representing a translation of at least part of the input into an output type that is associated with the selected formalism type; and

wherein if the selected formalism type is an equation, the translation is a solution to the equation or a source code function representing the equation;

wherein if the selected formalism type is a state diagram, the translation is a state chart model representing the state diagram;

wherein if the selected formalism type is a plot, the translation is a source code command that when executed generates the plot or a table of data represented in the plot;

wherein if the selected formalism type is a table, the translation is a plot or a chart representing the table;

wherein if the selected formalism type is a chart, the translation is a table of data represented in the chart;

wherein if the selected formalism type is a UML diagram, the translation is a source code class representing the UML diagram;

wherein if the selected formalism type is a chemical formula, the translation is a name of a chemical represented by the chemical formula;

wherein if the selected formalism type is a block diagram model, the translation is a source code representation of the block diagram model; and

wherein if the selected formalism type represents source code, the format is a block diagram model.

16. A system comprising:

a non-transitory computer readable medium storing an input, the input;

comprising a plurality of input elements, and

being uninterpretable by an engine associated with a selected formalism type; and

a processor programmed with instructions that, when executed, cause the processor to;

parse the input to identify;

a first element from among the plurality of input elements, and

a second element from among the plurality of input elements,

wherein the first element and the second element are represented according to

a formalism type defining a syntax for organizing elements of the formalism type;

access a library comprising entries for different formalism types, the different formalism types comprising the selected formalism type associated with the engine;

analyze the first element and the second element, the analyzing;

examining the first element in context with the second element to evaluate a likelihood that the first element and the second element coexist together in the selected formalism type, and

selecting, based on the examining, the selected formalism that is consistent with a coexistence of the first element and the second element;

wherein the selected formalism type is one of the following;

an equation, a state diagram, a plot, a chart, a table, a Unified Modeling Language (UML) diagram, a chemical formula, a block diagram model, a type of diagram, a multi-body modeling formalism, a computer-aided design formalism, a physical formalism or source code,

generate an output, the output;

comprising the first element and the second element, and

at least partly represented in a format corresponding to the selected formalism type,

wherein the output is interpretable by the engine associated with the selected formalism; and

wherein if the selected formalism type is an equation, the format is a solution to the equation or a source code function representing the equation;

wherein if the selected formalism type is a state diagram, the format is a state chart model representing the state diagram;

wherein if the selected formalism type is a plot, the format is a source code command that when executed generates the plot or a table of data represented in the plot;

wherein if the selected formalism type is a table, the format is a plot or a chart representing the table;

wherein if the selected formalism type is a chart, the format is a table of data represented in the chart;

wherein if the selected formalism type is a UML diagram, the format is a source code class representing the UML diagram;

wherein if the selected formalism type is a chemical formula, the format is a name of a chemical represented by the chemical formula;

wherein if the selected formalism type is a block diagram model, the format is a source code representation of the block diagram model; and

wherein if the selected formalism type represents source code, the format is a block diagram model.