Method and apparatus for correlating and viewing disparate data

• US 10,068,024 B2
• Filed: 12/18/2015
• Issued: 09/04/2018
• Est. Priority Date: 02/01/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Sawhney, Harpreet Singh, Eledath, Jayakrishnan, Divakaran, Ajay, Bansal, Mayank, Cheng, Hui
• Primary Examiner(s): To, Baoquoc N
• Application Number:
  US14/974,871
• Publication Number:
  US 20160110433A1
• Time in Patent Office: 991 Days
• Field of Search: 707705, 707706, 709206, 725 46
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06Q 50/26 Government or public services business processes related to the transportation industry G06Q50/40

Abstract

Methods and apparatuses of the present invention generally relate to generating actionable data based on multimodal data from unsynchronized data sources. In an exemplary embodiment, the method comprises receiving multimodal data from one or more unsynchronized data sources, extracting concepts from the multimodal data, the concepts comprising at least one of objects, actions, scenes and emotions, indexing the concepts for searchability; and generating actionable data based on the concepts.

Claims

1. A method for generating data to provide situational awareness or decision-making assistance to a user in relation to a physical environment, the method comprising, with a computer system:

processing input data comprising at least data associated with the physical environment; and

when a need for situational awareness or decision-making assistance is detected based on the input data, generating response data, the response data derived from multimodal data from a plurality of electronic data streams comprising audio, visual and textual information, the data streams received from a plurality of data sources, wherein generating the response data comprises;

determining a characteristic of the need for situational awareness or decision-making assistance;

extracting semantic information from the audio, visual and textual information;

correlating the extracted semantic information in accordance with the characteristic;

selecting a subset of the audio, visual and textual information based on the correlation of the extracted semantic information with the characteristic; and

outputting at least a portion of the selected subset as the response data.

2. The method of claim 1, comprising receiving at least some of the input data from a sensor of a vehicle.

3. The method of claim 2, wherein the input data comprises an inquiry relating to navigation of the physical environment.

4. The method of claim 1, wherein the multimodal data comprises at least two of:

image, audio, video and textual data.

5. The method of claim 1, comprising generating a route by which the user may navigate the physical environment.

6. The method of claim 1, comprising generating data indicative of one or more obstacles in a route by which the user may navigate the physical environment.

7. The method of claim 1, comprising extracting affect information from one or more of the audio, visual and textual information.

8. The method of claim 7, further comprising inferring a context from one or more of the audio, visual, and textual information.

9. The method of claim 1, wherein the input comprises data indicative of a location or a direction of movement.

10. The method of claim 1, wherein the input comprises a natural language query.

11. The method of claim 1, comprising:

extracting one or more concepts from the audio, visual and textual information, the one or more concepts comprising at least one of objects, actions, scenes and emotions;

indexing the extracted concepts; and

generating the response data using the indexed concepts.

12. The method of claim 1, comprising creating the subset of the audio, visual and textual information at least partly by performing a gating technique.

13. The method of claim 1, comprising:

receiving a natural language query;

extracting semantic information from the natural language query;

searching the correlated information using at least some of the semantic information extracted from the natural language query; and

formulating the response data based on a result of the searching.

14. The method of claim 1, comprising:

extracting concepts from the audio, visual and textual information, the extracted concepts comprising at least one of objects, actions, scenes and emotions;

grouping the concepts based on relationships of the concepts to particular events;

aligning related concepts within the groups of concepts; and

generating a measure of accuracy of the groups of concepts based on the aligning.

15. The method of claim 14, further comprising:

retrieving the audio, visual and textual information from unsynchronized data sources; and

generating a measure of a trustworthiness of the unsynchronized data sources based on the unsynchronized data sources.

16. A method for generating informational assistance for a user in relation to a physical environment, the method comprising, with a computer system:

processing input data to identify a need for informational assistance relating to the physical environment;

determining a characteristic of the physical environment, in addition to a geographical location, associated with the need for informational assistance;

extracting semantic information from a plurality of electronic data streams comprising audio, visual and textual information, the data streams received from a plurality of data sources;

correlating the extracted semantic information with the characteristic and the geographical location;

selecting a subset of the audio, visual and textual information based on the correlation of the extracted semantic information with the physical characteristic and the geographical location;

generating the informational assistance comprising at least a portion of the selected subset; and

outputting the informational assistance to the user.

17. The method of claim 16, comprising temporally aligning at least two of the audio, visual, and textual information of the informational assistance in the output to the user.

18. The method of claim 16, comprising geo-spatially aligning at least two of the audio, visual, and textual information of the informational assistance in the output to the user.

19. The method of claim 16, comprising semantically aligning at least two of the audio, visual, and textual information of the informational assistance in the output to the user.

20. The method of claim 16, comprising generating natural language text output based on at least a portion of the selected subset of audio, visual, and textual information.