Determining related data points from multi-modal inputs

• US 9,396,433 B2
• Filed: 05/29/2013
• Issued: 07/19/2016
• Est. Priority Date: 05/29/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Subramaniam, L. Venkata, Mehta, Sameep, Krishnapuram, Raghuram, Faruquie, Tanveer A.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): COUGHLAN, PETER D
• Application Number:
  US13/904,489
• Publication Number:
  US 20140358843A1
• Time in Patent Office: 1,147 Days
• Field of Search: 706/58
• US Class Current: 1/1
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Techniques, systems, and articles of manufacture for determining related data points from multi-modal inputs. A method includes collecting multiple items of multi-modal data comprising at least one dimension from multiple data sources, wherein said at least one dimension comprises a geographic dimension, a temporal dimension and/or an event-related dimension, determining a window of relevance for each of the multiple items of multi-modal data with respect to the at least one dimension, and identifying two or more of the multiple items of multi-modal data as related, by determining an overlap of the window of relevance corresponding to each of the two or more items of multi-modal data with respect to the at least one dimension.

Claims

1. A method comprising:

collecting multiple items of multi-modal data comprising a plurality of dimensions from multiple data stream originators, wherein said plurality of dimensions comprises at least a geographic dimension, a temporal dimension and an event-related dimension;

annotating each of the multiple items of collected multi-modal data with information, wherein the information comprises at least a veracity score associated with the data stream originator of the item of data;

determining multiple windows of relevance for the multiple items of multi-modal data with respect to the plurality of dimensions, wherein said multiple windows of relevance comprise at least (i) a temporal window of relevance, (ii) a spatial window of relevance, and (iii) and a semantic window of relevance; and

identifying two or more of the multiple items of multi-modal data as related based on said veracity score associated with the data stream originator of each item of data and by determining an overlap of at least one of the multiple windows of relevance corresponding to each of the two or more items of multi-modal data with respect to at least one of the plurality of dimensions;

wherein said collecting, said annotating, said determining, and said identifying are carried out by a computer device.

2. The method of claim 1, wherein said collecting comprises collecting multiple items of multi-modal data from multiple data stream originators over two or more periods of time.

3. The method of claim 1, wherein said collecting comprises collecting multiple items of multi-modal data from multiple data stream originators corresponding to two or more spatial locations within the geographic region.

4. The method of claim 1, wherein said multi-modal data comprises text message data, speech data, global positioning system data, audio data, mobile call detail records, internet content, and/or mobile-specific content.

5. The method of claim 1, wherein said collecting comprises collecting multiple items of multi-modal data via one or more physical sensors.

6. The method of claim 5, wherein the one or more physical sensors comprise mobile network mobile call detail records, global positioning system sensors, audio sensors, and/or traffic sensors.

7. The method of claim 1, wherein said information further comprises a time corresponding to said collection, a location corresponding to said collection, a semantic relevance to a given topic, a weight, and/or a relevance score associated with a given topic.

8. The method of claim 1, wherein said determining multiple windows of relevance comprises manually defining each respective one of the multiple windows of relevance with respect to at least a respective one of the plurality of dimensions based on domain knowledge.

9. The method of claim 1, wherein said determining multiple windows of relevance comprises automatically determining the multiple windows of relevance using labeled historical data associated with at least a respective one of the plurality of dimensions.

10. The method of claim 1, wherein said determining an overlap of the window of relevance corresponding to each of the two or more items of multi-modal data with respect to at least one of the plurality of dimensions comprises determining a physical distance between each of the two or more items of multi-modal data.

11. The method of claim 1, wherein said determining an overlap of the window of relevance corresponding to each of the two or more items of multi-modal data with respect to at least one of the plurality of dimensions comprises determining a syntactic distance between each of the two or more items of multi-modal data.

12. The method of claim 11, wherein said determining the syntactic distance comprises measuring the syntactic distance via an edit distance measure, a phonetic distance measure, a pre-fix matching measure, and/or an abbreviation expansion measure.

13. The method of claim 1, wherein said determining an overlap of the window of relevance corresponding to each of the two or more items of multi-modal data with respect to at least one of the plurality of dimensions comprises determining a semantic distance between each of the two or more items of multi-modal data.

14. The method of claim 13, wherein said determining the semantic distance comprises measuring the semantic distance based a relatedness of words contained and/or derived from the two or more items of multi-modal data.

15. The method of claim 1, comprising:

storing the identification of the two or more of the multiple items of multi-modal data as related with respect to at least one of the plurality of dimensions.

16. An article of manufacture comprising a non-transitory computer readable storage medium having computer readable instructions tangibly embodied thereon which, when implemented, cause a computer to carry out a plurality of method steps comprising:

collecting multiple items of multi-modal data comprising a plurality of dimensions from multiple data stream originators, wherein said plurality of dimensions comprises at least a geographic dimension, a temporal dimension and an event-related dimension;

annotating each of the multiple items of collected multi-modal data with information, wherein the information comprises at least a veracity score associated with the data stream originator of the item of data;

determining multiple windows of relevance for the multiple items of multi-modal data with respect to the plurality of dimensions, wherein said multiple windows of relevance comprise at least (i) a temporal window of relevance, (ii) a spatial window of relevance, and (iii) and a semantic window of relevance; and

identifying two or more of the multiple items of multi-modal data as related based on said veracity score associated with the data stream originator of each item of data and by determining an overlap of at least one of the multiple windows of relevance corresponding to each of the two or more items of multi-modal data with respect to at least one of the plurality of dimensions.

17. A system comprising:

a memory; and

at least one processor coupled to the memory and configured for;

collecting multiple items of multi-modal data comprising a plurality of dimensions from multiple data stream originators, wherein said plurality of dimensions comprises at least a geographic dimension, a temporal dimension and an event-related dimension;

annotating each of the multiple items of collected multi-modal data with information, wherein the information comprises at least a veracity score associated with the data stream originator of the item of data;

determining multiple windows of relevance for the multiple items of multi-modal data with respect to the plurality of dimensions, wherein said multiple windows of relevance comprise at least (i) a temporal window of relevance, (ii) a spatial window of relevance, and (iii) and a semantic window of relevance; and

identifying two or more of the multiple items of multi-modal data as related based on said veracity score associated with the data stream originator of each item of data and by determining an overlap of at least one of the multiple windows of relevance corresponding to each of the two or more items of multi-modal data with respect to at least one of the plurality of dimensions.

18. The system of claim 17, wherein said multi-modal data comprises text message data, speech data, global positioning system data, audio data, mobile call detail records, internet content, and/or mobile-specific content.

19. A method comprising:

collecting multiple items of multi-modal data comprising multiple dimensions from multiple data stream originators, wherein said multiple dimensions comprise a geographic dimension, a temporal dimension and an event-related dimension;

annotating each of the multiple items of collected multi-modal data with information, wherein the information comprises at least a veracity score associated with the data stream originator of the item of data;

determining multiple windows of relevance for the multiple items of multi-modal data with respect to each of the multiple dimensions, wherein said multiple windows of relevance comprise at least (i) a temporal window of relevance, (ii) a spatial window of relevance, and (iii) and a semantic window of relevance;

identifying two or more of the multiple items of multi-modal data as related based on said veracity score associated with the data stream originator of each item of data and by determining an overlap of at least one of the multiple windows of relevance corresponding to each of the two or more items of multi-modal data with respect to at least two of the multiple dimensions; and

generating an alert based on the content of the two or more related items of multi-modal data;

wherein said collecting, said annotating, said determining, said identifying, and said generating are carried out by a computer device.