ASSOCIATION OF VISUAL LABELS AND EVENT CONTEXT IN IMAGE DATA

• US 20150058348A1
• Filed: 08/26/2013
• Published: 02/26/2015
• Est. Priority Date: 08/26/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chang, Yuan-Chi, Nguyen, Quoc-Bao, Cao, Liangliang
• Granted Patent: US 9,734,166 B2
• Time in Patent Office: Days
• US Class Current: 707/740
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858

Abstract

A first set of contextual dimensions is generated from one or more textual descriptions associated with a given event, which includes one or more examples. A second set of contextual dimensions is generated from one or more visual features associated with the given event, which includes one or more visual example recordings. A similarity structure is constructed from the first set of contextual dimensions and the second set of contextual dimensions. One or more of the textual descriptions is matched with one or more of the visual features based on the similarity structure.

Claims

1. A method, comprising:

generating a first set of contextual dimensions from one or more textual descriptions associated with a given event, which comprise one or more examples;

generating a second set of contextual dimensions from one or more visual features associated with the given event, which comprise one or more visual example recordings;

constructing a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions; and

matching one or more of the textual descriptions with one or more of the visual features based on the similarity structure;

wherein the generating, constructing and matching steps are performed via one or more processing devices.

2. The method of claim 1, wherein the step of generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises parsing the one or more textual descriptions associated with the given event by identifying one or more terms or one or more sets of terms appearing in one or more taxonomies or one or more ontologies.

3. The method of claim 2, wherein the step of generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises mapping the one or more identified terms or one or more identified sets of terms to one or more textual objects in the one or more taxonomies or the one or more ontologies.

4. The method of claim 3, wherein the step of generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises classifying the one or more textual objects into one or more classes.

5. The method of claim 4, wherein the step of generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises arranging the one or more classified textual objects in a time sequence describing the given event in one or more event taxonomy graphs.

6. The method of claim 5, wherein the step of generating a second set of contextual dimensions for one or more visual features associated with the given event further comprises extracting the one or more audio-visual features associated with the given event from one or more images or one or more videos.

7. The method of claim 6, wherein the step of generating a second set of contextual dimensions for one or more visual features associated with the given event further comprises classifying the one or more visual features into one or more visual concepts associated with one or more taxonomies or one or more ontologies.

8. The method of claim 7, wherein the step of constructing a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions further comprises constructing a visual and textual concept relationship network.

9. The method of claim 8, wherein the step of constructing a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions further comprises forming the relationship network by associating each of the one or more visual concepts to the one or more event taxonomy graphs.

10. The method of claim 9, wherein the step of matching one or more of the textual descriptions with one or more of the visual features based on the similarity structure further comprises assigning a relevant one of the one or more textual descriptions to one of the one or more images or the one or more videos based on the formed relationship.

11. The method of claim 10, wherein the step of classifying the one or more textual objects and the step of classifying the one or more visual features further comprise selecting from context classes, object classes and activity classes.

12. A computer program product comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by the one or more processing devices implement steps of:

generating a first set of contextual dimensions from one or more textual descriptions associated with a given event, which comprise one or more examples;

generating a second set of contextual dimensions from one or more visual features associated with the given event, which comprise one or more visual example recordings;

constructing a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions; and

matching one or more of the textual descriptions with one or more of the visual features based on the similarity structure.

13. An apparatus, comprising:

a memory; and

a processor operatively coupled to the memory and configured to;

generate a first set of contextual dimensions from one or more textual descriptions associated with a given event, which comprise one or more examples;

generate a second set of contextual dimensions from one or more visual features associated with the given event, which comprise one or more visual example recordings;

construct a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions; and

match one or more of the textual descriptions with one or more of the visual features based on the similarity structure.

14. The apparatus of claim 13, wherein generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises parsing the one or more textual descriptions associated with the given event by identifying one or more terms or one or more sets of terms appearing in one or more taxonomies or one or more ontologies.

15. The apparatus of claim 14, wherein generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises mapping the one or more identified terms or one or more identified sets of terms to one or more textual objects in the one or more taxonomies or the one or more ontologies.

16. The apparatus of claim 15, wherein generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises classifying the one or more textual objects into one or more classes.

17. The apparatus of claim 16, wherein generating a first set of contextual dimensions for one or more textual descriptions associated with a given event further comprises arranging the one or more classified textual objects in a time sequence describing the given event in one or more event taxonomy graphs.

18. The apparatus of claim 17, wherein generating a second set of contextual dimensions for one or more visual features associated with the given event further comprises extracting the one or more visual features associated with the given event from one or more images or one or more videos.

19. The apparatus of claim 18, wherein generating a second set of contextual dimensions for one or more visual features associated with the given event further comprises classifying the one or more visual features into one or more visual concepts associated with one or more taxonomies or one or more ontologies.

20. The apparatus of claim 19, wherein constructing a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions further comprises constructing a visual and textual concept relationship network.

21. The apparatus of claim 20, wherein constructing a similarity structure from the first set of contextual dimensions and the second set of contextual dimensions further comprises forming the relationship network by associating each of the one or more visual concepts to the one or more event taxonomy graphs.

22. The apparatus of claim 21, wherein matching one or more of the textual descriptions with one or more of the visual features based on the similarity structure further comprises assigning a relevant one of the one or more textual descriptions to one of the one or more images or the one or more videos based on the formed relationship.

23. The apparatus of claim 22, wherein classifying the one or more textual objects and the step of classifying the one or more visual features further comprise selecting from context classes, object classes and activity classes.