Automated rich presentation of a semantic topic

• US 20070094251A1
• Filed: 10/21/2005
• Published: 04/26/2007
• Est. Priority Date: 10/21/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Li, Zhiwei, Lu, Lie
• Granted Patent: US 8,572,088 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums

Abstract

Automated rich presentation of a semantic topic is described. In one aspect, respective portions of multimodal information corresponding to a semantic topic are evaluated to locate events associated with the semantic topic. The probability that a document belongs to an event is determined based on document inclusion of one or more of persons, times, locations, and keywords, and document distribution along a timeline associated with the event. For each event, one or more documents objectively determined to be substantially representative of the event are identified. One or more other types of media (e.g., video, images, etc.) related to the event are then extracted from the multimodal information. The representative documents and the other media are for presentation to a user in a storyboard.

Claims

1. A computer-implemented method comprising:

determining a semantic topic;

evaluating respective portions of multimodal information corresponding to the semantic topic to identify events, each event being associated with one or more of person, time, location, and keyword;

for each document in the respective portion, calculating probability that the document belongs to an event of the events based on the generative model and document distribution along a timeline associated with the event;

for each event in least a subset of the events;

objectively identifying one or more representative documents that are of greater relevance to the event as compared to other documents;

extracting other media corresponding to the representative documents from the multimodal information, the representative documents and the other media being objectively most representative of the semantic topic; and

wherein the one or more representative documents and the other media are for presentation to a user in a storyboard.

2. The method of claim 1, wherein the at least a subset of the events is an event summary.

3. The method of claim 1, wherein calculating the probability further comprises:

identifying a salient number of events corresponding to the semantic topic, the salient number of events being less than a total number of the events; and

calculating the probability using the salient number of events.

4. The method of claim 1, wherein calculating the probability further comprises representing the probability associated with time as a function of where a date of the document lies with respect to event duration

5. The method of claim 1, wherein calculating the probability further comprises:

independently estimating, for respective person, location, and keyword models, model parameters by iterative expectation and maximization operations; and

calculating the probability using the models in view of temporal continuity of the event and any overlap of two or more of the events.

6. The method of claim 1, further comprising, for each document in the representative documents, removing any resource associated with an advertisement provided an entity other than an entity associated with a web site from which the document was obtained.

7. The method of claim 1, further comprising, for each event in the at least a subset of the events, if the representative documents include multiple documents, removing any duplicate image from the multiple documents.

8. The method of claim 1, wherein the one or more representative documents and the other media are representative content, and wherein the method further comprises integrating the representative content into the storyboard for presentation to a user, the storyboard providing a concise overview of salient event(s) and associated multimodal information regarding the semantic topic.

9. The method of claim 1, further comprising synchronizing the representative content of the storyboard layout with music.

10. The method of claim 9, wherein the synchronizing further comprises:

identifying music sub-clips and a corresponding timeline; and

synchronizing event slide transitions and the storyboard layout with the music sub-clips using the timeline.

11. The method of claim 9, wherein the synchronizing further comprises:

identifying music sub-clips and a corresponding timeline;

synchronizing event slide transitions and the storyboard layout with the music sub-clips using the timeline; and

wherein a length of an event slide is equal to a corresponding length of a respective music sub-clip.

12. The method of claim 9, wherein the synchronizing further comprises:

identifying music sub-clips and a corresponding timeline;

synchronizing event slide transitions and the storyboard layout with the music sub-clips using the timeline; and

wherein each event associated with the event slide transitions is objectively determined to belong to a set of events in the at least a subset of events that are more important than other events in the at least a subset of events.

13. A computer implemented method comprising:

determining a semantic topic;

extracting, from multimedia data, multimodal information relevant to the semantic topic;

evaluating respective portions of the multimodal information to identify events, each event being associated with one or more of person, time, location, and keyword;

for each document in the respective portion, calculating probability that the document belongs to an event of the events based on a generative model and document distribution along a timeline associated with the event;

generating an event summary summarizing the events;

for each event in the event summary;

objectively identifying one or more representative documents that are of greater relevance to the event as compared to other documents; and

extracting other media corresponding to the representative documents from the multimodal information, the representative documents and the other media being representative content;

integrating the representative content into a storyboard layout for presentation to a user, the storyboard layout providing a concise overview of salient event(s) and associated multimodal information regarding the semantic topic.

14. The method of claim 13, wherein calculating the probability further comprises:

identifying a salient number of events corresponding to the semantic topic, the salient number of events being less than a total number of the events; and

calculating the probability using the salient number of events.

15. The method of claim 13, wherein calculating the probability further comprises:

independently estimating, for respective person, location, and keyword models, model parameters by iterative expectation and maximization operations; and

calculating the probability using the models in view of temporal continuity of the event and any overlap of two or more of the events.

16. The method of claim 13, further comprising removing, from a document of the one or more representative documents, one or more of a duplicate image and an advertisement associated with any entity other than an entity associated with a web site from which the document was obtained.

17. The method of claim 13, further comprising synchronizing the representative content of the storyboard layout with music.

18. A computing device comprising processing means for:

identifying, from multi-modal data sources, representative multi-modal content of a semantic topic, the representative multi-modal content comprising one or more of images, news article summaries, video, and locations corresponding to one or more salient events associated with the semantic topic; and

integrating the representative content into a storyboard layout for presentation to a user; and

presenting the storyboard layout to a user for one or more of review and navigation of the representative content.

19. The computing device of claim 18, wherein identifying the representative multi-modal content further comprises calculating the probability that a document includes content representative of an event as a function of where a date of the document lies with respect to event duration.

20. The computing device of claim 18, wherein integrating the representative multi-modal content into the storyboard layout further comprises integrating the representative multi-modal content with incidental music for presentation to the user.