System for organizing videos based on closed-caption information

• US 6,580,437 B1
• Filed: 06/26/2000
• Issued: 06/17/2003
• Est. Priority Date: 06/26/2000
• Status: Active Grant

• Current Assignee: Siemens AG
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Das, Madirakshi, Liou, Shih-Ping, Toklu, Candemir
• Primary Examiner(s): dela Torre, Crescelle N.
• Application Number:
  US09/602,721
• Time in Patent Office: 1,086 Days
• Field of Search: 345/700, 345/716, 345/719-721, 345/723, 345/727, 345/764, 345/853, 348/461, 348/462, 348/465, 348/467, 348/468, 386/52, 386/55, 707/102, 707/104.1, 725/137
• US Class Current: 715/719
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/9027 Trees

Abstract

A system for organizing digital videos to archive and access them at different levels of abstraction uses data available from a closed-caption text along with off-the-shelf natural language processing tools to segment the video into self-contained story sections and speaker blocks. If the subject changes are marked, the system uses these points to divide the video into distinct stories which are represented as nodes attached to the root node in a tree structure and groups speaker segments belonging to a story under the story node as its children.

Claims

1. A system for indexing video comprising:

a video directory;

a shot editor connected to said video directory for browsing and editing the video directory to produce a shotlist;

a producer connected to said shot editor for producing a mergelist and a keyframelist from the shotlist;

a shot group editor connected to said producer for editing the mergelist and the keyframelist;

a closed-caption decision connected to said shot group editor for determining if closed-caption text is available;

a processor connected to said closed-caption decision for processing the closed-caption text to extract data, wherein said data includes change signals; and

a closed-caption generator connected to said processor for generating subject segments and speaker segments using the data, wherein said closed-caption generator further defines shots and stories using said change signals.

2. The system of claim 1, wherein the video includes audio and visual information.

3. The system of claim 1, wherein the video directory comprises a video shot list and representative frames.

4. The system of claim 1, wherein the closed-caption text comprises lines of text and frame numbers, wherein said lines of text include speaker markers for marking a beginning of different speaker segments.

5. The system of claim 1, wherein the producer produces the mergelist and the keyframelist by finding groups of visually similar shots in the shotlist.

6. The system of claim 1, wherein the shot group editor edits the mergelist and the keyframelist to check groups of visually similar shots for correctness, to move shots in and out of groups, and to create new groups of visually similar shots.

7. The system of claim 1, wherein the processor processes the closed-caption text by aligning the closed-caption text to the video, generating sentences from the closed-caption text, extracting noun phrases and proper nouns from the sentences, and storing said noun phrases and proper nouns in a noun data file.

8. A system for indexing video comprising:

a video directory;

a shot editor connected to said video directory for browsing and editing the video directory to produce a shotlist;

a producer connected to said shot editor for producing a mergelist and a keyframelist from the shotlist;

a shot group editor connected to said producer for editing the mergelist and the keyframelist;

a closed-caption decision connected to said shot group editor for determining if closed-caption text is available;

a processor connected to said closed-caption decision for processing the closed-caption text to extract data, wherein the closed-caption text comprises lines of text and frame numbers, wherein said lines of text include speaker markers for marking a beginning of different speaker segments; and

a closed-caption generator connected to said processor for generating subject segments and speaker segments using the data, wherein the closed-caption generator generates the subject segments and the speaker segments by determining whether different subject segments in the data are marked by subject markers, wherein if it is determined that different subject segments in the data are marked by subject markers, further comprising;

a separator for separating the different subject segments into separate nodes using the subject markers, and for separating the different speaker segments into separate speaker segments using the speaker markers; and

an organized tree creator for creating an organized tree comprising the separate nodes and attaching corresponding separate speaker segments to each separate node.

9. The system of claim 8, wherein if it is determined that the different subject segments in the data are not marked by subject markers, further comprising:

a related segments finder for finding related segments in the different speaker segments using common elements and grouping the related segments into separate nodes.

10. The system of claim 9, wherein the common elements comprise proper nouns in the closed-caption text.

11. A method of indexing video comprising the steps of:

providing a video directory;

browsing and editing said video directory using a shot editor to produce a shotlist;

producing a mergelist file and a keyframelist from the shotlist using a producer;

editing the mergelist file and the keyframelist using a shot group editor;

determining if closed-caption text of said video is available, wherein if closed-caption text is determined to be available;

processing said closed-caption text to extract data, wherein said data includes change signals; and

segmenting the video into subject segments and speaker segments using the data, wherein segmenting further includes defining shots and stories using said change signals.

12. The method of claim 11, wherein the video includes audio and visual information.

13. The method of claim 11, wherein the video directory comprises a video shot list and representative frames.

14. The method of claim 11, wherein the closed-caption text comprises lines of text and frame numbers, wherein said lines of text include speaker markers for marking a beginning of different speaker segments.

15. The method of claim 11, wherein the step of processing includes the steps of:

aligning the closed-caption text to the video;

generating sentences from the closed-caption text;

extracting noun phrases and proper nouns from the sentences; and

storing said noun phrases and proper nouns in a noun data file.

16. The method of claim 11, wherein the mergelist file comprises groups of visually similar shots.

17. The method of claim 11, wherein the keyframelist comprises additional distinct images from the shotlist.

18. The method of claim 11, wherein the shotlist comprises pairs of begin and end frames.

19. A method of indexing video comprising the steps of:

providing a video directory;

browsing and editing said video directory using a shot editor to produce a shotlist;

producing a mergelist file and a keyframelist from the shotlist using a producer;

editing the mergelist file and the keyframelist using a shot group editor;

determining if closed-caption text of said video is available, wherein if closed-caption text is determined to be available;

processing said closed-caption text to extract data, wherein the closed-caption text comprises lines of text and frame numbers, wherein said lines of text include speaker markers for marking a beginning of different speaker segments; and

segmenting the video into subject segments and speaker segments using the data, wherein the step of segmenting includes the steps of

determining whether different subject segments in the data are marked by subject markers, wherein if it is determined that different subject segments in the data are marked by subject markers, further comprising the steps of;

separating the different subject segments into nodes using the subject markers;

separating the different speaker segments using the speaker markers to form separate speaker segments; and

creating an organized tree comprising the nodes and attaching the separate speaker segments to each corresponding node.

20. The method of claim 19, wherein if it is determined that the different subject segments in the data are not marked by subject markers, the step of segmenting further comprises the steps of:

determining common elements in the different speaker segments; and

matching the common elements within the different speaker segments to form related segments, wherein the related segments are grouped into corresponding nodes.

21. The method of claim 20, wherein the common elements comprise proper nouns in the closed-caption text.

22. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform the method steps for indexing video, the method comprising the steps of:

providing a video directory;

browsing and editing said video directory using a shot editor to produce a shotlist;

producing a mergelist file and a keyframelist from the shotlist using a producer;

editing the mergelist file and the keyframelist using a shot group editor;

determining if closed-caption text of said video is available, wherein if closed-caption text is determined to be available;

processing said closed-caption text to extract data, wherein said data includes change signals; and

segmenting the video into subject segments and speaker segments using the data, wherein segmenting further includes defining shots and stories using said change signals.

23. The program storage device of claim 22, wherein the closed-caption text comprises lines of text and frame numbers, wherein said lines of text include speaker markers for marking a beginning of different speaker segments.

24. The program storage device of claim 22, wherein the instructions for performing the step of processing includes instructions for:

aligning the closed-caption text to the video;

generating sentences from the closed-caption text;

extracting noun phrases and proper nouns from the sentences; and

storing said noun phrases and proper nouns in a noun data file.

25. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform the method steps for indexing video, the method comprising the steps of:

providing a video directory;

browsing and editing said video directory using a shot editor to produce a shotlist;

producing a mergelist file and a keyframelist from the shotlist using a producer;

editing the mergelist file and the keyframelist using a shot group editor;

determining if closed-caption text of said video is available, wherein if closed-caption text is determined to be available;

processing said closed-caption text to extract data, wherein the closed-caption text comprises lines of text and frame numbers, wherein said lines of text include speaker markers for marking a beginning of different speaker segments; and

segmenting the video into subject segments and speaker segments using the data, wherein the instructions for performing the step of segmenting includes instructions for;

determining whether different subject segments in the data are marked by subject markers, wherein if it is determined that different subject segments in the data are marked by subject markers, further comprising instructions for;

separating the different subject segments into nodes using the subject markers;

separating the different speaker segments using the speaker markers to form separate speaker segments; and

creating an organized tree comprising the nodes and attaching the separate speaker segments to each corresponding node.

26. The program storage device of claim 25, wherein if it is determined that the different subject segments in the data are not marked by subject markers, the instructions for segmenting further comprises instructions for performing the steps of:

determining common elements in the different speaker segments; and

matching the common elements within the different speaker segments to form related segments, wherein the related segments are grouped into corresponding nodes and the common elements comprise proper nouns in the closed-caption text.