Video summary description scheme and method and system of video summary description data generation for efficient overview and browsing

• US 7,181,757 B1
• Filed: 09/29/2000
• Issued: 02/20/2007
• Est. Priority Date: 10/11/1999
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Jin Woong, Kim, Munchurl, Kim, Jae Gon, Chang, Hyun Sung
• Primary Examiner(s): BUI, KIEU OANH T
• Application Number:
  US09/675,984
• Time in Patent Office: 2,335 Days
• Field of Search: 725/40, 725/46, 725/61, 725/25, 725/35, 725/38, 345/716, 345/719, 345/720, 345/723, 348/563, 348/564
• US Class Current: 725/61
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

A video summary description system for describing video summary by meta data to provide overview functionality that makes it feasible to understand the overall contents of the original video within a short time with navigation and browsing functionalities, and that makes it feasible to search the desired video contents efficiently. The system includes a HierarchicalSummary Description Scheme (DS) for describing a video summary that includes a HighlightLevel DS having at least one HighlightSegment DS. The HighlightSegment DS describes information on a highlight segment corresponding to one of the video summary intervals, and the HighlightSegment DS includes a VideoSegmentLocator DS for describing time information or the video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment.

Claims

1. A HierarchicalSummary Description Scheme (DS) for describing a video summary, the HierarchicalSummary DS comprising:

a HighlightLevel DS includes at least one HighlightSegment DS, the HighlightSegment DS configured to describe information on a highlight segment corresponding to one of a plurality of video summary intervals, the HighlightSegment DS further including a VideoSegmentLocator DS describing time information or a video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment.

2. The HierarchicalSummary DS of claim 1 wherein said HighlightLevel DS further comprises at least one lower level HighlightLevel DS.

3. The HierarchicalSummary DS of claim 1 wherein said HighlightSegment DS further comprises a SoundLocator DS that describes the representative sound information of said corresponding highlight segment.

4. The HierarchicalSummary DS of claim 1 wherein said HighlightSegment DS further comprises an AudioSegmentLocator DS that describes the audio segment information constituting an audio summary of said corresponding highlight segment.

5. The HierarchicalSummary DS of claim 4 wherein the HierarchicalSummary DS includes an attribute of a SummaryComponentList that describes and enumerates SummaryComponentTypes representing types of summary.

6. The HierarchicalSummary DS of claim 5 wherein said SummaryComponentType comprises keyFrames representing a key frame summary composed of representative frames, keyVideoClips representing a key video clip summary composed of key video segment'"'"' sets, keyEvents representing a summary of a video interval corresponding to either an event or a subject, and keyAudioClips representing a key audio clip summary composed of representative audio intervals'"'"' sets, and an unconstraint representing the type of summary defined by users except for said summaries.

7. The HierarchicalSummary DS of claim 1 wherein the HierarchicalSummary DS further includes a SummaryThemeList DS enumerating the event or subject comprised in the summary and enabling users to execute summarizing and browsing based on the event.

8. The HierarchicalSummary DS of claim 7 wherein the SummaryThemeList DS comprises an arbitrary number of SummaryThemes as elements, and the SummaryTheme includes an attribute of and ID representing a corresponding event or subject.

9. The HierarchicalSummary DS of claim 8 wherein the SummaryTheme further includes an attribute of parentID that describes the ID of the event or subject of the upper level.

10. The HierarchicalSummary DS of claim 8 wherein the HighlightLevel DS comprises an attribute of themeIDs describing an attribute of the ID, when the HighlightSegments or HighlightLevels that constitute a corresponding highlight level have common events or subjects, the ID of the common events or the subjects is described in the themeID.

11. The HierarchicalSummary DS of claim 8 wherein the HighlightSegment DS comprises an attribute of themeIDs describing the event or subject of the highlight segment using the attribute of ID described in the SummaryThemeList DS.

12. A computer-readable recording medium where a HierarchicalSummary DS for describing a video summary is stored therein, wherein the HierarchicalSummary DS comprises a HighlightLevel DS that comprises at least one HighlightSegment DS,the HighlightSegment DS describes information on a highlight segment corresponding to one of a plurality of video summary intervals, and the HighlightSegment DS further comprises a VideoSegmentLocator DS describing time information or a video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment.

13. A method for generating video summary description data according to a video summary description scheme by inputting original video, comprising:

(a) analyzing the input original video and producing video analysis result;

(b) a summary rule for selecting the video summary interval;

(c) selecting the video summary interval capable of summarizing video contents from the original video based on the original video analysis result and the summary rule and constituting video summary interval information;

(d) extracting a representative frame based on the video summary interval information; and

(e) generating video summary description data according to the HierarchicalSummary DS enabling execution of hierarchical browsing based on the video summary interval information and the representative frame,

wherein the HeirarchicalSummary DS comprises a HighlightLevel DS that comprises at least one HighlightSegment DS, and the HighlightSegment DS describes information on a highlight segment corresponding to one of a plurality of video summary intervals, and the HighlightSegment DS comprises a VideoSegmentLocator DS describing time information or a video itself of the highlight segment.

14. The method of claim 13 wherein step (a) comprises the steps of:

extracting features from the input original video and outputting the types of features and video time interval at which those features are detected;

detecting key events included in the original video based on the types of features and video time interval at which those features are detected; and

detecting episode by dividing the original video into story flow on the basis of the detected event.

15. The method of claim 13 wherein the step (d) comprises the step of extracting a representative sound from the video summary interval information.

16. The method of claim 13 wherein the HighlightSegment DS further comprises an ImageLocator DS describing a representative frame of the highlight segment.

17. The method of claim 13 wherein the HighlightSegment DS further comprises SoundLocator DS describing a representative sound information of the highlight segment.

18. The method of claim 13 wherein the HighlightSegment DS further comprises AudioSegmentLocator DS describing the audio segment information constituting an audio summary of the highlight segment.

19. The method of claim 13 wherein the HierarchicalSummary DS further includes SummaryThemeList DS enumerating the event or subject comprised in the summary and enabling user'"'"'s to execute summarizing and browsing based on the event.

20. The method of claim 18 wherein the SummaryThemeList DS includes arbitrary number of SummaryThemes as elements, and the SummaryTheme includes an attribute of ID representing the corresponding event or subject.

21. The method of claim 20 wherein the HighlightSegment DS includes an attribute of themeIDs describing the event or subject of the highlight segment using the attribute of ID described in the SummaryThemeList DS.

22. The method of claim 20 wherein the SummaryTheme further includes an attribute of parentID describing the ID of the event or subject of the upper level.

23. The method of 20 wherein the HighlightLevel DS includes an attribute of themeIDs describing attribute of the ID, when the HighlightSegments or HighlightLevels which are constituting corresponding highlight level have common events or subjects, the ID of the common events or the subjects is described in the themeID.

24. A system for generating video summary description data according to a video summary description scheme by inputting original video, comprising:

video analyzing means for analyzing the original video and producing video analysis result;

summary rule defining means for defining the summary rule for selecting the video summary interval;

video summary interval selecting means for selecting the video interval capable of summarizing the video contents of the original video and outputting video summary interval information based on the video analysis result from the video analyzing means and the summary rule from the summary rule defining means;

representative frame extracting means for outputting a representative frame representing video summary interval based on the video summary interval information from the video summary interval selecting means; and

video summary describing means for generating video summary description data with HierarchicalSummary DS by inputting the video summary interval information from the video summary interval selecting means and the representative frame information from the representative frame extracting means,

wherein the HierarchicalSummary DS comprises a HighlightLevel DS that comprises at least one HighlightSegment DS, and the HighlightSegment DS describes information on segment information corresponding to one of a plurality of video summary intervals, and the HighlightSegment DS comprises a VideoSegmentLocator DS describing time information or a video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment, the HierarchicalSummary DS providing direct access to content of the original video through the representative frame information.

25. The system of claim 24 wherein said video analyzing means comprises:

feature extracting means for extracting features from the original video and producing the types of features and video time interval at which those features are detected;

event detecting means for detecting key events included in the original video by inputting the types of features and video time interval at which those features are detected; and

episode detecting means for detecting episode by dividing the original video into story flow base on the basis of said detected event.

26. The system of claim 24, the system further comprises representative sound extracting means for extracting a representative sound by inputting the video summary interval information and providing the extracted representative sound to the video summary describing means.

27. The system of claim 24 wherein the HighlightSegment DS further comprises SoundLocator DS describing a representative sound information of the highlight segment.

28. The system of claim 24 wherein the HighlightSegment DS further comprises AudioSegmentLocator DS describing the audio segment information constituting an audio summary of the highlight segment.

29. The system of claim 24 wherein the HierarchicalSummary DS further includes SummaryThemeList DS enumerating the event or subject comprised in the summary and enabling user'"'"'s to execute summarizing and browsing based on the event.

30. An apparatus for browsing video summary description data, wherein the video summary description data has a HierarchicalSummary Description Scheme (DS) for describing a video summary from an original video, the HierarchicalSummary DS includes:

a HighlightLevel DS which comprises at least one HighlightSegment DS describing information on highlight segment corresponding one of video summary intervals, and a SummaryThemeList DS enumerating the event or subject comprised in the summary and enabling user'"'"'s to execute summarizing and browsing based on the event, wherein the HighlightSegment DS includes a VideoSegmentLocator DS describing time information or video itself of the highlight segment, and an ImageLocator DS describing a representative frame of the highlight segment, wherein the browsing apparatus comprising;

a video playing part for playing an original video or the video summary;

an original video representative frame part for playing a representative frame of the original video;

a first video summary representative frame part for playing a first summary level of video interval,

a second video summary representative frame part for playing a second summary level of video interval, wherein the second summary level is summarized more finely than the first summary level;

a level selecting part for selecting the first summary level or the second summary level thereby enabling the video playing part to play the selected summary level; and

an event selecting part for enumerating the event or the subject provided in the SummaryThemeList DS for a user to browse desired event; and

a direct access part to provide direct access to the contents of the original video through the original video representative frame part when playing the first summary level of video interval or the second summary level of video interval.

31. The apparatus of claim 30 wherein the first video summary representative frame part plays the first summary level of sound information, and the second video summary representative frame part plays the first summary level of sound information.

32. The apparatus of claim 31 wherein the HighlightSegment DS further comprises:

a SoundLocator DS describing a representative sound information of the highlight segment; and

an AudioSegmentLocator DS describing the audio segment information constituting an audio summary of the highlight segment.

33. A method of browsing video summary description data, and accessing original video, the video summary description data having a HierarchicalSummary Description Scheme (DS) for describing a video summary,

the HierarchicalSummary DS having a HighlightLevel DS which comprises at least one HighlightSegment DS describing information on highlight segment corresponding one of video summary intervals, and

the HighlightSegment DS having a VideoSegmentLocator DS describing time information or video itself of the highlight segment, and an ImageLocator DS describing a representative frame of the highlight segment,

the browsing method comprising:

(a) playing a first summary level of video summary;

(b) playing a video interval of the original video corresponding to the representative frame when a desired scene is found through the video summary representative frame at the step (a); and

(c) directly accessing the original video through the video summary representative frame when playing the first summary level of video summary or when playing the video interval of the original video corresponding to the representative frame.

34. The method of claim 33, further comprising:

(c) playing a second summary level of video summary when a desired scene is not found through the video summary representative frame at the step (a), wherein the second summary level is summarized more finely than the first summary level.

35. The method of claim 33 wherein the HighlightSegment DS further comprises a SoundLocator DS describing a representative sound information of the highlight segment, the step (b) comprising the step of recognizing the desired scene to be found at the step (a) through the video summary representative sound information.

36. The method of claim 33 wherein the HierarchicalSummary DS further includes SummaryThemeList DS enumerating the event or subject comprised in the summary and enabling user'"'"'s to execute summarizing and browsing based on the event.

37. A Video Summary Description Scheme (DS) for describing a video summary of an original video, comprising:

at least one HighlightSegment DS describing information on a highlight segment corresponding to at least one video summary interval, wherein the HighlightSegment DS comprises a VideoSegmentLocator DS describing time information or video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment;

the HighlightSegment DS providing direct access to the original video throught the representative frame.

38. The Video Summary DS of claim 37 wherein the HighlightSegment DS further comprises SoundLocator DS describing a representative sound information of the highlight segment.

39. The Video Summary DS of claim 37 wherein the HighlightSegment DS further comprises AudioSegmentLocator DS describing the audio segment information constituting an audio summary of the highlight segment.

40. A method of browsing video summary description data of an original video, the video summary description data having at least one HighlightSegment DS describing information on a highlight segment corresponding to one of a plurality of video summary intervals, the HighlightSegment DS including a VideoSegmentLocator DS describing time information or video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment, the browsing method comprising:

(a) playing a first summary level of the video summary;

(b) playing a video interval of the original video corresponding to the representative frame when a desired scene is found through the video summary representative frame at the step (a); and

(c) directly accessing the contents of the original video when playing the first summary level of the video summary or playing the video interval of the original video.

41. The method of claim 40 further comprising:

(c) playing a second summary level of video summary when a desired scene is not found through the video summary representative frame at the step (a), wherein the second summary level is summarized more finely than the first summary level.

42. The method of claim 40 wherein the HighlightSegment DS further comprises a SoundLocator DS describing a representative sound information of the highlight segment, the step (b) comprising the step of recognizing the desired scene to be found at the step (a) through the video summary representative sound information.

43. The method of claim 40, wherein directly accessing comprises directly accessing the contents of the original video through the representive frame.