Methods and systems for real-time storyboarding with a web page and graphical user interface for automatic video parsing and browsing

• US 6,647,535 B1
• Filed: 03/18/1999
• Issued: 11/11/2003
• Est. Priority Date: 03/18/1999
• Status: Expired due to Term

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Harrington, Steven J., Bozdagi, Gozde, Yu, Hong Heather, Campanelli, Michael R., Bryll, Robert
• Primary Examiner(s): Hong, Stephen S.
• Assistant Examiner(s): Huynh, Thu V.
• Application Number:
  US09/271,867
• Time in Patent Office: 1,699 Days
• Field of Search: 707/530, 707/500, 707/500.1, 707/501.1, 715/501.1, 715/500, 715/530, 715/500.1, 348/460, 348/468, 348/700
• US Class Current: 715/255
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/8453 by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/8543 using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80
• CPC Class Codes: H04N 7/0885 for the transmission of subtitles
• CPC Class Codes: H04N 7/0887 for the transmission of programme or channel identifying signals
• CPC Class Codes: H04N 7/165 Centralised control of user terminal subsequent to an upstream request signal H04N7/17345; Registering at central by two-way working H04N7/17309

Abstract

Systems and methods to enable real-time and near real-time storyboarding on the World Wide Web in addition to a graphical user interface for video parsing and browsing the of the storyboard. Specifically, storyboarding can be accomplished on the World Wide Web by parsing an input video into representative or key frames. These frames then can be posted to a web document, or the like, for subsequent viewing by a user. This allows a video to be distilled down to the essential frames thus eliminating storage and bandwidth problems as well as eliminating the need for a user to view the entirety of the video. Furthermore, the graphical user interface allows a user to visually interact with an input video signal to determine the key or representative frames, or to retrieve video segments associated with already determined key frames. Furthermore, the interface allows manipulation of these frames including, but not limited to, playing of the entire segment represented by that key or significant frame as well as actual determining of the cuts between significant segments.

Claims

1. A real-time storyboarding system, comprising:

a frame capture device;

a command detector that detects if the input multimedia image data signal contains a command data that indicates key representative frames, the command data is embedded in a closed-caption portion of an input multimedia image data signal in addition to a closed-caption data to be displayed;

a command decoder that decodes the detected command data; and

an image significance determiner that determines in response to the command detector if there is at least one representative frame in the input multimedia image data signal, and that, if at least one representative frame is found, outputs the at least representative frame.

2. The system of claim 1, wherein the command data identifies at least one representative frame.

3. The system of claim 1, further comprising a frame difference determiner that determines a change between at least two frames of the input multimedia image data signal.

4. The system of claim 3, wherein a change between the at least two frames is a change in intensity between at least two frames of the input multimedia image data signal.

5. The system of claim 4, wherein the change in intensity E(t_i) for a current frame occurring at a time t=t_i, relative to a next frame occurring at time t=t_i+1, is:

6. The system of claim 5, wherein, if the change in intensity between the at least two frames is greater than a predefined threshold, one of the current frame and the next frame is identified as a representative frame.

7. The system of claim 1, wherein the at least one representative frame is output to a device capable of displaying the at least one representative frame.

8. The system of claim 1, wherein the at least one representative frame is output to a printer.

9. The system of claim 1, wherein the at least one representative frame is output to a web document.

10. The system of claim 9, wherein the web document is updated once a threshold number of representative frames are determined.

11. The system of claim 1, wherein at least a portion of the input multimedia image data signal is analyzed before the at least one representative frame is output.

12. The system of claim 11, wherein the entire input multimedia image data signal is analyzed before the at least one representative frame is output.

13. A method for real-time storyboarding, comprising:

selecting at least a portion of an input multimedia image data signal;

determining whether a command data is present such that the command data is embedded in a closed-caption portion of the input multimedia image data signal in addition to a closed-caption data to be displayed;

the command data indicates key representative frames, wherein if command data is present, decoding the command data to determine at least one representative frame;

determining in response to the command data is present the at least one representative frame of the selected portion of the input multimedia image data signal;

adding the at least one representative frame to a document; and

outputting the document.

14. The method of claim 13, wherein:

determining whether command data is present comprises determining if speaker data identifying at least one representative frame is present; and

if speaker data identifying at least one representative frame is present, adding the at least one identified representative frame to the document.

15. The method of claim 13, wherein:

determining whether command data is present comprises determining if at least one special characters identifying at least one representative frame is present; and

if at least one special character identifying at least one representative frame is present, adding the at least one identified representative frame to the document.

16. The method of claim 13, wherein:

determining whether command data is present comprises determining if a number of at least one special character identifies at least one representative frame; and

if the number of the at least one special character identifies at least one representative frame, adding the at least one representative frame to the document.

17. The method of claim 13, wherein:

determining whether command data is present comprises determining if a time lapse in the selected portion of the input multimedia image data signal since a last representative frame is greater than a threshold;

if the time lapse is greater than the threshold, identifying at least one additional frame as a representative frame of the selected portion of the input multimedia image data signal; and

adding the at least one identified frame to the document.

18. The method of claim 13, further comprising:

determining a frame difference between a pair of frames; and

if the frame difference exceeds a threshold, identifying one of the pair of frames as a representative frame of the selected portion of the input multimedia image data signal.

19. The method of claim 18, wherein the threshold is an intensity threshold.

20. The method of claim 19, wherein the intensity E(t_i) for a current frame occurring at a time t=t_i, relative to a next frame occurring at time t=t_i+1, is:

21. The method of claim 20, wherein if a change in intensity between the current frame and the next frame is greater than a predefined threshold, at least one of the frames is identified as a representative frame.

22. The method of claim 13, further comprising:

determining whether a threshold number of representative frames for the document have been identified; and

if the threshold is reached, selecting a new document; and

adding the at least one representative frame exceeding the threshold to the new document.