System and method for extracting text captions from video and generating video summaries

• US 8,488,682 B2
• Filed: 12/19/2007
• Issued: 07/16/2013
• Est. Priority Date: 12/06/2001
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Chang, Shih-Fu, Zhang, Dongqing
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US11/960,424
• Publication Number:
  US 20080303942A1
• Time in Patent Office: 2,036 Days
• Field of Search: None
• US Class Current: 375/240.25
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/434 Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 5/93 Regeneration of the television signal or of selected parts thereof
• CPC Class Codes: H04N 7/025 Systems for the transmission of digital non-picture data, e.g. of text during the active part of a television frame transmission of digital non-picture data during the vertical blanking interval only H04N7/088
• CPC Class Codes: H04N 7/0882 for the transmission of character code signals, e.g. for teletext circuits for the digital non-picture data signal H04N7/035

Abstract

Caption boxes which are embedded in video content can be located and the text within the caption boxes decoded. Real time processing is enhanced by locating caption box regions in the compressed video domain and performing pixel based processing operations within the region of the video frame in which a caption box is located. The captions boxes are further refined by identifying word regions within the caption boxes and then applying character and word recognition processing to the identified word regions. Domain based models are used to improve text recognition results. The extracted caption box text can be used to detect events of interest in the video content and a semantic model applied to extract a segment of video of the event of interest.

Claims

1. A method of decoding a caption box in video content comprising:

determining at least one expected location of a caption box in a frame of the video content;

determining at least one caption box mask within the expected location;

identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location;

for at least a portion of the caption frames, identifying word regions within the confines of the expected location;

for each word region,

identifying text characters within the region; and

processing the identified text characters.

2. The method of decoding a caption box according to claim 1 further comprising:

comparing the text characters in the word region against a domain specific model to enhance word recognition,

determining a word region type for at least one of the identified word regions;

wherein the video content is of a sporting event and wherein the word region types are selected from the group consisting of data points that indicate the current state of the event.

3. The method of decoding a caption box according to claim 2 wherein the data point is a period.

4. The method of decoding a caption box according to claim 2 wherein the data point is a quarter.

5. The method of decoding a caption box according to claim 2 wherein the data point is field position.

6. The method of decoding a caption box according to claim 2 wherein the data point is the number of shots on goal.

7. A non-transitory computer-readable storage medium storing a program for causing a computer to implement a method of decoding a caption box in video content comprising:

determining at least one expected location of a caption box in a frame of the video content;

determining at least one caption box mask within the expected location;

identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location;

for at least a portion of the caption frames, identifying word regions within the confines of the expected location;

for each word region, identifying text characters within the region;

and processing the identified text characters.

8. The non-transitory computer-readable storage medium of claim 7 wherein the method includes the step of comparing the text characters in the word region against a domain specific model to enhance word recognition.

9. A system for decoding a caption box in video content comprising:

location means for determining at least one expected location of a caption box in a frame of the video content;

determining means, coupled to the location means and receiving the at least one expected location therefrom, for determining at least one caption box mask within the expected location;

frame identifying means, coupled to the determining means and location means and receiving the at least one caption box mask and the at least one expected location therefrom, for identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location;

word region identifying means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for at least a portion of the caption frames, identifying word regions within the confines of the expected location;

text character means, coupled to the word region identifying means and receiving the word regions therefrom, for each word region, identifying text characters within the region; and

processing the identified text characters.

10. The system of claim 9 wherein the system includes a comparing means, coupled to the text character means and receiving the text characters therefrom, for comparing the text characters in the word region against a domain specific model to enhance word recognition.

11. The system of claim 9 wherein the location means includes a features means for evaluating motion features of the video frame in the compressed domain, evaluating texture features of the video frame in the compressed domain, and identifying regions having low motion features and high texture features as candidate caption box regions.

12. The system of claim 10 wherein the location means includes a features means for evaluating motion features of the video frame in the compressed domain, evaluating texture features of the video frame in the compressed domain, and identifying regions having low motion features and high texture features as candidate caption box regions.

13. The system of claim 9 wherein the system includes a removal means, coupled to the frame identifying means, the determining means and the location means and receiving the identified caption frames, the at least one caption box mask and the at least one expected location therefrom, for evaluating the identified caption frames, within the caption box location, for changes in content and removing caption frames from word region processing which do not exhibit a change in content.

14. The system of claim 9 wherein the system includes an interval means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for selecting a subset of caption frames based on a predetermined time interval and sending that subset to the word region identifying means.

15. The system of claim 9 wherein the system includes a number means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for determining a subset of caption frames by selecting caption frames based on a predetermined number of intervening caption frames and sending that subset to the word region identifying means.