Method for identification of tokens in video sequences

• US 20040183825A1
• Filed: 03/15/2004
• Published: 09/23/2004
• Est. Priority Date: 03/19/2003
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Chupeau, Bertrand, Stauder, Jurgen
• Granted Patent: US 7,340,096 B2
• Time in Patent Office: Days
• US Class Current: 345/723
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00

Abstract

For identification of tokens in video sequences, first the sequence is scanned for boundaries between consecutive parts of the sequence, where appearance of the tokens is expected. After that, candidate regions are pre-selected and classified in parts of the video sequence adjacent to detected boundaries and tokens are located in the candidate regions. Information carried on the tokens is located, interpreted and merged into consistent sets of information after passing a confidence analysis. In parallel, changes in the visual appearance of the token signaling a special event are detected.

Claims

1. A method for automatically identifying tokens associated with an event in video sequences containing individual takes and extracting information contained thereon including the steps of:

classifying the candidate regions into token or non-token,

locating tokens in candidate regions,

locating information on the token,

interpreting the information on the token,

performing a confidence analysis on the information to ensure that the information was interpreted correctly, wherein the method further includes the steps of;

detecting boundaries between individual takes,

pre-selecting candidate regions in images of the video sequence following or preceding a detected boundary prior to the step of classifying the candidate regions, and that the method also further includes the steps of

merging information of consecutive images into a single, coherent set of information, and

detecting changes in the visual appearance of the token, the changes signalling a particular point of an event, after the step of performing the confidence analysis.

2. The method of claim 1, wherein after detecting a boundary between individual takes a timer is set, upon time-out thereof the identification process is terminated.

3. The method of claim 1, wherein detecting boundaries between individual takes includes the steps of:

creating a histogram from image properties for consecutive images,

calculating the distance between filtered histograms of consecutive images,

comparing the calculated distance to a preset threshold, and

issuing a signal indicating a boundary is detected upon the distance exceeding the threshold.

4. The method of claim 1, wherein pre-selecting candidate regions in candidate images includes the steps of:

defining a reference feature value set corresponding to tokens,

scanning the image at varying locations using a suitably shaped scanning window;

computing a feature value set for each scanning window location,

comparing each feature value set with the reference feature value set, and

ranking the scanning windows containing feature value sets according to their distance to the reference feature value set.

5. The method of claim 1, wherein classifying the candidate regions into token or non-token includes the steps of:

calculating feature values from the candidate regions,

comparing the calculated feature values to known classified feature values of reference images,

assigning classifiers to the respective candidate regions, and

assigning a classification confidence value to the classified candidate regions.

6. The method of claim 1, wherein locating tokens in the candidate regions includes the steps of:

scanning the candidate region using a suitably shaped scanning window,

calculating coefficients describing the correlation between the scanning window and a reference image of a token,

averaging the coefficients, thereby defining a matching confidence value,

selecting the scanning window having the highest confidence value as candidate window.

7. The method of claim 6, wherein the scanning window and/or the reference image is decimated or interpolated in its spatial resolution prior to calculating the correlation coefficients, resulting in a corresponding number of pixels for the scanning window and the reference image.

8. The method of claim 6, wherein the candidate region is re-classified to non-token if the confidence value is below a preset threshold.

9. The method of claim 1, wherein locating information on the token includes the steps of:

cutting a sub-image from the candidate image using size and position data from the token localization process,

constructing a probability map of the sub-image describing the probability that a pixel of the sub-image belongs to information on the token, and

selecting an area of the sub-image with maximum probability values.

10. The method of claim 9, wherein the probability is obtained by comparing pixel properties of the sub-image to pre-defined pixel properties belonging to information elements.

11. The method of claims 1 and 9, wherein interpreting the information contained on the token includes the steps of:

rotating the selected sub-image area with maximum probability to bring the information contained therein into horizontal orientation,

binarizing the probability map values of the sub-image area,

filtering the binarized map, and

performing an optical character recognition on the filtered map.

12. The method of claim 1, wherein a confidence analysis is effected by checking information extracted from consecutive images for consistency.

13. The method of claim 1, wherein merging information elements includes replacing mismatching information elements and/or information elements having low confidence with interpolated information elements.

14. The method of claim 1, wherein detecting changes in the visual appearance of the token includes the steps of:

detecting and locating parts of the token subject to change by analysing visual features in candidate regions and comparing the visual features to pre-determined visual features of tokens,

monitoring the change in visual appearance of the parts subject to change by comparing detected visual features in consecutive images, and

outputting data describing the degree of change with regard to a pre-determined starting and/or end point.

15. The method of claim 14, wherein the visual features are translated into simplified models for analysis and monitoring.