TEMPORAL SEGMENT BASED EXTRACTION AND ROBUST MATCHING OF VIDEO FINGERPRINTS

• US 20090154806A1
• Filed: 12/17/2007
• Published: 06/18/2009
• Est. Priority Date: 12/17/2007
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Inventors: Chang, Jane Wen, Smith, John R., Natsev, Apostol Ivanov
• Granted Patent: US 9,177,209 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00

Abstract

A computer implemented method, apparatus, and computer program product code for temporal, event-based video fingerprinting. In one embodiment, events in video content are detected. The video content comprises a plurality of video frames. An event represents discrete points of interest in the video content. A set of temporal, event-based segments are generated using the events. Each temporal, event-based segment is a segment of the video content covering a set of events. A time series signal is derived from each temporal, event-based segment using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment. A temporal segment based fingerprint is extracted based on the time series signal for the each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content.

Claims

1. A computer implemented method for temporal, event-based video fingerprinting, the computer implemented method comprising:

detecting events in video content, wherein the video content comprises a plurality of video frames, and wherein an event represents discrete points of interest in the video content;

generating a set of temporal, event-based segments using the events, wherein each temporal, event-based segment is a segment of the video content covering a set of events;

deriving a time series signal from each temporal, event-based segment in the set of temporal, event-based segments using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment; and

extracting a segment fingerprint based on the time series signal for the each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content.

2. The computer implemented method of claim 1 wherein the set of temporal segment based fingerprints are used to determine if a portion of a first video clip is derived from a same original content as a second video clip, the computer implemented method further comprising:

comparing temporal segment based fingerprints for the first video clip with temporal segment based fingerprints generated for the second video clip;

identifying matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;

collecting all matching segments between the first video clip and the second video clip to form a set of matching segments;

selecting a subset of matching segments, wherein the subset of matching segments comprises matching segments associated with the first video clip that produces a good linear fit to matching segments associated with the second video clip;

identifying an overall video match score for the first video clip and the second video clip based on the selected matching segments; and

determining whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

3. The computer implemented method of claim 2 further comprising:

comparing the overall video match score to a threshold score; and

responsive to the overall video match score exceeding the threshold score, identifying the first video clip as a near-duplicate of the second video clip.

4. The computer implemented method of claim 1 wherein the temporal segment based fingerprints are used for at least one of content-based search, concept detection, content categorization, summarization, filtering, routing, or targeted advertising.

5. The computer implemented method of claim 1 wherein the content-based features comprises at least one of audio features and visual features, and wherein each event is detected based on at least one of video shot detection, scene changes, speaker changes, audio changes, frame intensity changes, or changes based on low-level content-based descriptors of color, texture, shape, edges, or motion.

6. The computer implemented method of claim 1 wherein the temporal, event-based segments span a set of frames in the plurality of frames, and wherein the set of frames covers only a subset of the video content.

7. The computer implemented method of claim 1 wherein a first temporal, event-based segment associated with the video content overlaps with a second temporal, event-based segment associated with the video content.

8. The computer implemented method of claim 1 wherein the time series signal is based on at least one of temporal tracking of overall frame intensity, tracking of frame region-based intensity sequences, tracking of grid-based intensity sequences, and tracking of adjacent frame differences.

9. The computer implemented method of claim 1 wherein the temporal segment based fingerprints are extracted from the segment time series based on at least one of uniform sampling, piece-wise linear approximation, Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT), and Discrete Fourier Transform (DFT).

10. The computer implemented method of claim 1 wherein a temporal segment based fingerprint comprises a fixed-dimensionality feature vector.

11. The computer implemented method of claim 1 wherein an event describes a significant change of state in at least one of audio content of the video, visual content of the video, and semantic content of the video.

12. A computer program product for temporal, event-based video fingerprinting, the computer program product comprising:

a computer readable medium;

program code stored on the computer readable medium for detecting events in video content, wherein the video content comprises a plurality of video frames, and wherein an event represents discrete points of interest in the video content;

program code stored on the computer readable medium for generating a set of temporal, event-based segments using the events, wherein each temporal, event-based segment is a segment of the video content covering a set of events;

program code stored on the computer readable medium for deriving a time series signal from each temporal, event-based segment in the set of temporal, event-based segments using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment; and

program code stored on the computer readable medium for extracting a segment fingerprint based on the time series signal for each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content.

13. The computer program product of claim 12 wherein the set of temporal segment based fingerprints are used to determine if a portion of a first video clip is derived from a same original content as a second video clip, the computer program product further comprising:

program code stored on the computer readable medium for comparing temporal segment based fingerprints for the first video clip with temporal segment based fingerprints generated for the second video clip;

program code stored on the computer readable medium for identifying matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;

program code stored on the computer readable medium for collecting all matching segments between the first video clip and the second video clip to form a set of matching segments;

program code stored on the computer readable medium for selecting a subset of matching segments, wherein the subset of matching segments comprises matching segments associated with the first video clip that produces a good linear fit to matching segments associated with the second video clip;

program code stored on the computer readable medium for identifying an overall video match score for the first video clip and the second video clip based on the selected matching segments in the subset of matching segments; and

program code stored on the computer readable medium for determining whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

14. The computer program product of claim 13 further comprising:

program code stored on the computer readable medium for comparing the overall video match score to a threshold score;

program code stored on the computer readable medium for identifying the first video clip as a near-duplicate of the second video clip in response to the overall video match score exceeding the threshold score.

15. The computer program product of claim 12 wherein the content-based features comprises at least one of audio features and visual features, and wherein each event is detected based on at least one of video shot detection, scene changes, speaker changes, audio changes, frame intensity changes, or changes based on low-level content-based descriptors of color, texture, shape, edges, or motion.

16. An apparatus for automatically detecting video piracy, the apparatus comprising:

a bus system;

a communications system coupled to the bus system;

a memory connected to the bus system, wherein the memory includes computer usable program code; and

a processing unit coupled to the bus system, wherein the processing unit executes the computer usable program code to detect events in video content, wherein the video content comprises a plurality of video frames, and wherein an event represents discrete points of interest in the video content;

generate a set of temporal, event-based segments using the events, wherein each temporal, event-based segment is a segment of the video content covering a set of events;

derive a time series signal from each temporal, event-based segment in the set of temporal, event-based segments using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment; and

extract a segment fingerprint based on the time series signal for each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content, wherein detection of the near-duplicate is used for content-based video piracy detection.

17. The apparatus of claim 16 wherein the processing unit further executes the computer usable program code to compare temporal segment based fingerprints for the first video clip with temporal segment based fingerprints generated for the second video clip;

identify matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;

collect all matching segments between the first video clip and the second video clip to form a set of matching segments;

select a subset of matching segments, wherein the subset of matching test segments produces a good linear fit to the matched video;

identify an overall video match score for the first video clip and the second video clip based on the selected matching segments in the subset of matching segments; and

determine whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

18. The apparatus of claim 17 wherein the processing unit further executes the computer usable program code to compare the overall video match score to a threshold score;

and identify the first video clip as a near-duplicate of the second video clip in response to the overall video match score exceeding the threshold score.

19. A computer implemented method for automatically detecting video piracy, the computer implemented method comprising:

comparing temporal segment based fingerprints for a first video clip with temporal segment based fingerprints generated for a second video clip;

identifying matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;

collecting all matching segments between the first video clip and the second video clip to form a set of matching segments;

selecting a subset of matching segments, wherein the subset of matching segments comprises matching segments associated with the first video clip that produces a good linear fit to matching segments associated with the second video clip;

identifying an overall video match score for the first video clip and the second video clip based on the selected matching segments in the subset of matching segments;

determining whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

20. The computer implemented method of claim 19 further comprising:

comparing the overall video match score to a threshold score; and

identifying the first video clip as a near-duplicate of the second video clip in response to the overall video match score exceeding the threshold score.