Endpoint based video fingerprinting

• US 9,135,674 B1
• Filed: 11/27/2013
• Issued: 09/15/2015
• Est. Priority Date: 06/19/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yagnik, Jay, Ioffe, Sergey, Rowley, Henry
• Primary Examiner(s): Pham, Luu
• Assistant Examiner(s): Kabir, Jahangir
• Application Number:
  US14/092,515
• Time in Patent Office: 657 Days
• Field of Search: 375/240.16, 375/240.01, 375/240.12, 375/240.25, 375/240.26
• US Class Current: 1/1
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/48 Matching video sequences

Abstract

A method and system generates and compares fingerprints for videos in a video library. The video fingerprints provide a compact representation of the temporal locations of discontinuities in the video that can be used to quickly and efficiently identify video content. Discontinuities can be, for example, shot boundaries in the video frame sequence or silent points in the audio stream. Because the fingerprints are based on structural discontinuity characteristics rather than exact bit sequences, visual content of videos can be effectively compared even when there are small differences between the videos in compression factors, source resolutions, start and stop times, frame rates, and so on. Comparison of video fingerprints can be used, for example, to search for and remove copyright protected videos from a video library. Furthermore, duplicate videos can be detected and discarded in order to preserve storage space.

Claims

1. A method for fingerprinting an input video having a sequence of video frames, the method performed by a computer system, the method comprising:

identifying, by the computer system, a plurality of discontinuities in features of audio-visual content of the input video, wherein the sequence of video frames includes frames associated with one or more associated discontinuities and frames not associated with discontinuities;

generating, by the computer system, an encoded data representation representing the sequence of video frames, wherein the frames associated with the one or more associated discontinuities are represented by first values in the encoded data representation, and wherein frames not associated with the discontinuities are represented by second values in the encoded data representation, the second values different than the first values, and wherein the first values and the second values are arranged in a sequence in the encoded data representation corresponding to the sequence of video frames;

segmenting, by the computer system, the encoded data representation;

generating, by the computer system, a plurality of fingerprints, each fingerprint comprising an encoded data segment corresponding to a segment of the input video;

indexing the plurality of fingerprints by;

generating a signature for a fingerprint of the plurality of fingerprints according to a first hash process; and

dividing the signature into a plurality of signature blocks, each signature block stored in a hash table according to a second hash process, wherein the first hash process and the second hash process are different; and

storing the hash table to a computer readable storage medium.

2. The method of claim 1, wherein a first discontinuity of the plurality of discontinuities comprises a shot boundary in the sequence of video frames.

3. The method of claim 1, wherein identifying a plurality of discontinuities comprises analyzing depicted motion between the frames and determining an endpoint frame where the motion is at a peak.

4. The method of claim 1, wherein identifying a plurality of discontinuities comprises analyzing volume of an audio stream associated with the sequence of video frames and determining an endpoint frame where the volume is below a predetermined threshold volume.

5. The method of claim 1, further comprising normalizing the sequence of video frames by reducing at least one of frame size and color information.

6. The method of claim 1, wherein the signature is uniquely associated with the fingerprint and comprises fewer bits than the first fingerprint.

7. The method of claim 6, wherein the first hash process comprises:

re-arranging bits of the fingerprint according to a predetermined number of pre-defined permutations; and

storing a location of the first bit value of “

1”

in the re-arranged bits following each permutation.

8. The method of claim 1, further comprising:

generating a fingerprint sequence comprising an ordered set of the plurality of fingerprints for the input video;

searching a reference database for reference fingerprint sequences that match the fingerprint sequence for the input video; and

determining at least one reference video corresponding to at least one matching reference fingerprint sequence.

9. The method of claim 8, further comprising removing a reference video corresponding to the at least one matching reference fingerprint sequence from a reference library storing the input video and the reference video.

10. A non-transitory computer-readable storage medium storing instructions for fingerprinting an input video having a sequence of video frames, the instructions when executed causing a processor to perform steps comprising:

identifying a plurality of discontinuities in features of audio-visual content of the input video, wherein the sequence of video frames includes frames associated with one or more associated discontinuities and frames not associated with discontinuities;

generating an encoded data representation representing the sequence of video frames, wherein the frames associated with the one or more associated discontinuities are represented by first values in the encoded data representation, and wherein frames not associated with the discontinuities are represented by second values in the encoded data representation, the second values different than the first values, and wherein the first values and the second values are arranged in a sequence in the encoded data representation corresponding to the sequence of video frames;

segmenting, by the computer system, the encoded data representation;

generating a plurality of fingerprints, each fingerprint comprising an encoded data segment corresponding to a segment of the input video;

indexing the plurality of fingerprints by;

generating a signature for a fingerprint of the plurality of fingerprints according to a first hash process; and

dividing the signature into a plurality of signature blocks, each signature block stored in a hash table according to a second hash process, wherein the first hash process and the second hash process are different; and

storing the hash table to a computer readable storage medium.

11. The non-transitory computer-readable storage medium of claim 10, wherein a first discontinuity of the plurality of discontinuities comprises a shot boundary in the sequence of video frames.

12. The non-transitory computer-readable storage medium of claim 10, wherein identifying a plurality of discontinuities comprises analyzing depicted motion between the frames and determining an endpoint frame where the motion is at a peak.

13. The non-transitory computer-readable storage medium of claim 10, wherein identifying a plurality of discontinuities comprises analyzing volume of an audio stream associated with the sequence of video frames and determining an endpoint frame where the volume is below a predetermined threshold volume.

14. The non-transitory computer-readable storage medium of claim 10, the instructions when executed by the processor further causing the processor to normalize the sequence of video frames by reducing at least one of frame size and color information.

15. The non-transitory computer-readable storage medium of claim 10, wherein the signature is uniquely associated with the fingerprint and comprises fewer bits than the fingerprint.

16. The non-transitory computer-readable storage medium of claim 15, wherein the first hash process comprises:

re-arranging bits of the fingerprint according to a predetermined number of pre-defined permutations; and

storing a location of the first bit value of “

1”

in the re-arranged bits following each permutation.

17. The non-transitory computer-readable storage medium of claim 10, the instructions when executed further causing the processor to perform steps of:

generating a fingerprint sequence comprising an ordered set of the plurality of fingerprints for the input video;

searching a reference database for reference fingerprint sequences that match the fingerprint sequence for the input video; and

determining at least one reference video corresponding to at least one matching reference fingerprint sequence.

18. The transitory computer-readable storage medium of claim 17, the instructions when executed further causing the processor to perform a step of:

removing a reference video corresponding to the at least one matching reference fingerprint sequence from a reference library storing the input video and the reference video.

19. A system for fingerprinting an input video having a sequence of video frames, the system comprising:

a non-transitory computer-readable medium storing executable instructions; and

a processor configured to execute the instructions to perform steps comprising;

identifying a plurality of discontinuities in features of audio-visual content of the input video, wherein the sequence of video frames includes frames associated with one or more associated discontinuities and frames not associated with discontinuities;

generating an encoded data representation representing the sequence of video frames, wherein the frames associated with the one or more associated discontinuities are represented by first values in the encoded data representation, and wherein frames not associated with the discontinuities are represented by second values in the encoded data representation, the second values different than the first values, and wherein the first values and the second values are arranged in a sequence in the encoded data representation corresponding to the sequence of video frames;

segmenting, by the computer system, the encoded data representation;

generating a plurality of fingerprints, each fingerprint comprising an encoded data segment corresponding to a segment of the input video;

indexing the plurality of fingerprints by;

generating a signature for a fingerprint of the plurality of fingerprints according to a first hash process; and

dividing the signature into a plurality of signature blocks, each signature block stored in a hash table according to a second hash process, wherein the first hash process and the second hash process are different; and

storing the hash table to a computer readable storage medium.