Video fingerprinting based on fourier transform of histogram

• US 10,390,083 B2
• Filed: 11/29/2017
• Issued: 08/20/2019
• Est. Priority Date: 02/22/2016
• Status: Active Grant

• Current Assignee: Roku, Inc.
• Original Assignee: Neilsen Company LLC
• Inventors: Chen, Juikun
• Primary Examiner(s): Mendoza, Junior O
• Application Number:
  US15/826,044
• Publication Number:
  US 20180084303A1
• Time in Patent Office: 629 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06V 10/431 Frequency domain transformation; Autocorrelation
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: H04N 21/222 Secondary servers, e.g. proxy server, cable television Head-end provisioning of proxy services in data packet switching networks H04L67/56
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/2407 Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/26258 for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list retrieval of multimedia data based on playlists G06F16/40
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/437 Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/440245 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A content device and method is disclosed to include a processing device to process streaming video content. A fingerprinter receives captured frames of the streaming video content and, for each frame of a plurality of the captured frames, generates a one-dimensional histogram function of pixel values and transforms the histogram function with a Fast Fourier Transform (FFT), to generate a plurality of complex values for the frame. The fingerprinter further, for each of the plurality of complex values, assigns a binary one (“1”) when a real part of the complex value is greater than zero (“0”) and assigns a binary zero (“0”) when the real part is less than or equal to zero, to generate a plurality of bits. The fingerprinter further concatenates a specific number of the bits to generate a fingerprint for the frame.

Claims

1. A content device comprising:

a non-transitory memory storing instructions; and

a processing device in communication with the non-transitory memory and configured to execute the instructions, the executed instructions causing the processing device to perform operations comprising;

receiving streaming video content from a content provider;

capturing a frame of the streaming video content; and

fingerprinting the captured frame of the streaming video content by;

generating a one-dimensional histogram function of pixel values based on pixel data for an entirety of the captured frame, the one-dimensional histogram function comprising an X axis representing specific pixel values and a Y axis representing a number of pixels at each respective specific pixel value;

transforming the histogram function with a Fast Fourier Transform (FFT) to generate a plurality of complex values for the captured frame;

for each of the plurality of complex values, generating a corresponding bit by assigning a binary one (“

1”

) when a real part of the respective complex value is greater than zero (“

0”

) and assigning a binary zero (“

0”

) when the real part of the respective complex value is less than or equal to zero; and

concatenating a number of the bits to generate a fingerprint for the captured frame.

2. The content device of claim 1, wherein each specific pixel value of the histogram function comprises a corresponding Y value of a YUV color space.

3. The content device of claim 1, wherein the operations further comprise, when fingerprinting the captured frame, indexing the plurality of complex values from zero (“

0”

) to 255.

4. The content device of claim 1, wherein the generated fingerprint is 128 bits long or 256 bits long.

5. The content device of claim 1, wherein the operations further comprise communicating the generated fingerprint to an automatic content recognition (ACR) server configured to store the generated fingerprint in relation to corresponding broadcast schedule metadata.

6. The content device of claim 1, wherein the streaming video comprises encoded video content.

7. The content device of claim 1, wherein the streaming video comprises raw video within a YUV 4:

2;

2 color space.

8. The content device of claim 1, wherein the operations further comprise converting the streaming video by scaling a resolution of the streaming video to a threshold resolution level.

9. The content device of claim 8, wherein the converting operation further comprises converting the streaming video to YUV 4:

2;

0 color space.

10. A method comprising:

receiving, at a processing device of a content device, streaming video content from a content provider;

capturing, by the processing device, a frame of the streaming video content; and

fingerprinting, by the processing device, the captured frame of the streaming video content by;

generating a one-dimensional histogram function of pixel values based on pixel data for an entirety of the captured frame, the one-dimensional histogram function comprises an X axis representing specific pixel values and a Y axis representing a number of pixels at each respective specific pixel value;

transforming the histogram function with a Fast Fourier Transform (FFT) to generate a plurality of complex values for the captured frame;

for each of the plurality of complex values, generating a corresponding bit by assigning a binary one (“

1”

) when a real part of the respective complex value is greater than zero (“

0”

) and assigning a binary zero (“

0”

) when the real part of the respective complex value is less than or equal to zero; and

concatenating a number of the bits to generate a fingerprint for the captured frame.

11. The method of claim 10, wherein each specific pixel value of the histogram function comprises a corresponding Y value of a YUV color space.

12. The method of claim 10, wherein when fingerprinting the captured frame, indexing the plurality of complex values from zero (“

0”

) to 255.

13. The method of claim 10, wherein the generated fingerprint is 128 bits long or 256 bits long.

14. The method of claim 10, further comprising communicating, by the processing device, the generated fingerprint to an automatic content recognition (ACR) server configured to store the generated fingerprint in relation to corresponding broadcast schedule metadata.

15. The method of claim 10, wherein the streaming video comprises encoded video content.

16. The method of claim 10, wherein the streaming video comprises raw video within a YUV 4:

2;

2 color space.

17. The method of claim 10, wherein fingerprinting, by the processing device, further comprises converting the streaming video by scaling a resolution of the streaming video to a threshold resolution level.

18. The method of claim 17, wherein converting further comprises converting the streaming video to YUV 4:

2;

0 color space.