Identifying video content via color-based fingerprint matching

• US 10,503,956 B2
• Filed: 10/06/2016
• Issued: 12/10/2019
• Est. Priority Date: 10/23/2013
• Status: Active Grant

• Current Assignee: Roku, Inc.
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Harron, Wilson, Cremer, Markus K.
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): Tran, Mai H
• Application Number:
  US15/286,787
• Publication Number:
  US 20170091524A1
• Time in Patent Office: 1,160 Days
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 40/1365 Matching; Classification
• CPC Class Codes: G06V 40/50 Maintenance of biometric data or enrolment thereof

Abstract

Systems and methods of identifying unknown video content are described. An example method may include receiving a first fingerprint and a second fingerprint. The first fingerprint may be a color-based fingerprint derived from colors in a portion of the unknown video content, and the second fingerprint may be at least partially based on a feature other than the colors of the unknown video content. A reference database of reference fingerprints may then be queried using one of the first fingerprint or the second fingerprint to obtain a candidate group of fingerprints. The candiate group of fingerprints may then be pried using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint. The unknown video content may then be identified using the at least one query fingerprint. For example, the second fingerprint may be a luminance-based fingerprint derived from luminance in the unknown video content.

Claims

1. A method comprising:

processing unknown video content to obtain a first fingerprint and a second fingerprint, the first fingerprint being a color-based fingerprint that includes a color-based value determined for each color of a plurality of colors in a portion of the unknown video content, the second fingerprint being a patch-based fingerprint that includes a patch-based value determined for each patch of a plurality of patches in the portion of the unknown video content, wherein each patch-based value is based on at least one of luminance, optical flow, or a Haar-like feature;

initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and

responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint.

2. The method of claim 1, further comprising receiving the unknown video content at a media playback device including a hardware processor, the query of the at least one reference database and the query of the candidate group of fingerprints being performed by the hardware processor.

3. The method of claim 2, wherein the at least one reference database is stored in memory of the media playback device.

4. The method of claim 1, wherein initiating the query of the at least one reference database of reference fingerprints comprises sending the first fingerprint and the second fingerprint to a remote server system to execute the query, the at least one reference database being part of the remote server system.

5. The method of claim 1, wherein the at least one reference database comprises:

a color-based fingerprint database including reference fingerprints that are color-based; and

a patch-based fingerprint database including reference fingerprints that are patch-based.

6. The method of claim 1, wherein the at least one reference database is queried with the first fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the second fingerprint to identify the at least one query fingerprint.

7. The method of claim 1, wherein the at least one reference database is queried with the second fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the first fingerprint to identify the at least one query fingerprint.

8. The method of claim 1, wherein the portion of the unknown video content corresponds to at least one frame of the unknown video content.

9. The method of claim 1, wherein initiating the query of the at least one reference database using one of the first fingerprint or the second fingerprint uses a fingerprint defined by a lower number of bits than that used to query the candidate group.

10. The method of claim 1, wherein initiating the query of the at least one reference database comprises querying the at least one reference database of reference fingerprints using the color-based fingerprint.

11. The method of claim 10, wherein querying the at least one reference database using the color-based fingerprint comprises determining that a similarity between the color-based fingerprint and the reference fingerprints satisfies a predetermined threshold associated with vector distance measurement.

12. The method of claim 1, wherein the patches are defined by regions of at least one frame of the unknown video content.

18. The method of claim 8, wherein the plurality of colors are the most dominant colors within the at least one frame.

19. The method of claim 8, wherein the color-based fingerprint includes 8 color-based values determined for the 8 most dominant colors within the at least one frame.

20. The method of claim 8, wherein the patch-based fingerprint includes 32 patch-based values determined for 32 patches in the at least one frame.

13. A computer comprising:

one or more hardware processors; and

memory including instructions that, when executed by the one or more processors, cause the computer to perform operations comprising;

processing unknown video content to obtain a first fingerprint and a second fingerprint, the first fingerprint being a color-based fingerprint that includes a color-based value determined for each color of a plurality of colors in a portion of the unknown video content, the second fingerprint being a patch-based fingerprint that includes a patch-based value determined for each patch of a plurality of patches in the portion of the unknown video content, wherein each patch-based value is based on at least one of luminance, optical flow, or a Haar-like feature;

initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and

responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint.

14. The computer of claim 13, wherein the computer is a media playback device including the at least one reference database, the media playback device configured to receive the unknown video content and query the at least one reference database and the candidate group of fingerprints to identify the at least one query fingerprint.

15. The computer of claim 13, wherein the at least one reference database is queried with the second fingerprint to obtain the candidate group of fingerprints, and the candidate group of fingerprints is then queried with the first fingerprint to identify the at least one query fingerprint.

16. A non-transitory computer-readable storage medium embodying instructions which, when executed by a machine, cause the machine to perform operations comprising:

processing unknown video content to obtain a first fingerprint and a second fingerprint, the first fingerprint being a color-based fingerprint that includes a color-based value determined for each color of a plurality of colors in a portion of the unknown video content, the second fingerprint being a patch-based fingerprint that includes a patch-based value determined for each patch of a plurality of patches in the portion of the unknown video content, wherein each patch-based value is based on at least one of luminance, optical flow, or a Haar-like feature;

initiating a query of at least one reference database of reference fingerprints using one of the first fingerprint to obtain a candidate group of fingerprints or the second fingerprint to obtain the candidate group of fingerprints, the candidate group of fingerprints being a subset of the reference fingerprints queried; and

responsive to the query, initiating a query of the candidate group of fingerprints using the other of the first fingerprint and the second fingerprint to identify at least one query fingerprint, wherein the unknown video content is identified using the at least one query fingerprint.

17. The non-transitory computer-readable storage medium of claim 16, wherein the machine is a media playback device, further comprising instructions to receive the unknown video content at the media playback device including a hardware processor, and query the at least one reference database and the candidate group of fingerprints using the hardware processor.