TV content segmentation, categorization and identification and time-aligned applications

• US 9,510,044 B1
• Filed: 12/15/2011
• Issued: 11/29/2016
• Est. Priority Date: 06/18/2008
• Status: Active Grant

• Current Assignee: Roku, Inc.
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Pereira, Jose Pio, Kulkarni, Sunil Suresh, Bolgarov, Oleksiy, Ramanathan, Prashant, Merchant, Shashank, Stojancic, Mihailo
• Primary Examiner(s): HUYNH, AN SON PHI
• Application Number:
  US13/327,359
• Time in Patent Office: 1,811 Days
• Field of Search: 705/14.69, 705/51, 713/176, 725/19
• US Class Current: 1/1
• CPC Class Codes: G06F 21/1011 to devices
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: H04H 20/106 Receiver-side switching
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

Content segmentation, categorization and identification methods are described. Content tracking approaches are illustrated that are suitable for large scale deployment. Time-aligned applications such as multi-language selection, customized advertisements, second screen services and content monitoring applications can be economically deployed at large scales. A client performs fingerprinting, scene change detection, audio turn detection, and logo detection on incoming video and gathers database search results, logos and text to identify and segment video streams into content, promos, and commercials. A learning engine is configured to learn rules for optimal identification and segmentation at each client for each channel and program. Content sensed at the client site is tracked with reduced computation and applications are executed with timing precision. A user interface for time-aligned publishing of content and subsequent usage and interaction on one or more displays is also described.

Claims

1. A method for time aligned identification of video content playing on an electronic client device, the method comprising:

receiving by the electronic client device, a video stream for presentation by the electronic client device, the video stream containing a first video content followed by a second video content;

generating by the electronic client device, a first set of signatures for the first video content;

identifying by the electronic client device, the first video content based on an analysis of the first set of signatures compared to reference signatures stored in a database;

detecting by the electronic client device, a scale ratio mapping between video frames of the identified first video content playing on the electronic client device and reference frames stored in the database by calculating a scale ratio along an x axis (Sx) and a difference scale ratio along a y axis (Sy) for a query signature in the first set of signatures and a reliable matching reference signature in the database, wherein the calculated scale ratios Sx and Sy indicate that the video frames of the identified first video content and the reference frames are not in alignment;

responsive to calculating the scale ratios, using the detected scale ratio mapping to generate subsequent signatures for subsequent incoming video frames; and

monitoring the subsequent incoming video frames for the second video content by the electronic client device to determine that the video stream has made a scene change from the first video content to the second video content.

2. The method of claim 1 further comprising:

replacing the second video content beginning at a time the scene change is verified with a third video that is different than the second video content.

3. The method of claim 2, wherein the second video content is an expected advertisement within an expected content or program stream.

4. The method of claim 1 further comprising:

displaying synchronous information on a selected device.

5. The method of claim 1 further comprising:

providing a user selection of display devices for synchronous display of data in place of the identified second video content;

providing a user choice of content for synchronous stream content in place of the identified second video content; and

providing a user choice of scrolling in time to select content choices including synchronous content and the identified second video content on the client device.

6. The method of claim 1, wherein identifying a time alignment comprises:

sequence matching and correlation between reference and query fingerprints.

7. The method of claim 1, wherein identifying a change in multimedia content comprises:

detecting a scene change.

8. The method of claim 1, wherein verifying a sample of multimedia content comprises:

matching the sample of multimedia content with a first frame or an opening sequence of an expected multimedia content.

9. The method of claim 1, wherein audio content is corrected for changes in pitch and changes in frequency response.

10. The method of claim 1, wherein Sx is equal to (QA(x)−

QB(x))/(RA(x)−

RB(x)) and wherein QA(x)−

QB(x) is a difference in query x coordinates of signature pairs A and B and RA(x)−

RB(x) is a difference in reference x coordinates of the signature pairs A and B.

11. A method of providing time-aligned services on broadcast non-recorded content, the method comprising:

analyzing by an electronic client device, incoming video of a broadcast live event based on comparisons with opening sequences of programs stored on the electronic client device to determine program details associated with the broadcast live event;

generating by the electronic client device, fingerprints of selected frames of the incoming video;

searching a reference database using the generated fingerprints to identify a likely match for content of the incoming video with content of an original video stored in the reference database; and

correcting by the electronic client device, frame alignment in the incoming video based on a scale ratio mapping by utilizing a scale ratio for an x axis (Sx) and a different scale ratio for a y axis (Sy) between the selected frames of the incoming video and corresponding frames of a matched original video selected from the reference database and for generating subsequent fingerprints based on the scale ratio mapping to enable a time aligned service to overlay on aligned video frames for display on a display of the electronic client device information personalized for the electronic client device.

12. The method of claim 11 further comprises:

searching on the electronic client device a subset reference database that is a subset of a remote database using the generated subsequent fingerprints in queries to minimize computational effort and improve accuracy in identification of the incoming video; and

searching on the remote database in response to a match not being found in the subset reference database.

13. The method of claim 11 further comprises:

using search reports and content analytics to determine a classification of content at time nodes and a similarity between the time nodes generated for the incoming video;

generating edge weights between the time nodes, wherein the edge weights represent the similarity between two edge connected time nodes; and

performing graph based segmentation to segment video based on the classification and similarity between the time nodes.

14. The method of claim 11 further comprises:

determining a geometric correlation of video signatures and associated data comprising geometric coordinates and scale information between the incoming video and the matched original video to represent transformation of the matched original video that approximate the incoming video.

15. The method of claim 11, wherein video segments of the incoming video represent chapters of a program, advertisements, and promotions.

16. The method of claim 11 further comprises:

generating the fingerprints of incoming video including a determined region'"'"'s location and scale information;

determining a reliable match of content from incoming video with the reference database; and

generating frame alignment information that includes pixel offsets along the x axis and along the y axis, the scale ratio for the x axis (Sx), and the scale ratio for the y axis (Sy) to represent transformation of the matched original video that approximate the incoming video.

17. The method of claim 16, wherein generating frame alignment information comprises:

generating pixel locations for each fingerprint generated for the incoming video;

calculating the scale ratios Sx and Sy based on pixel differences between a query signature pair and a reference signature pair along each axis;

summing up the scale ratios calculated on each axis and averaging while excluding outliers; and

calculating a pixel offset on each axis for matching pairs of signatures based on the scale ratio on that axis and the determined region'"'"'s location for the query signature pair and the reference signature pair.

18. A method to identify and track video content by an electronic user device, the method comprising:

detecting by the electronic user device, a scale ratio mapping between incoming video frames received by the electronic user device and reference video frames stored in a reference database by calculating a scale ratio along an x axis (Sx) and a different scale ratio along a y axis (Sy) for a query signature in an incoming video frame and a reliable matching reference signature in a corresponding reference video frame stored in the reference database, wherein the calculated scale ratio Sx and Sy indicate that the incoming video frame and the corresponding reference video frame are not in alignment;

responsive to calculating the scale ratio, using the detected scale ratio mapping to generate subsequent signatures for subsequent incoming video frames; and

processing by the electronic user device, the subsequent incoming video frames to identify and track video content based on the generated subsequent signatures,

receiving at the electronic user device, third party alternative content selected for display by the electronic user device to replace content at a specified time section in the identified and tracked video content, wherein the third party alternative content is selected from a menu of selectable options displayed on a second display screen associated with a user of the electronic user device.

19. The method of claim 18, wherein the third party alternative content includes a selection of content related information.

20. The method of claim 18, wherein the third party alternative content includes a selection of personal services.

21. The method of claim 18 further comprising:

correcting the incoming video frames utilizing the calculated scale ratios Sx and Sy to generate a corrected incoming video stream.

22. A computer readable non-transitory medium encoded with computer readable program data and code, the computer readable program data and code when executed to perform:

detecting by an electronic user device, a scale ratio mapping between incoming video frames received by the electronic user device and reference video frames stored in a reference database by calculating a scale ratio along an x axis (Sx) and a different scale ratio along a y axis (Sy) for a query signature in an incoming video frame and a reliable matching reference signature in a corresponding reference video frame stored in the reference database, wherein the calculated scale ratios Sx and Sy indicate that the incoming video frame and the corresponding reference video frame are not in alignment;

responsive to calculating the scale ratio, using the detected scale ratio mapping to generate subsequent signatures for subsequent incoming video frames;

process by the electronic user device, the subsequent incoming video frames on the user device to identify and track video content based on the generated subsequent signatures,

receiving at the electronic user device, third party alternative content selected for display by the electronic user device to replace content at a specified time section in the identified and tracked video content, wherein the third party alternative content is selected from a menu of selectable options displayed on a second display screen associated with a user of the electronic user device.