TV Content Segmentation, Categorization and Identification and Time-Aligned Applications

• US 20170201793A1
• Filed: 10/19/2016
• Published: 07/13/2017
• Est. Priority Date: 06/18/2008
• Status: Abandoned Application

• Current Assignee: Roku, Inc.
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Kulkarni, Sunil Suresh, Ramanathan, Prashant, Merchant, Shashank, Stojancic, Mihailo M., Pereira, Jose Pio, Bolgarov, Oleksiy
• Application Number:
  US15/297,658
• Publication Number:
  US 20170201793A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 21/1011 to devices
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: H04H 20/106 Receiver-side switching
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A content segmentation, categorization and identification method on consumer devices (clients) is described. Methods for content tracking are illustrated that are suitable for large scale deployment and applications such as broadcast monitoring, novel content publishing and interaction. Time-aligned (synchronous) applications such as multi-language selection, customized advertisements, second screen services and content monitoring applications can be economically deployed at large scales. The client performs fingerprinting, scene change detection, audio turn detection, and logo detection on incoming video and gathers database search results, logos and text to identify and segment video streams into content, promos, and commercials. A learning engine is configured to learn rules for optimal identification and segmentation at each client for each channel and program. Content sensed at the client site is tracked with reduced computation and applications are executed with timing precision. A method and user interface for time-aligned publishing of content and subsequent usage and interaction on one or more displays is described.

Claims

1-20. -20. (canceled)

21. A method to provide content publishing services that are personalized for different user devices, the method comprising:

detecting a logo in an incoming video;

identifying the incoming video based on the identified logo;

collecting a first set of information regarding the identified incoming video that is personalized for use by a first user device;

collecting a second set of information regarding the identified incoming video that is personalized for use by a second user device; and

adjusting content publishing services for the first user device and for the second user device according to the personalized information received by each user device.

22. The method of claim 21 further comprising:

generating video signatures on initial frames of the incoming video; and

searching for the generated video signatures in opening sequences of programs stored in a reference database to further identify the incoming video.

23. The method of claim 21, wherein the incoming video is a live broadcast.

24. The method of claim 21 further comprising:

generating first time alignment information by calculating locations in position and in time for pixel offsets along an x-axis and along a y-axis between signatures generated for incoming video frames received on the first user device and corresponding matching signatures generated for an opening sequence of frames from a program found in a reference database, wherein the first time alignment information is used on the first user device to synchronize publishing content selected from the reference database with the incoming video frames.

25. The method of claim 21 further comprising:

generating second time alignment information by calculating locations in position and in time for pixel offsets along an x-axis and along a y-axis between signatures generated for incoming video frames received on the second user device and corresponding matching signatures generated for an opening sequence of frames from a program found in a reference database, wherein the second time alignment information is used on the second user device to synchronize publishing content selected from the reference database with the incoming video frames.

26. The method of claim 21, wherein the content publishing services for the first user device comprise a first language customization.

27. The method of claim 26, wherein first time alignment information is generated in the first user device to synchronize the first language customization with the incoming video.

28. The method of claim 21, wherein the content publishing services for the second user device further comprise a second language customization.

29. The method of claim 28, wherein second time alignment information is generated in the second user device to synchronize the second language customization with the incoming video.

30. A method to provide time aligned language presentations personalized for user devices, the method comprising:

identifying audio content and video content based on a logo detected in broadcast content of a broadcast program;

determining an audio time alignment of the audio content and a video time alignment of the video content on a first user device relative to the broadcast content;

synchronizing the audio content with the video content according to the audio time alignment and the video time alignment on the first user device;

substituting the synchronized audio content with a first selected language on the first user device; and

overlaying text and the logo with customized text and a customized logo in the first selected language in the synchronized video content on the first user device.

31. The method of claim 30 further comprising:

determining an audio time alignment of the audio content and a video time alignment of the video content on a second user device relative to the broadcast content;

synchronizing the audio content with the video content according to the audio time alignment and the video time alignment on the second user device;

substituting the synchronized audio content with a second selected language on the second user device; and

overlaying text and the logo with customized text and a customized logo in the second selected language in the synchronized video content on the second user device.

32. The method of claim 30 further comprising:

generating audio and video signatures on initial frames of the incoming video; and

searching for the generated audio and video signatures in opening sequences of programs stored in a reference database to further identify the audio content and the video content.

33. The method of claim 30 further comprising:

using a program guide to further identify the audio content and the video content.

34. The method of claim 30 further comprising:

generating first time alignment information by calculating locations in position and in time for pixel offsets along an x-axis and along a y-axis between signatures generated for frames of the broadcast content received on the first user device and corresponding matching signatures generated for an opening sequence of frames from a program found in a reference database, wherein the first time alignment information is used on the first user device to synchronize time aligned language presentations selected from the reference database with the incoming video frames.

35. The method of claim 30 further comprising:

identifying a logo change as an indicator of and a time of a scene change;

substituting the synchronized audio content with a selected language at the time of the scene change; and

overlaying text and the logo with customized text and a customized logo in the synchronized video content.

36. The method of claim 30 further comprising:

monitoring content defined by the detected logo and identified audio content and video content on a plurality of user devices; and

sending the monitored content to a server to determine audience measurements of contents watched on the plurality of user devices.

37. The method of claim 36, wherein the monitored content includes broadcast programs.

38. The method of claim 36, wherein the monitored content includes advertisements specified as having a content logo.

39. A computer readable non-transitory medium encoded with computer readable program data and code, the computer readable program data and code when executed perform a method to provide content publishing services that are personalized for different user devices, the method comprising:

detecting a logo in an incoming video;

identifying the incoming video based on the identified logo;

collecting a first set of information regarding the identified incoming video that is personalized for use by a first user device;

collecting a second set of information regarding the identified incoming video that is personalized for use by a second user device; and

adjusting content publishing services for the first user device and for the second user device according to the personalized information received by each user device.

40. The computer readable non-transitory medium method of claim 39 further comprising:

generating video signatures on initial frames of the incoming video; and

searching for the generated video signatures in opening sequences of programs stored in a reference database to further identify the incoming video.