System and method for commercial detection in digital media environments

• US 10,200,748 B2
• Filed: 02/06/2017
• Issued: 02/05/2019
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Spotify AB (Spotify Technology SA)
• Original Assignee: Spotify AB (Spotify Technology SA)
• Inventors: Helferty, Nikhil, Hoffert, Eric
• Primary Examiner(s): Huynh, An Son P
• Application Number:
  US15/425,878
• Publication Number:
  US 20170150211A1
• Time in Patent Office: 729 Days
• Field of Search: 725 32, 725 34- 36
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2747 Remote storage of video programs received via the downstream path, e.g. from the server
• CPC Class Codes: H04N 21/4383 Accessing a communication channel
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4542 Blocking scenes or portions of the received content, e.g. censoring scenes
• CPC Class Codes: H04N 21/8113 comprising music, e.g. song in MP3 format
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

In accordance with an embodiment, described herein is a system and method for providing commercial (advertisement) detection in digital media environments. A media stream, for example, a video stream of television shows or movies, can be analyzed either in advance or in realtime, to determine commercial segments or blocks within the media stream, and to distinguish them from (non-commercial) content segments or blocks. In accordance with an embodiment, frame features, black spaces, cue points, or other segment or block information can be used to improve the accuracy and precision with which commercials are identified. Particular commercials can be identified, together with information as to the location of those commercials within the media stream. In accordance with an embodiment, the media streams can be used within an on-demand cloud-based media environment, which allows a user to select within a plurality of media channels, or to locate or skip particular commercials.

Claims

1. A system for providing commercial detection in digital media environments, comprising:

a computer including a processor and a media server that determines within a media stream associated with a media content, a plurality of blocks of commercial content and non-commercial media content, including, for each media stream of one or more media streams;

determining, by reference to a media analysis model that is trained to associate frame features with classifiers, that the media stream comprises a plurality of content blocks that are commercial blocks or non-commercial blocks based upon a multimodal analysis, including sampling the media stream together with its commercial blocks or non-commercial blocks therein as a plurality of samples, and

processing a cumulative frame data for a plurality of media frames within each sample to determine whether a respective content block is one of a commercial block or a noncommercial block, wherein the multimodal analysis weights frame features in each sample of the plurality of samples, in determining a likelihood that its respective content block includes commercial content or non-commercial media content;

wherein the media server provides media content to a client device, wherein the client device includes a user interface and displays media options associated with media content that can be selected and provided by the media server for playing at one or more of the client device, or a controlled device; and

wherein the determination, by reference to the media analysis model, of commercial blocks, and non-commercial content blocks, within each of the one or more media streams, is used by the media server in controlling playback of selected media content to the client device or the controlled device, including;

receiving, by the media server, a media control request comprising an indication of the selected media content and an identification of a destination device being one of the client device or the controlled device, and

controlling communication of a media stream including the selected media content, to the destination device, based on the determination of commercial blocks, and non-commercial content blocks, within that media stream and the identification of the destination device included in the media control request.

2. The system of claim 1, wherein the media analysis logic is provided at the media server, and determines in real time the plurality of blocks of commercial content and non-commercial media content, for use providing the media content to the client device or controlled device.

3. The system of claim 2, wherein the media server is provided within a cloud-based environment, which receives media content to be provided to the client device or controlled device as a plurality of channels having media streams, and receives interaction requests from the client device to switch between the channels.

4. The system of claim 3, wherein the controlled device is a television or other viewing device, and wherein the client device is used to communicate interaction requests to the media server to switch between or select particular channels to be played at the television or other viewing device;

wherein the media server provides a digital video recorder function that provides the media content or the channels to the client device or the controlled device as a plurality of thumbnails, including, upon receiving a request to skip to a media content represented by a thumbnail, allowing the player to slip to an end of a commercial block within that media content.

5. The system of claim 1, wherein each of a plurality of features found in the media stream are weighted by relative importance, and components summed, to determine an overall likelihood that a particular segment or block is commercial, or noncommercial, content.

6. The system of claim 1, wherein the media analysis logic is adapted to analyze characteristics of the media frame data, including black spaces, cue points, or other segment or block information.

7. The system of claim 1, wherein the media frames are cropped to provide a cropped frame area, prior to processing of media frame data.

8. The system of claim 1, wherein the destination device is the controlled device, and wherein playback of the media stream including the selected media content is performed at the controlled device.

9. The system of claim 1, wherein the one or more media streams are provided within an on-demand cloud-based media environment that enables a user to select within a plurality of media channels, including viewing or skipping particular commercials.

10. The system of claim 1, wherein during training of the media analysis model to associate frame features with classifiers, the media server writes a training analysis to a file, including a section of the file for each sample associated with a particular time period, which sections are then associated with block classifications.

11. The system of claim 1, wherein the user interface includes a slide device that enables moving forward and backward within a playback of a media content, and whereupon a viewing point being selected within a commercial segment associated with the media content, the media server instead causes playback to begin at a non-commercial segment associated with the media content.

12. The system of claim 1, wherein a plurality of different training media streams are provided for each of different regional requirements, or different types of media channels.

13. method for providing commercial detection in digital media environments, comprising:

determining, by a computer including a processor and a media server, within a media stream associated with a selected media content, a plurality of blocks of commercial content and non-commercial media content, including, for each media stream of one or more media streams;

determining, by reference to a media analysis model that is trained to associate frame features with classifiers, that the media stream comprises a plurality of content blocks that are commercial blocks or non-commercial blocks based upon a multimodal analysis, including sampling the media stream together with its commercial blocks or non-commercial blocks therein as a plurality of samples, and

processing a cumulative frame data for a plurality of media frames within each sample to determine whether a respective content block is one of a commercial block or a noncommercial block, wherein the multimodal analysis weights frame features in each sample of the plurality of samples, in determining a likelihood that its respective content block includes commercial content or non-commercial media content;

receiving requests for media content from a client device that includes a user interface and displays media options associated with media content that can be selected and provided by the media server for playing at one or more of the client device, or a controlled device; and

controlling playback, by the media server and the determination, by reference to the media analysis model, of commercial blocks, and non-commercial content blocks, within each of the one or more media streams, of selected media content to the client device or the controlled device, including;

receiving, by the media server, a media control request comprising an indication of the selected media content and an identification of a destination device being one of the client device or the controlled device, and

controlling communication of a media stream including the selected media content, to the destination device, based on the determination of commercial blocks, and non-commercial content blocks, within that media stream and the identification of the destination device included in the media control request.

14. The method of claim 13, wherein the media server determines in real time the plurality of blocks of commercial content and non-commercial media content, for use providing the media content to the client device or controlled device.

15. The method of claim 14, wherein the media server is provided within a cloud-based environment, which receives media content to be provided to the client device or controlled device as a plurality of channels having media streams, and receives interaction requests from the client device to switch between the channels.

16. The method of claim 15, wherein the controlled device is a television or other viewing device, and wherein the client device is used to communicate interaction requests to the media server to switch between or select particular channels to be played at the television or other viewing device;

wherein the media server provides a digital video recorder function that provides the media content or the channels to the client device or the controlled device as a plurality of thumbnails, including, upon receiving a request to skip to a media content represented by a thumbnail, allowing the player to slip to an end of a commercial block within that media content.

17. A non-transitory computer readable storage medium, including instructions stored thereon which, when executed by a computer, perform the method comprising:

determining, by a computer including a processor and a media server, within a media stream associated with a selected media content, a plurality of blocks of commercial content and non-commercial media content, including, for each media stream of one or more media streams;

determining, by reference to a media analysis model that is trained to associate frame features with classifiers, that the media stream comprises a plurality of content blocks that are commercial blocks or non-commercial blocks based upon a multimodal analysis, including sampling the media stream together with its commercial blocks or non-commercial blocks therein as a plurality of samples, and

processing a cumulative frame data for a plurality of media frames within each sample to determine whether a respective content block is one of a commercial block or a noncommercial block, wherein the multimodal analysis weights frame features in each sample of the plurality of samples, in determining a likelihood that its respective content block includes commercial content or non-commercial media content;

receiving requests for media content from a client device that includes a user interface and displays media options associated with media content that can be selected and provided by the media server for playing at one or more of the client device, or a controlled device; and

controlling playback, by the media server and the determination, by reference to the media analysis model, of commercial blocks, and non-commercial content blocks, within each of the one or more media streams, of selected media content to the client device or the controlled device, including;

receiving, by the media server, a media control request comprising an indication of the selected media content and an identification of a destination device being one of the client device or the controlled device, and

controlling communication of a media stream including the selected media content, to the destination device, based on the determination of commercial blocks, and non-commercial content blocks, within that media stream and the identification of the destination device included in the media control request.

18. The non-transitory computer readable storage medium of claim 17, wherein the media server determines in real time the plurality of blocks of commercial content and non-commercial media content, for use providing the media content to the client device or controlled device.

19. The non-transitory computer readable storage medium of claim 18, wherein the media server is provided within a cloud-based environment, which receives media content to be provided to the client device or controlled device as a plurality of channels having media streams, and receives interaction requests from the client device to switch between the channels.

20. The non-transitory computer readable storage medium of claim 19, wherein the controlled device is a television or other viewing device, and wherein the client device is used to communicate interaction requests to the media server to switch between or select particular channels to be played at the television or other viewing device;

wherein the media server provides a digital video recorder function that provides the media content or the channels to the client device or the controlled device as a plurality of thumbnails, including, upon receiving a request to skip to a media content represented by a thumbnail, allowing the player to slip to an end of a commercial block within that media content.