System and method for commercial detection in digital media environments

• US 9,565,456 B2
• Filed: 09/29/2014
• Issued: 02/07/2017
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Spotify AB (Spotify Technology SA)
• Original Assignee: Spotify AB (Spotify Technology SA)
• Inventors: Helferty, Nikhil, Hoffert, Eric
• Primary Examiner(s): HUYNH, AN SON PHI
• Application Number:
  US14/500,561
• Publication Number:
  US 20160094863A1
• Time in Patent Office: 862 Days
• Field of Search: 725/32, 725 34- 36
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2747 Remote storage of video programs received via the downstream path, e.g. from the server
• CPC Class Codes: H04N 21/4383 Accessing a communication channel
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4542 Blocking scenes or portions of the received content, e.g. censoring scenes
• CPC Class Codes: H04N 21/8113 comprising music, e.g. song in MP3 format
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

In accordance with an embodiment, described herein is a system and method for providing commercial (advertisement) detection in digital media environments. A media stream, for example, a video stream of television shows or movies, can be analyzed either in advance or in realtime, to determine commercial segments or blocks within the media stream, and to distinguish them from (non-commercial) content segments or blocks. In accordance with an embodiment, frame features, black spaces, cue points, or other segment or block information can be used to improve the accuracy and precision with which commercials are identified. Particular commercials can be identified, together with information as to the location of those commercials within the media stream. In accordance with an embodiment, the media streams can be used within an on-demand cloud-based media environment, which allows a user to select within a plurality of media channels, or to locate or skip particular commercials.

Claims

1. A system for providing commercial detection in digital media environments, comprising:

a computer including a processor and a media server, and further including

a media analysis logic, which determines within a media stream associated with a media content, a plurality of blocks of commercial content and non-commercial media content, based upon a multimodal analysis, within the media stream, of a plurality of samples selected from the media stream, wherein each of the plurality of samples comprises media frames,

wherein the multimodal analysis weights a plurality of features in each of the samples, and

wherein the weights of each of the plurality of features in each of the samples are used in a linear classifier in determining the plurality of blocks of commercial content and non-commercial media content,

a media delivery module and server-side media content buffer to be populated with media content, to be returned as media streams to a requesting device or to a controlled device, and

a buffering logic, that buffers a current media content and one or more additional media content;

wherein the media server provides media content to a client device which includes a user interface and a client-side media content buffer, and which displays an array of media options associated with media content selectable at the client device and provided by the media server for playing at the client device, or at the controlled device, or at the client device and the controlled device, wherein each media option can be associated with a portion of the buffered media content;

wherein the media server controls the playing at the client device, or at the controlled device, or at the client device and the controlled device, based upon input indicating a user selection of a media option received at the client device;

wherein information describing commercial blocks, and non-commercial content blocks, within the media stream, is used in providing the media content to the client device or the controlled device;

wherein the controlling the playing at the client device, or at the controlled device, or at the client device and the controlled device comprises

receiving indication of a selected media content, the selected media content associated with the input indicating a user selection of a media option, and

in response to receiving the indication of selected media content, controlling the playing of the selected media content based on the multimodal analysis and a determination of the plurality of commercial content blocks and non-commercial content blocks;

wherein receiving indication of a selected media content further comprises receiving information identifying a destination device, the destination device being one of the client device and the controlled device, the information identifying the destination device indicating the destination device to which the media server should provide the selected media content;

wherein the media server further comprises a context database, the context database storing data associated with the presentation of media content at a client device, the data associated with the presentation of media content comprising a current position within the displayed media content;

wherein the information identifying the destination device indicates a change in destination device from a current destination device to a new destination device; and

wherein the media server transmits the data associated with the presentation of media content to the new destination device.

2. The system of claim 1, wherein the media analysis logic is provided at the media server, and determines in real time the plurality of blocks of commercial content and non-commercial media content, for use in providing the media content to the client device or controlled device.

3. The system of claim 2, wherein the media server is provided within a cloud-based environment, which receives media content to be provided to the client device or controlled device as a plurality of channels having media streams, and receives interaction requests from the client device to switch between the channels.

4. The system of claim 3, wherein the controlled device is a television or other viewing device, and wherein the client device is used to communicate interaction requests to the media server to switch between or select particular channels to be played at the television or other viewing device.

5. The system of claim 4, wherein the media server provides a digital video recorder function that provides the media content or the channels to the client device or the controlled device as a plurality of thumbnails, including, upon receiving a request to skip to a media content represented by a thumbnail, allowing the player to slip to an end of a commercial block within that media content.

6. The system of claim 1, wherein each of the displayed media options associated with media content displays a media representation, the displayed media representations being associated with one of the media options;

each media representation comprising a portion of non-commercial media content.

7. The system of claim 1, wherein the media analysis logic performs the multimodal analysis, within the media stream, of a plurality of samples selected from the media stream, wherein each of the plurality of samples comprises media frames, wherein the analysis comprises:

comparing the media frames with a training set, the training set comprising a plurality of classified samples; and

identifying, based at least upon the comparison of the media frames with the training set, commercial media frames indicating the plurality of blocks of commercial content and non-commercial media frames indicating the plurality of blocks of non-commercial content.

8. The system of claim 7, further comprising:

a database, wherein the training set comprising a plurality of classified samples is stored, wherein the samples are classified as commercial or non-commercial samples;

wherein the media analysis logic classifies additional samples as commercial or non-commercial samples based upon the identification of the commercial media frames and the non-commercial media frames;

wherein the media analysis logic stores the additional samples at the database; and

wherein the additional samples are added to the training data.

9. The system of claim 1, wherein the media analysis logic selects the plurality of samples from media stream at a sampling frequency, wherein the sampling frequency is variable and is adjusted based on the determination of the plurality of blocks of commercial content and non-commercial media content.

10. The system of claim 1, wherein the media analysis logic creates a media analysis model, the medial analysis model associating frame features with classifiers, and

wherein the media analysis logic uses to the media analysis model in determining the plurality of blocks of commercial content and non-commercial media content.

11. A method for providing commercial detection in digital media environments, comprising:

providing, at a computer including a processor and a media server,

a media analysis logic, which determines within a media stream associated with a media content, a plurality of blocks of commercial content and non-commercial media content, based upon a multimodal analysis, within the media stream, of a plurality of samples selected from the media stream, wherein each of the plurality of samples comprises media frames,

wherein the multimodal analysis weights a plurality of features in each of the samples, and

wherein the weights of each of the plurality of features in each of the samples are used in a linear classifier in determining the plurality of blocks of commercial content and non-commercial media content,

a media delivery module and server-side media content buffer to be populated with media content, to be returned as media streams to a requesting device or to a controlled device, and

a buffering logic, that buffers a current media content and one or more additional media content;

providing media content to a client device which includes a user interface and a client-side media content buffer, and which displays an array of media options associated with media content selectable at the client device and provided by the media server for playing at the client device, or at the controlled device, or at the client device and the controlled device, wherein each media option can be associated with a portion of the buffered media content;

controlling the playing at the client device, or at the controlled device, or at the client device and the controlled device, based upon input indicating a user selection of a media option received at the client device, including using information describing commercial blocks, and non-commercial content blocks, within the media stream, in providing the media content to the client device or the controlled device;

wherein the controlling the playing comprises

receiving indication of a selected media content, the selected media content associated with the input indicating a user selection of a media option, and

in response to receiving the indication of selected media content, controlling the playing of the selected media content based on the multimodal analysis and a determination of the plurality of commercial content blocks and non-commercial content blocks;

wherein receiving indication of a selected media content further comprises receiving information identifying a destination device, the destination device being one of the client device and the controlled device, the information identifying the destination device indicating the destination device to which the media server should provide the selected media content;

wherein the media server further comprises a context database, the context database storing data associated with the presentation of media content at a client device, the data associated with the presentation of media content comprising a current position within the displayed media content;

wherein the information identifying the destination device indicates a change in destination device from a current destination device to a new destination device; and

wherein the media server transmits the data associated with the presentation of media content to the new destination device.

12. The method of claim 11, wherein the media server determines in real time the plurality of blocks of commercial content and non-commercial media content, for use in providing the media content to the client device or controlled device.

13. The method of claim 12, wherein the media server is provided within a cloud-based environment, which receives media content to be provided to the client device or controlled device as a plurality of channels having media streams, and receives interaction requests from the client device to switch between the channels.

14. The method of claim 13, wherein the controlled device is a television or other viewing device, and wherein the client device is used to communicate interaction requests to the media server to switch between or select particular channels to be played at the television or other viewing device.

15. The method of claim 14, wherein the media server provides a digital video recorder function that provides the media content or the channels to the client device or the controlled device as a plurality of thumbnails, including, upon receiving a request to skip to a media content represented by a thumbnail, allowing the player to slip to an end of a commercial block within that media content.

16. The method of claim 11, wherein each of the displayed media options associated with media content displays a media representation, the displayed media representations being associated with one of the media options;

each media representation comprising a portion of non-commercial media content.

17. The method of claim 11, wherein the multimodal analysis further comprises:

comparing the media frames with a training set, the training set comprising a plurality of classified samples; and

identifying, based at least upon the comparison of the media frames with the training set, commercial media frames indicating the plurality of blocks of commercial content and non-commercial media frames indicating the plurality of blocks of non-commercial content.

18. The method of claim 17, further comprising:

storing the training set on a database;

classifying additional samples as commercial or non-commercial samples based upon the identification of the commercial media frames and the non-commercial media frames;

storing the additional samples at the database; and

adding the additional samples to the training set.

19. The method of claim 11, further comprising:

selecting the plurality of samples from the media stream at a sampling frequency, wherein the sampling frequency is variable and is adjusted based on the determination of the plurality of blocks of commercial content and non-commercial media content.

20. The method of claim 11, further comprising:

creating, by the media analysis logic, a media analysis model, the medial analysis model associating frame features with classifiers, and

wherein the media analysis logic uses the media analysis model in determining the plurality of blocks of commercial content and non-commercial media content.

27. The non-transitory computer readable storage medium of claim 11 wherein the multimodal analysis further comprises:

comparing the media frames with a training set, the training set comprising a plurality of classified samples; and

identifying, based at least upon the comparison of the media frames with the training set, commercial media frames indicating the plurality of blocks of commercial content and non-commercial media frames indicating the plurality of blocks of non-commercial content.

28. The non-transitory computer readable storage medium of claim 27, the steps further comprising:

storing the training set on a database;

classifying additional samples as commercial or non-commercial samples based upon the identification of the commercial media frames and the non-commercial media frames;

storing the additional samples at the database; and

adding the additional samples to the training set.

21. A non-transitory computer readable storage medium, including instructions stored thereon which perform the steps comprising:

providing, at a computer including a processor and a media server,

a media analysis logic, which determines within a media stream associated with a media content, a plurality of blocks of commercial content and non-commercial media content, based upon a multimodal analysis, within the media stream, of a plurality of samples selected from the media stream, wherein each of the plurality of samples comprises media frames,

wherein the multimodal analysis weights a plurality of features in each of the samples, and

wherein the weights of each of the plurality of features in each of the samples are used in a linear classifier in determining the plurality of blocks of commercial content and non-commercial media content,

a media delivery module and server-side media content buffer to be populated with media content, to be returned as media streams to a requesting device or to a controlled device, and

a buffering logic, that buffers a current media content and one or more additional media content;

providing media content to a client device which includes a user interface and a client-side media content buffer, and which displays an array of media options associated with media content selectable at the client device and provided by the media server for playing at the client device, or at the controlled device, or at the client device and the controlled device, wherein each media option can be associated with a portion of the buffered media content;

controlling the playing at the client device, or at the controlled device, or at the client device and the controlled device, based upon input indicating a user selection of a media option received at the client device, including using information describing commercial blocks, and non-commercial content blocks, within the media stream, in providing the media content to the client device or the controlled device;

wherein the controlling the playing comprises

receiving indication of a selected media content, the selected media content associated with the input indicating a user selection of a media option, and

in response to receiving the indication of selected media content, controlling the playing of the selected media content based on the multimodal analysis and a determination of the plurality of commercial content blocks and non-commercial content blocks;

wherein receiving indication of a selected media content further comprises receiving information identifying a destination device, the destination device being one of the client device and the controlled device, the information identifying the destination device indicating the destination device to which the media server should provide the selected media content;

wherein the media server further comprises a context database, the context database storing data associated with the presentation of media content at a client device, the data associated with the presentation of media content comprising a current position within the displayed media content;

wherein the information identifying the destination device indicates a change in destination device from a current destination device to a new destination device; and

wherein the media server transmits the data associated with the presentation of media content to the new destination device.

22. The non-transitory computer readable storage medium of claim 21, wherein the media server determines in real time the plurality of blocks of commercial content and non-commercial media content, for use in providing the media content to the client device or controlled device.

23. The non-transitory computer readable storage medium of claim 22, wherein the media server is provided within a cloud-based environment, which receives media content to be provided to the client device or controlled device as a plurality of channels having media streams, and receives interaction requests from the client device to switch between the channels.

24. The non-transitory computer readable storage medium of claim 23, wherein the controlled device is a television or other viewing device, and wherein the client device is used to communicate interaction requests to the media server to switch between or select particular channels to be played at the television or other viewing device.

25. The non-transitory computer readable storage medium of claim 24, wherein the media server provides a digital video recorder function that provides the media content or the channels to the client device or the controlled device as a plurality of thumbnails, including, upon receiving a request to skip to a media content represented by a thumbnail, allowing the player to slip to an end of a commercial block within that media content.

26. The non-transitory computer readable storage medium of claim 21, wherein each of the displayed media options associated with media content displays a media representation, the displayed media representations being associated with one of the media options;

each media representation comprising a portion of non-commercial media content.

29. The non-transitory computer readable storage medium of claim 21, the steps further comprising:

selecting the plurality of samples from the media stream at a sampling frequency, wherein the sampling frequency is variable and is adjusted based on the determination of the plurality of blocks of commercial content and non-commercial media content.

30. The non-transitory computer readable storage medium of claim 21, the steps further comprising:

creating, by the media analysis logic, a media analysis model, the medial analysis model associating frame features with classifiers, and

wherein the media analysis logic uses the media analysis model in determining the plurality of blocks of commercial content and non-commercial media content.