CONTENT FILTERING WITH CONVOLUTIONAL NEURAL NETWORKS

• US 20170140260A1
• Filed: 11/17/2016
• Published: 05/18/2017
• Est. Priority Date: 11/17/2015
• Status: Abandoned Application

• Current Assignee: Malibu Entertainment Incorporated
• Original Assignee: Rcrdclub Corporation (Block, Inc. (f/k/a Square, Inc.))
• Inventors: Manning, Damian Franken, Shams, Omar Emad
• Application Number:
  US15/354,377
• Publication Number:
  US 20170140260A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/635 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G10L 25/30 using neural networks
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Systems and techniques are provided for content filtering with convolutional neural networks. A spectrogram generated from audio data may be received. A convolution may be applied to the spectrogram to generate a feature map. Values for a hidden layer of a neural network may be determined based on the feature map. A label for the audio data may be determined based on the determined values for the hidden layer of the neural network. The hidden layer may include a vector including the values for the hidden layer. The vector may be stored as a vector representation of the audio data.

Claims

1. A computer-implemented method performed by a data processing apparatus, the method comprising:

receiving a spectrogram generated from audio data;

applying a convolution to the spectrogram to generate a feature map;

determining values for a hidden layer of a neural network based on the feature map; and

determining a label for the audio data based on the determined values for the hidden layer of the neural network.

2. The computer-implemented method of claim 1, wherein the hidden layer comprises a vector comprising the values for the hidden layer, and further comprising:

storing the vector as a vector representation of the audio data.

3. The computer-implemented method of claim 1, wherein determining a label for the audio data based on the determined values for the hidden layer of the neural network further comprises determining values for an activation layer of the neural network based on the determined values for the hidden layer of the neural network.

4. The computer-implemented method of claim 1, wherein the spectrogram is a mel spectrogram or a mel-frequency cepstrum.

5. The computer-implemented method of claim 1, wherein applying a convolution comprises applying to the spectrogram one or more of:

a one-dimensional convolution, a two-dimensional convolution, and a three-dimensional convolution.

6. The computer-implemented method of claim 1, wherein the neural network comprises a convolutional neural network trained to identify a genre of a song based on a spectrogram generated from the song, and wherein the label identifies a genre of a song in the audio data.

7. The computer-implemented method of claim 2, further comprising:

receiving, for one or more songs, a vector representation for each of the one or more songs;

comparing the vector representation of the audio data to the vector representations for each of the one or more songs; and

generating a playlist comprising one or more of the one or more songs and a song represented by the audio data based on the comparing of the vector representation of the audio data to the vector representations for each of the one or more songs.

8. The computer-implemented method of claim 1, wherein comparing the vector representation of the audio data to the vector representations for each of the one or more songs comprises determining the dot products of the vector representation of the audio data and the vector representations for each of the one or more songs.

9. A computer-implemented system for content filtering with convolutional neural networks, comprising:

a storage comprising audio data; and

a processor that implements a convolutional neural network that receives a spectrogram generated from audio data, applies a convolution to the spectrogram to generate a feature map, determines values for a hidden layer of the convolutional neural network based on the feature map, and determines a label for the audio data based on the determined values for the hidden layer of the neural network.

10. The computer-implemented system of claim 9, wherein the hidden layer comprises a vector comprising the values for the hidden layer, and wherein the processor that implements the convolutional neural network further stores the vector in the storage as a vector representation of the audio data.

11. The computer-implemented system of claim 9, wherein the processor implementing the convolutional neural network further determines a label for the audio data based on the determined values for the hidden layer of the neural network further by determining values for an activation layer of the neural network based on the determined values for the hidden layer of the neural network.

12. The computer-implemented system of claim 9, wherein the spectrogram is a mel spectrogram or a mel-frequency cepstrum.

13. The computer-implemented system of claim 9, wherein the processor implementing the convolutional neural network applies a convolution by applying to the spectrogram one or more of:

a one-dimensional convolution, a two-dimensional convolution, and a three-dimensional convolution.

14. The computer-implemented system of claim 9, wherein the convolutional neural network is trained to identify a genre of a song based on a spectrogram generated from the song, and wherein the label identifies a genre of a song in the audio data.

15. The computer-implemented system of claim 10, wherein the processor further receives, for one or more songs, a vector representation for each of the one or more songs, compares the vector representation of the audio data to the vector representations for each of the one or more songs, and generates a playlist comprising one or more of the one or more songs and a song represented by the audio data based on the comparing of the vector representation of the audio data to the vector representations for each of the one or more songs.

16. The computer-implemented system of claim 9, wherein the processor compares the vector representation of the audio data to the vector representations for each of the one or more songs by determining the dot products of the vector representation of the audio data and the vector representations for each of the one or more songs.

17. A system comprising:

one or more computers and one or more storage devices storing instructions which are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a spectrogram generated from audio data;

applying a convolution to the spectrogram to generate a feature map;

determining values for a hidden layer of a neural network based on the feature map; and

determining a label for the audio data based on the determined values for the hidden layer of the neural network.

18. The system of claim 17, wherein the instructions further cause the one or more computers to perform operations comprising:

storing the vector as a vector representation of the audio data.

19. The system of claim 17, wherein the instructions further cause the one or more computers to perform operations comprising:

receiving, for one or more songs, a vector representation for each of the one or more songs;

comparing the vector representation of the audio data to the vector representations for each of the one or more songs; and

generating a playlist comprising one or more of the one or more songs and a song represented by the audio data based on the comparing of the vector representation of the audio data to the vector representations for each of the one or more songs.

20. The system of claim 17, wherein the instructions that cause the one or more computer to perform operations comprising comparing the vector representation of the audio data to the vector representations for each of the one or more songs further cause the one or more computers to perform operations comprising determining the dot products of the vector representation of the audio data and the vector representations for each of the one or more songs.