Automated image processing and content curation

• US 10,382,373 B1
• Filed: 08/31/2016
• Issued: 08/13/2019
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Yang, Jianchao, Zhu, Yuke, Xu, Ning, Tang, Kevin Dechau, Li, Jia
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): Maniwang, Joseph R
• Application Number:
  US15/253,492
• Time in Patent Office: 1,077 Days
• Field of Search: 709206
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 67/01 : Protocols
• CPC Class Codes: H04L 67/55 : Push-based network services
• CPC Class Codes: H04W 4/18 : Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems, devices, methods, media, and instructions for automated image processing and content curation are described. In one embodiment a server computer system receives a content message from a first content source, and analyzes the content message to determine one or more quality scores and one or more content values associated with the content message. The server computer system analyzes the content message with a plurality of content collections of the database to identify a match between at least one of the one or more content values and a topic associated with at least a first content collection of the one or more content collections and automatically adds the content message to the first content collection based at least in part on the match. In various embodiments, different content values, image processing operations, and content selection operations are used to curate content collections.

Claims

1. A method comprising:

receiving, at a server computer system, a content message from a first content source of a plurality of content sources, the content message comprising media content;

analyzing, by the server computer system, the content message to determine context values comprising one or more quality scores and one or more content values associated with the content message;

storing the content message in a database of the server computer system along with the one or more quality scores and the one or more content values;

analyzing, by the server computer system, the content message with a plurality of content collections of the database to identify a match between at least one of the content values and a topic associated with at least a first content collection of the plurality of content collections; and

automatically adding, by the server computer system, the content message to the first content collection based at least in part on the match;

communicate, to a curation tool, the first content collection following transmission of the first content collection to a first set of client devices;

receive, from the curation tool, a set of content collection adjustments;

update the content collection in response to the set of content collection adjustments from the curation tool; and

transmitting the content collection as updated to a second set of client devices.

2. The method of claim 1 further comprising:

receiving, at the server computer system, one or more content messages from each content source of the plurality of content sources, each content message comprising media content;

analyzing, by the server computer system, each of the one or more content messages from each content source to determine corresponding one or more quality scores and one or more content values for each content message;

storing the one or more content messages from each content source in the database of the server computer system along with the one or more quality scores and the one or more content values;

analyzing, by the server computer system, the database to identify one or more content values shared by a first set of content messages in the database;

automatically generating the first content collection based on the identification of the one or more content values shared by the first set of content messages; and

automatically transmitting, by the server computer system, a notification associated with the first content collection in response to generation of the first content collection.

3. The method of claim 2 wherein analyzing, by the server computer system, each of the one or more content messages, comprises:

processing each content message of the one or more content message to identify one or more objects in the content message;

associating one or more keywords with each content message based on the one or more objects in the content message; and

storing each content message in the database with the one or more keywords.

4. The method of claim 3 wherein the one or more content values comprise the one or more keywords;

and

wherein the match between the at least one of the one or more content values and the topic associated with at least the first content collection of the one or more content collections is based on an association between at least a portion of the one or more keywords and the topic.

5. The method of claim 4 wherein the context values further comprise a time, a location, a user account associated with the content message.

6. The method of claim 5 further comprising processing one or more images of the content message to determine one or more quality values determined by analyzing the one or more images using a set of content quality metrics.

7. The method of claim 6 wherein the context values comprise an audio content value selected based on an analysis of audio data from the content message;

and

wherein the one or more quality values comprises an audio quality value determined based at least in part on an analysis to determine an audio noise level.

8. The method of claim 7 wherein the audio content value further is based on a speech activity detection analysis of the content message.

9. The method of claim 6 wherein the one or more quality values comprises an interestingness value based on predicting an expected user interest in the content message.

10. The method of claim 9 wherein the interestingness value is generated using a neural network generated using a training set of content messages identified as interesting within the server computer system.

11. The method of claim 10 wherein the neural network further comprises a convolutional neural network with a feature map including a set of content features and a set of quality features.

12. The method of claim 10 wherein the interestingness value is further generated based on feedback training data comprising feedback messages from users rating selected content messages of the training set of content messages based on previous content collections communicated to the users.

13. The method of claim 10 wherein the interestingness value is further generated based on rating inputs received at the curation tool integrated with the server computer system to provide operator selected training sets for the neural network.

14. The method of claim 13 further comprising deleting the content message from the database based on a deletion trigger associated with the content message;

wherein the content message as received at the server computer system is associated with the deletion trigger.

15. A server computer system comprising:

a memory; and

one or more processors coupled to the memory and configured to;

receive a content message from a first content source of a plurality of content sources, the content message comprising media content;

analyze the content message to determine context values comprising one or more quality scores and one or more content values associated with the content message;

store the content message in a database of the server computer system along with the one or more quality scores and the one or more content values;

analyze the content message with a plurality of content collections of the database to identify a match between at least one of the one or more content values and a topic associated with at least a first content collection of the plurality of content collections; and

automatically add the content message to the first content collection based at least in part on the match;

wherein the one or more processors are further configured to;

communicate, to a curation tool, the first content collection following transmission of the first content collection to a first set of client devices;

receive, from the curation tool, a set of content collection adjustments;

update the content collection in response to the set of content collection adjustments from the curation tool; and

transmitting the content collection as updated to a second set of client devices.

16. The system of claim 15 wherein the one or more processors are further configured to:

receive one or more content messages from each content source of the plurality of content sources, each content message comprising media content;

analyze each of the one or more content messages from each content source to determine corresponding one or more quality scores and one or more content values for each content message;

store the one or more content messages from each content source in the database of the server computer system along with the one or more quality scores and the one or more content values;

analyze the database to identify one or more content values shared by a first set of content messages in the database;

automatically generate the first content collection based on the identification of the one or more content values shared by the first set of content messages; and

automatically transmit a notification associated with the first content collection in response to generation of the first content collection.

17. The system of claim 15 wherein configuration of the one or more processors to analyze the content message comprises configuring the one or more processors to:

analyze the content message to identify one or more objects in the content message;

associate one or more keywords with the content message based on the one or more objects in the content message; and

store the content message in the database with the one or more keywords.

18. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause a server computer system to perform a method comprising:

receiving, at the server computer system, one or more content messages from each content source of a plurality of content sources, each content message comprising media content;

analyzing, by the server computer system, each of the one or more content messages from each content source to determine corresponding one or more quality scores and one or more content values for each content message;

storing the one or more content messages from each content source in the database of the server computer system along with the one or more quality scores and the one or more content values;

analyzing, by the server computer system, the database to identify one or more content values shared by a first set of content messages in the database;

automatically generating a first content collection based on the identification of the one or more content values shared by the first set of content messages; and

automatically transmitting, by the server computer system, a notification associated with the first content collection in response to generation of the first content collection;

wherein automatically generating the first content collection based on the identification of the one or more content values shared by the first set of content messages comprises;

identifying a first content message of the first set of content messages as meeting a first set of scoring thresholds;

determining that a threshold number of content messages have associated values within a threshold distance of the first set of scoring thresholds; and

selecting a set of closest content messages to the first content message as the first set of content messages for the first content collection in response to the determination that the threshold number of content messages is met.

19. The non-transitory computer readable medium of claim 18 wherein automatically generating the first content collection based on the identification of the one or more content values shared by the first set of content messages comprises:

determining a first content topic for the first set of content messages;

analyzing the one or more content messages from each content source using a k-means analysis to identify the first set of content messages as a first cluster; and

generating the first content collection using the first set of content messages in response to the k-means analysis.

20. The non-transitory computer readable medium of claim 18 wherein analyzing, by the server computer system, each of the one or more content messages, comprises:

processing each content message of the one or more content message to identify one or more objects in the content message;

associating one or more keywords with each content message based on the one or more objects in the content message; and

storing each content message in the database with the one or more keywords.