Generating data in a messaging system for a machine learning model

• US 10,643,104 B1
• Filed: 12/01/2017
• Issued: 05/05/2020
• Est. Priority Date: 12/01/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Xue, Zehao, Ren, Zhou
• Primary Examiner(s): Couso, Yon J
• Application Number:
  US15/829,572
• Time in Patent Office: 886 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

Systems and methods are provided for analyzing location data associated with a location of a computing device to determine that a media content item is captured near a food-related venue or event, presenting interactive features to capture input related to food associated with the food-related venue or event, receiving the input in response to the presented interactive features, sending the media content item and the input in response to the interactive features to a computing system to incorporate the media content item and input into a machine learning model for food detection, and updating a messaging application to update a food detector functionality of the messaging application to comprise an updated machine learning model for food detection based on the media content item and input in response to the interactive features.

Claims

1. A method comprising:

detecting, by a computing device, capture of an image or video by the computing device using a messaging application;

analyzing, by the computing device, location data associated with a location of the computing device to determine that the image or video is captured near a food-related venue or event;

presenting, by the computing device, interactive features on a display of the computing device to capture an input to the computing device related to food associated with the food-related venue or event;

receiving, by the computing device, the input related to food associated with the food-related venue or event in response to the presented interactive features;

sending, by the computing device, the image or video and the input related to food associated with the food-related venue or event in response to the interactive features, to a computing system to incorporate the image or video and input into a machine learning model for food detection; and

updating, by the computing device, the messaging application to update a food detector functionality of the messaging application to comprise an updated machine learning model for food detection based on the image or video and the input related to food associated with the food-related venue or event in response to the interactive features.

2. The method of claim 1, wherein after analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event, the method further comprises:

determining that the image or video comprises food using the machine learning model for food detection.

3. The method of claim 2, wherein determining that the image or video comprises food using the machine learning model for food detection comprises:

inputting the image or video into the machine learning model for food detection;

generating a prediction value for the image or video;

determining whether the prediction value exceeds a predetermined threshold value; and

determining that the image or video comprises food based on the prediction value exceeding the predetermined threshold value.

4. The method of claim 1, wherein analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event comprises comparing the location data to map data to determine that the location of the computing device is near the food-related venue or event.

5. The method of claim 1, wherein the interactive features comprise a media overlay, a lens, or a request for a review of the food-related venue or event.

6. The method of claim 1, wherein the interactive features comprise a request for an image of food from the food-related venue or event, and the image or video sent to the computing system is the requested image.

7. The method of claim 1, wherein the computing system receives the image or video and associated input from the computing device and stores the received image or video and associated input in a database with a plurality of captured images and videos and associated input from a plurality of computing devices.

8. The method of claim 1, wherein the updated machine learning model for food detection based on the image or video and the input in response to the interactive features is further based on a plurality of captured images and videos and input from a plurality of users in response to interactive features sent by a plurality of computing devices.

9. The method of claim 1, wherein after analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event, the method further comprises:

determining that the image or video does not comprise food, using the trained machine learning model for food detection; and

wherein the interactive features presented comprise features to incentivize capture of an image of food associated with the food-related venue or event.

10. A computing device comprising:

one or more hardware processors; and

a computer-readable medium coupled with the one or more hardware processors, the computer-readable medium comprising instructions stored thereon that are executable by the one or more hardware processors to cause the computing device to perform operations comprising;

detecting capture of an image or video by the computing device using a messaging application;

analyzing location data associated with a location of the computing device to determine that the image or video is captured near a food-related venue or event;

presenting interactive features on a display of the computing device to capture an input to the computing device related to food associated with the food-related venue or event;

receiving the input related to food associated with the food-related venue or event in response to the presented interactive features;

sending the image or video and the input related to food associated with the food-related venue or event in response to the interactive features, to a computing system to incorporate the image or video and input into a machine learning model for food detection; and

updating the messaging application to update a food detector functionality of the messaging application to comprise an updated machine learning model for food detection based on the image or video and the input related to food associated with the food-related venue or event in response to the interactive features.

11. The computing device of claim 10, wherein after analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event, the operations further comprise:

determining that the image or video comprises food using a machine learning model for food detection.

12. The computing device of claim 11, wherein determining that the image or video comprises food using the trained machine learning model for food detection comprises:

inputting the image or video into the machine learning model for food detection;

generating a prediction value for the image or video;

determining whether the prediction value exceeds a predetermined threshold value; and

determining that the image or video comprises food based on the prediction value exceeding the predetermined threshold value.

13. The computing device of claim 10, wherein analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event comprises comparing the location data to map data to determine that the location of the computing device is near the food-related venue or event.

14. The computing device of claim 10, wherein the interactive features comprise a media overlay, a lens, or a request for a review of the food-related venue or event.

15. The computing device of claim 10, wherein the interactive features comprise a request for an image of food from the food-related venue or event, and the image or video sent to the computing system is the requested image.

16. The computing device of claim 10, wherein the computing system receives the image or video and associated input from the computing device and stores the received image or video and associated input in a database with a plurality of captured images and videos and associated input from a plurality of computing devices.

17. The computing device of claim 10, wherein the updated machine learning model for food detection based on the image or video and the input in response to the interactive features is further based on a plurality of captured images and videos and input from a plurality of users in response to interactive features sent by a plurality of computing devices.

18. The computing device of claim 10, wherein after analyzing the location data associated with the location of the computing device to determine that the image or video is being captured near the food-related venue or event, the operations further comprise:

determining that the image or video does not comprise food, using the trained machine learning model for food detection; and

wherein the interactive features presented comprise features to incentivize capture of an image of food associated with the food-related venue or event.

19. A non-transitory computer-readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

detecting capture of an image or video by the computing device using a messaging application;

analyzing location data associated with a location of the computing device to determine that the image or video is captured near a food-related venue or event;

presenting interactive features on a display of the computing device to capture an input to the computing device related to food associated with the food-related venue or event;

receiving the input related to food associated with the food-related venue or event in response to the presented interactive features;

sending the image or video and the input related to food associated with the food-related venue or event in response to the interactive features, to a computing system to incorporate the image or video and input into a machine learning model for food detection; and

updating the messaging application to update a food detector functionality of the messaging application to comprise an updated machine learning model for food detection based on the image or video and the input related to food associated with the food-related venue or event in response to the interactive features.

20. The non-transitory computer-readable medium of claim 19, wherein the updated machine learning model for food detection based on the image or video and input in response to the interactive features is further based on a plurality of captured images and videos and input from a plurality of users in response to interactive features sent by a plurality of computing devices.