Media overlay selection system

• US 10,482,327 B1
• Filed: 11/02/2017
• Issued: 11/19/2019
• Est. Priority Date: 01/18/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Ouimet, Kirk, Gong, Yunchao
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US15/801,853
• Time in Patent Office: 747 Days
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

A computing system receives, from a client device, image data describing an image captured by an optical sensor of the client device. The computing system compares the image to a set of reference images that include associated metadata describing a real-world feature depicted by the respective reference image. The computing system determines, based on the comparison, a subset of reference images that are similar to the image, and then determines, based on associated metadata of the subset of reference images, that the image captured by the optical sensor of the client device depicts a first real-world feature. The computing system selects a subset of media overlays related to the first real-world feature based on metadata associated with each media overlay that describes the respective media overlay. The computing system transmits the subset of media overlays to the client device.

Claims

1. A method comprising:

receiving, from a client device, image data describing an image captured by an optical sensor of the client device;

comparing, based on the image data, the image captured by the optical sensor of the client device to a set of reference images, yielding a comparison, each reference image from the set of reference images including associated metadata describing a real-world feature depicted by the respective reference image;

determining, based on the comparison, a subset of reference images that are similar to the image captured by the optical sensor of the client device;

determining, based on associated metadata of the subset of reference images, that the image captured by the optical sensor of the client device depicts a first real-world feature;

receiving, from the client device, motion data representing movement of the client device;

selecting, from a set of media overlays, a subset of media overlays related to the first real-world feature, the subset of media overlays selected based on metadata associated with each media overlay that describes the respective media overlay and based on the motion data representing movement of the client device, wherein each media overlay is a digital image that can be annotated over the image captured by the optical sensor of the client device; and

transmitting the subset of media overlays to the client device.

2. The method of claim 1, wherein comparing the image captured by the optical sensor of the client device to the set of reference images comprises:

generating, based on the image data, a target vector representing the image captured by the optical sensor of the client device, the target vector including numeric values describing features of the image captured by the optical sensor of the client device; and

determining, using a distance function, a distance between the target vector and a set of reference vectors, yielding a distance determination, each reference vector from the set of reference vectors representing a respective reference image from the set of reference images and including numeric values describing features of the respective reference image.

3. The method of claim 2, wherein determining the subset of reference images that are similar to the image captured by the optical sensor of the client device comprises:

determining, based on the distance determination, a subset of reference vectors that are closest to the target vector, the subset of reference vectors representing the subset of reference images that are similar to the image captured by the optical sensor of the client device.

4. The method of claim 1, wherein determining that the image captured by the optical sensor of the client device depicts a first real-world feature comprises:

determining, based on associated metadata of the subset of reference images, a percentage of the subset of reference images associated with metadata indicating that the respective reference image depicts an image of the first real-world feature;

comparing the percentage to a threshold percentage; and

in response to determining the percentage meets or exceeds the threshold percentage, determining that the image depicts the first real-world feature.

5. The method of claim 1, wherein selecting a subset of media overlays related to the first real-world feature comprises:

determining, based on metadata associated with each media overlay that describes the respective media overlay, media overlays that depict the first real-world feature; and

selecting at least one media overlay from the set of media overlays based on popularity, wherein the popularity indicates how often each of the media overlays in the set are used to annotate an image.

6. The method of claim 1, further comprising:

determining, based on associated metadata of the subset of reference images, a number that represents how many of the subset of reference images is associated with metadata indicating that the respective reference image depicts an image of the first real-world feature;

comparing the number to a threshold;

in response to determining the number meets or exceeds the threshold, determining that the image depicts the first real-world feature;

determining a size of the first real-world feature as depicted in the first image; and

adjusting a size of a first media overlay that depicts the first real-world feature based on the size of the first real-world feature as depicted in the first image.

7. The method of claim 1, wherein selecting a subset of media overlays related to the first real-world feature comprises:

determining a category associated with the first-real world object; and

determining, based on metadata associated with each media overlay that describes the respective media overlay, media overlays that depict real-world features that are associated with the category.

8. The method of claim 1, further comprising:

receiving, from the client device, contextual data captured by a sensor of the client device.

9. The method of claim 8, wherein selecting a subset of media overlays related to the first real-world feature comprises:

determining, based on the first real-world feature and the contextual data received from the client device and, a contextual category of the image; and

determining, based on metadata associated with each media overlay that describes the respective media overlay, media overlays that depict real-world features that are associated with the contextual category.

10. The method of claim 1, wherein after the subset of media overlays are received by the client device:

the client device receives user input designating a place within the image to position a first overlay from the subset of media overlays; and

the client device, in response to receiving the user input, causes the first media overlay to be appended over the designated place within the image, yielding an annotated image.

11. A system comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising;

receiving, from a client device, image data describing an image captured by an optical sensor of the client device;

comparing, based on the image data, the image captured by the optical sensor of the client device to a set of reference images, yielding a comparison, each reference image from the set of reference images including associated metadata describing a real-world feature depicted by the respective reference image;

determining, based on the comparison, a subset of reference images that are similar to the image captured by the optical sensor of the client device;

determining, based on associated metadata of the subset of reference images, that the image captured by the optical sensor of the client device depicts a first real-world feature;

receiving, from the client device, motion data representing movement of the client device;

selecting, from a set of media overlays, a subset of media overlays related to the first real-world feature, the subset of media overlays selected based on metadata associated with each media overlay that describes the respective media overlay and based on the motion data representing movement of the client device, wherein each media overlay is a digital image that can be annotated over the image captured by the optical sensor of the client device; and

transmitting the subset of media overlays to the client device.

12. The system of claim 11, wherein comparing the image captured by the optical sensor of the client device to the set of reference images comprises:

generating, based on the image data, a target vector representing the image captured by the optical sensor of the client device, the target vector including numeric values describing features of the image captured by the optical sensor of the client device; and

determining, using a distance function, a distance between the target vector and a set of reference vectors, yielding a distance determination, each reference vector from the set of reference vectors representing a respective reference image from the set of reference images and including numeric values describing features of the respective reference image.

13. The system of claim 12, wherein determining the subset of reference images that are similar to the image captured by the optical sensor of the client device comprises:

determining, based on the distance determination, a subset of reference vectors that are closest to the target vector, the subset of reference vectors representing the subset of reference images that are similar to the image captured by the optical sensor of the client device.

14. The system of claim 11, wherein determining that the image captured by the optical sensor of the client device depicts a first real-world feature comprises:

determining, based on associated metadata of the subset of reference images, a percentage of the subset of reference images associated with metadata indicating that the respective reference image depicts an image of the first real-world feature;

comparing the percentage to a threshold percentage; and

in response to determining the percentage meets or exceeds the threshold percentage, determining that the image depicts the first real-world feature.

15. The system of claim 11, wherein selecting a subset of media overlays related to the first real-world feature comprises:

determining, based on metadata associated with each media overlay that describes the respective media overlay, media overlays that depict the first real-world feature; and

selecting at least one media overlay from the set of media overlays based on popularity, wherein the popularity indicates how often each of the media overlays in the set are used to annotate an image.

16. The system of claim 11, the operations further comprising:

determining, based on associated metadata of the subset of reference images, a number that represents how many of the subset of reference images is associated with metadata indicating that the respective reference image depicts an image of the first real-world feature;

comparing the number to a threshold;

in response to determining the number meets or exceeds the threshold, determining that the image depicts the first real-world feature;

determining a size of the first real-world feature as depicted in the first image; and

adjusting a size of a first media overlay that depicts the first real-world feature based on the size of the first real-world feature as depicted in the first image.

17. The system of claim 11, wherein selecting a subset of media overlays related to the first real-world feature comprises:

determining a category associated with the first-real world object; and

determining, based on metadata associated with each media overlay that describes the respective media overlay, media overlays that depict real-world features that are associated with the category.

18. The system of claim 11, the operations further comprising:

receiving, from the client device, contextual data captured by a sensor of the client device.

19. The system of claim 18, wherein selecting a subset of media overlays related to the first real-world feature comprises:

determining, based on the first real-world feature and the contextual data received from the client device, a contextual category of the image; and

determining, based on metadata associated with each media overlay that describes the respective media overlay, media overlays that depict real-world features that are associated with the contextual category.

20. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing device, cause the computing device to perform operations comprising:

receiving, from a client device, image data describing an image captured by an optical sensor of the client device;

comparing, based on the image data, the image captured by the optical sensor of the client device to a set of reference images, yielding a comparison, each reference image from the set of reference images including associated metadata describing a real-world feature depicted by the respective reference image;

determining, based on the comparison, a subset of reference images that are similar to the image captured by the optical sensor of the client device;

determining, based on associated metadata of the subset of reference images, that the image captured by the optical sensor of the client device depicts a first real-world feature;

receiving, from the client device, motion data representing movement of the client device;

selecting, from a set of media overlays, a subset of media overlays related to the first real-world feature, the subset of media overlays selected based on metadata associated with each media overlay that describes the respective media overlay and based on the motion data representing movement of the client device, wherein each media overlay is a digital image that can be annotated over the image captured by the optical sensor of the client device; and

transmitting the subset of media overlays to the client device.