Device location based on machine learning classifications

• US 9,980,100 B1
• Filed: 08/31/2017
• Issued: 05/22/2018
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Charlton, Ebony James, Hanumante, Sumant, Ren, Zhou, Sagar, Dhritiman
• Primary Examiner(s): Patel, Ajit
• Application Number:
  US15/692,990
• Time in Patent Office: 264 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/36 Indoor scenes
• CPC Class Codes: G06V 20/38 Outdoor scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/21 for social networking applications
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings
• CPC Class Codes: H04W 88/02 Terminal devices

Abstract

A venue system of a client device can submit a location request to a server, which returns multiple venues that are near the client device. The client device can use one or more machine learning schemes (e.g., convolutional neural networks) to determine that the client device is located in one of specific venues of the possible venues. The venue system can further select imagery for presentation based on the venue selection. The presentation may be published as ephemeral message on a network platform.

Claims

1. A method comprising:

transmitting, using one or more processors of a client device, a request for venue data to one or more servers, the request comprising geolocation data generated by the client device;

receiving, from the one or more servers, a venue dataset comprising a plurality of venues, each venue of the plurality of venues associated with tags that describe the venue;

identifying an image generated by the client device;

classifying an object depicted in the image using a machine learning scheme;

selecting a venue from the venue dataset based at least in part on the classified object matching a tag associated with the venue;

selecting one or more display elements that are pre-associated with the selected venue; and

displaying, on the client device, a presentation comprising the one or more display elements and the image.

2. The method of claim 1, further comprising:

determining environment data for the image using an additional machine learning scheme, the environment data describing whether the image is of an outside environment or inside environment.

3. The method of claim 2, wherein the selection of the venue is further based at least in part on the environment data of the image matching a tag associated with the venue.

4. The method of claim 1, wherein the image is part of an active video feed generated by the client device;

and

the presentation displays the one or more display elements overlaid on the active video feed.

5. The method of claim 1, wherein the plurality of venues are categorized, on the client device, into categories and subcategories, and the tags are metadata of the categories or subcategories.

6. The method of claim 5, wherein the one or more display elements are categorized into the categories and subcategories, and the one or more display elements are pre-associated with the selected venue in that the one or more display elements and the selected elements share at least one of a same category or subcategory.

7. The method of claim 1, wherein the one or more display elements include an avatar of a user of the client device.

8. The method of claim 1, wherein the geolocation data includes global positioning system (GPS) data generated by a GPS sensor on the client device.

9. The method of claim 1, wherein the venue dataset is a subset of a larger venue dataset that includes venues that are not near to a location indicated by the geolocation data of the client device.

10. The method of claim 1, wherein the machine learning scheme comprises one or more neural networks trained to classify physical objects using an image set, the image set comprising images of a plurality of physical images.

11. The method of claim 2, wherein the additional machine learning scheme comprises one or more neural networks trained to classify environments as being inside or outside, the one or more neural networks trained on an image set comprising images of a plurality of images of outside and inside environments.

12. The method of claim 1, further comprising:

receiving, on the client device, an instruction to store the presentation.

13. The method of claim 12, further comprising:

publishing, to a social network site, the presentation as an ephemeral message.

14. A system comprising:

one or more processors of a machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

transmitting a request for venue data to one or more servers, the request comprising geolocation data generated by a client device;

receiving, from the one or more servers, a venue dataset comprising a plurality of venues, each venue of the plurality of venues associated with tags that describe the venue;

identifying an image generated by the client device;

classifying an object depicted in the image using a machine learning scheme;

selecting a venue from the venue dataset based at least in part on the classified object matching a tag associated with the venue;

selecting one or more display elements that are pre-associated with the selected venue; and

displaying, on the client device, a presentation comprising the one or more display elements and the image.

15. The system of claim 14, the operations further comprising:

determining environment data for the image using an additional machine learning scheme, the environment data describing whether the image is of an outside environment or inside environment.

16. The system of claim 15, wherein the selection of the venue is further based at least in part on the environment data of the image matching a tag associated with the venue.

17. The system of claim 14, wherein the image is part of an active video feed generated by the client device;

and

the presentation displays the one or more display elements overlaid on the active video feed.

18. The system of claim 14, wherein the plurality of venues are categorized, on the client device, into categories and subcategories, and the tags are metadata of the categories or subcategories.

19. A machine-readable storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

transmitting a request for venue data to one or more servers, the request comprising geolocation data generated by the client device;

receiving, from the one or more servers, a venue dataset comprising a plurality of venues near, each venue of the plurality of venues associated with tags that describe the venue;

identifying an image generated by the client device;

classifying an object depicted in the image using a machine learning scheme;

selecting a venue from the venue dataset based at least in part on the classified object matching a tag associated with the venue;

selecting one or more display elements that are pre-associated with the selected venue; and

displaying, on the client device, a presentation comprising the one or more display elements and the image.

20. The machine-readable storage medium of claim 19, the operations further comprising:

determining environment data for the image using an additional machine learning scheme, the environment data describing whether the image is of an outside environment or inside environment.