Device location based on machine learning classifications

• US 10,264,422 B2
• Filed: 04/30/2018
• Issued: 04/16/2019
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Charlton, Ebony James, Hanumante, Sumant, Ren, Zhou, Sagar, Dhritiman
• Primary Examiner(s): Patel, Ajit
• Application Number:
  US15/967,201
• Publication Number:
  US 20190069147A1
• Time in Patent Office: 351 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/36 Indoor scenes
• CPC Class Codes: G06V 20/38 Outdoor scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/21 for social networking applications
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings
• CPC Class Codes: H04W 88/02 Terminal devices

Abstract

A venue system of a client device can submit a location request to a server, which returns multiple venues that are near the client device. The client device can use one or more machine learning schemes (e.g., convolutional neural networks) to determine that the client device is located in one of specific venues of the possible venues. The venue system can further select imagery for presentation based on the venue selection. The presentation may be published as ephemeral message on a network platform.

Claims

1. A method comprising:

transmitting, to one or more servers, a request comprising location data generated by a client device;

receiving, from the one or more servers, a venue dataset comprising a plurality of venues;

identifying, in memory of the client device, an image of a real-world environment;

classifying, using a machine learning scheme, an image attribute of the image depicting the real-world environment;

selecting a venue from the plurality of venues based at least in part on the venue matching the classified attribute of the image;

selecting one or more display elements that are pre-associated with the selected venue; and

displaying, on the client device, a presentation comprising the one or more display elements.

2. The method of claim 1, wherein the classified image attribute indicates that the real-world environment depicted in the image is an outside environment or an inside environment.

3. The method of claim 2, wherein the selection of the venue comprises excluding one or more other venues in the plurality of venues based at least in part on the classified image attribute not matching the one or more other venues.

4. The method of claim 1, wherein the machine learning scheme is trained on images of a plurality of physical places, and the classified image attribute indicates that the real-world environment depicted in the image is one of the plurality of physical places, wherein the selected venue matches the one of the plurality of physical places.

5. The method of claim 1, wherein the one or more display elements includes one or more words.

6. The method of claim 1, wherein the plurality of venues are categorized, on the client device, into categories described by metadata tags.

7. The method of claim 6, wherein the one or more display elements are categorized into the categories, and the one or more display elements are pre-associated with the selected venue in that the one or more display elements share at least one of a same category.

8. The method of claim 1, wherein the one or more display elements include an avatar of a user of the client device.

9. The method of claim 1, wherein the location data comprises at least one of:

global positioning system (GPS) data generated by a GPS sensor on the client device, Internet Protocol (IP) networks detected by a network sensor on the client device.

10. The method of claim 1, wherein the venue dataset is a subset of a larger venue dataset stored on the one or more servers, the larger venue dataset including venues that are not near to a location indicated by the location data of the client device.

11. The method of claim 1, wherein the machine learning scheme comprises one or more convolutional neural networks.

12. The method of claim 1, wherein the classified image attribute indicates an recognized physical object depicted in the image, the venue being selected based on a category of the venue being pre-associated with the recognized physical object.

13. The method of claim 12, further comprising:

publishing, to a social network site, the presentation as an ephemeral message.

14. The method of claim 1, wherein the presentation displays the one or more display elements as an overlay on the image generated by the client device.

15. The method of claim 1, further comprising:

generating the image using an image sensor on the client device; and

storing the image in the memory of the client device.

16. The method of claim 1, wherein the image is from a video sequence captured using the client device.

17. A system comprising:

one or more processors of a machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

transmitting, to one or more servers, a request comprising location data generated by a client device;

receiving, from the one or more servers, a venue dataset comprising a plurality of venues;

identifying, in the memory, an image of a real-world environment;

classifying, using a machine learning scheme, an image attribute of the image depicting the real-world environment;

selecting a venue from the plurality of venues based at least in part on the venue matching the classified attribute of the image;

selecting one or more display elements that are pre-associated with the selected venue; and

displaying, on the client device, a presentation comprising the one or more display elements.

18. The system of claim 17, wherein the classified image attribute indicates that the real-world environment depicted in the image is an outside environment or an inside environment;

and wherein

the selection of the venue comprises excluding one or more other venues in the plurality of venues based at least in part on the classified image attribute not matching the one or more other venues.

19. A machine-readable storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

transmitting, to one or more servers, a request comprising location data generated by a client device;

receiving, from the one or more servers, a venue dataset comprising a plurality of venues;

identifying an image of a real-world environment;

classifying, using a machine learning scheme, an image attribute of the image depicting the real-world environment;

selecting a venue from the plurality of venues based at least in part on the venue matching the classified attribute of the image;

selecting one or more display elements that are pre-associated with the selected venue; and

displaying, on the client device, a presentation comprising the one or more display elements.

20. The machine-readable storage medium of claim 19, wherein the classified image attribute indicates that the real-world environment depicted in the image is an outside environment or an inside environment;

and wherein

the selection of the venue comprises excluding one or more other venues in the plurality of venues based at least in part on the classified image attribute not matching the one or more other venues.