Virtual vision system

• US 10,565,795 B2
• Filed: 07/19/2017
• Issued: 02/18/2020
• Est. Priority Date: 03/06/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Charlton, Ebony James, Dargis, Jokubas, Pilipski, Eitan, Sagar, Dhritiman, Shaburov, Victor
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US15/654,429
• Publication Number:
  US 20180253901A1
• Time in Patent Office: 944 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences

Abstract

A context based augmented reality system can be used to display augmented reality elements over a live video feed on a client device. The augmented reality elements can be selected based on a number of context inputs generated by the client device. The context inputs can include location data of the client device and location data of nearby physical places that have preconfigured augmented elements. The preconfigured augmented elements can be preconfigured to exhibit a design scheme of the corresponding physical place.

Claims

1. A system comprising:

one or more processors of a machine;

a camera;

a display device;

one or more sensors; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

receive, from a network platform, a plurality of lens virtual objects corresponding to different geographic locations, each of the plurality of lens virtual objects comprising one or more navigational display elements uploaded to the network platform by submitting client devices;

generate location data using the one or more sensors;

select a lens virtual object from the plurality of lens virtual objects based on the location data corresponding to a geographic location of the selected lens virtual object;

determine heading data using the geographic location of the selected lens virtual object, the heading data specifying a direction of the geographic location relative to a current orientation of the system; and

cause a presentation, on live video generated by the camera and displayed on the display device, of a navigational direction display element in the uploaded one or more navigational display elements associated with the selected lens virtual object, the navigational direction display element indicating the direction to the geographic location relative to the current orientation of the system.

2. The system of claim 1, the operations further comprising:

identify an update to the heading data that indicates a physical place is in a field of view of the live video, the update to the heading data caused in response to physical movement of the machine; and

responsive to the physical place being in the field of view of the live video, display one or more navigational physical place display elements of the selected lens virtual object, the one or more navigational physical place display elements indicating location of the physical place within the live video.

3. The system of claim 1, wherein the plurality of lens virtual objects comprises a first lens virtual object and a second lens virtual object, the first lens virtual object comprising a first set of one or more navigational display elements uploaded by a first submitting client device, the second lens virtual object comprising a second set of one or more navigational display elements uploaded by a second submitting client device.

4. The system of claim 3, wherein the first set of one or more navigational display elements includes a first text in a first font and the second set of one or more navigational display elements includes a second text in a second font that is different from the first font.

5. The system of claim 3, wherein the first set of one or more navigational display elements includes a first visual logo and the second set of one or more navigational display elements have a second visual logo that is different than the first visual logo.

6. The system of claim 2, wherein the physical place is obfuscated in the field of view by one or more physical objects, and wherein the one or more navigational physical place display elements are overlaid over the one or more physical objects in the live video.

7. The system of claim 1, wherein the one or more sensors comprise a Global Positioning System (GPS) sensor and a compass sensor, and wherein the location data is generated by the GPS sensor and the heading data is generated by the compass sensor.

8. The system of claim 1, wherein the location data includes latitude and longitude data generated by the one or more sensors.

9. The system of claim 1, wherein the navigational direction display element is at least one of:

an arrow or a direction pointer.

10. The system of claim 7, wherein the operations that determine the heading data include operations comprising:

identifying an absolute bearing of the geographic location; and

dynamically updating the heading data using the compass sensor as the compass sensor is physically moved.

11. The system of claim 2, wherein the update indicates that the current orientation coincides with a bearing of the geographic location.

12. The system of claim 2, wherein the operations that indicate the physical place is in the live video use operations comprising:

determining a difference between the current orientation of the system and a bearing of the geographic location.

13. The system of claim 2, wherein the one or more navigational physical place display elements are dynamically displayed on frames of the live video.

14. A method comprising:

receiving, by a client device from a network Platform, a plurality of lens virtual objects corresponding to different geographic locations, each of the plurality of lens virtual objects comprising one or more navigational display elements uploaded to the network platform by submitting client devices;

generating location data using one or more sensors of the client device;

selecting a lens virtual object from the plurality of lens virtual objects based on the location data generated by the one or more sensors of the client device corresponding to a geographic location of the selected lens virtual object;

determining heading data using the geographic location of the selected lens virtual object, the heading data specifying a direction of the geographic location relative to a current orientation of the client device; and

causing a presentation, on live video generated and displayed by the client device, of a navigational direction display element in the uploaded one or more navigational display elements associated with the selected lens virtual object, the navigational direction display element indicating the direction to the geographic location relative to the current orientation of the client device.

15. The method of claim 14, further comprising:

identifying an update to the heading data that indicates a physical place is in a field of view of the live video, the update to the heading data caused in response to physical movement of the client device; and

responsive to the physical place being in the field of view of the live video, displaying one or more navigational physical place display elements of the selected lens virtual object, the one or more navigational physical place display elements indicating a location of the physical place within the live video.

16. The method of claim 14, wherein the plurality of lens virtual objects comprises a first lens virtual object and a second lens virtual object, the first lens virtual object comprising a first set of one or more navigational display elements uploaded by a first submitting client device, the second lens virtual object comprising a second set of one or more navigational display elements uploaded by a second submitting client device.

17. The method of claim 16, wherein the navigational direction display element is at least one of:

an arrow, a direction pointer.

18. The method of claim 16, wherein the first set of one or more navigational display elements includes a first visual logo and the second set of one or more navigational display elements have a second visual logo that is different than the first visual logo.

19. A machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

receive, from a network platform, a plurality of lens virtual objects corresponding to different geographic locations, each of the plurality of lens virtual objects comprising one or more navigational display elements uploaded to the network platform by submitting client devices;

generate location data using one or more sensors;

select a lens virtual object from the plurality of lens virtual objects based on the location data corresponding to a geographic location of the selected lens virtual object;

determine heading data using the geographic location of the selected lens virtual object, the heading data specifying a direction of the geographic location relative to a current orientation; and

cause a presentation, on live video generated by a camera and displayed on a display device, of a navigational direction display element in the uploaded one or more navigational display elements associated with the selected lens virtual object, the navigational direction display element indicating the direction to the geographic location relative to the current orientation.

20. The machine-readable storage device of claim 19, the operations further comprising:

identifying an update to the heading data that indicates the geographic location is in a field of view of the live video, the update the heading data caused in response to physical movement; and

responsive to the geographic location being in the field of view of the live video, displaying one or more navigational physical place display elements of the selected lens virtual object, the one or more navigational physical place display elements indicating a location of a physical place within the live video.