Presenting an augmented reality within a custom graphic

• US 10,380,394 B1
• Filed: 02/09/2018
• Issued: 08/13/2019
• Est. Priority Date: 09/12/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Boyd, Nathan Kenneth, Luc, Christopher Jonathan, Ouimet, Kirk, Sheth, Rahul Bhupendra
• Primary Examiner(s): Gudorf, Laura A
• Application Number:
  US15/893,165
• Time in Patent Office: 550 Days
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06K 19/06037 multi-dimensional coding
• CPC Class Codes: G06K 19/06103 the marking being embedded in a human recognizable image, e.g. a company logo with an embedded two-dimensional code
• CPC Class Codes: G06K 19/06112 the marking being simulated using a light source, e.g. a barcode shown on a display or a laser beam with time-varying intensity profile
• CPC Class Codes: G06K 7/1417 2D bar codes
• CPC Class Codes: G06K 7/1443 locating of the code in an image
• CPC Class Codes: G06K 7/1447 extracting optical codes from image or text carrying said optical code
• CPC Class Codes: G06K 7/1456 determining the orientation of the optical code with respect to the reader and correcting therefore
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Systems and methods for presenting an augmented reality within a custom graphic are provided. In example embodiments, one or more hardware processors may receive an image, identify candidate shape features in the image, identify a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic, identify an optical barcode based on the identified custom graphic, identify an augmented reality based on the optical barcode, and transmit the augmented reality to the client device.

Claims

1. A method comprising:

receiving, by one or more hardware processors, an image from a client device;

identifying, by the one or more hardware processors, candidate shape features from the image;

identifying, by the one or more hardware processors, a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic, the comparing comprising;

computing an area score by comparing a first candidate shape feature of the candidate shape features that represents an area value with a reference area of the reference shape;

determining that the area score fails to exceed an area rule threshold; and

analyzing a second candidate shape feature of the candidate shape features to identify the custom graphic in response to determining that the area score fails to exceed the area rule threshold; and

identifying, by the one or more hardware processors, an optical barcode based on the identified custom graphic;

identifying, by the one or more hardware processors, an augmented reality based on the optical barcode; and

transmitting, by the one or more hardware processors, the augmented reality to the client device

identifying, by the one or more hardware processors, a user account based on the optical barcode;

receiving, by the one or more hardware processors, an indication that the image including the optical barcode was selected at the client device; and

transmitting, to the client device, a user interface item associated with the user account in response to the indication.

2. The method of claim 1, further comprising determining an orientation of the custom graphic based on the image, wherein the identification of the optical barcode is based on the orientation.

3. The method of claim 1, further comprising generating the augmented reality to include a video, the video configured to be displayed within the custom graphic.

4. The method of claim 1, further comprising:

generating the augmented reality via the identified user account;

storing the augmented reality in conjunction with the user account on a server;

determining that the user account, identified based on the optical barcode, is associated with the augmented reality;

retrieving the augmented reality in response to determining that the user account, identified based on the optical barcode, is associated with the augmented reality; and

transmitting, to the client device from the server, the retrieved augmented reality in response to determining that the user account, identified based on the optical barcode, is associated with the augmented reality.

5. The method of claim 4, further comprising generating a user interface item to include an interface for connecting with the user account via an instant messaging application.

6. The method of claim 5, further comprising generating the user interface item to include a link to a webpage associated with or provided by the user account.

7. The method of claim 1, the augmented reality being transmitted to the client device for display within the custom graphic of the image.

8. The method of claim 1 further comprising:

generating a low resolution version of the image;

modifying the low resolution version of the image;

extracting the candidate shape features from the modified low resolution version of the image; and

generating a high resolution version of a particular portion of the image corresponding to the extracted candidate shape features.

9. The method of claim 1, wherein the custom graphic is included in a portion of the image and comprises a first region bounded by a second region, wherein the optical barcode is in a portion of the custom graphic between the first and second regions and is excluded from the first region, and wherein the first region includes visual content visible that is part of the image received from the client device.

10. The method of claim 1 further comprising aligning the custom graphic based on a distinctive feature of the custom graphic.

11. A non-transitory machine-readable medium comprising instructions which, when executed by one or more hardware processors of a client device, cause the client device to implement operations comprising:

identifying candidate shape features in an image;

identifying a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic, the comparing comprising;

computing an area score by comparing a first candidate shape feature of the candidate shape features that represents an area value with a reference area of the reference shape;

determining that the area score fails to exceed an area rule threshold; and

analyzing a second candidate shape feature of the candidate shape features to identify the custom graphic in response to determining that the area score fails to exceed the area rule threshold; and

identifying an optical barcode based on the identified custom graphic;

transmitting the image to a server based on identification of the optical barcode;

receiving, from the server, an augmented reality associated with the optical barcode; and

displaying the augmented reality on an electronic display of the client device;

transmitting an indication to the server that the image including the optical barcode was selected;

receiving, from the server, a user interface item associated with a user account associated with the optical barcode in response to the transmitting of the indication; and

displaying the user interface item on the electronic display.

12. The machine-readable medium of claim 11, further comprising receiving an input indicating the selection was of a predetermined shape within the image.

13. The machine-readable medium of claim 11, the operations further comprising:

receiving a selection of the user interface item; and

carrying out an action corresponding to the selection of the user interface item.

14. The machine-readable medium of claim 11, the operations further comprising capturing the image via a camera of the client device.

15. The machine-readable medium of claim 14, the operations further comprising:

identifying a user account of an instant messaging application based on the optical barcode; and

capturing the image via the instant messaging application of the client device.

16. The machine-readable medium of claim 15, the operations further comprising playing a video indicated by the augmented reality.

17. The machine-readable medium of claim 16, wherein the user interface item comprises an interface for connecting with the user account via the instant messaging application, the operations further comprising connecting with the user account of the instant messaging application via the interface, and playing the video based on the connection.

18. The machine-readable medium of claim 17, wherein the user interface item comprises a link to a webpage associated with or provided by the user account, and wherein the connecting with the user account is based on the link.

19. The machine-readable medium of claim 18, the operations further comprising displaying at least a portion of the image external to the custom graphic while displaying the augmented reality within the custom graphic of the image.

20. The machine-readable medium of claim 19, wherein the portion of the image comprises the optical barcode.

21. A system comprising:

one or more hardware processors; and

a memory storing instructions which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising;

identifying candidate shape features from an image;

identifying a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic, the comparing comprising;

computing an area score by comparing a first candidate shape feature of the candidate shape features that represents an area value with a reference area of the reference shape;

determining that the area score fails to exceed an area rule threshold; and

analyzing a second candidate shape feature of the candidate shape features to identify the custom graphic in response to determining that the area score fails to exceed the area rule threshold; and

identifying an optical barcode based on the identified custom graphic;

identifying an augmented reality based on the optical barcode; and

transmitting, by the one or more hardware processors, the augmented reality to the client device;

identifying, by the one or more hardware processors, a user account based on the optical barcode;

receiving, by the one or more hardware processors, an indication that the image including the optical barcode was selected at the client device; and

transmitting, to the client device, a user interface item associated with the user account in response to the indication.