Presenting an augmented reality within a custom graphic

• US 9,922,226 B1
• Filed: 09/12/2016
• Issued: 03/20/2018
• Est. Priority Date: 09/12/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Boyd, Nathan Kenneth, Luc, Christopher Jonathan, Ouimet, Kirk, Sheth, Rahul Bhupendra
• Primary Examiner(s): Gudorf, Laura
• Application Number:
  US15/262,942
• Time in Patent Office: 554 Days
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06K 19/06037 multi-dimensional coding
• CPC Class Codes: G06K 19/06103 the marking being embedded in a human recognizable image, e.g. a company logo with an embedded two-dimensional code
• CPC Class Codes: G06K 19/06112 the marking being simulated using a light source, e.g. a barcode shown on a display or a laser beam with time-varying intensity profile
• CPC Class Codes: G06K 7/1417 2D bar codes
• CPC Class Codes: G06K 7/1443 locating of the code in an image
• CPC Class Codes: G06K 7/1447 extracting optical codes from image or text carrying said optical code
• CPC Class Codes: G06K 7/1456 determining the orientation of the optical code with respect to the reader and correcting therefore
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Systems and methods for presenting an augmented reality within a custom graphic are provided. In example embodiments, a server receives a scanned image from a client device, the scanned image including a custom graphic and an optical barcode. The server identifies an augmented reality and a user interface item based on the optical barcode. The server transmits the augmented reality to the client device for display, at the client device, within a custom graphic of the scanned image. The server receives an indication that the scanned image was selected at the client device. The server transmits, to the client device, the user interface item for display at the client device.

Claims

1. A method comprising:

receiving, at a server, a scanned image from a client device;

extracting candidate shape features from the scanned image,

identifying a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic;

identifying an optical barcode based on the custom graphic;

identifying an augmented reality and a user interface item based on the optical barcode;

transmitting the augmented reality to the client device for display, at the client device, within the custom graphic of the scanned image;

receiving, at the server, an indication that the scanned image was selected at the client device; and

transmitting, to the client device, the user interface item for display at the client device.

2. The method of claim 1, further comprising determining an orientation of the optical barcode using the scanned image.

3. The method of claim 1, wherein the augmented reality comprises a video to be displayed within the custom graphic.

4. The method of claim 1, wherein the optical barcode identifies a user account of an instant messaging application, wherein the augmented reality is generated via the user account of the instant messaging application.

5. The method of claim 4, wherein the user interface item comprises an interface for connecting with the user account via the instant messaging application.

6. The method of claim 4, wherein the user interface item comprises a link to a webpage associated with or provided by the user account.

7. The method of claim 1, wherein the scanned image includes a predetermined shape, and wherein the custom graphic and the optical barcode reside within the predetermined shape.

8. The method of claim 1, the augmented reality being transmitted to the client device for display, at the client device, within the custom graphic of the scanned image, while the client device continues to display at least a portion of the scanned image external to the custom graphic.

9. The method of claim 8, wherein the portion of the scanned image comprises the optical barcode.

21. The method of claim 1, further comprising comparing the candidate shape features with multiple custom graphics to determine whether each of the candidate shape features match any of the multiple custom graphics.

22. The method of claim 1, further comprising searching for both the custom graphic and an outer box surrounding the custom graphic, and identifying the custom graphic based on the combination of the custom graphic and the outer box.

23. The method of claim 1, further comprising:

determining, for each of the candidate shape features, a probability that the candidate shape feature is included in a predetermined set of shape features for the custom graphic; and

excluding candidate shape features with a probability below a threshold from the comparison.

24. The method of claim 23, further comprising:

determining whether one or more of the candidate shape features include an enclosed line; and

excluding the candidate shape features that do not include the enclosed line from the comparison.

25. The method of claim 1, further comprising identifying a user account based on the optical barcode, wherein the user interface item is configured to include a button for adding the user of the user account as a friend based on the user account and the optical barcode.

10. A non-transitory machine-readable medium comprising instructions which, when executed by one or more hardware processors of a client device, cause the client device to implement operations comprising:

scanning an image;

determining that the scanned image including a custom graphic and an optical barcode by;

extracting candidate shape features from the scanned image,

identifying the custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic, and

identifying the optical barcode based on the custom graphic;

transmitting the scanned image to a server;

receiving, from the server and in response to the scanned image, an augmented reality for display within the custom graphic of the scanned image;

providing for display of the augmented reality within the custom graphic of the scanned image;

receiving a user selection of the scanned image;

transmitting, to the server, an indication that the scanned image was selected;

receiving, from the server, a user interface item in response to the indication that the scanned image was selected; and

providing for display of the user interface item.

11. The machine-readable medium of claim 10, wherein the scanned image includes a predetermined shape, wherein the custom graphic and the optical barcode reside within the predetermined shape, and wherein receiving the selection of the scanned image comprises receiving a selection of the predetermined shape within the scanned image.

12. The machine-readable medium of claim 10, the operations further comprising:

receiving a selection of the user interface item; and

carrying out an action corresponding to the selection of the user interface item.

13. The machine-readable medium of claim 10, wherein the image is scanned using an instant messaging application of the client device, and wherein the optical barcode identifies a user account of the instant messaging application.

14. The machine-readable medium of claim 13, wherein the augmented reality comprises a video generated via the user account of the instant messaging application.

15. The machine-readable medium of claim 13, wherein the user interface item comprises an interface for connecting with the user account via the instant messaging application.

16. The machine-readable medium of claim 13, wherein the user interface item comprises a link to a webpage associated with or provided by the user account.

17. The machine-readable medium of claim 10, the operations further comprising:

providing for display of at least a portion of the scanned image external to the custom graphic while providing for display of the augmented reality within the custom graphic of the scanned image.

18. The machine-readable medium of claim 17, wherein the portion of the scanned image comprises the optical barcode.

19. A system comprising:

one or more hardware processors; and

a memory storing instructions which, when executed, cause the one or more hardware processors, to implement operations comprising;

receiving a scanned image from a client device;

extracting candidate shape features from the scanned image;

identifying a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic;

identifying an optical barcode based on the custom graphic;

identifying an augmented reality and a user interface item based on the optical barcode;

transmitting the augmented reality to the client device for display at the client device within the custom graphic of the scanned image;

receiving an indication that the scanned image was selected at the client device; and

transmitting, to the client device, the user interface item for display at the client device.

20. The system of claim 19, the operations further comprising determining an orientation of the optical barcode using the custom graphic.