Presenting an augmented reality within a custom graphic

• US 10,607,053 B1
• Filed: 06/21/2019
• Issued: 03/31/2020
• Est. Priority Date: 09/12/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Boyd, Nathan Kenneth, Luc, Christopher Jonathan, Ouimet, Kirk, Sheth, Rahul Bhupendra
• Primary Examiner(s): Gudorf, Laura A
• Application Number:
  US16/448,796
• Time in Patent Office: 284 Days
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06K 19/06037 multi-dimensional coding
• CPC Class Codes: G06K 19/06103 the marking being embedded in a human recognizable image, e.g. a company logo with an embedded two-dimensional code
• CPC Class Codes: G06K 19/06112 the marking being simulated using a light source, e.g. a barcode shown on a display or a laser beam with time-varying intensity profile
• CPC Class Codes: G06K 7/1417 2D bar codes
• CPC Class Codes: G06K 7/1443 locating of the code in an image
• CPC Class Codes: G06K 7/1447 extracting optical codes from image or text carrying said optical code
• CPC Class Codes: G06K 7/1456 determining the orientation of the optical code with respect to the reader and correcting therefore
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Systems and methods for presenting an augmented reality within a custom graphic are provided. In example embodiments, one or more hardware processors may receive an image, identify candidate shape features in the image, identify a custom graphic by comparing the candidate shape features with a reference shape feature of the custom graphic, identify an optical barcode based on the identified custom graphic, identify an augmented reality based on the optical barcode, and transmit the augmented reality to the client device.

Claims

1. A method comprising:

identifying, by one or more hardware processors, shape features from an image received from a client device;

analyzing, by the one or more hardware processors, a first shape feature of the shape features;

determining, by the one or more hardware processors, based on analyzing the first shape feature, whether the first shape feature satisfies a first rule;

in response to determining that the first shape feature fails to satisfy the first rule, analyzing, by the one or more hardware processors, a second shape feature of the shape features;

determining, by the one or more hardware processors, based on analyzing the second shape feature, whether the second shape feature satisfies a second rule;

identifying, by the one or more hardware processors, a custom graphic in response to determining that the second shape feature satisfies the second rule;

identifying, by the one or more hardware processors, an optical barcode based on the identified custom graphic; and

transmitting, by the one or more hardware processors, a media item to the client device that corresponds to the identified optical barcode.

2. The method of claim 1, wherein the first rule comprises an area rule, and further comprising:

comparing an area value represented by the first shape feature with a reference area of the area rule;

computing an area score based on comparing the area value with the reference area; and

determining that the first shape feature fails to satisfy the first rule when the area score fails to exceed an area threshold.

3. The method of claim 1, wherein the second rule comprises a path rule, and further comprising:

generating a path having a starting point and an ending point based on the second shape feature;

determining that the starting point and the ending point are the same; and

determining that the second shape feature satisfies the second rule in response to determining that the starting point and the ending point are the same.

4. The method of claim 1, further comprising:

identifying a user account based on the optical barcode; and

receiving an indication that the image including the optical barcode was selected at the client device.

5. The method of claim 1, further comprising determining an orientation of the custom graphic based on the image, wherein the identification of the optical barcode is based on the orientation.

6. The method of claim 1, further comprising causing the media item to be displayed within the custom graphic.

7. The method of claim 1, wherein the media item comprises augmented reality, and further comprising:

storing the augmented reality in conjunction with a user account on a server;

determining that the user account, identified based on the optical barcode, is associated with the augmented reality;

retrieving the augmented reality in response to determining that the user account, identified based on the optical barcode, is associated with the augmented reality; and

transmitting, to the client device, the retrieved augmented reality in response to determining that the user account, identified based on the optical barcode, is associated with the augmented reality.

8. The method of claim 1, further comprising:

generating a low resolution version of the image;

modifying the low resolution version of the image;

extracting the shape features from the modified low resolution version of the image; and

generating a high resolution version of a particular portion of the image corresponding to the extracted shape features.

9. The method of claim 1, wherein the custom graphic is included in a portion of the image and comprises a first region bounded by a second region, wherein the optical barcode is in a portion of the custom graphic between the first and second regions and is excluded from the first region, and wherein the first region includes visual content visible in part of the image received from the client device.

10. The method of claim 1, further comprising aligning the custom graphic based on a distinctive feature of the custom graphic.

11. The method of claim 1, further comprising displaying at least a portion of the image external to the custom graphic while displaying the media item within the custom graphic of the image.

12. A non-transitory machine-readable medium comprising instructions which, when executed by one or more hardware processors of a device, cause the device to implement operations comprising:

identifying shape features from an image received from a client device;

analyzing a first shape feature of the shape features;

determining based on analyzing the first shape feature, whether the first shape feature satisfies a first rule;

in response to determining that the first shape feature fails to satisfy the first rule, analyzing a second shape feature of the shape features;

determining, based on analyzing the second shape feature, whether the second shape feature satisfies a second rule;

identifying a custom graphic in response to determining that the second shape feature satisfies the second rule;

identifying an optical barcode based on the identified custom graphic; and

transmitting a media item to the client device that corresponds to the identified optical barcode.

13. The non-transitory machine-readable medium of claim 12, wherein the first rule comprises an area rule, and the operations further comprising:

comparing an area value represented by the first shape feature with a reference area of the area rule;

computing an area score based on comparing the area value with the reference area; and

determining that the first shape feature fails to satisfy the first rule when the area score fails to exceed an area threshold.

14. The non-transitory machine-readable medium of claim 12, wherein the second rule comprises a path rule, and the operations further comprising:

generating a path having a starting point and an ending point based on the second shape feature;

determining that the starting point and the ending point are the same; and

determining that the second shape feature satisfies the second rule in response to determining that the starting point and the ending point are the same.

15. The non-transitory machine-readable medium of claim 12, the operations further comprising:

identifying a user account based on the optical barcode; and

receiving an indication that the image including the optical barcode was selected at the client device.

16. The non-transitory machine-readable medium of claim 12, the operations determining an orientation of the custom graphic based on the image, and wherein the identification of the optical barcode is based on the orientation.

17. The non-transitory machine-readable medium of claim 12, the operations further comprising:

generating a low resolution version of the image;

modifying the low resolution version of the image;

extracting the shape features from the modified low resolution version of the image; and

generating a high resolution version of a particular portion of the image corresponding to the extracted shape features.

18. The non-transitory machine-readable medium of claim 12, wherein the custom graphic is included in a portion of the image and comprises a first region bounded by a second region, wherein the optical barcode is in a portion of the custom graphic between the first and second regions and is excluded from the first region, and wherein the first region includes visual content visible in part of the image received from the client device.

19. The non-transitory machine-readable medium of claim 12, the operations further comprising aligning the custom graphic based on a distinctive feature of the custom graphic.

20. A system comprising:

one or more hardware processors; and

a memory storing instructions which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising;

identifying shape features from an image received from a client device;

analyzing a first shape feature of the shape features;

determining based on analyzing the first shape feature, whether the first shape feature satisfies a first rule;

in response to determining that the first shape feature fails to satisfy the first rule, analyzing a second shape feature of the shape features;

determining, based on analyzing the second shape feature, whether the second shape feature satisfies a second rule;

identifying a custom graphic in response to determining that the second shape feature satisfies the second rule;

identifying an optical barcode based on the identified custom graphic; and

transmitting a media item to the client device that corresponds to the identified optical barcode.