Augmented reality content selection and display based on printed objects having security features

• US 10,699,124 B1
• Filed: 09/09/2019
• Issued: 06/30/2020
• Est. Priority Date: 09/09/2019
• Status: Active Grant

• Current Assignee: Techinvest Company Limited
• Original Assignee: AR
• Inventors: Herzberg, Stefan W., Herzberg, Megan
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US16/565,234
• Time in Patent Office: 295 Days
• Field of Search: None
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Systems, methods and techniques for automatically recognizing two-dimensional real world objects with an augmented reality display device, and augmenting or enhancing the display of such real world objects by superimposing virtual images such as a still or video advertisement, a story or other virtual image presentation. In non-limiting embodiments, the real world object includes visible features including visible security features and a recognition process takes the visible security features into account when recognizing the object and/or displaying superimposed virtual images.

Claims

1. A method of presenting image information to a user comprising:

enabling capture of an image of a printed item including at least one visible security feature encoding a value;

displaying the captured image;

recognizing the captured image using at least one of color and grey scale information within the captured image wherein the recognizing does not require recognizing the value encoded by the at least one visible security feature;

selecting a media item based at least in part on the recognition; and

using augmented reality to superimpose the selected media item onto the displayed image as if it were part of the printed item,

wherein the enabling capture comprises downloading the image from social media over a digital network.

2. The method of claim 1 wherein the recognizing comprises recognizing the image within a predefined recognition area that excludes the at least one visible security feature.

3. The method of claim 1 wherein the recognizing comprises recognizing the image within a predefined recognition area that includes the at least one visible security feature.

4. The method of claim 1 wherein the recognizing takes the at least one visible security feature into account but not the value the at least one visible security feature encodes.

5. A method of presenting image information to a user comprising:

enabling capture of an image of a printed item including at least one visible security feature encoding a value;

displaying the captured image;

recognizing the captured image using at least one of color and grey scale information within the captured image wherein the recognizing does not require recognizing the value encoded by the at least one visible security feature;

selecting a media item based at least in part on the recognition; and

using augmented reality to superimpose the selected media item onto the displayed image as if it were part of the printed item,

wherein the printed item comprises a banknote or other government issued document, and the at least one visible security feature includes at least three items selected from the group consisting of holograms, woven three-dimensional security ribbon(s), distinctive image pattern(s) such as the EURion constellation, glossy stripe(s) that reveal additional information when viewed on an angle, watermark(s), raised printing, ultraviolet ink which reveals additional image information when viewed under ultraviolet light, thermochromatic ink the color of which is dependent on temperature, color changing or color shifting ink, see through indicia with complementary front and obverse portions, microprinting, unique printing fonts, embedded security threads, perforations, plastic, cotton fiber or other material matted or other textured surface(s), foil stamps, photographs including clear window photographs, serial numbers and barcodes.

6. A method of providing a political message comprising:

pointing a camera at a billboard, flyer or other representation of a political campaign;

capturing and displaying an image;

uploading indicia of the captured image to a server;

recognizing the billboard, flyer or other representation of the political campaign based on the uploaded indicia; and

providing an augmented reality overlay for the displayed image, the overlay providing a message associated with the political campaign.

7. The method of claim 6 wherein the camera is disposed on a wireless wearable display device comprising a wearable display, the displaying displays the captured image on the wearable display, and the providing provides an augmented reality display on the wearable display.

8. The method of claim 7 wherein the wearable display device comprises wirelessly-connected augmented reality enabled glasses.

9. The method of claim 6 wherein the recognizing comprises recognizing a predefined recognition area using at least one color and grey scale information.

10. The method of claim 6 wherein uploading indicia comprises uploading a compressed format including coordinates of pattern features.

11. A system for presenting image information to a user comprising:

enabling a smart device to capture an image of a printed item including at least one visible security feature encoding a value and to display the captured image;

at least one processor configured to recognize the captured image based on at least one of color and grey scale information wherein the recognition does not require recognizing the encoded value; and

in response to the recognized captured image, enabling the smart device to use augmented reality to superimpose a media item selected based at least in part on the recognized captured image onto the displayed image as if it were part of the printed item,

wherein the printed item comprises a banknote or other government issued document, and the at least one visible security feature includes at least three items selected from the group consisting of holograms, woven three-dimensional security ribbon(s), distinctive image pattern(s) such as the EURion constellation, glossy stripe(s) that reveal additional information when viewed on an angle, watermark(s), raised printing, ultraviolet ink which reveals additional image information when viewed under ultraviolet light, thermochromatic ink the color of which is dependent on temperature, color changing or color shifting ink, see through indicia with complementary front and obverse portions, microprinting, unique printing fonts, embedded security threads, perforations, plastic, cotton fiber or other material matted or other textured surface(s), foil stamps, photographs including clear window photographs, serial numbers and barcodes.

12. The system of claim 11 wherein the recognizing comprises recognizing the image within a predefined recognition area that excludes the at least one visible security feature.

13. The system of claim 11 wherein the recognizing comprises recognizing the image within a predefined recognition area that includes the at least one visible security feature.

14. The system of claim 11 wherein the recognizing takes the visible security feature into account but not the value the at least one visible security feature encodes.

15. The system of claim 11 wherein the smart device comprises a handheld device.

16. The system of claim 11 wherein the smart device comprises a wearable device.

17. A method of operating a portable or handheld display device comprising:

in response to capture and/or display of an image, initiating an operation to recognize the image based on at least one of color and grey scale information, the operation to recognize not requiring recognition of an encoded security value within the image;

upon recognition of the image, displaying a virtual media item superimposed onto the image, the media item including controls that enable a user to control at least some aspects of presentation of the media item; and

recognizing denomination of a banknote without requiring recognition of a serial number of the banknote.

18. The method of claim 17 wherein the controls enable the user to control video playback sequence.

19. The method of claim 17 wherein the controls enable the user to control audio playback.

20. The method of claim 17 wherein upon the user controlling the display device to capture and/or display a changed perspective of the image, the superimposing controls the display device to display the virtual media item from the changed perspective.

21. The method of claim 17 further including selecting the media item based on recognizing a predefined recognition area of the image.

22. The method of claim 17 wherein the image of the banknote is an image of a copy of the banknote.

23. The method of claim 17 wherein the banknote comprises a counterfeit or specimen of a banknote.

24. The method of claim 17 wherein the banknote comprises an inauthentic banknote.

25. The method of claim 17 wherein the display device comprises a smartphone.

26. The method of claim 17 wherein the display device comprises augmented reality enabled glasses.

27. A system for presenting image information to a user comprising:

enabling a smart device to capture an image of a printed item including at least one visible security feature encoding a value and to display the captured image;

at least one processor configured to recognize the captured image based on at least one of color and grey scale information wherein the recognition does not require recognizing the encoded value; and

in response to the recognized captured image, enabling the smart device to use augmented reality to superimpose a media item selected based at least in part on the recognized captured image onto the displayed image as if it were part of the printed item,

wherein the smart device comprises a wearable device,

wherein the wearable device comprises wirelessly-connected augmented reality enabled glasses.