System and method for providing augmented reality on mobile devices

• US 10,127,724 B2
• Filed: 01/06/2014
• Issued: 11/13/2018
• Est. Priority Date: 01/04/2013
• Status: Active Grant

• Current Assignee: Vuezr, Inc.
• Original Assignee: Vuezr, Inc.
• Inventors: Carre, Guillaume, Mutricy, Alain
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US14/759,093
• Publication Number:
  US 20150348329A1
• Time in Patent Office: 1,772 Days
• Field of Search: 345631-633, 345473
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: H04N 21/25841 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/25858 involving client software characteristics, e.g. OS identifier
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/42684 Client identification by a unique number or address, e.g. serial number, MAC address, socket ID network arrangements, protocols or services for addressing or naming H04L61/00
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/443 OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/472 : End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/4722 : for requesting additional data associated with the content
• CPC Class Codes: H04N 21/47805 : Electronic banking banking in general G06Q30/02
• CPC Class Codes: H04N 21/47815 : Electronic shopping payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12
• CPC Class Codes: H04N 21/4782 : Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4784 : receiving rewards
• CPC Class Codes: H04N 21/4786 : e-mailing message switching systems, e.g. electronic mail systems H04L51/00
• CPC Class Codes: H04N 21/4788 : communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6175 : involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/6547 : comprising parameters, e.g. for client setup
• CPC Class Codes: H04N 21/6581 : Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/816 : involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/8186 : specially adapted to be executed by a peripheral of the client device, e.g. by a reprogrammable remote control
• CPC Class Codes: H04N 21/8586 : by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

A system and method for providing augmented reality on a mobile device is herein disclosed. According to one embodiment, the computer-implemented method includes providing a targeting advice area in a camera preview of an application running on a user device and recognizing a target using the targeting advice area. The computer-implemented method further provides an event via the camera preview based on a target recognition.

Claims

1. A computer-implemented method, comprising:

providing a targeting advice area in a camera preview of an application running on a user device, the user device including one or more of a mobile phone and a tablet;

providing user interface suggestions in the targeting advice area to direct the user device to a target, wherein the user interface suggestions include one or more of directions, an arrow, symbols, text, pictures, diagrams, movies, and maps;

recognizing the target using the targeting advice area; and

providing content via the camera preview based on a target recognition, the content includes an animated character that behaves in response to a user'"'"'s interaction with the user device, the user'"'"'s interaction includes aiming at a live target in a real scene and providing a voice command.

2. The computer-implemented method of claim 1, wherein the target is a computer code selected from a group comprising a bar code, a matrix code, a flash code, a data matrix code, a quick response (QR) code, a MICROSOFT®

, code, and a proprietary code.

3. The computer-implemented method of claim 1, further comprising sending QR code information associated with the target to a server based on a user'"'"'s action as if a QR code has been scanned.

4. The computer-implemented method of claim 1, further comprising:

recognizing a uniform resource locator (URL) associated with the target;

making a web request to a server based on the URL; and

receiving an augmented reality event from the server based on the URL.

5. The computer-implemented method of claim 4, further comprising:

receiving a ghost image from the server, and

displaying the ghost image in a transparency layer of a camera preview to guide the user to aim at the target.

6. The computer-implemented method of claim 5, wherein the transparency layer for the camera preview includes a button or a user instruction.

7. The computer-implemented method of claim 5, wherein the ghost image is selected from a group comprising an outline of an object, a silhouette, a local environment view.

8. The computer-implemented method of claim 4, further comprising determining that the URL is recognizable by the application.

9. The computer-implemented method of claim 4, further comprising receiving an error message from the server if the URL is not recognizable by the server.

10. The computer-implemented method of claim 1, further comprising displaying a notification based on a location of the user device.

11. The computer-implemented method of claim 10, wherein the notification is triggered by the application if the user opens the application or the application runs in a wake-mode in an OS background of the user device.

12. The computer-implemented method of claim 10, wherein the location of the user device is provided by one or more of a GPS position data, Wi-Fi data, BTLE data, RFID or NFC tagging, audio signal, e-compass data, an accelerometer signal, and a gyroscope signal.

13. The computer-implemented method of claim 10, further comprising displaying the notification when the application is launched at the location.

14. The computer-implemented method of claim 13, wherein the application stores a list of locations to determine to display the notification.

15. The computer-implemented method of claim 1, wherein the user'"'"'s interaction includes aiming at a target, a selection of a button, and a motion with the user device.

16. The computer-implemented method of claim 1, wherein the content is changed and refreshed depending on a status of the user device before, during, and after an augmented reality event.

17. The computer-implemented method of claim 1, further comprising:

identifying a reference image of the target;

determining a projection plane of the reference image in the camera preview by calculating a position, an orientation, and scaling of the reference image;

resizing the targeting advice area on the projection plane; and

projecting the content on the projection plane.

18. The computer-implemented method of claim 1, wherein the event is an augmented reality (AR) event.

19. The computer-implemented method of claim 18, wherein the AR event is selected from a group comprising a video, an audio, an instruction for a user action, web, a URL, a message, a telephone call, a logo, marketing material, a coupon, a purchase order, and a payment.

20. The computer-implemented method of claim 18, further comprising streaming a video of the augmented reality event from a URL of the target.

21. The computer-implemented method of claim 18, wherein the AR event includes a transparency video displayed in a transparency layer of a camera preview to provide an augmented reality over the real scene.

22. The computer-implemented method of claim 21, wherein the transparency video is selected from a group comprising a green chromakey, a chromakey movie, alpha left transparency, and an alpha movie.

23. A non-transitory computer readable medium containing computer-readable instructions stored therein for causing a computer processor to perform operations to:

provide a targeting advice area in a camera preview of an application running on a user device, the user device including one or more of a mobile phone and a tablet;

providing user interface suggestions in the targeting advice area to direct the user device to a target, wherein the user interface suggestions include one or more of directions, an arrow, symbols, text, pictures, diagrams, movies, and maps;

recognizing the target using the targeting advice area; and

provide content via the camera preview based on a target recognition, the content includes an animated character that behaves in response to a user'"'"'s interaction with the user device, the user'"'"'s interaction includes aiming at a live target in a real scene and providing a voice command.

24. The non-transitory computer readable medium of claim 23, wherein the computer-readable instructions stored therein for causing the computer processor to perform operations further to:

recognize a uniform resource locator (URL) associated with the target;

make a web request to a server based on the URL; and

receive an augmented reality event from the server based on the URL.