BARCODE VISUALIZATION IN AUGMENTED REALITY

• US 20140247278A1
• Filed: 03/01/2013
• Published: 09/04/2014
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Layar BV (Candy Capital Ltd.)
• Original Assignee: Layar BV (Candy Capital Ltd.)
• Inventors: Groten, Dirk, Samara, Anatoliy, Hofmann, Klaus Michael
• Granted Patent: US 9,946,963 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06K 17/0016 Selecting or retrieving of images by means of their associated code-marks, e.g. coded microfilm or microfiche microfilm reading G03B21/11; computers therefor G06F16/00; teaching devices G09B5/02; accessing record carriers G11B27/002; static digital stores using optical elements G11C13/04; intermediate storage of facsimile picture signals H04N1/21
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation

Abstract

Disclosed herein is an improved method for providing content associated with barcodes in augmented reality in addition or in combination with providing content associated with target objects in augmented reality. The improved method advantageously provides a augmented reality client that a user may use to view the respective content associated with barcodes and target objects while in camera view to improve usability. Advantageously, the user is not unexpectedly taken out of camera view to view the content associated with the barcode and the user experience provided is consistent between barcodes and target objects. Furthermore, the improved method integrates barcodes and a visualization of the barcode within augmented reality, without disrupting the real-time augmented reality experience.

Claims

1. A method for providing visualization of text encoded in an optically machine-readable barcode to a user in augmented reality, said augmented reality provided by a user system configured to render at least one augment for display, said at least one augment overlaying a view of the real world, the method comprising:

receiving an image frame captured by a digital imaging part of the user system;

detecting, from the image frame, whether the barcode is present in the image frame;

if a barcode is detected, rendering a first augment associated with the barcode for display using a display part of the user system based on first pose parameters of the barcode, wherein, when the first augment is displayed to a user, the first augment appears in position with the barcode;

detecting, from the image frame, whether a target object is present in the image frame, said target object not being a barcode;

if a target object is detected, rendering a second augment associated with the target object for display using the display part of the user system based on second pose parameters of the target object, wherein, when the second augment is displayed to the user, the first augment appears in position with the target object.

2. The method according to claim 1, further comprising:

if a barcode is detected;

decoding the barcode to determine the text embedded in the barcode;

constructing the first augment based on at least part of the text;

estimating, from the image frame, the first pose parameters of the barcode.

3. The method according to claim 1, wherein detecting whether a target object is present comprises:

extracting query features from the image frame, said query features suitable for object recognition and/or pose estimation;

determining whether the query features matches reference features of the target object, said reference features suitable for object recognition and/or pose estimation.

4. The method according to claim 1, further comprising:

if the target object is present;

constructing the second augment, said second augment associated with the target object;

estimating, from the query features and the reference features, the second pose parameters of the target object.

5. The method according to claim 1, wherein the barcode comprises a one-dimensional barcode or a two-dimensional barcode, or a three-dimensional barcode.

6. The method according to claim 1, wherein the barcode comprises a Quick Response (QR) code or a matrix barcode.

7. The method according to claim 2, wherein the first augment visually indicates an action performable by the user system based on the text.

8. The method according to claim 7, further comprising:

receiving user input through an input part of the user system indicating a desire to perform the action;

executing the action on the user system based on the text.

9. The method according to claim 2, wherein the first augment comprises at least a part of visual content indicated and/or directed by the text.

10. The method according to claim 2, wherein constructing the first augment based on the text comprises:

retrieving visual content indicated and/or directed by the text;

generating the first augment based on at least a part of the visual content.

11. The method according to claim 2, wherein constructing the first augment based on the text comprises:

determining the type of information contained in and/or indicated by the text;

selecting a graphics template depending on the type;

generating the first augment based on the graphics template and the at least a part of the text.

12. The method according to claim 11, wherein the graphics template is retrieved from a storage local to the user system, and the first augment is generated locally at the user system.

13. The method according to claim 11, wherein the type of information comprises at least one of the following:

contact information, calendar event information, electronic mail (e-mail) address, telephone number, uniform resource locator (URL), information having a particular content format, geographical location, network identifier, Short Message Service (SMS) messaging information.

14. The method according to claim 1, wherein the detecting step detects whether the barcode is present in the image frame from only a portion of the image frame, and the portion of the image frame is determined based on pose parameters and/or locations of position detection patterns of the barcode that are previously determined from an prior image frame captured before the image frame.

15. An augmented reality client of a user system configured to providing visualization of text encoded in an optically machine-readable barcode to a user in augmented reality, said augmented reality provided by a user system configured to render at least one augments for display, said at least one augments overlaying a view of the real world, the augmented reality client configured to:

receive an image frame captured by a digital imaging part of the user system;

detect, from the image frame, whether the barcode is present in the image frame;

if a barcode is detected, render a first augment associated with the barcode for display using a display part of the user system based on first pose parameters of the barcode, wherein, when the first augment is displayed to a user, the first augment appears in position with the barcode;

detect, from the image frame, whether a target object is present in the image frame, said target object not being a barcode;

if a target object is detected, render a second augment associated with the target object for display using the display part of the user system based on second pose parameters of the target object, wherein, when the second augment is displayed to the user, the first augment appears in position with the target object.

16. The augmented reality client according to claim 15 is further configured to:

if a barcode is detected;

decode the barcode to determine the text embedded in the barcode;

construct the first augment based on at least part of the text;

estimate, from the image frame, the first pose parameters of the barcode.

17. The augmented reality client according to claim 15 is further configured to:

determine the type of information contained in the text;

select a graphics template depending on the type;

generate the first augment based on the graphics template and the at least a part of the text.

20. A user system comprising:

a digital imaging part configured to capture one or more image frames;

a processor configured to execute functions of an augmented reality client according to claim 15; and

a display part configured to display at least the first augment and/or the second augment overlaying a view of the real world.

18. The augmented reality client according to claim 18, wherein the graphics template is retrieved from a storage local to the user system, and the first augment is generated completely locally at the user system.