COMPUTER-VISION BASED AUGMENTED REALITY SYSTEM

• US 20150070347A1
• Filed: 08/18/2011
• Published: 03/12/2015
• Est. Priority Date: 08/18/2011
• Status: Abandoned Application

• Current Assignee: Layar BV (Candy Capital Ltd.)
• Original Assignee: Layar BV (Candy Capital Ltd.)
• Inventors: Hofmann, Klaus Michael, Van Der Lingen, Ronald
• Application Number:
  US14/239,190
• Publication Number:
  US 20150070347A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/18 Commands or executable codes

Abstract

Methods for providing a graphical user interface through an augmented reality service provisioning system. A panel is used as a template to enable content providers to provide configurations for a customizable graphical user interface. The graphical user interface is displayable in perspective with objects in augmented reality through the use of computer vision techniques.

Claims

1. A method for generating an augmented reality content item on a user device comprising a digital imaging part, a display output, a user input part and an augmented reality client, said client comprising a computer-vision based tracker for tracking an object in said display on the basis of at least an image of the object from the digital imaging part, said method comprising:

receiving an object identifier associated with an object in an image;

on the basis of said object identifier retrieving panel data from a panel database and tracking resources from a tracking resources database, said panel data comprising at least location information for retrieving a content item;

on the basis of said tracking resources said computer-vision based tracker generating three-dimensional pose information associated with said object;

on the basis of said panel data requesting at least part of said content item; and

,

on the basis of said three-dimensional pose information rendering said content item for display in the display output such that the content rendered matches the three-dimensional pose of said object in the display output.

3. The method according to claim 1, further comprising:

receiving an image frame from a digital imaging device of the augmented reality system;

transmitting the image frame to an object recognition system;

receiving, in response to transmitting the image frame, identification information for the tracked object from an object recognition system if the transmitted image frame matches the tracked object; and

storing the identification information for the tracked object as state data in the tracker.

4. The method according to claim 1, further comprising:

receiving, at the tracker, an image frame from a camera of the augmented reality system;

estimating, in the tracker, the three-dimensional pose of the tracked object from at least the image frame; and

storing the estimated three-dimensional pose of the tracked object as state data in the tracker.

5. The method according to claim 4, wherein estimating the three-dimensional pose of the tracked object from at least the image frame comprises:

obtaining reference features from a reference features database based on the identification information in the state data;

extracting candidate features from the image frame;

searching for a match between the candidate features and reference features, said reference features associated with the tracked object in the image frame;

estimating a two-dimensional translation of the tracked object in the image frame in response to a finding a match from searching for the match between candidate and reference features;

estimating a three-dimensional pose of the tracked object in the image frame based at least in part on the camera parameters and the estimated two-dimensional translation of the tracked object.

6. The method according to claim 1, wherein said three-dimensional pose information is generated using homogeneous transformation matrix H and a homogeneous camera projection matrix P, said homogeneous transformation matrix H comprising rotation and translation information associated with the camera relative to the object and said homogeneous camera projection matrix defining the relation between the coordinates associated with the three-dimensional world and the two-dimensional image coordinates.

12. The method according to claim 1, wherein said panel data further comprise:

the user interactivity configuration information comprises a function for displaying at least part of the backside of an augmented reality content item or an augmented reality graphical user interface in response to receiving a first user input interacting;

location information comprising instructions for fetching a further content item and/or a further graphical user interface associated with the backside of said augmented reality content item or said augmented reality graphical user interface;

said method further comprising;

on the basis of said content layout information, said user interactivity configuration information, said location information and said three-dimensional pose information, rendering a further graphical user interface for display in the display output such that said further graphical user interface rendered matches the three-dimensional pose of said object in the display output.

13. The method according to any claim 12, wherein said panel database and said tracking resources database are hosted on one or more servers, and wherein said augmented reality client is configured to communicate with said one or more servers.

2. A method for generating an augmented reality graphical user interface on a user device comprising a digital imaging part, a display output, a user input part and an augmented reality client, said client comprising a computer-vision based tracker for tracking an object in said display on the basis of at least an image of the object from the digital imaging part, said method comprising:

receiving an object identifier associated with an object in an image;

on the basis of said object identifier retrieving panel data from a panel database and tracking resources from a tracking resources database, said panel data comprising at least location information for retrieving a content item and user interactivity configuration information, said content item and said user interactivity information defining a graphical user interface;

on the basis of said tracking resources said computer-vision based tracker generating three-dimensional pose information associated with said object;

on the basis of said panel data, requesting at least part of said content item; and

,

on the basis of said user interactivity configuration information and said three-dimensional pose information, rendering said graphical user interface for display in the display output such that the graphical user interface rendered matches the three-dimensional pose of said object in the display output.

7. The method according to claim 2, wherein content layout data comprises visual attributes for elements of the graphical user interface.

8. The method according to claim 2, wherein the user interactivity configuration data comprises at least one user input event variable and at least one function defining an action to be performed responsive to a value of the user input event variable.

9. The method according to any claim 2, further comprising:

receiving a first user input interacting with the graphical user interface;

retrieving a further content item on the basis of said location information in said panel data, said further content item and said user interactivity configuration information defining a further graphical user interface;

on the basis of said user interactivity configuration information and said three-dimensional pose information, rendering said further graphical user interface for display in the display output such that the graphical user interface rendered matches the three-dimensional pose of said object in the display output.

10. The method according to claim 2, wherein said three-dimensional pose information is generated using a homogeneous transformation matrix H, said homogeneous transformation matrix H comprising rotation and translation information of the camera relative to the object, said method further comprising:

receiving a first user input interacting with said graphical user interface for generating a further graphical user interface;

providing a further a second homogeneous transformation matrix H′

only comprising a static translation component;

generating further three-dimensional pose information on the bases of said homogeneous transformation matrix H′

;

on the basis of said user interactivity configuration information and said further three-dimensional pose information, rendering said further graphical user interface for display in the display output such that said further graphical user interface rendered is detached from the three-dimensional pose of said object in the display output and positioned at a fixed distance behind the camera.

11. The method according to claim 2, wherein said panel data further comprise:

content layout information for specifying the display of a subset of content items from a plurality of content items in a predetermined spatial arrangement;

user interactivity configuration information comprises a function for displaying a next subset of content items from said plurality of images in response to receiving a first user input interacting; and

location information comprising instructions for fetching at least one additional content items of said next subset of content items from a location,

said method further comprising;

on the basis of said content layout information, said user interactivity configuration information, said location information and said three-dimensional pose information, rendering a further graphical user interface for display in the display output such that said further graphical user interface rendered matches the three-dimensional pose of said object in the display output.

14. A client for generating an augmented reality content item on a user device comprising a digital imaging part, a display output, a user input part, said client comprising a computer-vision based tracker for tracking an object in said display on the basis of at least an image of the object from the digital imaging part, said being configured for:

receiving an object identifier associated with an object in an image;

on the basis of said object identifier retrieving panel data from a panel database and tracking resources from a tracking resources database, said panel data comprising at least location information for retrieving a content item;

on the basis of said tracking resources said computer-vision based tracker generating three-dimensional pose information associated with said object;

on the basis of said panel data requesting at least part of said content item; and

,

on the basis of said three-dimensional pose information rendering said content item for display in the display output such that the content rendered matches the three-dimensional pose of said object in the display output.

16. A user device comprising a client according to claim 14.

15. A client for generating an augmented reality graphical user interface on a user device comprising a digital imaging part, a display output, a user input part, said client comprising a computer-vision based tracker for tracking an object in said display on the basis of at least an image of the object from the digital imaging part, said client further being configured for:

receiving an object identifier associated with an object in an image;

on the basis of said object identifier retrieving panel data from a panel database and tracking resources from a tracking resources database, said panel data comprising at least location information for retrieving a content item and user interactivity configuration information, said content item and said user interactivity information defining a graphical user interface;

on the basis of said tracking resources said computer-vision based tracker generating three-dimensional pose information associated with said object;

on the basis of said panel data, requesting at least part of said content item; and

,

on the basis of said user interactivity configuration information and said three-dimensional pose information, rendering said graphical user interface for display in the display output such that the graphical user interface rendered matches the three-dimensional pose of said object in the display output.

17. A vision-based augmented reality system comprising:

one or more servers configured to host a panel database, a tracking resources database and a object recognition system; and

a user device configured communicatively connect to said one or more servers and to generate an augmented reality content item comprising a digital imaging part, a display output, a user input part said client comprising a computer-vision based tracker for tracking an object in said display on the basis of at least an image of the object from the digital imaging part, said being configured for;

receiving an object identifier associated with an object in an image;

on the basis said object identifier retrieving panel data from a panel database and tracking resources from a tracking resources database, said panel data comprising at least location information for retrieving a content item;

on the basis of said tracking resources said computer-vision based tracker generating three-dimensional pose information associated with said object;

on the basis of said panel data requesting at least part of said content item; and

,

on the basis of said three-dimensional pose information rendering said content item for display in the display output such that the content rendered matches the three-dimensional pose of said object in the display output.

18. A graphical user interface for a user device comprising a digital imaging part, a display output, a user input part and an augmented reality client, said graphical user interface being associated with a object displayed in said display output;

said graphical user interface being rendered on the basis of panel data from a panel database and three-dimensional pose information associated with said object, said panel data comprising at least location information for retrieving a content item,

wherein said graphical user interface comprises said content item and at least one user input area, wherein said content item and said at least one user input area match the three-dimensional pose of said object.

19. A data structure stored in a storage medium, said data structure controlling the generation of a graphical user interface in a user device, said data structure comprising:

content layout information for specifying the display of a content item in said graphical user interface, user interactivity configuration information for configuring one or more user-input functions used by said graphical user interface and location information comprising instructions for fetching a content item from a content source.

20. A computer program product, implemented on computer-readable non-transitory storage medium, the computer program product configured for, when run on a computer, generating an augmented reality content item on a user device comprising a digital imaging part, a display output, a user input part and an augmented reality client, said client comprising a computer-vision based tracker for tracking an object in said display on the basis of at least an image of the object from the digital imaging part, said method comprising:

receiving an object identifier associated with an object in an image;

on the basis of said object identifier retrieving panel data from a panel database and tracking resources from a tracking resources database, said panel data comprising at least location information for retrieving a content item;

on the basis of said tracking resources said computer-vision based tracker generating three-dimensional pose information associated with said object;

on the basis of said panel data requesting at least part of said content item; and

,

on the basis of said three-dimensional pose information rendering said content item for display in the display output such that the content rendered matches the three-dimensional pose of said object in the display output.