METHODS AND SYSTEMS FOR ENABLING CREATION OF AUGMENTED REALITY CONTENT

• US 20150040074A1
• Filed: 08/18/2011
• Published: 02/05/2015
• Est. Priority Date: 08/18/2011
• Status: Abandoned Application

• Current Assignee: Layar BV (Candy Capital Ltd.)
• Original Assignee: Layar BV (Candy Capital Ltd.)
• Inventors: Hofmann, Klaus Michael, Van Der Klein, Raimo Jahani, Van Der Lingen, Ronald, Van De Zandschulp, Klasien
• Application Number:
  US14/239,230
• Publication Number:
  US 20150040074A1
• Time in Patent Office: Days
• US Class Current: 715/852
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods and systems for enabling creation of augmented reality content on a user device including a digital imaging part, a display, a user input part and an augmented reality client, wherein said augmented reality client is configured to provide an augmented reality view on the display of the user device using an live image data stream from the digital imaging part are disclosed. User input is received from the user input part to augment a target object that is at least partially seen on the display while in the augmented reality view. A graphical user interface is rendered to the display part of the user device, said graphical user interface enabling a user to author augmented reality content for the two-dimensional image.

Claims

1. A method for enabling creation of user-generated content on a user device associated with a digital imaging part, a display output, a user input part and an augmented reality client, wherein said augmented reality client is configured to provide an augmented reality view on the display output using image data from the digital imaging part, the method comprising:

receiving a first user input from the user input part to select a target object displayed in said display output;

providing a first graphical user interface comprising a two-dimensional image of at least part of the target object, said graphical user interface being configured to receive second user input associated with user-generated content, preferably said user-generated content being aligned with said two-dimensional image;

receiving a third user input from the user input part to attach said user-generated content to said target object;

determining, in a tracker part of the augmented reality client, three-dimensional pose information associated with said selected target object on the basis of at least an image of the target object from the digital imaging part; and

rendering said user-generated content for display in the display output, on the basis of said three-dimensional pose information such that the user-generated content is displayed in perspective with the target object, said user-generated content rendered matching the three-dimensional pose of said selected target object in the display output.

2. The method according to claim 1, wherein the graphical user interface for enabling the user to author the user-generated content on top of the two-dimensional image using the graphical user interface is a what-you-see-is-what-you-get editor that enables the capture of the spatial relationship between the second user input and the two-dimensional image of the target object.

3. The method according to claim 1, wherein determining the three dimensional pose information comprises:

processing an image frame from a live image data stream captured by the digital imaging part on the user device to extract tracking features; and

wherein the determining of three-dimensional pose information is performed on the user device on the basis of the extracted tracking features and the image data stream.

4. The method according to claim 3, wherein:

the extracted tracking features have a quality that is lower than the quality of other tracking features that are associated with the target object and are extracted by an object recognition system remote from the user device; and

the processing of the image frame is performed if the tracking features from the object recognition system are not available at the user device.

5. The method according to claim 1, wherein determining the three dimensional pose information comprises:

transmitting an image frame from a live image data stream captured by the digital imaging part or a derivation of the image frame to an object recognition system remote from the user device;

receiving an identifier associated with the target object, tracking features and the two-dimensional image from the object recognition system; and

wherein the determining of the three-dimensional pose information is performed on the user device on the basis of the received tracking features and the image data stream.

6. The method according to claim 1, wherein the rendering of said user-generated content comprises:

transforming the user-generated content by scaling, rotating and translating the user-generated content based on three-dimensional pose information in the tracker part to generate a graphics object having the transformed user-generated content; and

rendering the graphics object for display in the display output, the graphics object appearing in perspective with the target object in the augmented reality view.

7. The method according to claim 2, wherein the user device further includes a tracker part, wherein the rendering of said user-generated content comprises:

transforming the user-generated content by scaling, rotating and translating the user-generated content based on (1) three-dimensional pose information in the tracker part and (2) the spatial relationship, to generate a graphics object having the transformed user-generated content; and

rendering the graphics object for display in the display output, the graphics object appearing in perspective with the target object in the augmented reality view.

8. The method according to claim 1, the method further comprising:

receiving the second user input associated with the user-generated content through the graphical user interface object from the user input part;

storing the second user input or a derivation of the second user input as a graphics data file in a non-transient computer readable medium;

associating the graphics data file with the target object.

9. The method according to claim 8, wherein the storing of the derivation of the second user input data comprises:

deriving a scalable vector graphic of the user-generated content based on the second user input;

using the scalable vector graphic as the derivation of the user input data.

10. The method according to claim 6, wherein the method further comprises:

associating the graphics data file with a user profile associated with the user device.

11. The method according to claim 1, wherein the graphical user interface comprises at least one of the following interactive parts for augmenting the target object:

a drawing part for drawing on the two-dimensional image displayed on the display output;

a stamping part for adding a copy of a stored image onto the two-dimensional image displayed on the display output;

a three-dimensional drawing part for adding a three dimensional object to the target object;

a text part for adding a text onto the two-dimensional image displayed on the display output.

12. The method according to any of the claim 1, further comprising:

receiving a third user input to flip the two-dimensional image; and

animating the two-dimensional image on the display output by showing an effect of flipping over the two-dimensional image and displaying content associated with the target object.

13. The method according to claim 6, further comprising:

receiving a third user input to flip the graphics object; and

animating the graphics object on the display output by showing an effect of flipping over the user-generated content and displaying other content associated with the target object.

14. The method according to claim 6, further comprising:

receiving a third user input to detach the graphics object from the target object; and

updating the graphics object by scaling, transforming, and rotating the graphics object to having a pose where the graphics object has a two-dimensional plane substantially parallel to the plane of the display output.

15. The method according to claim 6, further comprising:

receiving a third user input to attach the graphics object to the tracked object;

receiving updated three-dimensional pose information of the tracked object from the tracker part; and

updating the graphics object for display on the display output by scaling, rotating and translating the graphical object based on the updated three-dimensional pose information.

16. The method of claim 6, further comprising:

receiving a fifth user input to flip the graphics object, the graphics object having a first pose, such that the graphics object is rotated from the first pose to a second pose by substantially 180 degrees around an axis lying in the plane of the graphics object from the first pose to a second pose;

retrieving back-side content to be displayed on the display output for a back-side of the graphics object;

updating the back-side of the graphics object to include the back-side content; and

generating an animated sequence for the graphics object from the first pose to the second pose by scaling, rotating and translating the graphics object.

17. An augmented reality client configured to enable creation of user-generated content on a user device having a digital imaging part, a display output and a user input part, the augmented reality client comprising:

a first user input listener configured to receive a first user input from the user input part to select a target object displayed in said display output;

a graphics engine configured to provide a first graphical user interface comprising a two-dimensional image of at least part of the target object, said graphical user interface being configured to receive second user input associated with user-generated content;

a second user input listener configured to receive a third user input from the user input part to attach said user-generated content to said target object;

a tracker part configured to determine three-dimensional pose information associated with said selected target object on the basis of at least an image of the target object from the digital imaging part; and

the graphics engine further configured to render said user-generated content for display in the display output, on the basis of said three-dimensional pose information such that the user-generated content is displayed in perspective with the target object, said user-generated content rendered matching the three-dimensional pose of said selected target object in the display output.

18. A computer program product, implemented on computer-readable non-transitory storage medium, the computer program product configured for, when run on a computer, enable creation of user-generated content on a user device associated with a digital imaging part, display output, a user input part and an augmented realty client, wherein said augmented reality client is configured to provide an augmented reality view on the display output using image data from the digital imaging part, the method comprising:

receiving a first user input from the user input part to select a target object displayed in said display output;

providing a first graphical user interface comprising a two-dimensional image of at least part of the target object, said graphical user interface being configured to receive second user input associated with user-generated content;

receiving a third user input from the user input part to attach said user-generated content to said target object;

determining, in a tracker part of the augmented reality client, three-dimensional pose information associated with said selected target object on the basis of at least an image of the target object from the digital imaging part; and

rendering said user-generated content for display in the display output, on the basis of said three-dimensional pose information such that the user-generated content is displayed in perspective with the target object, said user-generated content rendered matching the three-dimensional pose of said selected target object in the display output.

19. A graphical user interface for enabling the creation of user-generated content on a user device having a digital imaging part, a display output and a user input part, comprising:

a first display state comprising a first user input listener configured to receive a first user input from the user input part to select a target object displayed in said display output;

a second display state, having a first transition from the first state in response to receiving the first user input, comprising;

a two-dimensional image of at least part of the target object;

a second user input listener being configured to receive second user input associated with user-generated content;

a third user input listener to receive a third user input from the user input part to attach said user-generated content to said target object;

a third display state, having a second transition from the second state in response to receiving the third user input, comprising said user-generated content for display in the display output, said user-generated content;

being rendered on the basis of said three-dimensional pose information such that the user-generated content is displayed in perspective with the target object;

matching the three-dimensional pose of said selected target object in the display output, said three-dimensional pose determined by a tracker part of the augmented reality client.

20. A user device having an augmented reality client configured to enable creation of user-generated content, said user device having a digital imaging part, a display output and a user input part, the augmented reality client of the user device comprising:

a first user input listener configured to receive a first user input from the user input part to select a target object displayed in said display output;

a graphics engine configured to provide a first graphical user interface comprising a two-dimensional image of at least part of the target object, said graphical user interface being configured to receive second user input associated with user-generated content;

a second user input listener configured to receive a third user input from the user input part to attach said user-generated content to said target object;

a tracker part configured to determine three-dimensional pose information associated with said selected target object on the basis of at least an image of the target object from the digital imaging part; and

the graphics engine further configured to render said user-generated content for display in the display output, on the basis of said three-dimensional pose information such that the user-generated content is displayed in perspective with the target object, said user-generated content rendered matching the three-dimensional pose of said selected target object in the display output.