METHOD FOR VISUALLY AUGMENTING A REAL OBJECT WITH A COMPUTER-GENERATED IMAGE

• US 20150042678A1
• Filed: 08/09/2013
• Published: 02/12/2015
• Est. Priority Date: 08/09/2013
• Status: Abandoned Application

• Current Assignee: Metaio GmbH (Apple Inc.)
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Alt, Thomas
• Application Number:
  US13/963,736
• Publication Number:
  US 20150042678A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/147 using display panels

Abstract

A method and system for visually augmenting a real object with a computer-generated image that includes sending a virtual model in a client-server architecture from a client computer to a server via a computer network, receiving the virtual model at the server, instructing a 3D printer to print at least a part of the real object according to the virtual model, generating an object detection and tracking configuration configured to identify at least a part of the real object, receiving an image captured by a camera representing at least part of an environment in which the real object is placed, determining a pose of the camera with respect to the real object, and overlaying at least part of a computer-generated image with at least part of the real object.

Claims

1. A method for visually augmenting a real object with a computer-generated image comprising:

sending a virtual model in a client-server architecture from a client computer to a server via a computer network;

receiving the virtual model at the server;

instructing a 3D printer to print the real object or a part of the real object according to at least part of the virtual model received at the server;

generating an object detection and tracking configuration configured to identify the real object or a part of the real object;

receiving an image captured by a camera representing at least part of an environment in which the real object is placed;

determining a pose of the camera with respect to the real object according to the object detection and tracking configuration and at least part of the image; and

overlaying at least part of a computer-generated image with at least part of the real object according to the object detection and tracking configuration and the pose of the camera.

2. The method according to claim 1, wherein the object detection and tracking configuration is or contains an identification code.

3. The method according to claim 2, wherein the object detection and tracking configuration is based on at least one of:

a reference image of at least part of the real object and a virtual model of at least part of the real object.

4. The method according to claim 1, further comprising determining the pose of the camera with respect to the real object based on an image captured by the camera and a visual marker fixed relative to the real object.

5. The method according to claim 4, wherein the visual marker encodes an identification code.

6. The method according to claim 1, further comprising determining the pose of the camera with respect to the real object based on an image captured by the camera and a virtual model of at least part of the real object.

7. The method according to claim 6, wherein edge features are used to determine the pose of the camera.

8. The method according to claim 1, wherein the camera is part of or associated with a mobile device.

9. The method according to claim 8, wherein the mobile device is the client computer.

10. The method according to claim 8, further comprising the mobile device obtaining the object detection and tracking configuration by a manual user input, or by receiving it from the server, or by downloading it from a cloud of computers.

11. The method according to claim 8, wherein the mobile device comprises a display adapted to display an image captured by the camera, and the step of overlaying at least part of the computer-generated image with at least part of the real object comprises overlaying the at least part of the computer-generated image with an image of the at least part of the real object captured by the camera and displayed on the display of the mobile device.

12. The method according to claim 8, wherein the mobile device comprises semi-transparent glasses, and the step of overlaying at least part of the computer-generated image with at least part of the real object comprises blending in the at least part of the computer-generated image in the semi-transparent glasses with a view of the least part of the real object captured by a user'"'"'s eye.

13. The method according to claim 12, wherein the camera has a known position relative to the user'"'"'s eye.

14. The method according to claim 1, further comprising a projector, and the step of overlaying at least part of the computer-generated image with at least part of the real object comprises projecting the at least part of computer-generated image onto the at least part of the real object by the projector.

15. The method according to claim 13, wherein the camera has a known position relative to the projector.

16. A computer program product comprising a computer readable storage medium having computer readable software code sections embodied in the medium, which software code sections are configured to perform a method according to claim 1.

17. A system for visually augmenting a real object with a computer-generated image, comprising:

a client computer in communication with a server via a computer network;

a 3D printer; and

a camera;

wherein the client computer is adapted to selectively send a virtual model in a client-server architecture to the server via the computer network, and the server is adapted to receive the virtual model; and

wherein the server is adapted to instruct the 3D printer to print the real object or a part of the real object according to at least part of the virtual model received at the server; and

wherein one of the client computer or the server is adapted to generate an object detection and tracking configuration configured to identify the real object or a part of the real object; and

wherein one of the client computer or server is adapted to receive an image captured by the camera, which image represents at least part of an environment in which the real object is placed, and to determine a pose of the camera with respect to the real object according to the object detection and tracking configuration and at least part of the image, and to overlay at least part of the computer-generated image with at least part of the real object according to the object detection and tracking configuration and the pose of the camera.