AUGMENTED REALITY METHOD AND DEVICES USING A REAL TIME AUTOMATIC TRACKING OF MARKER-FREE TEXTURED PLANAR GEOMETRICAL OBJECTS IN A VIDEO STREAM

• US 20100220891A1
• Filed: 01/18/2008
• Published: 09/02/2010
• Est. Priority Date: 01/22/2007
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Total Immersion
• Inventors: Lefevre, Valentin, Livet, Nicolas
• Granted Patent: US 8,315,432 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

The invention relates to a method and to devices for the real-time tracking of one or more substantially planar geometrical objects of a real scene in at least two images of a video stream for an augmented-reality application. After receiving a first image of the video stream (300), the first image including the object to be tracked, the position and orientation of the object in the first image are determined from a plurality of previously determined image blocks (320), each image block of said plurality of image blocks being associated with an exposure of the object to be tracked. The first image and the position and the orientation of the object to be tracked in the first image define a key image. After receiving a second image from the video stream, the position and orientation of the object to be tracked in the second image are evaluated from the key image (300). The second image and the corresponding position and orientation of the object to be tracked can be stored as a key image. If the position and the orientation of the object to be tracked cannot be found again in the second image from the key

Claims

1. A method for tracking in real time at least one substantially plane geometrical object from a real scene in at least two substantially consecutive images of at least one video stream, in an augmented reality application, the initialization of the method being automatic, said method comprising the following steps:

reception of a first image from said at least one video stream (300), said first image including said at least one object to be tracked;

determination of the position and the orientation of said at least one object to be tracked in said first image (320) on the basis of a plurality of image blocks determined beforehand, each image block of said plurality of image blocks being associated with a pose of said at least one object to be tracked;

creation of a key image including said first image and the position and the orientation of said at least one object to be tracked in said first image;

reception of a second image from said at least one video stream (330), said second image including said at least one object to be tracked; and

evaluation of the position and the orientation of said at least one object to be tracked in said second image on the basis of said key image (330).

2. The method according to claim 1, further comprising a preliminary step of determination of said plurality of image blocks and said associated poses on the basis of at least one textured image of said object to be tracked.

3. The method according to claim 1, wherein the steps of reception of an image and evaluation of the position and the orientation of said at least one object to be tracked in said received image are repeated for images of said at least one video stream after said second image.

4. The method according to claim 1, wherein if the position and the orientation of said at least one object to be tracked cannot be evaluated in a received image on the basis of said key image, the position and the orientation of said at least one object to be tracked in said received image are evaluated on the basis of said plurality of image blocks (320).

5. The method according to claim 1, wherein the values of said key image are replaced by a received image and by the evaluated position and orientation of said object to be tracked in said received image.

6. The method according to claim 1, further comprising a step of evaluation of the pose of the image sensor from which said at least one video stream is obtained in a frame of reference linked to said real scene on the basis of the evaluated position of said at least one object to be tracked.

7. The method according to claim 6, further comprising a step of determination of the movement of said image sensor.

8. The method according to claim 1, characterized in that said at least one object to be tracked includes an area of uniform color adapted to use the chromakey technology.

9. The method according to claim 1, wherein said at least one object to be tracked is tracked simultaneously in said at least one video stream and in at least one other video stream, said at least one video stream and said at least one other video stream coming from at least two different image sensors, said method further comprising a step of estimation of the relative position of said at least two image sensors.

10. The method according to claim 1, further comprising a step of insertion of at least one element in at least one of said received images according to the evaluated position and orientation of said at least one object to be tracked in said received image, said at least one element being selected from a list comprising at least one representation of at least one virtual object and at least one second video stream.

11. The method according to claim 1, further comprising a step of determination of at least one control command, said at least one control command being determined according to the variation of the pose of said object to be tracked.

12. The method according to claim 1, further comprising a step of transmission of at least one indication relating to the variation of the pose of said object to be tracked.

13. The method according to claim 1, further comprising a step of acquisition of said first and second images and a step of at least partial display of said first and second images, said acquisition and display steps being executed in a device separate from the device executing said other steps.

14. A computer program stored on a computer readable storage medium comprising instructions for executing of each of the steps of the method according to claim 1 when executed by a computer.

15. A device for tracking in real time at least one substantially plane geometrical object from a real scene in at least two substantially consecutive images of at least one video stream, in an augmented reality application, the initialization of said tracking being automatic, said device comprising:

means for receiving a first image from said at least one video stream, said first image including said at least one object to be tracked;

IDBans for storing said first image in first storage means;

means for determining the position and the orientation of said at least one object to be tracked in said first image on the basis of a plurality of image blocks stored beforehand in second storage means, each image block of said plurality of image blocks being associated with a pose of said at least one object to be tracked, said pose being stored in said second storage means, the position and the orientation of said at least one object to be tracked in said first image being stored in said first storage means;

means for receiving a second image from said at least one video stream, said second image including said at least one object to be tracked; and

means for evaluating the position and the orientation of said at least one object to be tracked in said second image on the basis of data stored in said first storage means.

16. The device according to claim 15, further comprising means for determining if the position and the orientation of said at least one object to be tracked can be evaluated in said second image on the basis of data stored in said first storage means, said means for determining the position and the orientation of said at least one object to be tracked in said first image on the basis of data stored in said second storage means being adapted to evaluate the position and the orientation of said at least one object to be tracked in said second image on the basis of data stored in said second storage means.

17. The device according to claim 15, further comprising means for storing said second image and the position and the orientation of said at least one object to be tracked in said second image in said first storage means.

18. The device according to claim 15, further comprising transformation means adapted to determine the pose of said at least one object to be tracked or of the image sensor from which said at least one video stream is obtained in one of the frames of reference linked to said real scene, to said at least one object to be tracked, or to said image sensor.

19. The device according to claim 15, further comprising means for inserting at least one element into at least one of said received images according to the evaluated position and orientation of said at least one object to be tracked in said received image, said at least one element being chosen in a list comprising at least one representation of at least one virtual object and at least one second video stream.

20. The device according to claim 15, further comprising means for receiving said first and second images from at least one mobile telephone.