Coded visual markers for tracking and camera calibration in mobile computing systems

• US 20030076980A1
• Filed: 10/02/2002
• Published: 04/24/2003
• Est. Priority Date: 10/04/2001
• Status: Abandoned Application

• Current Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Zhang, Xiang, Navab, Nassir
• Application Number:
  US10/262,693
• Publication Number:
  US 20030076980A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components

Abstract

A method for determining a pose of a user is provided including the steps of capturing a video image sequence of an environment including at least one coded marker; detecting if the coded marker is present in the video images; if the marker is present, extracting feature correspondences of the coded marker; determining a code of the coded marker using the feature correspondences; and comparing the determined code with a database of predetermined codes to determine the pose of the user. According to an embodiment, the coded marker includes four color blocks arranged in a square formation and the determining a code of the at least one marker further includes determining a color of each of the four blocks. According to another embodiment, the marker includes a coding matrix and a code of the marker being determined by numbered squares of the coding matrix being covered by a circle.

Claims

1. A method for determining a pose of a user comprising the steps of:

capturing a video image sequence of an environment including at least one coded marker;

detecting if the at least one coded marker is present in the video images;

if the at least one marker is present, extracting feature correspondences of the at least one coded marker;

determining a code of the at least one coded marker using the feature correspondences; and

comparing the determined code with a database of predetermined codes to determine the pose of the user.

2. The method as in claim 1, wherein the at least one coded marker comprises four color blocks arranged in a square formation.

3. The method as in claim 2, wherein the detecting step further comprises applying a watershed transformation to the at least one coded marker to extract a plurality of closed-edge strings that form a contour of the at least one marker.

4. The method as in claim 3, wherein the detecting step further comprises grouping at least four closed-edge strings with a least maximum mutual distance.

5. The method as in claim 4, wherein the extracting step further comprises;

locating a weight center for each of the at least four closed-edge strings; and

using the weight centers as a correspondence of each of the four blocks to compute a homography from the at least one coded marker to an image of the marker.

6. The method as in claim 5, wherein the extracting step further comprises projecting eight lines onto the marker image using the homography to locate the four blocks of the at least one marker.

7. The method as in claim 6, wherein the extracting step further comprises applying a 1-d Canny edge detection to locate the edge points of the four blocks.

8. The method as in claim 2, wherein the determining a code of the at least one marker further comprises determining a color of each of the four blocks.

9. The method as in claim 8, wherein the at least one of the four blocks of the at least one marker includes a white patch.

10. The method as in claim 1, wherein the at least one marker comprises a coding matrix including a plurality of columns and rows with a numbered square at intersections of the columns and rows, the coding matrix being surrounded by a rectangular frame and a code of the at least one marker being determined by the numbered squares being covered by a circle.

11. The method as in claim 10, wherein the coding matrix includes m columns and n rows, where m and n are whole number, resulting in 3×

2<sup>m×

n−

4</sup> codes.

12. The method as in claim 10, wherein the detecting step further comprises applying a watershed transformation to the at least one coded marker to extract a plurality of closed-edge strings that form a contour of the at least one marker.

13. The method as in claim 12, wherein the detecting step further comprises locating at least two closed-edge strings that have close weight centers.

14. The method as in claim 13, wherein the detecting step further comprises locating a corner of the rectangular frame of the at least one marker by determining a cross-point of the at least two closed-edge strings.

15. The method as in claim 12, wherein the detecting step comprises locating corners of the rectangular frame of the at least one marker by locating cross-points of the plurality of closed-edge strings.

16. The method as in claim 15, wherein the extracting step further comprises applying a 1-d Canny edge detection to locate the edge points of the rectangular frame.

17. The method as in claim 16, wherein the extracting step further comprises

computing a homography from the corners and edge points;

extracting image feature correspondences of the at least one marker; and

determining locations of the circles in the coding matrix by the image correspondences.

18. The method as in claim 17, wherein the extracting image feature correspondences is performed by linear interpolation.

19. The method as in claim 17, further comprising the step of calibrating a camera used to capture the video image sequence with the image correspondences of the least one marker.

20. The method as in claim 19, further comprising the step of determining a position and orientation of the camera relative to the at least one marker.

21. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for determining a pose of a user, the method steps comprising:

capturing a video image sequence of an environment including at least one coded marker;

detecting if the at least one coded marker is present in the video images;

if the at least one marker is present, extracting feature correspondences of the at least one coded marker;

determining a code of the at least one coded marker using the feature correspondences; and

comparing the determined code with a database of predetermined codes to determine the pose of the user.

22. The program storage device as in claim 21, further comprising the step of determining a location of the user based on the pose of the user and a position of the at least one marker.

23. A system comprising:

a plurality of coded markers located throughout an environment, each of the plurality of coded markers relating to a location in the environment, codes of the plurality of coded markers being stored in a database;

a camera for capturing a video image sequence of the environment, the camera coupled to a processor; and

the processor adapted for detecting if at least one coded marker is present in the video images, if the at least one marker is present, extracting feature correspondences of the at least one coded marker, determining a code of the at least one coded marker using the feature correspondences, and comparing the determined code with the database to determine the pose of the user.

24. The system as in claim 23, wherein the at least one coded marker comprises four color blocks arranged in a square formation and a code of the at least one marker being determined by a color sequence of the blocks.

25. The system as in claim 23, wherein the at least one marker comprises a coding matrix including a plurality of columns and rows with a numbered square at intersections of the columns and rows, the coding matrix being surrounded by a rectangular frame and a code of the at least one marker being determined by the numbered squares being covered by a circle.

26. The system as in claim 23, wherein the camera and processor are mobile devices.

27. The system as in claim 23, wherein the camera and processor are housed in an integral mobile device.

28. The system as in claim 23, wherein based on a first location of the at least one marker, the processor being adapted to direct the user to a second location.

29. The system as in claim 23, further comprising a display device, wherein the display device will provide to the user information relative to the location of the at least one marker.

30. The system as in claim 23, further comprising a display device, wherein based on a first location of the at least one marker, the display device will provide to the user information to direct the user to a second location.

31. The system as in claim 23, further comprising an external database of information relative to a plurality of items located throughout the environment, wherein when the user is in close proximity to at least one of the plurality of items, the processor provides the user with access to the external database.

32. The system as in claim 31, further comprising a display device for displaying information of the external database to the user.

33. The system as in claim 31, further comprising a display device for displaying virtual objects overlaid on the at least one item.

34. The system as in claim 31, further comprising a head-mounted display for overlaying information of the at least one item in a view of the user.