Augmented reality technology

• US 6,625,299 B1
• Filed: 04/06/1999
• Issued: 09/23/2003
• Est. Priority Date: 04/08/1998
• Status: Expired due to Fees

• Current Assignee: Jeffrey Meisner, Richard Roosen, Walter P. Donnelly
• Original Assignee: Jeffrey Meisner, Richard Roosen, Walter P. Donnelly
• Inventors: Donnelly, Walter P., Roosen, Richard, Meisner, Jeffrey
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/286,817
• Time in Patent Office: 1,631 Days
• Field of Search: 382/100, 382/154, 382/103, 382/252, 382/191, 382/306, 382/115, 380/28, 380/287, 380/252, 380/135, 380/253, 380/54, 380/202, 380/200, 348/169, 348/170, 348/171, 348/172, 702/153, 702/152
• US Class Current: 382/103
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A tracker system for determining the relative position between a sensor and an object surface, generally comprising a sensor or sensors for detecting a pattern of fiducials disposed on an object surface and a processor connected to the at least one sensor. An augmented reality system generally comprising a pattern of fiducials disposed on an object surface, a computer having a processor and a memory, a user interface for receiving input and presenting augmented reality output to a user, and a tracker for detecting the pattern of fiducials. A method for tracking the position and orientation of an object generally comprising the steps of scanning across an object to detect fiducials and form video runs, clumping video runs to detect a pattern of fiducials, acquiring estimated values for a set of tracking parameters by comparing a detected pattern of fiducials to a reference pattern of fiducials, and iterating the estimated values for the set of tracking parameters until the detected pattern of fiducials match the reference pattern of fiducials to within a desired convergence. A method for augmenting reality generally comprising the steps of disposing a pattern of fiducials on an object surface, tracking the position and orientation of the object, retrieving and processing virtual information stored in a computer memory according to the position and orientation of the object, and presenting the virtual information with real information to a user in near real time.

Claims

1. A fiducial-based tracker system, comprising:

(a) at least one optical sensor for detecting a pattern of fiducials disposed on an object surface, said pattern of fiducials including a hard fiducial pattern and a soft fiducial pattern, both of said hard fiducial and said soft fiducial patterns having a known shape and arrangement, said at least one optical sensor having a position and an orientation with respect to said pattern of fiducials on said object surface, said at least one optical sensor scanning said pattern of fiducials and detecting fiducial edges; and

(b) a processor connected to said at least one optical sensor, said processor identifying said position and said orientation of said at least one optical sensor with respect to said object surface by matching a detected pattern of fiducials with a reference pattern of fiducials.

4. A method for tracking the position and orientation of an object of claim 1, comprising the step of

(a) scanning across an object to detect fiducials and form runs;

(b) clumping video runs to detect a pattern of fiducials;

wherein the step of clumping video runs includes the steps of;

(i) combining adjacent video runs; and

(ii) extracting relevant information about the runs;

(c) acquiring estimated values for a set of tracking parameters by comparing a detected pattern of fiducials to a reference pattern of fiducials, wherein the detected and reference patterns of fiducials includes a geometrically consistent pattern of hard fiducials and a pseudo random pattern of soft fiducials, wherein the step of acquiring estimated values for a set of tracking parameters includes the steps of;

(i) corresponding a predetermined number of detected hard fiducials with the reference pattern of fiducials to estimate phi, theta and psi orientation parameters and to estimate a distance position parameter; and

(ii) electing at least one of the soft fiducials with the reference pattern of fiducials to estimate x-bar and y-bar position parameters; and

(d) iterating the estimated values for the set of tracking parameters until the detected pattern of fiducials match the reference pattern of fiducials to within a desired convergence.

2. A fiducial-based tracker system for registering virtual information to real information within an augmented reality system, comprising:

(a) at least one optical sensor for detecting a pattern of fiducials disposed on an object surface, said at least one sensor producing a signal corresponding to said detected pattern of fiducials, said pattern of fiducials including a hard fiducial pattern and a soft fiducial pattern, both of said hard fiducial and said soft fiducial patterns having a known shape and arrangement, said hard fiducial pattern being a geometrically consistent, rectilinear pattern, said soft fiducial pattern being a pseudo random pattern, wherein said orientation of said at least one sensor with respect to said pattern of fiducials is defined by angular parameters including phi, theta and psi, wherein said position of said at least one sensor with respect to said pattern of fiducials is defined by linear parameters including X-bar, Y-bar, and distance, said hard fiducial pattern being used to identify said phi, theta, psi and distance parameters and said soft fiducial pattern being used to identify said X-bar and Y-bar parameters, said at least one optical sensor having a position and an orientation with respect to said pattern of fiducials on said object surface, said at least one optical sensor scanning said pattern of fiducials and detecting fiducial edges;

(b) a processor connected to said at least one optical sensor, said processor identifying said position and said orientation of said at least one optical sensor with respect to said object surface by matching a detected pattern of fiducials with a reference pattern of fiducials; and

(c) a video feature extractor for reducing bandwidth to quickly process said signal and for removing noise from said signal.

3. An augmented reality system incorporating a fiducial-based tracking system for registering virtual information to real information, comprising:

(a) a pattern of fiducials disposed on an object surface, said pattern of fiducials including a geometrically consistent hard fiducial pattern and a pseudo-random soft fiducial pattern,

(b) a computer having a processor and a memory;

(c) a user interface for receiving input and presenting augmented information to a user, said user interface including a head mounted display on which said computer presents said augmented information, said head mounted display presenting said augmented information at an update rate of at least 60 Hertz and a latency below 16 milliseconds; and

(d) a tracker for detecting said pattern of fiducials, said tracker including at least one optical sensor for detecting said pattern of fiducials disposed on said object surface, said at least one optical sensor being electrically connected to said computer, said pattern of fiducials including a hard fiducial pattern and a soft fiducial pattern, both of said hard fiducial and said soft fiducial patterns having a known shape and arrangement, said at least one optical sensor having a position and an orientation with respect to said pattern of fiducials on said object surface, said at least one optical sensor scanning said pattern of fiducials and detecting fiducial edges, said processor being connected to said at least one optical sensor and identifying said position and said orientation of said at least one optical sensor with respect to said object surface by matching a detected pattern of fiducials with a reference pattern of fiducials stored in said memory of said computer.

5. A method for augmenting visual reality, comprising the steps of:

(a) disposing a pattern of fiducials on an object surface, wherein the pattern of fiducials are disposed on the object surface in a geometrically consistent hard fiducial pattern and in a pseudo random soft fiducial pattern;

(b) tracking the position and orientation of the object, including;

(i) scanning across the object to detect the fiducials, wherein a video run is formed by a scan;

(ii) clumping video runs to detect the pattern of fiducials;

(iii) acquiring estimated values for a set of tracking parameters by comparing a detected pattern of fiducials to a reference pattern of fiducials; and

(iv) iterating the estimated values for the set of tracking parameters until the detected pattern of fiducials match the reference pattern of fiducials to within a desired convergence;

(c) processing virtual information stored in a computer memory according to the position and orientation of the object; and

(d) presenting the virtual information with real information to a user in near real time, wherein said steps of tracking the position and orientation of the object, retrieving virtual information, and presenting the virtual information with real information are performed with an update rate of at least 60 Hertz and a latency below 16 milliseconds.

6. A method according to claim 5, wherein steps (a) through (c) are performed with a self-contained, mobile system that comprises a wearable computer.

7. A method according to claim 5, wherein step (a) is performed with a self-contained, mobile system that comprises a head mounted display.

8. A method according to claim 5, wherein step (c) comprises presenting virtual aural information to the user.

9. A method according to claim 5, wherein step (c) comprises presenting virtual kinesthetic information to the user.

10. A method according to claim 5, wherein at least one of steps (b) and (c) are performed by electronically reducing said virtual information to a reduced amount of information for manageable processing.

11. A method according to claim 10, wherein said step of electronically reducing said virtual information comprises electronically filtering said information to eliminate background information.

12. A method according to claim 10, wherein said step of electronically reducing said virtual information comprises electronically filtering said information to eliminate noise.

13. A method according to claim 12, wherein said step of electronically filtering said information to eliminate noise comprises electronically correcting to compensate for known aberrations of at least one hardware element.

14. A method according to claim 10, wherein said step of electronically reducing said virtual information comprises removing at least one synchronization signal from said information.

15. A method according to claim 10, wherein said step of electronically reducing said virtual information comprises a step of determining a centroid position of at least one of said fiducials.

16. A method according to claim 15, wherein said step of electronically reducing said virtual information further comprises utilizing said centroid position in subsequent electronic processing.

17. A method according to claim 15, wherein said step of determining a centroid position of at least one of said fiducials comprises steps of determining at least two boundary locations of a fiducial and calculating the centroid position based at least in part on said boundary locations.

18. A method according to claim 17, wherein said step of determining a centroid position of at least one of said fiducials comprises detecting a beginning pixel and an ending pixel of the fiducial during a video raster run;

extracting the line number, the beginning pixel number and the number of continuous pixels after the beginning pixel;

processing such information obtained from different line numbers of the video raster run to form individual fiducial representations; and

further processing this information to determine the centroid position.

19. A method according to claim 5, wherein step (a) comprises tracking the position and orientation of at least one hard fiducial pattern having a known geometric shape and further comprises tracking the position and orientation of at least one soft fiducial pattern.

20. A method according to claim 19, wherein at least one fiducial of said soft fiducial pattern is located within said known geometric shape.

21. A method according to claim 19, wherein step (b) comprises a step of locating said hard fiducial pattern, and wherein said step of locating said hard fiducial pattern is aided by utilizing a reference pattern of fiducials corresponding to said known geometric shape.

22. A method according to claim 19, wherein step (b) further comprises locating said soft fiducial pattern, and wherein said step of locating said soft fiducial pattern is aided by said prior determination of said hard fiducial pattern.

23. A method according to claim 19, wherein step (b) comprises electronically calculating the position and orientation of the object based on the determined location of the hard fiducial pattern and the determined location of the soft fiducial pattern.

24. A method according to claim 23, wherein said step of electronically calculating the position and orientation of the object is performed based on a relative position of at least one fiducial within said soft fiducial pattern that is located within said known geometric shape of said hard fiducial pattern.

25. A method according to claim 5, wherein step (a) comprises iteratively tracking the position and orientation of a number of fiducials on said object.

26. A method according to claim 25, wherein said step of iteratively tracking the position and orientation of a number of fiducials comprises a quick acquire routine for calculating a theoretical parameter value for at least one fiducial based on a previous position of the fiducial.

27. A method according to claim 26, wherein said quick acquire routine tracks movement of the system with respect to the object and utilizes data pertaining to said movement to predict likely future locations of said fiducial.

28. A method according to claim 25, wherein step (b) comprises receiving a signal of information based on said iterative tracking and processing said signal to eliminate nonessential data, thereby minimizing bandwidth requirements.