Augmented-reality tool employing scen e-feature autocalibration during camera motion

• US 20020191862A1
• Filed: 09/25/2001
• Published: 12/19/2002
• Est. Priority Date: 03/07/2001
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Neumann, Ulrich, You, Suya
• Granted Patent: US 6,765,569 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

An augmented reality tool makes use of autocalibrated features for rendering annotations into images of a scene as a camera moves about relative to the scene. The autocalibrated features are used for positioning the annotations and for recovery of tracking, correspondences and camera pose. An improved method of autocalibration for autocalibrating structured sets of point features together is also described. The augmented reality tool makes use of manual, semi-automatic and automatic methods employing autocalibrated features.

Claims

1. A method for producing an augmented image by combining a computer-generated virtual-image with a real-world view of a real scene captured by an imaging device, comprising the steps of:

capturing a view of the real scene with the imaging device;

detecting features in the view corresponding to calibration features in a calibration database containing 3-dimensional coordinates of the calibration features relative to a calibration coordinate frame;

corresponding the detected features to the calibration features to provide the 3-dimensional coordinates of the detected features;

determining a pose of the imaging means from the corresponded features;

changing the pose of the imaging device to a different pose and capturing a different view of the real scene wherein the different view contains at least some of the corresponded features;

detecting additional features in the different view of the real scene;

calibrating the additional features using the positions of at least some of the corresponded features within the different view to produce a set of autocalibrated features;

determining correspondences between at least some of the set of autocalibrated features and features in a model database containing 3-dimensional coordinates of the computer-generated virtual-image relative to a model coordinate frame different from the calibration coordinate frame;

determining a transformation between the calibration coordinate frame and the model coordinate frame based on the correspondences between the autocalibrated features and the model database features; and

combining the computer-generated virtual-image with the different view of the real scene in an output device by using the transformation between the calibration coordinate frame and the model coordinate frame and the determined different pose.

2. The method of claim 1, further comprising the steps of:

changing the pose of the imaging device to an additional different pose and capturing an additional different view of the real scene containing at least some of the set of autocalibrated features;

determining the additional different pose of the imaging device using the positions of at least some of the set of autocalibrated features within the additional different view; and

combining the computer-generated virtual-image with the additional different view of the real scene in an output device by using the transformation between the calibration coordinate frame and the model coordinate frame and the determined additional different pose.

3. The method of claim 1, wherein at least three features having their 3-dimenional coordinates known in both the calibration frame and the model frame are corresponded.

4. The method of claim 1, wherein the correspondences between at least some of the set of autocalibrated features and the features in the model database are determined by interactively selecting the autocalibrated and model database features.

5. The method of claim 4, wherein a user is prompted to interactively select certain points.

6. The method of claim 3, wherein the three features are selected according to characteristics selected from the group comprising color, shape and texture.

7. The method of claim 3, wherein at least one of the features is a corner of an object.

8. The method of claim 1, further comprising the step of tracking in 2-dimensions at least some of the detected features and additional features as the camera pose changes to the different pose.

9. The method of claim 1, wherein the computer-generated virtual-image is combined with the different view of the real scene in the output device by using the equation:

10. An imaging tool for producing an augmented image by combining at least one computer-generated virtual-image with a real-world view of a real scene captured by an imaging means, comprising:

a processor;

a data storage means;

a means for transferring data between the storage device and the processor;

an imaging device for capturing views of the real scene;

a means for changing the pose of the imaging device;

a calibration database stored by the data storage means and containing 3-dimensional coordinates relative to a calibration coordinate frame of calibration features corresponding to features detected in views of the real scene;

a set of calibrated features having known 3-dimensional coordinates relative to the calibration coordinate frame obtained by corresponding features in the views of the real scene with the calibration features;

a set of autocalibrated features having known 3-dimensional coordinates relative to the calibration coordinate frame obtained from a view containing at least some of the set of calibrated features and from the pose of the imaging device determined from at least some of the set of calibrated features;

a model database stored by the data storage means and containing 3-dimensional coordinates of the computer-generated virtual-images and of at least some of the autocalibrated features relative to a model coordinate frame so that the processor determines a transformation between the calibration coordinate frame and the model coordinate frame;

a display for combining at least some of the set of computer-generated virtual-images with a real-world view of a real scene based on a determination by the processor of the pose of the imaging device and based upon the transformation by the processor to match the model and calibration coordinate frames.

11. The imaging tool of claim 10, wherein the transformation between the calibration coordinate frame and the model frame is given by:

12. The imaging tool of claim 10, wherein the transformation between the calibration coordinate frame and the model coordinate frame is determined by corresponding at least three autocalibrated features having their 3-dimmensional coordinates known in both the calibration frame and the model frame.

13. The imaging tool of claim 10, wherein the processor determines the transformation between the calibration coordinate frame and the model coordinate frame by interactively corresponding features having their 3-dimmensional coordinates known in both the calibration frame and the model frame.

14. The imaging tool of claim 13, wherein a user is prompted to interactively select certain points to correspond the features.

15. The imaging tool of claim 12, wherein the three features are selected according to characteristics selected from the group comprising color, shape and texture.

16. The imaging tool of claim 12, wherein at least one of the features is a corner of an object.

17. The imaging tool of claim 10, wherein the processor tracks at least some of the set of autocalibrated features in 2-dimensions as the camera pose changes.

18. The imaging tool of claim 10, wherein the calibration database and autocalibration database are stored together in a single file and the data storage means consists of a single memory storage device.

19. The imaging tool of claim 10, wherein the calibration database, autocalibration database and model database are stored together in the data storage means, the data storage means comprised of several separate memory storage devices.

20. A method for producing an augmented image by combining a computer-generated virtual-image with a real-world view of a real scene captured by an imaging device, comprising the steps of:

capturing a view of the real scene with the imaging device;

detecting features in the view corresponding to calibration features in a calibration database containing 3-dimensional coordinates of the calibration features relative to a calibration coordinate frame;

corresponding the detected features to the calibration features to provide the 3-dimensional coordinates of the detected features;

determining the pose of the imaging means from the corresponded features;

changing the pose of the imaging device to a different pose and capturing a different view of the real scene wherein the different view contains at least some of the corresponded features;

detecting additional features in the different view of the real scene;

calibrating the additional features using the positions of at least some of the corresponded features within the different view to produce autocalibrated features;

storing the autocalibrated features;

losing tracking of the stored autocalibrated features;

recovering tracking of the stored autocalibrated features by corresponding detected features to the stored autocalibrated features; and

combining the computer-generated virtual-image with the different view of the real scene in an output device using the stored autocalibrated features.

21. The method of claim 20, wherein the stored autocalibrated features are part of a larger set of stored autocalibrated features.

22. The method of claim 20, further comprising the steps of:

changing the pose of the imaging device to an additional different pose and capturing an additional different view of the real scene containing the stored autocalibrated features;

determining the additional different pose of the imaging device using the stored autocalibrated features within the additional different view;

combining the computer-generated virtual-image with the additional different view of the real scene in an output device using the stored autocalibrated features.

23. The method of claim 20, wherein at least three of the detected features are corresponded to three of the stored autocalibrated features.

24. The method of claim 20, wherein a detected feature is corresponded to a stored autocalibrated feature to recover lost feature tracking by:

projecting a stored autocalibrated feature onto another view of the real scene;

determining whether the autocalibrated feature falls within a determined distance of the detected feature; and

corresponding the detected feature to the autocalibrated feature when it is determined that the autocalibrated feature falls within the determined distance of the detected feature.

25. The method of claim 20, wherein at least one detected feature is corresponded to a stored autocalibrated feature by:

estimating a position of the detected feature using autocalibration;

comparing the estimated position to a position of a stored autocalibrated feature having similar characteristics;

corresponding the detected feature to the stored autocalibrated feature when the positions fall within a determined distance to each other; and

updating the estimated position with the position of the stored autocalibrated feature when a correspondence is determined.

26. The method of claim 20, further comprising the step of recovering a lost pose of the imaging device.

27. The method of claim 26, wherein the lost pose is recovered by corresponding at least three detected features to the stored autocalibrated features.

28. The method of claim 26, wherein the lost pose is recovered by interactively matching a previous frame with a later frame.

29. The method of claim 28, wherein the previous and later frames are displayed so that they are blended together with one transparent relative to the other.

30. The method of claim 26, wherein the lost pose of the imaging device is determined at least in part using a gyroscope, a GPS sensor or an ultrasonic sensor.

31. The method of claim 20, further comprising the step of tracking in 2-dimensions at least some of the detected features and additional features as the camera pose changes to the different pose.

32. An imaging tool for producing an augmented image by combining at least some of a set of computer-generated virtual-images with a real-world view of a real scene captured by an imaging means, comprising:

a processor;

a data storage means;

a means for transferring data between the storage device and the processor;

an imaging device for capturing views of the real scene;

a means for changing the pose of the imaging device;

a calibration database stored by the data storage means and containing 3-dimensional coordinates relative to a calibration coordinate frame of calibration features corresponding to features detected in views of the real scene;

a set of calibrated features having known 3-dimensional coordinates relative to the calibration coordinate frame obtained by corresponding features in the views of the real scene with the calibration features;

autocalibrated features having known 3-dimensional coordinates relative to the calibration coordinate frame obtained from a view containing at least some of the set of calibrated features and from the pose of the imaging device determined from at least some of the set of calibrated features;

an autocalibration database storing the autocalibrated features and, in cooperation with the processor, corresponding detected features captured by the imaging device to the autocalibrated features in the autocalibration database to recover lost autocalibrated feature tracking;

a model database stored by the data storage means and containing 3-dimensional coordinates of a set of computer-generated virtual-images; and

a display for combining at least some of the set of computer-generated virtual-images with a real-world view of a real scene based, based at least in part, on a determination by the processor of the pose of the imaging device.

33. The imaging tool of claim 32, wherein the stored autocalibrated features are part of a larger set of stored autocalibrated features.

34. The imaging tool of claim 32, wherein at least three of the detected features are corresponded to three of the autocalibrated features to recover lost autocalibrated feature tracking.

35. The imaging tool of claim 32, wherein the processor corresponds a detected feature to a stored autocalibrated feature by:

projecting a stored autocalibrated feature onto another view of the real scene;

determining whether the autocalibrated feature falls within a determined distance of detected feature; and

corresponding the detected feature to the autocalibrated feature when it is determined that the autocalibrated feature falls within the determined distance of the detected feature.

36. The imaging tool of claim 32, wherein the processor corresponds a detected feature to a stored autocalibrated feature by:

estimating a position of the detected feature;

comparing the estimated position to a position of a stored autocalibrated feature;

corresponding the detected feature to the stored autocalibrated feature when the positions fall within a determined distance to each other;

updating the estimated position with the position of the stored autocalibrated feature when a correspondence is determined.

37. The imaging tool of claim 32, wherein a lost pose of the imaging device is recovered.

38. The imaging tool of claim 37, wherein the lost pose is recovered by corresponding at least three detected features to at least three of the stored autocalibrated features.

39. The imaging tool of claim 37, wherein the lost pose is recovered by interactively matching a previous frame with a later frame.

40. The imaging tool of claim 39, wherein the previous and later frames are displayed so that they are blended together with one transparent relative to the other.

41. The imaging tool of claim 37, further comprising a gyroscope, a GPS sensor or an ultrasonic sensor for determining the lost pose of the imaging device.

42. The imaging tool of claim 32, wherein the calibration database and autocalibration database are stored together in a single storage device.

43. A method for producing an augmented image by combining a computer-generated virtual-image with a real-world view of a real scene captured by an imaging device, comprising the steps of:

capturing a view of the real scene with the imaging device;

detecting features in the view corresponding to calibration features in a calibration database containing 3-dimensional coordinates of the calibration features relative to a calibration coordinate frame;

corresponding the detected features to the calibration features to provide the 3-dimensional coordinates of the detected features;

determining a true pose of the imaging means relative to the calibration coordinate frame from the corresponded features;

detecting a shape feature in the view corresponding to a calibration shape feature element in a shape feature database containing 3-dimensional coordinates of the shape feature relative to a shape feature coordinate frame;

corresponding the shape feature to the calibration shape feature element to provide the 3-dimensional coordinates of the detected shape feature;

determining a second pose of the imaging means relative to the shape feature coordinate frame using the corresponded shape feature coordinates;

determining the difference between the true and second poses to determine a transformation between the calibration and shape feature coordinate frames;

autocalibrating the second shape feature using the transformation; and

combining the computer-generated virtual-image with the view of the real scene in an output device based on a view of the shape feature.

44. The method of claim 43, wherein the step of combining the computer-generated virtual-image with the view of the real scene in an output device based on a view of the shape feature also includes:

determining correspondences between the autocalibrated shape feature and features in a model database containing 3-dimensional coordinates of the computer-generated virtual-image relative to a model coordinate frame different from the calibration coordinate frame;

determining a transformation between the calibration coordinate frame and the model coordinate frame based on the correspondences between the autocalibrated shape feature and the model database features; and

combining the computer-generated virtual-image with the view of the real scene in the output device using the transformation.

45. The method of claim 44, further comprising the step of storing the autocalibrated shape feature.

46. The method of claim 45, further comprising the step of recovering lost feature tracking by corresponding detected features to the stored autocalibrated shape feature.

47. The method of claim 43, wherein the shape feature is autocalibrated using the equation:

49. The imaging tool of claim 48, wherein the shape feature is autocalibrated using the equation:

50. The imaging tool of claim 48, wherein:

the model database also contains the coordinates of the autocalibrated shape feature relative to a model coordinate frame so that the processor determines a transformation between the calibration coordinate frame and the model coordinate frame for combining the computer-generated virtual-image with the view of the real scene.

51. The imaging tool of claim 48, further comprising an autocalibration database storing the set of autocalibrated shape features and, in cooperation with the processor, corresponding detected features captured by the imaging device to the autocalibrated shape feature in the autocalibration database to recover lost feature tracking.

48. An imaging tool for producing an augmented image by combining at least one computer-generated virtual-image with a real-world view of a real scene captured by an imaging means, comprising:

a processor;

a data storage means;

a means for transferring data between the storage device and the processor;

an imaging device for capturing views of the real scene;

a means for changing the pose of the imaging device;

a calibration database stored by the data storage means and containing 3-dimensional coordinates relative to a calibration coordinate frame of calibration features corresponding to features detected in views of the real scene;

calibrated features having known 3-dimensional coordinates relative to the calibration coordinate frame obtained by corresponding features in the views of the real scene with the calibration features;

a shape feature database stored by the data storage means and containing 3-dimensional coordinates relative to a shape feature coordinate frame of at least one shape feature detected in views of the real scene;

at least one autocalibrated shape feature determined by autocalibrating the shape feature, the autocalibrating performed by determining a transformation between a true pose of the imaging device calculated from the calibrated features relative to the calibration coordinate frame and a pose of the imaging device calculated from the shape feature relative to the shape feature coordinate frame;

a model database stored by the data storage means and containing 3-dimensional coordinates of the computer-generated virtual-image; and

a display for combining the computer-generated virtual-image with a real-world view of a real scene based, at least in part, on a determination by the computer processor of the pose of the imaging device based on an image of autocalibrated shape feature.

52. A method for producing an augmented image by combining a computer-generated virtual-image with a real-world view of a real scene captured by an imaging device, comprising the steps of:

capturing a view of the real scene with the imaging device;

detecting features in the view corresponding to calibration features in a calibration database containing 3-dimensional coordinates of the calibration features relative to a calibration coordinate frame;

corresponding the detected features to the calibration features to provide the 3-dimensional coordinates of the detected features;

determining the pose of the imaging means from the corresponded features;

selecting a scene attachment point in the view for autocalibration;

changing the pose of the imaging device to a different pose and capturing a different view of the real scene wherein the different view contains the scene attachment point;

selecting the same scene attachment point in the different view;

calibrating the 3D coordinates of the scene attachment point by triangulating the same selected scene attachment point positions of the view and different view; and

attaching the computer-generated virtual-image at the 3D coordinates of the calibrated scene attachment point for display by the imaging device.