METHOD AND SYSTEM FOR HAPTIC INTERACTION IN AUGMENTED REALITY

• US 20090195538A1
• Filed: 02/03/2009
• Published: 08/06/2009
• Est. Priority Date: 02/04/2008
• Status: Active Grant

• Current Assignee: Gwangju Institute of Science and Technology
• Original Assignee: Gwangju Institute of Science and Technology
• Inventors: Lee, Yong-Gu, Hwang, Sun-Uk, Ryu, Jeha, Lee, Beom-Chan, Park, Hyeshin
• Granted Patent: US 8,243,099 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2210/21 Collision detection, intersection
• CPC Class Codes: G06T 2210/28 Force feedback
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G09G 2320/02 Improving the quality of display appearance
• CPC Class Codes: G09G 2320/103 Detection of image changes, e.g. determination of an index representative of the image change
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/06 using colour palettes, e.g. look-up tables

Abstract

The present invention relates to a method and system for haptic interaction in augmented reality. More specifically, the present invention relates to a method and system for haptic interaction in augmented reality that can effectively remove noise from real images captured by an image capturing device and minimize discontinuity of force generated in the haptic interaction so as to stably and smoothly perform the haptic interaction in the augmented reality.

The present invention provides an augmented reality system and a method for providing augmented reality therefor, the augmented reality system comprising: a marker detecting unit that receives images for a real environment including markers and detects the markers included in the images; a marker position extracting unit that extracts the positions of the detected markers; a noise removing unit that removes noise from positional information of the markers extracted from the marker position extracting unit; a visual rendering unit including a virtual object augmenting unit that augments virtual objects corresponding to the markers and outputs them to the positions of the markers; a motion estimating unit that estimates the motion of the markers over a time step so as to perform the haptic rendering on the basis of the positional information of the markers transferred from the noise removing unit; a collision detecting unit that detects whether the virtual objects corresponding to the positions of the markers estimated by the motion estimating unit collide with end points of the haptic device; and a haptic rendering unit including reaction force calculating unit that calculates reaction force to be provided through the haptic device according to a determination of the collision detecting unit.

Claims

1. An augmented reality system including haptic interaction, comprising:

a marker detecting unit that receives images for a real environment including markers and detects the markers included in the images;

a marker position extracting unit that extracts the positions of the detected markers;

a noise removing unit that removes noise from positional information of the markers extracted from the marker position extracting unit;

a visual rendering unit including a virtual object augmenting unit that augments virtual objects corresponding to the markers and outputs them to the positions of the markers;

a motion estimating unit that estimates the motion of the markers over a time step so as to perform the haptic rendering on the basis of the positional information of the markers transferred from the noise removing unit;

a collision detecting unit that detects whether the virtual objects corresponding to the positions of the markers estimated by the motion estimating unit collide with end points of the haptic device; and

a haptic rendering unit including reaction force calculating unit that calculates reaction force to be provided through the haptic device according to a determination of the collision detecting unit.

2. The augmented reality system according to claim 1, wherein the positional information of the marker includes three-dimensional positional information and orientation information of the marker.

3. The augmented reality system according to claim 1, wherein the noise removing unit is configured of any one of a low pass filter, an FIR filter, and an IIR filter or a filter equivalent thereto.

4. The augmented reality system according to claim 1, wherein the noise removing unit is configured of any one of a least square estimator (MSE), a minimum mean squared estimator, a maximum likelihood estimator, a maximum A posteriori estimator, Bayes estimator, Kalman filter, a particle filter, a Markov chain Monte Carlo estimator, a minimum variance unbiased estimator, and a best linear unbiased estimator or an estimator equivalent thereto, such that it can simultaneously perform the noise removal and the motion estimation of the markers.

5. The augmented reality system according to claim 1, wherein the noise removing unit is implemented by an extended Kalman filter that filters and expects the noise from the positional information of the marker by continuously estimating and correcting the noise from the initial value.

6. The augmented reality system according to claim 5, wherein in the extended Kalman filter, a model for the marker tracking is represented by a state vector including position, velocity, and acceleration and system noise caused in the tracking process and is continuously estimated based on actually measured values.

7. The augmented reality system according to claim 5 or 6, wherein the extended Kalman filter estimates a priori estimate of the state vector and a priori estimate of error covariance from initial conditions and then obtains a Kalman gain and obtains a posteriori estimate of the state vector by correcting the priori estimate of the state vector using the actually measured values and the Kalman gain.

8. The augmented reality system according to any one of claim 1 to 6, wherein the motion estimating unit estimates the motion of the marker by an adaptive extrapolation that determines the increment of position of the marker for the haptic rendering after a current time based on the difference between the position of the marker, which is finally estimated by applying the extrapolation method based on the previous time image, and the actual position of the marker in a current time image.

9. The augmented reality system according to claim 8, wherein the position of the marker, which is finally estimated based on the previous time image, is continuously monitored for every image frame.

10. The augmented reality system according to 8, wherein any one of the visual rendering unit and the haptic rendering unit further includes a coordinate synchronizing unit that synchronizes visual coordinates for the visual rendering with haptic coordinates for the haptic rendering.

11. A method for proving augmented reality including haptic interaction, comprising:

(a) receiving images for a real environment including markers;

(b) detecting the markers in the images and extracting the positions of the detected markers;

(c) removing noise included in the positional information of the makers;

(d) estimating the motion of the markers over a time step so as to perform the haptic rendering on the basis of the positional information of the markers;

(e) detecting whether the virtual objects corresponding to the markers collide with end points of the haptic device, calculating reaction force and outputting it to the haptic device; and

(f) augmenting the virtual objects at the marker positions for the real images and outputting them to the visual display.

12. The method for proving augmented reality according to claim 11, wherein the positional information of the marker includes three-dimensional positional information and orientation information of the markers.

13. The method for proving augmented reality according to claim 11, wherein the step (c) is performed by any one of a low pass filter, an FIR filter, and an IIR filter or a filter equivalent thereto.

14. The method for proving augmented reality according to claim 11, wherein the step (c) is performed by any one of a least square estimator (MSE), a minimum mean squared estimator, a maximum likelihood estimator, a maximum A posteriori estimator, Bayes estimator, Kalman filter, a particle filter, a Markov chain Monte Carlo estimator, a minimum variance unbiased estimator, and a best linear unbiased estimator or an estimator equivalent thereto, such that it can simultaneously perform the noise removal and the motion estimation of the markers.

15. The method for proving augmented reality according to claim 11, wherein the step (c) is performed by an extended Kalman filter that filters and expects the noise from the positional information of the marker by continuously estimating and correcting the noise from the initial value.

16. The method for proving augmented reality according to claim 11, wherein the step (c) includes:

(c1) estimating a priori estimate for a state vector including position, velocity, and acceleration of the marker and a priori estimate of error covariance from initial conditions; and

(c2) obtaining a Kalman gain and obtaining a posteriori estimate of the state vector by correcting the priori estimate of the state vector using the actually measured values and the Kalman gain.

17. The method for proving augmented reality according to any one of claims 11 to 16, wherein the step d) estimates the motion of the marker by an adaptive extrapolation that determines the increment of position of the marker for the haptic rendering after a current time based on the difference between the position of the marker, which is finally estimated by applying the extrapolation method based on the previous time image, and the actual position of the marker in a current time image.

18. The method for proving augmented reality according to claim 17, wherein the position of the marker, which is finally estimated based on the previous time image, is continuously monitored for every image frame.