Augmented reality user interaction system

• US 20080266323A1
• Filed: 04/25/2007
• Published: 10/30/2008
• Est. Priority Date: 04/25/2007
• Status: Abandoned Application

• Current Assignee: The Board of Trustees of Michigan State University (Michigan State University)
• Original Assignee: The Board of Trustees of Michigan State University (Michigan State University)
• Inventors: Owen, Charles B., Biocca, Frank
• Application Number:
  US11/789,488
• Publication Number:
  US 20080266323A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

An augmented reality user interaction system includes a wearable computer equipped with at least one camera to detect one or more fiducial markers worn by a user. A user-mounted visual display worn by the user is employed to display visual 3D information. The computer detects in an image a fiducial marker worn by the user, extracts a position and orientation of the fiducial marker in the image, and superimposes on the image a visual representation of a user interface component directly on or near the user based on the position and orientation.

Claims

1. An augmented reality user interaction system, comprising:

a wearable computer equipped with at least one camera to detect one or more fiducial markers worn by a user; and

a user-mounted visual display worn by the user and employed to display visual 3D information,

wherein said computer operably detects in an image including at least part of the user one or more fiducial markers worn by the user, extracts a position and orientation of a fiducial marker in the image, and superimposes on the image a visual representation of a user interface component directly on or near the user based on the position and orientation.

2. The system of claim 1, wherein said computer operably displays visual 3D information to the user including the image having the visual representation of the user interface component.

3. The system of claim 2, wherein said computer operably constructs a frame associated with the fiducial marker having a quadrilateral edge containing a valid square image, and operably renders graphics into the frame, including the visual representation of the user interface component.

4. The system of claim 3, wherein said computer operably searches the image for edges, determines whether there are any candidate edges, and traces a candidate edge.

5. The system of claim 4, wherein said computer operably determines whether there are any candidate edges by scanning across rows of the image and looking for local transitions from a background color to a predefined edge color.

6. The system of claim 3, wherein said computer operably determines whether a traced edge approximates a quadrilateral, and computes corners of an edge that has been determined to approximate a quadrilateral.

7. The system of claim 6, wherein said computer operably determines whether a traced edge approximates a quadrilateral by determining if a quadrilateral can be overlaid on the edge such that the edge does not deviate from the quadrilateral by more than a maximum distance determined by a noise tolerance of a capture process employed to capture the image.

8. The system of claim 6, wherein said computer operably computes corners of an edge that has been determined to approximate a quadrilateral by employing all pixels along all four edges of the quadrilateral to determine an optimal line fitting in a least-squared sense by computing four such lines and interpreting intersections of the four lines as the corners of the quadrilateral.

9. The system of claim 3, wherein said computer operably processes an interior of the quadrilateral edge into a square image, and determining whether the square image is valid.

10. The system of claim 9, wherein said computer operably processes an interior of the quadrilateral edge into a square image by warping a portion of the image enclosed by the quadrilateral edge into the square image by determining a mapping of pixels within the quadrilateral edge to corresponding locations in the square image.

11. The system of claim 9, wherein said computer operably determines whether the square image is valid by determining whether the square image matches one of plural predefined images selected to have low correlations to random image data.

12. The system of claim 1, further comprising detecting user interaction with the visual representation of the user interface component based on detected position and orientation of another fiducial marker worn by the user.

13. The system of claim 12, wherein said computer operably superimposes a visual representation of a cursor at or near the other fiducial marker;

and detects user interaction with the visual representation of the user interface component when the user manipulates the other fiducial marker to cause the visual representation of the cursor to appear to the user to interact with the visual representation of the user interface component in a predetermined fashion.

14. The system of claim 12, wherein said computer operably detects the fiducial marker on or near one hand of the user, and detects the other fiducial marker on a ring worn on another hand of the user.

15. The system of claim 1, wherein said computer operably detects user interaction with the visual representation of the user interface component based on user occlusion of the fiducial marker with respect to which the visual representation of the user interface component is visually rendered.

16. The system of claim 1, wherein said computer operably triggers a predetermined interaction when the user interacts with the visual representation of the user interface component.

17. The system of claim 1, wherein said computer operably extracts and recognizes image content located inside or near the fiducial marker, and determines which of plural user interface components to display in relation to the fiducial marker based on the image content.

18. The system of claim 1, wherein said computer operably detects the fiducial marker attached to skin of at least one of a hand, wrist, or arm of the user.

19. The system of claim 1, wherein said computer operably detects the fiducial marker on clothing worn by the user.

20. The system of claim 1, wherein said computer operably detects the fiducial marker on at least one of a watch or jewelry worn by the user.

21. An augmented reality user interaction method, comprising:

visually detecting in an image, including at least part of the user, one or more fiducial markers worn by the user;

extracting a position and orientation of a fiducial marker in the image; and

superimposing on the image a visual representation of a user interface component directly on or near the user based on the position and orientation.

22. The method of claim 21, further comprising displaying visual 3D information to the user including the image having the visual representation of the user interface component.

23. The method of claim 22, further comprising:

constructing a frame associated with the fiducial marker having a quadrilateral edge containing a valid square image; and

rendering graphics into the frame, including the visual representation of the user interface component.

24. The method of claim 23, further comprising:

searching the image for edges;

determining whether there are any candidate edges; and

tracing a candidate edge.

25. The method of claim 24, wherein determining whether there are any candidate edges includes scanning across rows of the image and looking for local transitions from a background color to a predefined edge color.

26. The method of claim 23, further comprising:

determining whether a traced edge approximates a quadrilateral; and

computing corners of an edge that has been determined to approximate a quadrilateral.

27. The method of claim 26, wherein determining whether a traced edge approximates a quadrilateral includes determining if a quadrilateral can be overlaid on the edge such that the edge does not deviate from the quadrilateral by more than a maximum distance determined by a noise tolerance of a capture process employed to capture the image.

28. The method of claim 26, wherein computing corners of an edge that has been determined to approximate a quadrilateral includes employing all pixels along all four edges of the quadrilateral to determine an optimal line fitting in a least-squared sense by computing four such lines and interpreting intersections of the four lines as the corners of the quadrilateral.

29. The method of claim 23, further comprising:

processing an interior of the quadrilateral edge into a square image; and

determining whether the square image is valid.

30. The method of claim 29, wherein processing an interior of the quadrilateral edge into the square image includes warping a portion of the image enclosed by the quadrilateral edge into the square image by determining a mapping of pixels within the quadrilateral edge to corresponding locations in the square image.

31. The method of claim 29, wherein determining whether the square image is valid includes determining whether the square image matches one of plural predefined images selected to have low correlations to random image data.

32. The method of claim 21, further comprising detecting user interaction with the visual representation of the user interface component based on detected position and orientation of another fiducial marker worn by the user.

33. The method of claim 32, further comprising:

superimposing a visual representation of a cursor at or near the other fiducial marker; and

detecting user interaction with the visual representation of the user interface component when the user manipulates the other fiducial marker to cause the visual representation of the cursor to appear to the user to interact with the visual representation of the user interface component in a predetermined fashion.

34. The method of claim 32, further comprising:

detecting the fiducial marker on or near one hand of the user; and

detecting the other fiducial marker on a ring worn on another hand of the user.

35. The method of claim 21, further comprising detecting user interaction with the visual representation of the user interface component based on user occlusion of the fiducial marker with respect to which the visual representation of the user interface component is visually rendered.

36. The method of claim 21, further comprising triggering a predetermined interaction when the user interacts with the visual representation of the user interface component.

37. The method of claim 21, further comprising employing a camera worn by the user to capture the image including at least part of the user.

38. The method of claim 21, further comprising employing a visual display worn by the user to display visual 3D information to the user including the image having the visual representation of the of the user interface component.

39. The method of claim 21, further comprising:

extracting and recognizing image content located at least one of inside or near the fiducial marker; and

determining which of plural user interface components to display in relation to the fiducial marker based on the image content.

40. The method of claim 21, further comprising detecting the fiducial marker attached to skin of at least one of a hand, wrist, or arm of the user.

41. The method of claim 21, further comprising detecting the fiducial marker on clothing worn by the user.

42. The method of claim 21, further comprising detecting the fiducial marker on at least one of a watch or jewelry worn by the user.