New input devices for augmented reality applications

• US 20040113885A1
• Filed: 05/22/2002
• Published: 06/17/2004
• Est. Priority Date: 05/31/2001
• Status: Active Grant

• Current Assignee: Siemens Corp. (Siemens AG)
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Genc, Yakup, Navab, Nassir
• Granted Patent: US 7,215,322 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A new interaction/input device for an augmented reality system is provided. In an augmented reality system using infrared video-based tracking, the interaction/input device is employed by placing markers, e.g., small disks, at a predetermined location in a scene viewed by a user which are augmented to simulate physical buttons. These augmented markers, as viewed through the augmented reality system, can then be physically manipulated by the user. The user will put their fingers on one of these markers, and in turn, the infrared video-based tracker will recognize this action and process it accordingly. The augmented reality system can also augment simulated menus in the user'"'"'s view giving the user the necessary feedback for interaction. By exploiting conventional tracking technology, the interaction/input device can be implemented with minimal additional hardware and minimal additional processing required by the augmented reality system.

Claims

1. A system for augmenting a user'"'"'s view of real-world objects with virtual objects to provide a composite augmented reality image, the system comprising:

a display device for displaying the composite augmented reality image to the user;

a video-based tracking system for locating real-world objects;

a processor for determining the position and orientation of the user'"'"'s view based on the location of the real-world objects and for projecting the virtual objects onto the display device; and

an input device including at least one marker placed at a predetermined location in the real world, wherein the tracking system locates the input device and the processor determines its functionality based on its location.

2. The system as in claim 1, wherein the video-based tracking system is an infrared video-based tracking system.

3. The system as in claim 2, wherein the infrared video-based tracking system comprises:

a camera for capturing video of the real world, the camera including an infrared-filter lens for capturing light in an infrared frequency; and

a plurality of infrared illumination light-emitting diodes (LEDs) for providing the infrared light.

4. The system as in claim 2, wherein the at least one marker of the input device is retroreflective.

5. The system as in claim 2, further comprising a marker board including a plurality of retroreflective markers surrounded by a continuous band of retroreflective material, the marker board being located in the real world at a known location to serve as a point of reference for the infrared video-based tracking system.

6. The system as in claim 1, wherein the at least one marker of the input device is augmented on the display device with a virtual object representing a button.

7. The system as in claim 1, wherein the at least one marker of the input device is augmented on the display device with a virtual object representing a menu of selectable options.

8. The system as in claim 1, wherein the input device further comprises a plurality of markers, each of the plurality of markers being augmented with different virtual objects.

9. The system as in claim 8, wherein when one of the plurality of markers is activated the remaining markers are augmented with virtual objects corresponding to options of the activated marker.

10. In a system for augmenting a user'"'"'s view of real-world objects with virtual objects to provide a composite augmented reality image, the system including a display device for displaying the composite augmented reality image to the user, a video-based tracking system for locating real-world objects, and a processor for determining the position and orientation of the user'"'"'s view based on the location of the real-world objects and for projecting the virtual objects onto the display device, an input device comprising:

at least one marker placed at a predetermined location in the real world, wherein the tracking system locates the input device and the processor determines its functionality based on its location.

11. The input device as in claim 10, wherein the video-based tracking system is an infrared video-based tracking system.

12. The input device as in claim 11, wherein the infrared video-based tracking system comprises:

a camera for capturing video of the real world, the camera including an infrared-filter lens for capturing light in an infrared frequency; and

a plurality of infrared illumination light-emitting diodes (LEDs) for providing the infrared light.

13. The input device as in claim 11, wherein the at least one marker is retroreflective.

14. The input device as in claim 11, wherein the infrared video-based tracking system comprises a marker board including a plurality of retroreflective markers surrounded by a continuous band of retroreflective material, the marker board being located in the real world at a known location to serve as a point of reference for the infrared video-based tracking system.

15. The input device as in claim 10, wherein the at least one marker is augmented on the display device with a virtual object representing a button.

16. The input device as in claim 10, wherein the at least one marker is augmented on the display device with a virtual object representing a menu of selectable options.

17. The input device as in claim 10, further comprising a plurality of markers, each of the plurality of markers being augmented with different virtual objects.

18. The input device as in claim 17, wherein when one of the plurality of markers is activated the remaining markers are augmented with virtual objects corresponding to options of the activated marker.

19. In a system for augmenting a user'"'"'s view of real-world objects with virtual objects to provide a composite augmented reality image, the system including a display device for displaying the composite augmented reality image to the user, a video-based tracking system for locating real-world objects, and a processor for determining the position and orientation of the user'"'"'s view based on the location of the real-world objects and for projecting the virtual objects onto the display device, a method for interacting with the system comprising the steps of:

providing an input device including at least one marker placed at a predetermined location in the real world;

capturing video of the real world by the video-based tracking system;

analyzing the captured video to determine if the at least one marker is visible;

if the at least one marker is visible, determining the real world location of the at least one marker; and

loading the input device'"'"'s functionality into the system to be available for the user to interact with the system.

20. The method as in claim 19, wherein the loading step further comprises the step of entering an input mode.

21. The method as in claim 20, further comprising the steps of:

determining if the at least one marker is not visible after entering the input mode; and

if the at least one marker is not visible, performing a function associated with the at least one marker.

22. The method as in claim 19, wherein the video-based tracking system is an infrared video-based tracking system and the at least one marker of the input device is retroreflective.

23. The method as in claim 19, further comprising the step of augmenting the at least one marker of the input device with a virtual object.

24. The method as in claim 19, further comprising the step of augmenting the at least one marker of the input device with a virtual object representing a menu of selectable options.

25. The method as in claim 19, wherein the input device comprises a plurality of markers, each of the plurality of markers being augments with different virtual objects, further comprising the step of, when at least one of the markers is activated, augmenting the remaining markers with virtual objects corresponding to options of the activated marker.

26. The method as in claim 19, further comprising the step of providing a marker board, the marker board being located at the real world in a known location to serve as a point of reference for the video-based tracking system.