Tools for interacting with virtual environments

• US 6,842,175 B1
• Filed: 12/07/1999
• Issued: 01/11/2005
• Est. Priority Date: 04/22/1999
• Status: Expired due to Term

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer USA Inc. (Fraunhofer-Institut für Nachrichtentechnik HHI)
• Inventors: Schmalstieg, Dieter, Bimber, Oliver, Encarnacão, L. Miguel
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/959,087
• Time in Patent Office: 1,862 Days
• Field of Search: 345/418, 345/419, 345/426, 345/427, 345/428, 345/619
• US Class Current: 345/427
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

The tools include components which are images produced by the system (103) that creates the virtual environment in response to inputs specifying the tool'"'"'s location and the point of view of the tool'"'"'s user. One class of the tools includes transparent components (121, 123); the image component of the tool is produced at a location in the virtual environment determined by the transparent component'"'"'s location and the point of view of the tool'"'"'s user. Tools in this class include a transparent pad (121) and a transparent stylus (123). The transparent pad (121) has an image component that may be used as a palette, as a magic lens, and as a device for selecting and transporting components of the virtual environment or even a portion of the virtual environment itself.

Claims

1. A virtual environment system of a type that produces a virtual environment in a display, the virtual environment being produced in the display as seen from a given direction and point of view and

the virtual environment system having the improvement comprising:

a device that is separate from the display, is used to view a portion of the display from the given direction and point of view, and whose position and orientation are received by the virtual environment system together with the given direction and point of view; and

the virtual environment system responding to the device position and orientation and to the given direction and point of view to produce a modification of the virtual environment in the portion of the display that is viewed using the device.

2. The virtual environment system set forth in claim 1 wherein:

the device reflects the portion of the display when viewed from the given direction and point of view.

3. The virtual environment system set forth in claim 2 wherein:

the device has a plurality of reflective surfaces; and

the modification produced by the virtual environment system varies with the reflective surface that is reflecting the portion.

4. The virtual environment system set forth in claim 2 wherein:

the virtual environment system responds to the device position and orientation and to the given direction and point of view to determine a second direction and point of view which the virtual environment system uses with the device position and orientation; and

the modification of the virtual environment in the portion is the virtual environment as seen from the second direction and point of view.

5. The virtual environment system set forth in claim 4 wherein:

the second direction and point of view is that which the reflection of the given direction and point of view in the device appears to have.

6. The virtual environment system set forth in claim 5 wherein:

the virtual environment in the portion differs from the virtual environment outside the portion in more than being seen from the second point of view, whereby the virtual environment in the portion operates as a magic lens in the virtual environment.

7. The virtual environment system set forth in claim 2 further comprising:

a ray tool that produces a ray for selecting a location in the virtual environment, the ray tool'"'"'s position and orientation being received in the virtual environment, and

the virtual environment responding when the ray tool'"'"'s position and orientation relative to the device is such that a reflection of the ray from the device falls within the virtual environment in the portion of the display that is reflected in the device by selecting the location in the virtual environment upon which the reflection of the ray falls.

8. The virtual environment system set forth in claim 1 wherein:

the device is transflective; and

the portion of the display is located relative to the device such that the modification of the virtual environment in the portion is reflected in the device when the device is seen from the given direction and point of view,

whereby the reflection of the modification is seen together with an object that is seen through the device.

9. The virtual environment system set forth in claim 8 wherein:

the virtual environment system further responds to the location of the object by locating the portion of the display such that the reflection of the modification of the virtual environment appears to be at a predetermined location relative to the object seen through the device.

10. The virtual environment system set forth in claim 9 wherein:

the object seen through the device is a virtual object;

the apparent location of the reflection of the object is that of the virtual object; and

the modification of the virtual environment in the portion is identical in appearance to the virtual object,

whereby the reflection of the modification appears to overlay an object that occludes the virtual object.

11. The virtual environment system set forth in claim 9 wherein:

the object seen through the device is a real object; and

the modification of the portion of the virtual environment is an augmentation of the real object.

12. The virtual environment system set forth in claim 1 wherein:

the portion of the display is viewed through the device.

13. The virtual environment system set forth in claim 12 wherein:

the device has at least two sides; and

the modification of the virtual environment produced by the virtual environment system in the portion depends on which side of the device is being seen from the given direction and point of view.

14. The virtual environment system set forth in claim 12 wherein:

the virtual environment system produces the modification of the virtual environment such that the portion has the appearance of being viewed through a transparent panel having the orientation of the device.

15. The virtual environment system set forth in claim 14 wherein:

the virtual environment system further alters the modification of the virtual environment depending on an object in the virtual environment which has the appearance of being viewed through the transparent panel,

whereby the modification is made context-sensitive.

16. The virtual environment system set forth in claim 12 wherein:

when an object is in the portion, the object is subject to manipulation.

17. The virtual environment system set forth in claim 12 wherein:

the device is movable by a user; and

an object that is in a volume of the virtual environment through which the device appears to sweep is thereby selected for manipulation.

18. The virtual environment system set forth in claim 12 wherein:

the device is movable by the user;

the virtual environment system further receives one or more mode inputs; and

the virtual environment system responds to a first mode input by binding a copy of an item in the virtual environment in the portion currently being viewed through the device to the device, whereby the copy becomes separate from the copied item and the virtual environment in the portion includes the copy.

19. The virtual environment system set forth in claim 18 wherein:

the virtual environment system responds to a second mode input by unbinding the copy from the device, the copy remaining in the virtual environment at the point where the copy was unbound.

20. The virtual environment system set forth in claim 19 wherein:

the virtual environment system responds to a third mode input by in the alternative binding the copy to the copied portion, so that a change in either the copy or the copied portion also takes place in the other or unbinding the copy from the copied portion, so that a change in either the copy or the copied portion does not affect the other.

21. The virtual environment system set forth in claim 12 wherein:

the device is movable and

the virtual environment in the portion includes one or more objects that move with the device.

22. The virtual environment system set forth in claim 21 further comprising:

a selection device that is separate from the display and movable independently of the device and that is capable of selecting objects in the virtual environment in the portion or elsewhere in the virtual environment.

23. The virtual environment system set forth in claim 22 wherein:

the selection device selects an object in the virtual environment in the portion for placement elsewhere in the virtual environment.

24. The virtual environment system set forth in claim 22 wherein:

the selection device selects an object elsewhere in the virtual environment for placement in the virtual environment in the portion.

25. The virtual environment system set forth in claim 22 wherein:

the object in the virtual environment in the portion represents an operation; and

the virtual environment system responds to a selection of the object by the selection device by performing the operation represented by the object.

26. The virtual environment system set forth in claim 25 wherein:

the selection device is a transparent device whose position and orientation are received by the virtual environment system; and

the virtual environment system responds to the position and orientation of the selection device and to the given direction and point of view to produce an image of the selection device in the virtual environment in the portion of the display viewed through the selection device.

27. The virtual environment system set forth in claim 26 wherein:

the image of the selection device includes a visual representation of the specified operation.

28. The virtual environment system set forth in claim 25 wherein:

the virtual environment system produces a plurality of types of modifications of the virtual environment; and

the specified operation is producing another type of modification.

29. The virtual environment system set forth in claim 25 wherein:

the specified operation is modifying a selected object.

30. The virtual environment system set forth in claim 29 wherein:

There is a plurality of objects that specify operations; and

the objects in the virtual environment in the portion vary according to the selected object.

31. The virtual environment system set forth in claim 30 wherein:

the objects in the virtual environment in the portion further vary according to a previously-selected operation.

32. The virtual environment system set forth in claim 25 wherein:

the object is a button that is activated by placing the image of the selection device within the confines of the button.

33. The virtual environment system set forth in claim 25 wherein:

the object is a slider that is moved by means of the image of the selection device.

34. The virtual environment system set forth in claim 21 wherein:

the stylus is transparent; and

the virtual environment system responds to the position and orientation of the stylus and to the given direction and point of view to produce an image of a stylus in the virtual environment in the portion of the display viewed through the stylus.

35. The virtual environment system set forth in claim 12 wherein:

the device includes a transparent panel;

the virtual environment system further includes a stylus, the stylus'"'"' position and orientation being received in the virtual environment; and

when a user of the virtual environment system moves the stylus'"'"' end on or near the surface of the transparent panel, the virtual environment system responds thereto by making corresponding marks in the virtual environment in the portion.

36. The virtual environment system set forth in claim 35 further comprising:

a gesture manager that interprets movements of the stylus'"'"' end on or near the surface of the transparent panel as a gesture specifying an operation and provides a specifier for the operation to the virtual environment system, the virtual environment system responding to the specifier by performing the operation.

37. The virtual environment system set forth in claim 36 wherein:

the gesture indicates creation of a type of object.

38. The virtual environment system set forth in claim 37 wherein:

the gesture further indicates a size and orientation for the object.

39. The virtual environment system set forth in claim 36 wherein:

the gesture indicates that a selected object in the portion is to be deleted.

40. The virtual environment system set forth in claim 36 wherein:

the gesture indicates that a previous operation is to be undone.

41. The virtual environment system set forth in claim 36 wherein:

the virtual environment system produces a plurality of types of modifications; and

the gesture indicates that the type of modification is to be changed.

42. The virtual environment system set forth in claim 36 wherein:

the gesture indicates that an object in the portion is to be selected.

43. The virtual environment system set forth in claim 36 wherein:

the gesture indicates a letter or a numeral that is to be input to the virtual environment system.

44. The virtual environment system set forth in claim 36 wherein:

the gesture manager learns for a gesture the operation specified by the gesture.

45. The virtual environment system set forth in claim 1 wherein:

depending on the orientation of the device relative to the virtual environment, the device is in the alternative transparent and reflective; and

when the orientation of the device indicates that the device is reflective, the virtual environment system responds to the device position and orientation and to the given direction and point of view such that the portion is reflected by the device when the device is viewed from the given direction and point of view; and

when the orientation of the device indicates that the device is transparent, the portion is viewed through the device from the given direction and point of view.

46. The virtual environment system set forth in claim 1 wherein:

the given direction and point of view have a fixed relationship to the device.

47. The virtual environment system set forth in claim 46 wherein:

the device is movable relative to the virtual environment.

48. The virtual environment system set forth in claim 1 wherein:

both the device and the given direction and point of view are movable relative to the virtual environment.

49. The virtual environment system set forth in claim 48 wherein:

the virtual environment system produces a plurality of types of modifications; and

which type of modification is determined by the orientation of the device.

50. The virtual environment system set forth in claim 48 wherein:

the given direction and point of view are those of a user of the virtual environment system; and

the user holds the device in the user'"'"'s hand.

51. The virtual environment system set forth in claim 1 wherein:

the virtual environment in the portion functions as a magic lens.

52. The virtual environment system set forth in claim 1 wherein:

the device is a transparent panel.

53. The virtual environment system set forth in claim 1 wherein:

the virtual environment system further receives a mode input; and

the virtual environment system produces a plurality of types of modifications and which type is produced is determined by the mode input.

54. The virtual environment system set forth in claim 53 wherein:

the types of modifications produced by the virtual environment system include at least a modification that functions as a magic lens into the virtual environment.

55. Apparatus for producing an image of a tool in a virtual environment system of a type that produces a virtual environment in a display, the virtual environment being produced in the display as seen from a given direction and point of view and the apparatus comprising:

a real object that is separate from the display and is held by a user of the virtual environment system, the object including at least a handle and the object'"'"'s position and orientation being provided to the virtual environment system; and

parameters for the tool'"'"'s image in the virtual environment system, the virtual environment system responding to the given direction and point of view, the parameters, and the object'"'"'s position and orientation by producing an image of the tool in the virtual environment at a position and orientation therein that corresponds to the position and orientation of the object and the given direction and point of view.

56. The apparatus set forth in claim 55 wherein:

the object further includes a transparent component extending beyond the handle.

57. The apparatus set forth in claim 56 wherein:

the extent of the transparent component corresponds to the extent of the image in the virtual environment.

58. The apparatus set forth in claim 57 wherein:

the apparatus produces images of more than one tool;

each tool has an object corresponding thereto; and

when one tool is being used in conjunction with another tool, the transparent component of the one object comes into contact with the transparent component of the other object,

whereby the user is provided with tactile feedback.

59. The apparatus set forth in claim 58 wherein:

one of the objects has a planar transparent extent and the image corresponding thereto is a planar surface;

another of the objects has a stylus-like transparent extent and the image corresponding thereto is a stylus; and

the user applies the stylus-like transparent extent to the planar transparent extent, whereby the stylus tool performs an operation on the planar surface tool.

60. The apparatus set forth in claim 56 wherein:

the transparent component is a transparent pad.

61. The apparatus set forth in claim 60 wherein:

the image of the tool is a modification of the virtual environment that corresponds to the position and orientation of the transparent pad and the given direction and point of view.

62. The apparatus set forth in claim 55 wherein:

the apparatus produces a plurality of images of tools, with the image being produced at a given point in time being dependent on the position and orientation of the real object.

63. The apparatus set forth in claim 55 wherein:

the parameters include a mode parameter;

the apparatus produces a plurality of images of tools, with the image being produced at a given point in time being dependent on the mode parameter.

64. The apparatus set forth in claim 55 wherein:

the apparatus produces a plurality of images of tools, with the image being produced at a given point in time being dependent on the image'"'"'s position in the virtual environment.