Wearable display and methods for controlling same

• US 6,127,990 A
• Filed: 01/21/1999
• Issued: 10/03/2000
• Est. Priority Date: 11/28/1995
• Status: Expired due to Fees

• Current Assignee: Rembrandt Portable Display Technologies, LP (Rembrandt IP Management, LLC)
• Original Assignee: Vega Vista, Inc.
• Inventors: Zwern, Arthur L.
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Lewis, David L.
• Application Number:
  US09/235,096
• Time in Patent Office: 621 Days
• Field of Search: 345/7, 345/8, 345/9, 345/156, 345/158, 345/127, 345/130, 345/150, 345/339, 345/427, 348/63, 348/115, 348/531, 348/121, 395/235, 364/708.1
• US Class Current: 345/8
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

A virtual computer monitor is described which enables instantaneous and intuitive visual access to large amounts of visual data by providing the user with a large display projected virtually in front of the user. The user wears a head-mounted display containing a head-tracker, which together allow the user to position an instantaneous viewport provided by the head-mounted display at any position within the large virtual display by turning to look in the desired direction. The instantaneous viewport further includes a mouse pointer, which may be positioned by turning the user'"'"'s head, and which may be further positioned using a mouse or analogous control device. A particular advantage of the virtual computer monitor is intuitive access to enlarged computer output for visually-impaired individuals.

Claims

1. A device for assisting a computer user in interacting with a computer or like data processing device, comprising:

a wearable display device coupled to said computer'"'"'s display output, said wearable display including an instantaneous viewing window;

a visual interface device resident on said computer, said visual interface device providing a virtual desktop whose size is larger than the instantaneous viewing window provided by said wearable display;

a tracking device for sensing motion of the wearable display device, converting said sensory input generated by said motion sensed into tracking data, and inputting said tracking data into said computer;

a magnification selection device responsive to said sensory input generated by said motion sensed into tracking data for selection of an enlargement factor applied to the enlarged data; and

,

a signal processing device resident on said computer for interpreting said tracking data for purposes of determining which portion of the virtual desktop will be displayed within said instantaneous viewing window at a particular point in time,

whereby the computer user can view any portion of the virtual desktop and control the magnification of the portion displayed in said instantaneous viewing window by controlling the position of the tracking device.

2. A device as described in claim 1, wherein a mouse pointer is made visible within said head-mounted display, and remains linked to the user'"'"'s line of sight as the user'"'"'s line of sight changes.

3. A device as in claim 2, further providing a secondary control device for positioning the mouse pointer, by which the user can perform fine adjustment of the mouse pointer location within the virtual desktop after moving to an approximate location using said head-tracking device.

4. A device as in claim 3, where said secondary mouse pointer positioning device is hand-operated.

5. A device as in claim 3, where said secondary mouse pointer positioning device is finger operated.

6. A device as described in claim 2, wherein said signal processing device further performs spatial field compression to reduce the total head orientation change required for redirecting the user'"'"'s line of sight from one location within the virtual desktop to another location.

7. A device as described in claim 3, wherein said signal processing device further performs line-of-sight snap-back processing to return the user'"'"'s instantaneous viewing window to a location within the virtual desktop determined by the user'"'"'s head orientation following temporary modification of said viewing window location by said secondary mouse pointer control function.

8. A device as described in claim 2, wherein said mouse pointer may be positioned at any user-selectable position within said head-mounted display'"'"'s field of view, and maintained at that position regardless of changes in the user'"'"'s head orientation.

9. A device as in claim 2, further including a spatialized audio cue generation device to provide audio signals to the user which appear to originate from portions of said virtual desktop which are not within the user'"'"'s instantaneous field of view, thereby alerting the user to turn and look in the direction of said spatialized audio cues.

10. A device as in claim 2, further including a device for accepting input of voice commands spoken by the user, and a device for converting said commands into internal computer formats through which individual computer functions may be controlled.

11. A device as in claim 10 wherein said computer functions include selection of a location or object within said virtual desktop.

12. A device as described in claim 10 wherein said computer functions include selection of specific locations for repositioning said instantaneous viewing window within said virtual desktop.

13. A device as described in claim 2 wherein said signal processing device further includes predictive algorithms to reduce apparent latency in repositioning of said instantaneous viewing window.

14. A device as described in claim 2 which further includes a device for generating multiple application windows within said virtual desktop, wherein the user may select an individual application window by head reorientation.

15. A device as described in claim 1, further including a real-time image sensor device, a real-time image digitization device, and a processing device for displaying the captured imagery within said virtual desktop.

16. A device as described in claim 15, where said captured imagery is displayed within one application window on said virtual desktop, and at least one computer application program is displayed within at least one additional application window on said virtual desktop.

17. A computer implemented method for assisting a user in the control and operation of a computer system having a wearable display device, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon any display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping a user with a wearable display device coupled to a computer system;

mapping visual feedback generated by the computer system into a virtual desktop suitable for display via the wearable display device;

displaying a certain portion of the virtual desktop via the wearable display device;

tracking movement of the wearable display device; and

updating the displayed certain portion of the virtual desktop in a manner correlated to the tracked movement of the wearable display device, the updated characteristics including virtual position of the displayed certain portion and virtual magnification of the displayed certain portion.

18. A computer implemented method as recited in claim 17 further comprising the step of redefining the orientation of the certain portion displayed via the wearable display device such that, without moving the wearable display device, the certain portion displayed via the wearable display device changes.

19. A computer implemented method as recited in claim 18 wherein the orientation of the certain portion displayed is redefined in response to a request by the user.

20. A computer implemented method as recited in claim 17 wherein the step of mapping the visual feedback into the virtual desktop includes the substep of performing spatial field compression to reduce the required motion of the wearable display device necessary for redirecting the displayed certain portion from one location within the virtual desktop to another location within the virtual desktop.

21. A computer implemented method as recited in claim 17 wherein the visual feedback generated by the computer system includes multiple application windows.

22. A computer implemented method as recited in claim 21 wherein a first window of the multiple application windows corresponds to a first application executing upon the computer system.

23. A computer implemented method as recited in claim 22 wherein the first application executing upon the computer system is a physical map application.

24. A computer implemented method as recited in claim 23 wherein the physical map application enables navigation through a physical map via user movement of the wearable display device.

25. A computer implemented method as recited in claim 22 further comprising the steps of:

monitoring a real scene in real space and time;

capturing images of the real scene; and

displaying within a first window of the multiple application windows the captured images of the real scene.

26. A computer implemented method as recited in claim 25 wherein a second window of the multiple application windows corresponds to an application program executing upon the computer system.

27. A computer implemented method for assisting a user in the control and operation of a computer system having a wearable display device, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon any display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping a user with a wearable display device coupled to a computer system;

mapping visual feedback generated by the computer system into a virtual desktop suitable for display via the wearable display device;

displaying a certain portion of the virtual desktop via the wearable display device;

tracking movement of the wearable display device;

updating the displayed certain portion of the virtual desktop in a manner correlated to the tracked movement of the wearable display device, the updated characteristics including virtual position of the displayed certain portion and virtual magnification of the displayed certain portion;

monitoring a real scene in real space and time;

capturing images of the real scene; and

providing the captured images of the real scene somewhere within the virtual desktop.

28. A computer implemented method as recited in claim 27 further comprising the step of displaying a pointer within the certain portion of the virtual desktop.

29. A computer implemented method as recited in claim 28 further comprising the steps of:

receiving a request to adjust a fixed relation between the pointer and the user'"'"'s line of sight; and

adjusting the fixed relation in accordance with the received request.

30. A computer implemented method as recited in claim 28 further comprising the step of providing the user a pointer control input device for controlling operation of the pointer, the pointer control input device coupled to the computer system.

31. A computer implemented method as recited in claim 30 further comprising the steps of:

receiving a create selection box command via the pointer control input device;

anchoring a first comer of a selection box at a first point where the pointer is disposed within the certain portion displayed upon receipt of the create selection box command;

sizing the selection box in conjunction with movement of the user'"'"'s line of sight;

receiving a close selection box command via the pointer control input device;

defining the selection box by anchoring a second comer of the selection box, the second corner opposing the first corner, the second corner anchored at a second point where the pointer is disposed within the certain portion displayed upon receipt of the close selection box command; and

marking as selected that portion of the virtual desktop which is bounded by the selection box.

32. A computer implemented method as recited in claim 31 wherein the create selection box command corresponds to the depression of a button located on the pointer control input device and the close selection box command corresponds to the release of the button located on the pointer control input device.

33. A computer implemented method as recited in claim 28 further comprising the step of redefining the orientation of the certain portion displayed within the virtual desktop such that, without adjusting the user'"'"'s line of sight, the certain portion displayed within the wearable display device changes.

34. A computer system particularly well-suited for use by a low-vision user, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon a display screen coupled with the computer system, the computer system comprising:

a central processing unit (CPU);

random access memory (RAM);

a wearable display device;

a motion-tracking device operable to track movement of the wearable display device;

a visual interface device resident upon within the computer system and executing upon the CPU, the visual interface device operable to a) map the visual feedback generated by the computer system into a virtual desktop suitable for display via the wearable display device, b) continuously display a certain portion of the virtual desktop within the wearable display device such that the low-vision user can view the certain portion of the virtual desktop corresponding to that portion of the virtual desktop which is located within the line of sight of the low-vision user, c) adjust a magnification of the virtual desktop in accordance with input from the low-vision user, d) update the wearable display device such that the certain portion displayed maintains the correspondence with that portion of the virtual desktop which is located within the line of sight of the low-vision user.

35. A computer system as recited in claim 34 further comprising:

a video camera coupled to the computer system;

a real-time imagery device implemented upon the computer system, the real-time imagery device operable to utilize the video camera in order to a) monitor a real scene in real space and time, b) capture images of the real scene, and c) provide the captured images of the real scene somewhere within the virtual desktop.