Virtual computer monitor

• US 6,084,556 A
• Filed: 03/09/1999
• Issued: 07/04/2000
• Est. Priority Date: 11/28/1995
• Status: Expired due to Term

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Vega Vista, Inc.
• Inventors: Zwern, Arthur L.
• Primary Examiner(s): Shalwala, Bipin H.
• Assistant Examiner(s): LEWIS, DAVID LEE
• Application Number:
  US09/264,799
• Time in Patent Office: 483 Days
• Field of Search: 345/7-9, 345/156-159, 345/127, 345/130, 345/339, 345/972
• US Class Current: 345/8
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

A virtual computer monitor is described which enables instantaneous and intuitive visual access to large amounts of visual data by providing the user with a large display projected virtually in front of the user. The user wears a head-mounted display containing a head-tracker, which together allow the user to position an instantaneous viewport provided by the head-mounted display at any position within the large virtual display by turning to look in the desired direction. The instantaneous viewport further includes a mouse pointer, which may be positioned by turning the user'"'"'s head, and which may be further positioned using a mouse or analogous control device. A particular advantage of the virtual computer monitor is intuitive access to enlarged computer output for visually-impaired individuals.

Claims

1. A device for assisting a computer user in interacting with a computer or like data processing device, comprising:

a head-mounted display device coupled to said computer'"'"'s display output;

a visual interface device providing a virtual desktop whose size is larger than the instantaneous viewing window provided by said head-mounted display, said virtual desktop containing enlarged data which occupies a larger angular extent of the user'"'"'s visual field than is provided by viewing the same data without enlargement;

a magnification selection device operable by the computer user for selection of an enlargement factor applied to the enlarged data;

a head-tracking device for sensing changes in the user'"'"'s line-of sight, converting said changes into tracking data, and inputting said tracking data into said computer; and

,

a signal processing device for interpreting said tracking data for purposes of determining which portion of the virtual desktop will be displayed within said head-mounted display at a particular point in time,

whereby the computer user can view any portion of the virtual desktop by turning to look in that portion'"'"'s direction.

2. A device as described in claim 1, wherein a mouse pointer is made visible within said head-mounted display, and remains linked to the user'"'"'s line of sight as the user'"'"'s line of sight changes.

3. A device as in claim 2, further providing a secondary control device for positioning the mouse pointer, by which the user can perform fine adjustment of the mouse pointer location within the virtual desktop after moving to an approximate location using said head-tracking device.

4. A device as in claim 3, where said secondary mouse pointer positioning device is hand-operated.

5. A device as in claim 3, where said secondary mouse pointer positioning device is finger operated.

6. A device as described in claim 2, wherein said signal processing device further performs spatial field compression to reduce the total head orientation change required for redirecting the user'"'"'s line of sight from one location within the virtual desktop to another location.

7. A device as described in claim 3, wherein said signal processing device further performs line-of-sight snap-back processing to return the user'"'"'s instantaneous viewing window to a location within the virtual desktop determined by the user'"'"'s head orientation following temporary modification of said viewing window location by said secondary mouse pointer control function.

8. A device as described in claim 2, wherein said mouse pointer may be positioned at any user-selectable position within said head-mounted display'"'"'s field of view, and maintained at that position regardless of changes in the user'"'"'s head orientation.

9. A device as in claim 2, further including a spatialized audio cue generation device to provide audio signals to the user which appear to originate from portions of said virtual desktop which are not within the user'"'"'s instantaneous field of view, thereby alerting the user to turn and look in the direction of said spatialized audio cues.

10. A device as in claim 2, further including a device for accepting input of voice commands spoken by the user, and a device for converting said commands into internal computer formats through which individual computer functions may be controlled.

11. A device as in claim 10 wherein said computer functions include selection of a location or object within said virtual desktop.

12. A device as described in claim 10 wherein said computer functions include selection of specific locations for repositioning said instantaneous viewing window within said virtual desktop.

13. A device as described in claim 2 wherein said signal processing device further includes predictive algorithms to reduce apparent latency in repositioning of said instantaneous viewing window.

14. A device as described in claim 2 which further includes a device for generating multiple application windows within said virtual desktop, wherein the user may select an individual application window by head reorientation.

15. A device as described in claim 1, further including a real-time image sensor device, a real-time image digitization device, and a processing device for displaying the captured imagery within said virtual desktop.

16. A device as described in claim 15, where said captured imagery is displayed within one application window on said virtual desktop, and at least one computer application program is displayed within at least one additional application window on said virtual desktop.

17. A computer implemented method for assisting a vision-impaired user in the control and operation of a computer system, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon a display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping the vision-impaired user with a head-mounted display device coupled to the computer system;

mapping the visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the head-mounted display device;

continually displaying a certain portion of the virtual desktop within the head-mounted display device such that the vision-impaired user can view the certain portion of the virtual desktop, the certain portion of the virtual desktop corresponding to that portion of the virtual desktop which is located within the line of sight of the vision-impaired user;

adjusting a magnification of the virtual desktop in accordance with input from the visually-impaired user, whereby the certain portion is correspondingly adjusted such that the certain portion displayed within the head-mounted device is a larger or a smaller percentage of the virtual desktop depending upon the vision-impaired user'"'"'s magnification adjustment;

tracking changes in the vision-impaired user'"'"'s line of sight; and

updating the head-mounted display device such that the certain portion displayed maintains the correspondence with that portion of the virtual desktop which is located within the line of sight of the vision-impaired user,

whereby the visually-impaired user can adjust the magnification of the visual feedback provided by the computer system to a magnification level improving comprehension of the visual feedback, and, even when the magnification level precludes simultaneous display of all the visual feedback within the head-mounted display device, the visually-impaired user may still access all the visual feedback by adjusting his or her line of sight.

18. A computer implemented method as recited in claim 17 further comprising the step of displaying a pointer within the certain portion of the virtual desktop, the pointer having a fixed relation with the vision-impaired user'"'"'s line of sight.

19. A computer implemented method as recited in claim 18 further comprising the steps of:

receiving a request to adjust the fixed relation between the pointer and the vision-impaired user'"'"'s line of sight; and

adjusting the fixed relation in accordance with the received request.

20. A computer implemented method as recited in claim 18 further comprising the step of providing the vision-impaired user a pointer control input device for controlling operation of the pointer, the pointer control input device coupled to the computer system.

21. A computer implemented method as recited in claim 20 further comprising the steps of:

receiving a create selection box command via the pointer control input device;

anchoring a first corner of a selection box at a first point where the pointer is disposed within the certain portion displayed upon receipt of the create selection box command;

sizing the selection box in conjunction with movement of the vision-impaired user'"'"'s line of sight;

receiving a close selection box command via the pointer control input device;

defining the selection box by anchoring a second corner of the selection box, the second corner opposing the first corner, the second corner anchored at a second point where the pointer is disposed within the certain portion displayed upon receipt of the close selection box command; and

marking as selected that portion of the virtual desktop which is bounded by the selection box.

22. A computer implemented method as recited in claim 21 wherein the create selection box command corresponds to the depression of a button located on the pointer control input device and the close selection box command corresponds to the release of the button located on the pointer control input device.

23. A computer implemented method as recited in claim 17 further comprising the step of redefining the orientation of the certain portion displayed within the virtual desktop such that, without adjusting the vision-impaired user'"'"'s line of sight, the certain portion displayed within the head-mounted display device changes.

24. A computer implemented method as recited in claim 23 wherein the orientation of the certain portion displayed is redefined in response to a request by the vision-impaired user.

25. A computer implemented method as recited in claim 17 wherein the step of mapping the visual feedback into the virtual desktop includes the substep of performing spatial field compression to reduce the total head orientation change required for redirecting the vision-impaired user'"'"'s line of sight from one location within the virtual desktop to another location within the virtual desktop.

26. A computer implemented method as recited in claim 17 wherein the visual feedback generated by the computer system and intended for display upon a display screen includes multiple application windows.

27. A computer implemented method as recited in claim 26 wherein a first window of the multiple application windows corresponds to a first application executing upon the computer system.

28. A computer implemented method as recited in claim 26 further comprising the steps of:

monitoring a real scene in real space and time;

capturing images of the real scene; and

displaying within a first window of the multiple application windows the captured images of the real scene.

29. A computer implemented method as recited in claim 28 wherein a second window of the multiple application windows corresponds to an application program executing upon the computer system.

30. A computer implemented method for assisting a vision-impaired user in the control and operation of a computer system, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon a display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping the vision-impaired user with a head-mounted display device coupled to the computer system;

mapping the visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the head-mounted display device;

continually displaying a certain portion of the virtual desktop within the head-mounted display device such that the vision-impaired user can view the certain portion of the virtual desktop, the certain portion of the virtual desktop corresponding to that portion of the virtual desktop which is located within the line of sight of the vision-impaired user;

adjusting a magnification of the virtual desktop in accordance with input from the visually-impaired user, whereby the certain portion is correspondingly adjusted such that the certain portion displayed within the head-mounted device is a larger or a smaller percentage of the virtual desktop depending upon the vision-impaired user'"'"'s magnification adjustment;

tracking changes in the vision-impaired user'"'"'s line of sight;

updating the head-mounted display device such that the certain portion displayed maintains the correspondence with that portion of the virtual desktop which is located within the line of sight of the vision-impaired user;

monitoring a real scene in real space and time;

capturing images of the real scene; and

providing the captured images of the real scene somewhere within the virtual desktop.

31. A computer implemented method as recited in claim 30 further comprising the step of displaying a pointer within the certain portion of the virtual desktop, the pointer having a fixed relation with the vision-impaired user'"'"'s line of sight.

32. A computer implemented method as recited in claim 31 further comprising the steps of:

receiving a request to adjust the fixed relation between the pointer and the vision-impaired user'"'"'s line of sight; and

adjusting the fixed relation in accordance with the received request.

33. A computer implemented method as recited in claim 31 further comprising the step of providing the vision-impaired user a pointer control input device for controlling operation of the pointer, the pointer control input device coupled to the computer system.

34. A computer implemented method as recited in claim 33 further comprising the steps of:

receiving a create selection box command via the pointer control input device;

anchoring a first corner of a selection box at a first point where the pointer is disposed within the certain portion displayed upon receipt of the create selection box command;

sizing the selection box in conjunction with movement of the vision-impaired user'"'"'s line of sight;

receiving a close selection box command via the pointer control input device;

defining the selection box by anchoring a second corner of the selection box, the second corner opposing the first corner, the second corner anchored at a second point where the pointer is disposed within the certain portion displayed upon receipt of the close selection box command; and

marking as selected that portion of the virtual desktop which is bounded by the selection box.

35. A computer implemented method as recited in claim 34 wherein the create selection box command corresponds to the depression of a button located on the pointer control input device and the close selection box command corresponds to the release of the button located on the pointer control input device.

36. A computer implemented method as recited in claim 30 further comprising the step of redefining the orientation of the certain portion displayed within the virtual desktop such that, without adjusting the vision-impaired user'"'"'s line of sight, the certain portion displayed within the head-mounted display device changes.

37. A computer system particularly well-suited for use by a low-vision user, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon a display screen coupled with the computer system, the computer system comprising:

a central processing unit (CPU);

random access memory (RAM);

a head-mounted display device;

a head-tracking device operable to sense changes in the low-vision user'"'"'s line of sight;

a visual interface device implemented upon the computer system, the visual interface device operable to a) map the visual feedback generated by the computer system into a virtual desktop suitable for display by the head-mounted display device, b) continuously display a certain portion of the virtual desktop within the head-mounted display device such that the low-vision user can view the certain portion of the virtual desktop corresponding to that portion of the virtual desktop which is located within the line of sight of the low-vision user, c) adjust a magnification of the virtual desktop in accordance with input from the low-vision user, d) update the head-mounted display device such that the certain portion displayed maintains the correspondence with that portion of the virtual desktop which is located within the line of sight of the low-vision user.

38. A computer system as recited in claim 37 further comprising:

a video camera coupled to the computer system;

a real-time imagery device implemented upon the computer system, the real-time imagery device operable to utilize the video camera in order to a) monitor a real scene in real space and time, b) capture images of the real scene, and c) provide the captured images of the real scene somewhere within the virtual desktop.