Intuitive control of portable data displays

• US RE42,336 E1
• Filed: 10/06/2009
• Issued: 05/10/2011
• Est. Priority Date: 11/28/1995
• Status: Expired due to Term

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Rembrandt Portable Display Technologies, LP (Rembrandt IP Management, LLC)
• Inventors: Zwern, Arthur L., Flack, James F., Fateh, Sina
• Primary Examiner(s): Liang; Regina
• Application Number:
  US12/574,607
• Time in Patent Office: 581 Days
• Field of Search: 345 7- 9, 345156-158, 715/863
• US Class Current: 345/8
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A virtual computer monitor is described which enables instantaneous and intuitive visual access to large amounts of visual data by providing the user with a large display projected virtually in front of the user. The user wears a head-mounted display or holds a portable display containing a head-tracker or other motion tracker, which together allow the user to position an instantaneous viewport provided by the display at any position within the large virtual display by turning to look in the desired direction. The instantaneous viewport further includes a mouse pointer, which may be positioned by turning the user'"'"'s head or moving the portable display, and which may be further positioned using a mouse or analogous control device. A particular advantage of the virtual computer monitor is intuitive access to enlarged computer output for visually-impaired individuals.

Claims

1. A computer implemented method for assisting a user in the control and operation of a computer system having a display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping the user with a portable display device coupled to the computer system;

mapping visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the portable display device;

continually displaying a certain portion of the virtual desktop within the portable display device such that the user can view the certain portion of the virtual desktop;

tracking motion of the portable display device including discrete motion gestures initiated by the user;

and

when the tracked motion corresponds to a request for a special discrete command, performing the special discrete command;

wherein the virtual desktop represents a text document; and

wherein the tracked motion is a shaking motion of the portable device and the special discrete command is an erase command.

2. A computer implemented method as recited in claim 1 wherein the detected motion is a forward motion and the special discrete command is an increase in magnification of the displayed certain portion of the virtual desktop.

3. A computer implemented method as recited in claim 2 wherein the degree of increase in magnification is a function of the acceleration of the detected motion.

4. A computer implemented method as recited in claim 1 wherein the detected motion is backward motion and the special discrete command is an decrease in magnification of the displayed certain portion of the virtual desktop.

5. A computer implemented method as recited in claim 4 wherein the degree of decrease in magnification is a function of the acceleration of the detected motion.

6. A computer implemented method as recited in claim 1 wherein the virtual display space represents a text document.

7. A computer implemented method as recited in claim 6 wherein the detected motion is an abrupt downward motion and the special discrete command is a scroll display down command.

8. A computer implemented method as recited in claim 6 wherein the detected motion is a rotating motion and the special discrete command is a page down function.

9. A computer implemented method as recited in claim 6 wherein the detected motion is a shaking motion of the portable device and the special discrete command is an erase command.

10. A computer implemented method as recited in claim 1 wherein the neutral display position is the image presented when the device is not in motion, and the special discrete command includes a step of setting the current field of view to the neutral position.

11. A computer implemented method for assisting a vision-impaired user in the control and operation of a computer system, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon a display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping the vision-impaired user with a portable display device coupled to the computer system;

mapping the visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the portable display device;

continually displaying a certain portion of the virtual desktop within the portable display device such that the vision-impaired user can view the certain portion of the virtual desktop, the certain portion of the virtual desktop corresponding to that portion of the virtual desktop which is located within the line of sight of the vision-impaired user;

adjusting a magnification of the virtual desktop in accordance with input from the visually-impaired user, whereby the certain portion is correspondingly adjusted such that the certain portion displayed within the portable device is a larger or a smaller percentage of the virtual desktop depending upon the vision-impaired user'"'"'s magnification adjustment, the adjust magnification input being a function of the portable display;

tracking changes in the vision-impaired user'"'"'s line of sight; and

updating the portable display device such that the certain portion displayed maintains the correspondence with that portion of the virtual desktop which is located within the line of sight of the vision-impaired user,

whereby the visually-impaired user can adjust the magnification of the visual feedback provided by the computer system to a magnification level improving comprehension of the visual feedback, and, even when the magnification level precludes simultaneous display of all the visual feedback within the portable display device, the visually-impaired user may still access all the visual feedback by adjusting his or her line of sight.

12. A computer implemented method as recited in claim 11 further comprising the step of displaying a pointer within the certain portion of the virtual desktop, the pointer having a fixed relation with the vision-impaired user'"'"'s line of sight.

13. A computer implemented method as recited in claim 12 further comprising the steps of:

receiving a request to adjust the fixed relation between the pointer and the vision-impaired user'"'"'s line of sight; and

adjusting the fixed relation in accordance with the received request.

14. A computer implemented method as recited in claim 12 further comprising the step of providing the vision-impaired user a pointer control input device for controlling operation of the pointer, the pointer control input device coupled to the computer system.

15. A computer implemented method as recited in claim 14 further comprising the steps of:

receiving a create selection box command via the pointer control input device;

anchoring a first comer corner of a selection box at a first point where the pointer is disposed within the certain portion displayed upon receipt of the create selection box command;

sizing the selection box in conjunction with movement of the vision-impaired user'"'"'s line of sight;

receiving a close selection box command via the pointer control input device;

defining the selection box by anchoring a second comer corner of the selection box, the second comer corner opposing the first comer corner, the second comer corner anchored at a second point where the pointer is disposed within the certain portion displayed upon receipt of the close selection box command; and

marking as selected that portion of the virtual desktop, which is bounded by the selection box.

16. A computer implemented method as recited in claim 15 wherein the create selection box command corresponds to the depression of a button located on the pointer control input device and the close selection box command corresponds to the release of the button located on the pointer control input device.

17. A computer implemented method as recited in claim 11 further comprising the step of redefining the orientation of the certain portion displayed within the virtual desktop such that, without adjusting the vision-impaired user'"'"'s line of sight, the certain portion displayed within the head-mounted display device changes.

18. A computer implemented method as recited in claim 17 wherein the orientation of the certain portion displayed is redefined in response to a request by the vision-impaired user.

19. A computer implemented method as recited in claim 11 wherein the step of mapping the visual feedback into the virtual desktop includes the substep of performing spatial field compression to reduce the total display orientation change required for redirecting the vision-impaired user'"'"'s line of sight from one location within the virtual desktop to another location within the virtual desktop.

20. A computer implemented method as recited in claim 11 wherein the visual feedback generated by the computer system and intended for display upon a display screen includes multiple application windows.

21. A computer implemented method as recited in claim 20 wherein a first window of the multiple application windows corresponds to a first application executing upon the computer system.

22. A computer implemented method as recited in claim 20 further comprising the steps of:

monitoring a real scene in real space and time;

capturing images of the real scene; and

displaying within a first window of the multiple application windows the captured images of the real scene.

23. A computer implemented method as recited in claim 22 wherein a second window of the multiple application windows corresponds to an application program executing upon the computer system.

24. A computer implemented method for assisting a user in the control and operation of a computer system having a display screen coupled with the computer system, the computer implemented method comprising the steps of:

equipping the user with a portable display device coupled to the computer system;

equipping the user with a wearable control device operable to monitor movement and gestures made by the user with the portable control device, the computer system storing a map corresponding certain gestures and movements to certain computer operating instructions;

mapping visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the portable display device;

continually displaying a certain portion of the virtual desktop within the portable display device such that the user can view the certain portion of the virtual desktop;

tracking motion of the portable control device;

determining that a user implemented gesture with the portable control device corresponds to a specific computer operating instruction; and

executing the specific computer operating instruction.

25. A computer implemented method as recited in claim 24 wherein the portable display device and the portable control device are formed as a single unit.

26. A computer implemented method as recited in claim 24 wherein the portable display device and the portable control device are separate devices.

27. A computer implemented method as recited in claim 24 wherein the computer system is housed within the portable display device.

28. A computer implemented method as recited in claim 24 wherein the computer system is housed within the portable control device.

29. A computer implemented method as recited in claim 24 wherein the computer system is remote from the portable display device.