Selectively controllable heads-up display system

• US 6,847,336 B1
• Filed: 10/02/1996
• Issued: 01/25/2005
• Est. Priority Date: 10/02/1996
• Status: Expired due to Fees

• Current Assignee: Jerome H. Lemelson, John H. Hiett
• Original Assignee: Jerome H. Lemelson, John H. Hiett
• Inventors: Lemelson, Jerome H., Hiett, John H.
• Primary Examiner(s): NGUYEN, CHANH DUY
• Application Number:
  US08/720,662
• Time in Patent Office: 3,037 Days
• Field of Search: 345/7, 345/8, 345/9, 345/156, 345/157, 345/145, 345/146, 345/340, 345/348, 345/32, 351/210, 378/41, 378/42, 378/99
• US Class Current: 345/8
• CPC Class Codes: A61B 1/00048 Constructional features of the display
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/7445 Display arrangements, e.g. multiple display units
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Systems and methods are disclosed for displaying data on a head'"'"'s-up display screen. Multiple forms of data can be selectively displayed on a semi-transparent screen mounted in the user'"'"'s normal field of view. The screen can either be mounted on the user'"'"'s head, or mounted on a moveable implement and positioned in front of the user. A user interface is displayed on the screen including a moveable cursor and a menu of computer control icons. An eye-tracking system is mounted proximate the user and is employed to control movement of the cursor. By moving and focusing his or her eyes on a specific icon, the user controls the cursor to move to select the icon. When an icon is selected, a command computer is controlled to acquire and display data on the screen. The data is typically superimposed over the user'"'"'s normal field of view.

Claims

1. A heads-up display system for use by a medical technician, the heads-up display system selectively controllable by the medical technician comprising:

a) a transparent heads-up display screen including a plurality of icons including a vital signs icon, the transparent heads-up display screen worn on the head of the medical technician, the transparent heads-up display screen positioned in the normal field of view of the medical technician;

b) a command control computer coupled to the transparent heads-up display screen and including an electronic storage device;

c) a communication network coupled to the command control computer and further comprising;

i. a wide-area network for accessing medical data bases to be accessed by the medical technician and for communication and consulting with other medical technicians; and

,

ii. signal transmitters and receivers for communication of information and images derived from the procedure for remote viewing, thus establishing a remote telepresence for consultation, education or other medical purposes;

d) an eye-tracking system integrated with the transparent heads-up display system, the eye-tracking system supported on the head of the medical technician and coupled to the command control computer and configured,

(i) to detect and track movement of one of the medical technician'"'"'s eyes as the medical technician views the icons on the transparent heads-up display screen, and

(ii) to communicate to the command control computer eye-tracking data corresponding to the movement of the medical technician'"'"'s eye to the icons on the transparent heads-up display; and

e) a user interface including;

(i) an icon control program stored in the electronic storage device and configured to cause the command control computer to generate and maintain on the display screen a plurality of selectable control icons, including a vital sign icon in a manner that is superimposed over the normal field of view;

(ii) a cursor control program stored in the electronic storage device and configured to cause the command control computer to;

(1) generate and maintain on the display screen a moveable cursor in a manner that is superimposed over the normal field of view;

(2) translate the eye-tracking data communicated from the eye-tracking system into cursor control data;

(3) apply the cursor control data to cause the cursor displayed on the display screen to move in a manner that tracks movement of the medical technician'"'"'s eye; and

(4) generate process control data when the cursor is tracked over and selects the vital sign icon; and

(iii) a process control program stored in the electronic storage device and configured to cause the command computer to initiate specific control actions corresponding to the process control data generated when the eye-tracking cursor is moved to the vital sign icon.

2. The system of claim 1 wherein the heads-up display includes a display driver coupled to the command computer and configured so as to project images onto the display screen.

3. The system of claim 2 wherein the command computer is coupled to the heads-up display by a radio communication link.

4. The system of claim 3 wherein the command computer is coupled to the heads-up display by a wire communication link.

5. The system of claim 2 comprising a voice recognition system including:

a) a microphone placed proximate the user'"'"'s mouth;

b) a speaker placed proximate at least one ear of the user;

c) a voice recognition circuit configured to recognize words spoken by the user and generate speech command signals corresponding thereto; and

d) wherein the process control program is configured to cause the command computer to initiate specific control actions corresponding to the speech command signals generated by the voice recognition circuit.

6. The system of claim 1 wherein the icon control program generates and maintains on the display screen the vital sign icon relating to data acquisition procedures.

7. A hands-free system for assisting medical practitioners in the performance of medical procedures on patients comprising:

a. a computer subsystem coupled to a patient medical data base subsystem, a communication network subsystem, a patient monitoring sensor subsystem, and a heads-up display (HUD) subsystem worn on the head of the medical practitioner;

b. the computer subsystem further comprising;

i. one or more computers programmed to respond to patient sensor inputs and to external HUD commands from medical practitioners;

to retrieve and record medical data base information;

to manage information flow to and from the external communication network;

to transmit medical images and information to the HUD; and

, to perform analysis of information and present results of the analysis;

ii. signal processing circuitry coupled to the computer subsystem for processing command signals and sensor information signals for input to the computer, and to process signals from the computer for audio, visual or other presentation to the medical practitioners;

iii. a first communication link for communicating with the heads-up display subsystem;

c. the heads-up display (HUD) subsystem further comprising;

i. a second communication link for communicating with the computer subsystem first communication link and with;

(1) a HUD camera and light subsystem, (2) a HUD eye-tracking subsystem, (3) a HUD display subsystem, (4) a HUD audio input subsystem; and

, (5) a HUD speaker subsystem;

ii. the HUD camera and light subsystem coupled to the second communication link;

iii. the HUD eye-tracking subsystem coupled to the second communication link;

iv. the HUD display subsystem coupled to the second communication link;

v. the HUD audio input subsystem coupled to the second communication link;

vi. the HUD speaker subsystem coupled to the second communication link;

d. the medical data base subsystem coupled to the computer subsystem and further comprising;

i. patient data base information including medical history, medical images, allergic reactions, family history and related data;

ii. treatment information for various maladies and recommended procedures;

iii. medication alternatives; and

,

iv. surgical recommendations for different patient conditions;

e. the communication network subsystem coupled to the computer subsystem and further comprising;

i. a wide-area network for accessing medical data bases to be queried by the medical practitioner and for communication and consulting with other medical practitioners; and

,

ii. signal transmitters and receivers for communication of information and images derived from the procedure for remote viewing, thus establishing a remote telepresence for consultation, education or other medical purposes;

f. the patient monitoring sensor subsystem coupled to the computer subsystem and further comprising;

i. vital sign sensors comprising blood pressure, heart rate, respiration rate, body temperature, or other patient status indicators;

ii. signal generators coupled to the vital sign sensors for generating coded signals representative of the measured vital signs; and

iii. communication circuitry coupled to the signal generators and the computer subsystem for formatting and transmitting sensor information to the computer sub-system;

g. the computer subsystem being further programmed to operate the HUD to provide a user selectable display options comprising;

i. patient images derived from the camera and lighting subsystem or from the computer data base information, including composite and overlay images,

ii. sensor icons displaying sensor measurements from the patient monitoring sensor subsystems;

iii. control icons or menus to control various subsystems including the camera and lighting subsystem panning, tilting, magnification or zooming and lighting;

the display subsystem; and

one or more computer subsystems;

iv. hierarchical icon or menu operation with layers of menus;

v. multiple simultaneously viewable display windows arranged in tiled format with selected images, patient information, text information, or other related information, the display windows being semitransparent or opaque under user or under program control, individual windows being further selectable for expanded viewing, resizing, hiding, or rearrangement,

vi. selectable, programmable warning indicators to visually indicate warnings at desired levels,

vii. the display options being user selectable and controllable using the HUD eye-tracking subsystem by tracking the medical practitioner'"'"'s eyes causing an eye tracking cursor to move over the display with user selection of the desired icon or menu by specific eye movements, a spoken command, or foot actuation of a system control to direct desired operations and displays.

8. The invention in accordance with claim 7 wherein the first communication link of the computer subsystem further comprises:

a wireless radio for transmission and reception of signals;

an antenna coupled to the wireless radio; and

signal routing and control circuitry for routing various signals to and from the computer subsystem.

9. The invention in accordance with claim 8 wherein the second communication link of the HUD subsystem comprises:

wireless radio for transmission and reception of signals;

an antenna coupled to the wireless radio;

couplings to the HUD eye-tracking subsystem, display subsystem, audio input subsystem, camera and light subsystem and speaker subsystem for receipt and delivery of messages and signals to, from and between these HUD subsystems; and

,

signal routing and control circuitry for routing various signals to and from respective subsystems.

10. The invention in accordance with claim 9 wherein the HUD camera and light subsystem coupled to the second communication link comprises:

at least one imaging device, such as a CCD camera, an x-ray scanner, a NMR scanner, a CAT scanner, a sonic scanner, or other medical imaging scanner, coupled to a pan driver, a tilt driver, a magnification or zoom driver and an illumination light for capturing images of selected areas of a subject patient for display, analysis or recording, the imaging device being further coupled to the second communication link for delivery of images of selected areas of the patient to the computer subsystem and to the HUD display subsystems;

the pan driver coupled to the second communication link to receive control signals and further coupled to the imaging device to enable controllable panning of the subject patient with the imaging device to view different areas of the patient;

the controllable tilt driver coupled to the second communication link to receive control signals and further coupled to the imaging device to enable varying the viewing angle of the selected patient area with the scanning device

the or zoom or magnification driver coupled to the second communication link to receive control signals and further coupled to the imaging device to enable controllable magnification of selected areas of a subject patient; and

,

the light receiving control signals to provide controllable illumination to selected areas of a subject patient to enhance user or imaging device visibility.

11. The invention in accordance with claim 10 wherein the HD eye-tracking subsystem coupled to the second communication link comprises:

a low powered laser coupled to the eye-tracking electronics for generating an infrared eye-tracking laser beam;

eye-tracking optics coupled to the eye-tracking electronics including a lens for focusing the laser beam onto a mirror positioned to reflect the beam onto the user'"'"'s eye(s), and a mirror to positioned to capture reflected energy from the user'"'"'s eye(s) and direct the reflected energy through a lens to an infra photo-detector camera;

eye-tracking electronics coupled to the infrared photo-detector camera for detecting distortion of the reflected laser beam energy corresponding to the eye direction vector, and generating data defining the eye direction vector for transmission through communication links to the computer subsystem; and

,

a calibration subsystem coupled to the eye-tracking subsystem elements for adjusting the elements for optimal performance through the use of computer controlled actuators.

12. The invention in accordance with claim 11 wherein the HUD display subsystem coupled to the second communication link comprises:

a display screen or transparent or semi-transparent surface mounted in the field of view of the medical practitioner using the HUD for the display of images;

display optics comprising mirrors or lenses for focusing of display images on the screen or surface for viewing by the medical practitioner, including capability for superimposing the display images over the real scene being viewed through the transparent or semitransparent HUD viewing surface; and

,

a display driver coupled to the second communication link for receiving display signal and generating the actual display image to be viewed by the medical practitioner.

13. The invention in accordance with claim 12 wherein the HUD audio input subsystem coupled to the second communication link comprises:

an audio input microphone mounted on the HUD to enable the user to use spoken commands to direct overall system operation or to record comments;

a speech recognition subsystem coupled to the audio microphone to recognize spoken commands and generate control signals; and

,

an input command interpreter subsystem coupled to the speech recognition subsystem and the second communication link to interpret commands from the speech recognition subsystem and generate computer commands for transmission to other system elements, including at least the computer subsystem.

14. The invention in accordance with claim 13 wherein the HUD speaker subsystem coupled to the second communication link comprises:

a speaker mounted on the HUD to enable the user to hear spoken or otherwise audible commands generated by the computer subsystem or relayed by that subsystem from the external communication subsystem or information database subsystem or from the patient; and

,

speaker control circuitry to adjust the speaker volume or other audio parameters for optimum performance in the working environment.