Computer-Aided System for 360º Heads Up Display of Safety/Mission Critical Data

• US 20130162632A1
• Filed: 11/12/2012
• Published: 06/27/2013
• Est. Priority Date: 07/20/2009
• Status: Active Grant

• Current Assignee: Real Time Companies, Llc
• Original Assignee: Real Time Companies, Llc
• Inventors: Varga, Kenneth A., Young, Joel, Hiett, John, Cove, Patty
• Granted Patent: US 9,728,006 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G09B 9/302 the image being transformed by computer processing, e.g. updating the image to correspond to the changing point of view
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A safety critical, time sensitive data system for projecting safety/mission critical data onto a display pair of Commercial Off The Shelf (COTS) light weight projection glasses or monocular creating an immersive omnidirectional HUD (Heads Up Display) system with 6 degrees of freedom movement with user immersion and user inputs & outputs. The system includes the display, haptic glove, haptic suit, vestibular interface, temperature emulation, smell emulation, and omnidirectional sound, the workstation, the application software, and inputs containing the safety/mission critical information (Current User Position, Total Collision Avoidance System—TCAS, Global Positioning System—GPS, Magnetic Resonance Imaging—MRI Images, CAT scan images. Weather data. Military troop data, real-time space type markings etc.). The workstation software processes the incoming safety/mission critical data and converts it into a three dimensional space for the user to immerse into the environment. Selecting any of the images may display available information about the selected item or may enhance the image, or moving hand or body will generate enhanced environmental perception & awareness, as well as inputs and outputs. Predicted position vectors may be displayed as well as 3D terrain.

Claims

1. A system for displaying and updating data in three dimensions, comprising:

a computer system connected to a communications network and configured to process data regarding multiple locations and create a virtual three dimensional orthogonal space wherein the data is associated with a multiple locations in the virtual three dimensional orthogonal space;

a wearable display device connected to the computer system;

a head-tracking sensing sensor system connected to the wearable display device and to the computer system, configured to sense the movement and orientation of the wearable display device in real-time;

an eye-tracking sensor system connected to the wearable display device and to the computer system, configured to sense the eye movement of a user wearing the wearable display device in real-time;

wherein the computer system is configured to receive real-time data from the head-tracking sensing system and the eye-tracking sensor system and to process said real-time data to generate an image of the virtual three dimensional orthogonal space for display on the wearable display device; and

a handheld control device that is configured to send a signal to the computer system, wherein the computer system is capable of displaying data related to any of the multiple locations.

2. The system of claim 1, further comprising a gimbaled zoom camera attached to the wearable display device.

3. The system of claim 1, further comprising a microphone, wherein the computer system is configured to recognize voice commands.

4. The system of claim 1, characterized by one or more of the following:

a. wherein the computer system is configured to allow a user to select an object on the display and deliver known information about the object from an accompanying database;

b. wherein the computer system is configured to display the attributes of an object in the virtual three dimensional orthogonal space;

c. wherein the system is an air traffic control system configured to display a plurality of moving aircraft flight plans;

d. wherein the system is an air traffic control system configured to track each objects predicted position vector and flight plan, if available, to determine if a collision is anticipated, either in the air or on the ground taxiway, and provide a warning when an incursion is projected;

e. wherein the system is configured to display the virtual three dimensional orthogonal space from the point of view of a selected object;

f. wherein the system is configured to update the image of the virtual three dimensional orthogonal space based on movement of the user'"'"'s head;

g. wherein the system is configured to allow selection and manipulation of the image data by combining eye tracking and head tracking;

h. wherein the system is a military battlefield display system and is configured to track an object'"'"'s predicted position vector and identify battlefield conflicts and spaces;

i. wherein the system is a military battlefield display system and is configured to identify friendly troops;

j. wherein the system is a medical display system configured to generate an image of a patient from the inside;

k. wherein the system is a medical display system configured to allow a doctor to practice a procedure before operating on a patient;

l. wherein the system is a medical display system configured to allow doctors to view the inside of a patient using arthroscopic camera technology;

m. wherein the system is configured to allow a vision impaired user to see;

n. wherein the computer system is configured to record the data of the virtual three dimensional orthogonal space over a specified time period.