Computer-aided system for 360° heads up display of safety/mission critical data

• US 9,728,006 B2
• Filed: 11/12/2012
• Issued: 08/08/2017
• Est. Priority Date: 07/20/2009
• Status: Expired due to Fees

• Current Assignee: Real Time Companies, Llc
• Original Assignee: Real Time Companies, Llc
• Inventors: Varga, Kenneth A.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Wu, Chong
• Application Number:
  US13/674,671
• Publication Number:
  US 20130162632A1
• Time in Patent Office: 1,730 Days
• Field of Search: 345419, 345 8, 345 32
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G09B 9/302 the image being transformed by computer processing, e.g. updating the image to correspond to the changing point of view
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A safety critical, time sensitive data system for projecting safety/mission critical data onto a display pair of Commercial Off The Shelf (COTS) light weight projection glasses or monocular creating an immersive omnidirectional HUD (Heads Up Display) system with 6 degrees of freedom movement with user immersion and user inputs & outputs. The system includes the display, haptic glove, haptic suit, vestibular interface, temperature emulation, smell emulation, and omnidirectional sound, the workstation, the application software, and inputs containing the safety/mission critical information (Current User Position, Total Collision Avoidance System—TCAS, Global Positioning System—GPS, Magnetic Resonance Imaging—MRI Images, CAT scan images. Weather data. Military troop data, real-time space type markings etc.). The workstation software processes the incoming safety/mission critical data and converts it into a three dimensional space for the user to immerse into the environment. Selecting any of the images may display available information about the selected item or may enhance the image, or moving hand or body will generate enhanced environmental perception & awareness, as well as inputs and outputs. Predicted position vectors may be displayed as well as 3D terrain.

Claims

1. A system for displaying and updating data in three dimensions, comprising:

a computer system connected to a communications network and configured to process data regarding multiple locations and create a virtual three dimensional orthogonal space wherein the data is associated with multiple locations in the virtual three dimensional orthogonal space;

a wearable display device connected to the computer system;

a head-tracking sensing sensor system connected to the wearable display device and to the computer system, configured to sense the movement and orientation of the wearable display device in real-time;

an eye-tracking sensor system connected to the wearable display device and to the computer system, configured to sense the eye movement of a user wearing the wearable display device in real-time;

wherein the computer system is configured to receive real-time data from the head-tracking sensing system and the eye-tracking sensor system and to process said real-time data to generate an image of the virtual three dimensional orthogonal space for display on the wearable display device; and

a handheld control device that is configured to send a signal to the computer system, wherein the computer system is capable of displaying data related to any of the multiple locations;

wherein the system is configured to track a first object'"'"'s predicted position vector, display a possibly collision point between the first object and the user based upon the first object'"'"'s predicted position vector, and predict the position the user must respond to in order to avoid collision with the object;

wherein the system is configured to highlight a surface terrain visible through the wearable display device; and

wherein the system displays out-of-sight indicators to alert the user regarding additional objects outside a view of the wearable display device.

2. The system of claim 1, further comprising a gimbaled zoom camera attached to the wearable display device.

3. The system of claim 1, further comprising a microphone, wherein the computer system is configured to recognize voice commands.

4. The system of claim 1, further comprising:

wherein the computer system is configured to allow a user to select an object on the display and deliver known information about the object from an accompanying database;

wherein the computer system is configured to display the attributes of an object in the virtual three dimensional orthogonal space;

wherein the system is configured to display a plurality of moving aircraft flight plans; and

wherein the system is configured to track each objects predicted position vector and flight plan, if available, to determine if a collision is anticipated, either in the air or on a ground taxiway, and provide a warning when an incursion is projected.

5. The system of claim 1, wherein the first object is an incoming aircraft.

6. The system of claim 1, wherein the system is further configured to track a predicted position vector for a second object.

7. The system of claim 1, wherein the computer system is configured to allow a user to select the first object on the display and deliver known information about the first object from an accompanying database.

8. The system of claim 1, wherein a known flight path of a first incoming aircraft and a predicted position vector of a second incoming aircraft are displayed to the user.

9. The system of claim 1, wherein the user can be displayed probable battle spaces to signify danger and probable safe spaces outside of the danger.

10. The system of claim 1, further comprising a military battlefield display system, wherein the military battlefield system is a first one of a plurality of military battlefield display systems which communicate and share information about a plurality of objects.

11. The system of claim 1, wherein the user can be displayed probable safe weather spaces.

12. The system of claim 1, wherein:

the system is configured to display the virtual three dimensional orthogonal space from the point of view of a selected object;

the system is configured to update the image of the virtual three dimensional orthogonal space based on movement of the user'"'"'s head;

the system is configured to allow selection and manipulation of the image data by combining eye tracking and head tracking;

the system is configured to allow a vision impaired user to see; and

the computer system is configured to record the data of the virtual three dimensional orthogonal space over a specified time period.