Virtual/live hybrid behavior to mitigate range and behavior constraints

• US 9,754,507 B1
• Filed: 07/02/2013
• Issued: 09/05/2017
• Est. Priority Date: 07/02/2013
• Status: Active Grant

• Current Assignee: Rockwell Collins, Inc. (Rtx Corporation)
• Original Assignee: Rockwell Collins, Inc. (Rtx Corporation)
• Inventors: Wenger, Jason C., Wolford, Brian R., Hoke, Jaclyn A.
• Primary Examiner(s): Yip, Jack
• Application Number:
  US13/933,870
• Time in Patent Office: 1,526 Days
• Field of Search: 345633, 345419, 345156, 382104, 359630, 434 38, 434 30, 715757
• CPC Class Codes: A63F 13/5378 for displaying an additional top view, e.g. radar screens or maps using two or more virtual cameras concurrently A63F13/5252
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 9/003 for military purposes and tactics
• CPC Class Codes: G09B 9/006 for locating or ranging of objects
• CPC Class Codes: G09B 9/54 Simulation of radar G09B9/40 takes precedence

Abstract

A system and method are disclosed for offering a trainee presentation blending live and virtual entities to create a training scenario unconstrained by live entity operational performance and geographical limitations. The system blends an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation. Outside of trainee local sensor range, the system presents a virtual entity to the trainee while occluding local sensor presentation of the live entity. As the scenario progresses to a lessor range or higher criticality, the system offers the live entity emulation information concerning characteristics of the virtual entity so the live entity may anticipate and begin to emulate the virtual. At a crossover point, the system determines if the live entity has successfully emulated the virtual and if so, discontinues presentation of the virtual while removing the occlusion allowing presentation of the live entity.

Claims

1. A method for blending an instance of a virtual entity with an actual presentation of a live entity within an aircraft flight trainee presentation, comprising:

receiving a training scenario having a defined fixed range that limits an ability of a trainee aircraft or its instrumentation;

generating the instance of the virtual entity in compliance with the training scenario;

generating the trainee presentation, the trainee presentation including the instance of the virtual entity, the trainee presentation further including an occlusion of the actual presentation of the live entity, wherein including the instance of the virtual entity and including the occlusion of the actual presentation of the live entity are independent of the defined range;

communicating the trainee presentation to a trainee via at least one of;

a graphic display and a pictorial display;

presenting emulation information to an operator of the live entity in compliance with the training scenario to mimic a behavior of the virtual entity;

correlating a characteristic of the live entity with a characteristic of the virtual entity;

blending, within the trainee presentation, the instance of the virtual entity with the actual presentation of the live entity based on the correlating;

removing the virtual entity from the trainee presentation; and

removing the occlusion of the actual presentation of the live entity from the trainee presentation, wherein the removing the virtual entity and the removing the occlusion occur as a result of the blending, the virtual entity being within the defined range, and as a result of the live entity being within a correlating tolerance.

2. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein the defined range is a first defined range, wherein receiving a training scenario having the defined range further includes receiving data associated with presentation of the virtual entity on a display that presents training data according to a second defined range, the second defined range being greater than the first defined range, and wherein the first defined range is a charted geographical section of airspace and the second defined range is at least one of:

outside the charted geographical section of airspace, beyond visual range (BVR) of the trainee aircraft, and beyond local sensor range of the trainee aircraft.

3. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein generating the instance of the virtual entity further comprises generating a plurality of virtual entities in compliance with the training scenario.

4. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein the trainee presentation is configured for presentation on an avionics display onboard the trainee aircraft.

5. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, further comprising:

returning the virtual entity and the occlusion to the trainee presentation when at least one of;

the virtual entity is outside the defined range and the live entity being is not within the correlating tolerance.

6. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein correlating a characteristic of the live entity with a characteristic of the virtual entity further comprises correlating at least one of:

a position, an altitude, an airspeed, and a heading of the virtual entity with the live entity.

7. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein correlating a characteristic of the live entity with a characteristic of the virtual entity further comprises receiving at least one characteristic associated with the live entity, comparing the at least one characteristic associated with the live entity to a similar characteristic of the virtual entity, and determining if the live entity is within the correlating tolerance.

8. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein blending, within the trainee presentation, the instance of the virtual entity with the actual presentation of the live entity further comprises removing the occlusion of the actual presentation of the live entity and discontinuing a presentation of the instance of the virtual entity, thereby replacing the discontinued presentation of the instance of the virtual entity with the actual presentation of the live entity.

9. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein removing the occlusion of the actual presentation of the live entity from the trainee presentation further comprises enabling an onboard sensor based display of information when the virtual entity is within the defined range and the live entity is within the correlating tolerance.

19. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 1, wherein presenting emulation information to an operator of the live entity in compliance with the training scenario further comprises presenting flight plan, flight directive, or flight vector information to the operator to mimic a behavior of the virtual entity.

20. The method for blending an instance of a virtual entity with an actual presentation of a live entity within a trainee presentation of claim 19, wherein blending, within the trainee presentation, the instance of the virtual entity with the actual presentation of the live entity based on the correlating further comprises blending based on a determination of compliance with the flight plan, flight directive, or flight vector information.

10. A non-transitory computer readable medium having non-transitory computer readable program code embodied therein for blending an instance of a virtual entity with an actual presentation of a live entity within an aircraft trainee presentation, the computer readable program code comprising instructions which, when executed by a computer device or processor, perform and direct the steps of:

receiving a training scenario having a defined fixed range;

generating the instance of the virtual entity in compliance with the training scenario;

generating the trainee presentation, the trainee presentation including the instance of the virtual entity, the trainee presentation further including an occlusion of the actual presentation of the live entity, wherein including the instance of the virtual entity and including the occlusion of the actual presentation of the live entity are independent of the defined range;

communicating the trainee presentation to a trainee via at least one of;

a graphic display and a pictorial display;

presenting emulation information to an operator of the live entity in compliance with the training scenario to enable the live entity to rendezvous or to join with the virtual entity;

correlating a characteristic of the live entity with a characteristic of the virtual entity;

blending, within the trainee presentation, the instance of the virtual entity with the actual presentation of the live entity based on the correlating;

removing the virtual entity from the trainee presentation; and

removing the occlusion of the actual presentation of the live entity from the trainee presentation, wherein the removing the virtual entity and the removing the occlusion occur as a result of the blending, the virtual entity being within the defined range, and as a result of the live entity being within a correlating tolerance.

11. The non-transitory computer readable medium of claim 10, wherein the defined range is a first defined range that limits an ability of a trainee aircraft or its instrumentation, and wherein receiving a training scenario having the defined range further includes receiving data associated with presentation of the virtual entity on a display that presents training data according to a second defined range, the second defined range being greater than the first defined range, and wherein the first defined range is a charted geographical section of airspace and the second defined range is at least one of:

outside the charted geographical section of airspace, beyond visual range (BVR) of the trainee aircraft, and beyond local sensor range of the trainee aircraft.

12. The non-transitory computer readable medium of claim 10, wherein generating the instance of the virtual entity further comprises generating a plurality of virtual entities in compliance with the training scenario.

13. The non-transitory computer readable medium of claim 10, wherein the trainee presentation is configured for presentation on an avionics display onboard a trainee aircraft.

14. The non-transitory computer readable medium of claim 10, wherein presenting emulation information to an operator of the live entity to enable the live entity to rendezvous or to join with the virtual entity further comprises presenting information to the live entity to mimic a behavior of the virtual entity.

15. The non-transitory computer readable medium of claim 10, wherein correlating a characteristic of the live entity with a characteristic of the virtual entity further comprises correlating at least one of:

a position, an altitude, an airspeed, and a heading of the virtual entity with the live entity.

16. The non-transitory computer readable medium of claim 10, wherein correlating a characteristic of the live entity with a characteristic of the virtual entity further comprises receiving at least two characteristics associated with the live entity, comparing the at least two characteristics to two similar characteristics of the virtual entity, and determining if the live entity is within the correlating tolerance, the correlating tolerance including a tolerance for each of the at least two characteristics, and wherein the tolerance for one characteristic of the at least two characteristics is more flexible than the other tolerance.

17. The non-transitory computer readable medium of claim 10, wherein blending, within the trainee presentation, the instance of the virtual entity with the actual presentation of the live entity further comprises removing the occlusion of the actual presentation of the live entity and discontinuing a presentation of the instance of the virtual entity, thereby replacing the discontinued presentation of the instance of the virtual entity with the actual presentation of the live entity.

18. The non-transitory computer readable medium of claim 10, wherein removing the occlusion of the actual presentation of the live entity from the trainee presentation further comprises enabling an onboard sensor based display of information when the virtual entity is within the defined range and the live entity is within the correlating tolerance.