Simulated training environments based upon foveated object events

• US 8,721,341 B2
• Filed: 03/02/2005
• Issued: 05/13/2014
• Est. Priority Date: 03/02/2004
• Status: Expired due to Fees

• Current Assignee: Brian T. Mitchell
• Original Assignee: Optimetrics, Inc (DCS Corporation)
• Inventors: Mitchell, Brian T.
• Primary Examiner(s): Thai, Xuan
• Assistant Examiner(s): CARLOS, ALVIN LEABRES
• Application Number:
  US11/070,038
• Publication Number:
  US 20050195165A1
• Time in Patent Office: 3,359 Days
• Field of Search: 434/118, 434/247, 434/322, 434/323, 434/350, 434/367, 345/156, 345/158
• US Class Current: 434/247
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09B 9/08 for teaching control of aircraft, e.g. Link trainer

Abstract

Objects associated with eye fixations are used to support training tasks that are characterized by operation within a dynamic 3D environment, problem-solving that involves user multitasking, short decision time periods, heavy workloads, potential for unexpected events, and the need for high-performance skills. The invention is ideally suited to measure the nuances associated with maintaining situation awareness in such problems where dynamic 3D environments that support problem solving with high cognitive fidelity have already been developed. Potential training application areas include urban warfighting, night vision goggle use, aircraft piloting, helicopter operation, remote control of vehicles and robots, night driving, and air traffic control.

Claims

1. A training method, comprising the steps of:

defining a set of instructional goals including training objectives;

simulating a learning environment using a user interface and a display with pixels representing 2- and 3-dimensional objects, wherein the location of each object is represented by all of the pixels of the object visible on the display screen, and wherein the objects can move and interact;

monitoring user interactions with the objects, keeping track of which pixels on the display correspond to the interacting objects upon which the user is fixated;

computing various measures based upon the user interactions, including eye events, and simulation state;

recording and time-stamping the computed measures;

recording current user skill level based upon the computed measures;

correlating current user skill level with the training objectives; and

modifying the learning environment in accordance with the correlation.

2. The method of claim 1, wherein the eye events are calculated based upon pixel values.

3. The method of claim 2, including the following steps to compute the eye events:

obtaining a display fixation point from an input device;

initializing an object pointer and fixation point pixel reference value;

updating the object pointer and the pixel reference value as the pixel at the display fixation point changes; and

constructing the eye event from the display fixation point and the object referenced by the latest object pointer.

4. The method of claim 3, wherein the input device is an eye tracker.

5. The method of claim 1, wherein the eye events are calculated based upon non-pixel values.

6. The method of claim 5, including the following steps to compute the eye events:

obtaining a display fixation pixel from an input device;

initializing an object table;

iterating through an object rendering loop and, for each iteration;

placing an object in the object table,

setting the object value, and

drawing the next object;

determining an object key by referencing storage bits in a non-display pixel associated with the display fixation pixel;

using the object key to identify the fixation object through lookup in the object table; and

constructing the eye event from the display fixation point and the resulting fixation object.

7. The method of claim 6, wherein the input device is an eye tracker.

8. The method of claim 1, wherein the eye events are calculated based upon ordered objects that intersect with an imaginary eye ray.

9. The method of claim 8, including the following steps to compute the eye events:

obtaining a display fixation pixel from an input device;

constructing an imaginary eye ray through the display fixation pixel;

intersecting the eye ray with all the graphics objects;

ordering the intersecting objects along the eye ray;

identifying the fixation object as the intersecting object closest to the eyepoint; and

constructing the eye event from the display fixation point and the resulting fixation object.

10. The method of claim 9, wherein the input device is an eye tracker.

11. The method of claim 1, wherein the simulated learning environment is built on top of a game engine.

12. The method of claim 1, wherein the computed measures that are derived using subject workload metrics.

13. The method of claim 1, wherein the simulated learning environment involves vehicle or aircraft control.

14. The method of claim 1, wherein the simulated learning environment is a computer application.

15. The method of claim 1, wherein the simulated learning environment is a computer game.

16. The method of claim 1, wherein the simulated learning environment is a cooperative problem solving environment.

17. A training system, comprising:

a set of instructional goals including training objectives;

a simulated learning environment including a user interface and a display with pixels representing 2- and 3-dimensional objects, wherein the location of each object is represented by all of the pixels of the object visible on the display screen, and wherein the objects can move and interact;

the simulated learning environment being further operative to monitor user interactions with the objects and keep track of which pixels on the display correspond to the interacting objects upon which the user is fixated;

a simulation manager for modifying the learning environment as a function of user interactions;

a set of computed measures based upon the user interactions, including eye events, and simulation state;

a data recorder for recording and time-stamping the computed measures;

a skill model that maintains a record of current user skill level based upon the computed measures; and

an instructor that correlates the current user skill level with the training objectives to modify the learning environment through the simulation manager.

18. The system of claim 17, wherein the eye events are calculated using information from display pixel values.

19. The system of claim 17, wherein the eye events are calculated using information from the non-display pixel values.

20. The system of claim 17, wherein the eye events are calculated using information from ordered objects that intersect an eye ray.

21. The system of claim 17, wherein the eye events are represented as a computer-language structure or a class object.

22. The system of claim 17, wherein the eye events contain data that represents the time the event occurred.

23. The system of claim 17, wherein:

the display is generated using a graphics-rendering loop; and

the eye events are constructed during each iteration of the loop.

24. The system of claim 17, wherein the eye events are based upon a user'"'"'s eye fixation or eye movement.

25. The system of claim 17, wherein the simulated learning environment is built on top of a game engine.

26. The system of claim 17, wherein:

the goals are research goals; and

the simulation manager is an experiment manager.

27. The system of claim 17, wherein the computed measures that are derived using subject workload metrics.

28. The system of claim 17, wherein the simulated learning environment involves vehicle or aircraft control.

29. The system of claim 17, wherein the simulated learning environment is a computer application.

30. The system of claim 17, wherein the simulated learning environment is a computer game.

31. The system of claim 17, wherein the simulated learning environment is a cooperative problem-solving environment.