Automated trainee monitoring and performance evaluation system

• US 7,949,295 B2
• Filed: 08/18/2005
• Issued: 05/24/2011
• Est. Priority Date: 08/18/2004
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Hanna, Keith J., Kumar, Rakesh, Paragano, Vincent, Cheng, Hui, Arpa, Aydin
• Primary Examiner(s): Thai; Xuan M
• Assistant Examiner(s): Hu; Kang
• Application Number:
  US11/206,678
• Publication Number:
  US 20060073449A1
• Time in Patent Office: 2,105 Days
• Field of Search: 434/247, 434/362, 434/1, 434/21, 434/76, 434/308, 434/307.R, 434/323, 434/325, 434/350, 482/1, 482/4, 482/8, 345/418, 345/441
• US Class Current: 434/307.00R
• CPC Class Codes: F41A 33/00 Adaptations for training adaptations of barrels for recoil reinforcement F41A21/26; Gun simulators teaching or practice apparatus for gun-aiming or gun-laying F41G3/26 ; shooting games A63F9/02; military simulation G09B9/003
• CPC Class Codes: F41G 3/26 Teaching or practice apparatus for gun-aiming or gun-laying shooting games A63F9/02; light- or radiation-emitting guns F41A33/02; targets therefor F41J; responders, transponders for radar, sonar, lidar G01S13/767, G01S15/74, G01S17/74
• CPC Class Codes: F41J 5/10 Cinematographic hit-indicating systems cinematographic targets F41J9/14
• CPC Class Codes: F41J 5/14 Apparatus for signalling hits or scores to the shooter, e.g. manually operated, or for communication between target and shooter; Apparatus for recording hits or scores
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A method and system for performing automated training environment monitoring and evaluation. The training environment may include a mixed reality elements to enhance a training experience.

Claims

1. A system for monitoring a training environment, comprising:

at least one first sensor for monitoring a training facility, the at least one first sensor being configured to measure geometric and reflective properties of arbitrary surfaces in the training facility and to compensate for color, texture, and form factor of the surfaces of the training facility by component projectors;

at least one second sensor, coupled to at least one trainee, for monitoring activities of the at least one trainee;

a control system, coupled to the at least one first sensor and the at least one second sensor for receiving monitoring data from the at least one first sensor and for receiving monitoring data from the at least one second sensor, respectively, the control system being configured to create two and three dimensional scriptable synthetic objects that comprise static and animated two and three-dimensional images of targets, opponents, and obstacles corresponding to objectives of training exercises, the control system being further configured to providing interaction of the two and three dimensional scriptable synthetic objects to the at least one trainee within the training facility, said control system being further configured to analyze the activities of the at least one trainee by evaluating performance criteria comprising movements, poses and positions of a head, torso, legs and arms of the at least one trainee, and relative movements, poses and positions of the at least one trainee to the two and three dimensional scriptable synthetic objects and positions of real objects based on the monitoring data received from the at least one first sensor and the at least one second sensor, said control system being further configured to process the at least one pose using projective warping so as to produce at least one warped image, the at least one warped image being applied to an occlusion computation to create augmented imagery within a scene from a point of view of the trainee, the augmented imagery being processed to form left and right eye views; and

the at least one component projector for projecting onto the training facility synthetic, the two and three dimensional scriptable synthetic objects to construct a seamless mixed reality scene comprising both real, live world and projected synthetic world within which the at least one trainee is immersed on a real-time basis, the two and three dimensional scriptable synthetic objects changing in response to the performance of the at least one trainee in the training facility and based on the monitoring data received from the at least one first sensor and the at least one second sensor.

2. The system of claim 1 wherein the control system evaluates a performance of the at least one trainee and alters the training environment in view of the performance evaluation.

3. The system of claim 1 wherein the at least one trainee comprises at least one of an individual, an apparatus operated by the at least one individual, a group of individuals, and a group of apparatuses operated by the group of individuals.

4. The system of claim 1 wherein the at least one second sensor are selected from the group consisting of at least one of a visible marker, an infrared marker, an inertial navigation sensor (INS), a video camera, a microphone, or a radio-frequency tag (RFID).

5. The system of claim 1 wherein the at least one second sensor are used to track the at least one trainee.

13. The apparatus of claim 1, wherein the control system is further configured to evaluate the performance of the at least one trainee in real-time that computes the performance of the at least one trainee from sensor data and a knowledgebase.

14. The apparatus of claim 13, wherein the control system is further configured to optimize and adaptively adjusts stimuli based on training objectives, behavior of the at least one trainee, performance of the at least one trainee, and the history of the stimuli.

15. The apparatus of claim 14, wherein the control system is further configured to adjust training objectives and difficulties based on sensor data in real-time and in subsequent training exercises.

16. The apparatus of claim 15, further comprising an after-action-review (AAR) subsystem that allows instructors and participants to search and retrieve events, behavior of interests, statistics, to replay a training exercise including videos taken and symbolic representation of the exercise and detected events or behaviors, and to provide a report generation function.

17. The apparatus of claim 16, further comprising a database for storing sensor information, performance, training objectives and stimuli for archiving and for providing after-action-review (AAR).

6. A computer implemented method of monitoring a training environment, comprising:

providing at least one first sensor for monitoring a training facility, the at least one first sensor being configured to measure geometric and reflective properties of arbitrary surfaces in the training facility and to compensate for color, texture, and form factor of the surfaces of the training facility by component projectors;

providing at least one second sensor, coupled to at least one trainee, for monitoring activities of the at least one trainee;

monitoring the training facility and the activities of the at least one trainee, using a computer processor to perform the steps of;

receiving monitoring data from the at least one first sensor and receiving monitoring data from the at least one second sensor;

creating two and three dimensional scriptable synthetic objects that comprise static and animated images of targets, opponents, and obstacles corresponding to objectives of training exercises;

providing interaction between the two and three dimensional scriptable synthetic objects and the at least one trainee within the training facility;

analyzing the activities of the at least one trainee by evaluating performance criteria comprising movements, poses and positions of a head, torso, legs and arms of the at least one trainee, and relative movements, poses and positions of the at least one trainee to the two and three dimensional scriptable synthetic objects and positions of real objects based on the monitoring data received from the at least one first sensor and the at least one second sensor;

processing the at least one pose using projective warping so as to produce at least one warped image, the at least one warped image being applied to an occlusion computation to create augmented imagery within a scene from a point of view of the trainee, the augmented imagery being processed to form left and right eye views; and

using the at least one component project to perform the steps of;

projecting onto the training facility the two and three dimensional scriptable synthetic objects to construct a seamless mixed reality scene comprising both real, live world and a projected synthetic world within which the at least one trainee is immersed on a real-time basis, the two and three dimensional scriptable synthetic objects changing in response to the performance of the at least one trainee in the training facility and based on the monitoring data received from the at least one first sensor and the at least one second sensor.

7. The method of claim 6 further comprising tracking the at last one trainee using the at least one second sensor.

8. The method of claim 6 further comprising:

evaluating the activities of the at least one trainee.

9. The method of claim 8 further comprising:

altering the training environment in view of the evaluation.

10. The method of claim 6 further comprising:

analyzing the activities of the at least one trainee and comparing a performance of the at least one trainee with a performance of an expert.

11. The method of claim 6 wherein the at least one trainee comprises at least one of an individual, an apparatus operated by the at least one individual, a group of individuals, and a group of apparatuses operated by the group of individuals.

12. The method of claim 6 wherein the at least one second sensor are selected from the group consisting of at least one of a visible marker, an infrared marker, an inertial navigation sensor (INS), a video camera, a microphone, or a radio-frequency tag (RFID).