System for providing a high-fidelity visual display coordinated with a full-scope simulation of a complex system and method of using same for training and practice

• US 7,117,135 B2
• Filed: 05/14/2002
• Issued: 10/03/2006
• Est. Priority Date: 05/14/2002
• Status: Active Grant

• Current Assignee: CAE Incorporated
• Original Assignee: CAE Incorporated
• Inventors: Gagnon, Pierre, Boisvert, Philippe, Cull, Christopher A., Daigle, Jean, Riel, Yves
• Primary Examiner(s): PALADINI, ALBERT WILLIAM
• Application Number:
  US10/143,868
• Publication Number:
  US 20030214533A1
• Time in Patent Office: 1,603 Days
• Field of Search: 703/6, 703/1, 345/419, 345/426, 345/427, 700/118, 700/95, 700/100, 700/114, 700/115
• US Class Current: 703/6
• CPC Class Codes: G09B 9/302 the image being transformed by computer processing, e.g. updating the image to correspond to the changing point of view

Abstract

Control panels of a simulated complex system are displayed as a texture mask over a three-dimensional visual representation of an environment in which the complex system is operating. The three-dimensional visual representation is displayed in an inactive window to a user practicing or training on the simulated system. An active window underlying the visual display is used to track user inputs to the control panels. Smart graphics translate the user inputs into data elements that are sent to a full-scope simulation, and simulation conditions are returned and used to update the display of the control panels and the visual representation of the environment.

Claims

1. A system for providing a high-fidelity visual display coordinated with a full-scope simulation instance of a complex system, the system comprising smart graphics processing that generates a two-dimensional image of displayed control panels of the simulated complex system, the two-dimensional image including embedded smart graphics that translate user input events representing user manipulations of virtual controls associated with the displayed control panels into data elements that are sent to the full-scope simulation instance, three-dimensional visuals processing for generating the high fidelity visual display using visual environment data, and a visual merge management function that synchronizes and coordinates the two-dimensional image and the three-dimensional visuals to present a single, coordinated view to the user, and the full-scope simulation instance is fully coordinated with the high fidelity visual display by exchanging data between the smart graphics processing, the three-dimensional visuals processing and the full-scope simulation instance to determine when a change to the single coordinated view presented to the user is required.

2. The system as claimed in claim 1 further comprising a high-fidelity three-dimensional database for dynamic modeling of an environment in which the complex system operates, the database providing the visuals environment data to the three-dimensional visuals processing.

3. The system as claimed in claim 1 wherein the three-dimensional visuals processing exchanges data with the full-scope simulation instance, so that the full-scope simulation instance is responsive to the visual environment data, and the three-dimensional visuals processing is coordinated with the full-scope simulation instance in real time.

4. The system as claimed in claim 1 further comprising courseware adapted to facilitate the learning of system knowledge and procedures by the user of the system.

5. The system as claimed in claim 1 wherein the smart graphics processing further stores the two-dimensional image in an off-screen buffer and maintains a shared memory device context for tracking the smart graphics associated with the two-dimensional image to enable translation of the user input events into changes to data element values that are supplied to the full-scope simulation instance.

6. The system as claimed in claim 5 wherein the smart graphics processing udates the two-dimensional image in response to a user input event that changes a viewport that displays the high-fidelity visual display.

7. The system as claimed in claim 6 wherein the visual merge management function communicates and coordinates changes in the two-dimensional image to the three-dimensional visuals processing.

8. The system as claimed in claim 7 wherein the visual merge management function receives window data and bitmap updates, and forwards the window data and the bitmap updates to the three-dimensional visuals processing.

9. The system as claimed in claim 8 further comprising a texture mask assembler adapted to receive the bitmap updates, and to convert the bitmap to a texture mask.

10. The system as claimed in claim 9 wherein three-dimensional processing generates the high-fidelity visual display by overlaying the texture mask on a three-dimensional image generated using the visual environment data.

11. The system as claimed in claim 10 wherein the smart graphics processing and the tree-dimensional visuals processing each dynamically subscribe to data elements used to exchange data with the instance of the full-scope simulation.

12. The system as claimed in claim 11 wherein the smart graphics processing changes its data element subscriptions each time a change to the two-dimensional image results in a change to smart graphics embedded in the two-dimensional image.

13. The system as claimed in claim 1 wherein the client computer further comprises courseware for providing training in any one of systems knowledge, integrated procedures, and skills required to operate or maintain the complex system.

14. A method of providing training and practice on a simulated complex system for personnel who are to operate or maintain the complex system, comprising:

generating a two-dimensional image of displayed control panels of the simulated complex system, the two-dimensional image including embedded smart graphics processing that translates user input events representing user manipulations of virtual controls associated with the displayed control panels into data elements that are sent to a full-scope simulation instance of the complex system;

generating a high fidelity visual display using three-dimensional visuals processing which processes visual environment data and a visual merge management function that synchronizes and coordinates the two-dimensional image and the three-dimensional visuals to present a single, coordinated view to the user; and

coordinating the high fidelity visual display with the full-scope simulation instance of the complex system by exchanging data between the smart graphics processing, the three-dimensional visuals processing and the full-scope simulation instance to determine when a change to the single coordinated view presented to the user is required.

15. The method as claimed in claim 14 further comprising retrieving the visual environment data from a database local to a client computer that performs the steps of generating and coordinating.

16. The method as claimed in claim 15 further comprising simulating the complex system using the instance of the simulation that runs on a remote server.

17. The method as claimed in claim 16 further comprising subscribing to data elements for exchanging data with the instance of the simulation to permit an exchange of data with the instance of the simulation using the data elements, in order to transfer the inputs generated by the manipulation of the virtual controls by the personnel to the instance of the simulation, and to receive feed back from the instance of the simulation for updating the two dimensional image.

18. The method as claimed in claim 16 further comprising subscribing to the data elements for exchanging data with the instance of the simulation, to permit an exchange of data with the instance of the simulation using the data elements, in order to transfer information retrieved from the visual environment database to the instance of the simulation and to coordinate the high fidelity visual display with a condition of the instance of the simulation.

19. The method as claimed in claim 14 wherein generating the two-dimensional image further comprises modifying the two-dimensional image so that areas that are to be transparent are assigned a chroma-key color attribute, and storing a bitmap image in a memory of the client computer.

20. The method as claimed in claim 19 further comprising transferring the bitmap image to three-dimensional visuals processing, which is responsible for generating the high fidelity visual display, each time the smart graphics is changed.

21. The method as claimed in claim 20 further comprising filtering the chroma-key values from the bitmap image and converting the bitmap image to three-dimensional texture data used by the three-dimensional visuals processing to create the texture mask.

22. The method as claimed in claim 20 further comprising providing courseware on the client computer for training the personnel in any one of systems knowledge, integrated procedures, operator skills, and maintenance skills required to operate or maintain the complex system.

23. The method as claimed in claim 22 further comprising associating courseware with learning management system records and tracking mechanisms for the purpose of granting training credits and complementing a course curriculum.

24. The method as claimed in claim 23 further comprising downloading any one of courseware, smart graphics objects and three-dimensional databases based on curriculum requirements determined by a learning management system.

25. A complex system training and practice apparatus, comprising a client computer provisioned with smart graphics processing adapted to generate a two-dimensional image of displayed control panels of the simulated complex system, the image including embedded smart graphics that translate user input events representing user manipulations of virtual controls associated with the displayed control panels into data elements that are sent to a full-scope simulation instance of the complex system, and three-dimensional visuals processing for generating a high fidelity visual display using visual environment data representing an environment in which the simulated complex system is operated, and combining the high fidelity visual display with the two-dimensional image for display on a display surface of the client computer, and an interface for permitting the client computer to communicate and exchange simulation data with a server computer that operates the full-scope simulation instance that simulates the complex system.

26. The apparatus as claimed in claim 25 wherein the client computer further comprises a database for storing the visual environment data.

27. The apparatus as claimed in claim 26 wherein the client computer further comprises a texture mask assembler for converting the two-dimensional image into visual texture data used by the three-dimensional visuals processing as a texture mask to generate the high fidelity visual display.

28. The apparatus as claimed in claim 27 wherein the three-dimensional visuals processing applies the textures mask to a full screen polygon at zero depth to a user, and the user views the high fidelity visual display of the environment through transparent areas of the texture mask.

29. The apparatus as claimed in claim 28 wherein the smart graphics processing generates the two-dimensional image with a chroma-key color attribute assigned to selected areas that are to be transparent, and the texture mask assembler converts the two-dimensional image to a texture mask with transparency in the selected areas.

30. A computer readable medium containing computer executable program instructions, comprising:

program instructions for smart graphics processing that generates a two-dimensional image of displayed control panels of a simulated complex system, the two-dimensional image including embedded smart graphics that accept user input events that represent user manipulations of virtual controls associated with the displayed control panels and translate the user input events into data elements that are sent to a full-scope simulation instance of the simulated complex system;

program instructions for three-dimensional visuals processing that generates a high fidelity visual display using visual environment data representations of an environment in which the simulated complex system is operated, and a visual merge management function that synchronizes and coordinates the two-dimensional image and the three-dimensional visuals to present a single, coordinated view to the user;

program instructions for providing the full-scope simulation instance of the complex system that is fully coordinated with the high-fidelity visual display by exchanging data between the smart graphics processing, the three-dimensional visuals processing and the full-scope simulation instance to determine when a change to the single coordinated view presented to the user is required.

31. The computer readable medium as claimed in claim 30 further comprising program instructions for providing the visual merge management function, which accepts the two-dimensional image and display window data from the smart graphics processing, passes the two-dimensional image and display window data to the three-dimensional visuals processing, and coordinates generation of the high fidelity visual display.

32. The computer readable medium as claimed in claim 30 further comprising program instructions for converting the two-dimensional image into a three-dimensional texture data.

33. The computer readable medium as claimed in claim 32 further comprising program instructions for filtering out regions of the two-dimensional image to provide transparent areas in the three-dimensional texture data through which the high fidelity visual display of the environment can be viewed.

34. The computer readable medium as claimed in claim 33 further comprising program instructions for applying the texture data to a full screen polygon laid over the high fidelity visual display at zero depth to a user, to permit the user to view the high fidelity visual display of the environment through the transparent areas of the texture data.

35. The computer readable medium as claimed in claim 30 further comprising courseware program instructions for training students in any one of systems knowledge, integrated procedures, operator skills, and maintenance skills required to operate or maintain the complex system.