AUGMENTED AND VIRTUAL REALITY SIMULATOR FOR PROFESSIONAL AND EDUCATIONAL TRAINING

• US 20170213473A1
• Filed: 03/07/2017
• Published: 07/27/2017
• Est. Priority Date: 09/08/2014
• Status: Active Grant

• Current Assignee: Simx, Inc.
• Original Assignee: Simx, Inc.
• Inventors: Ribeira, Ryan Joseph, Sarma, Karthik Venkataraman, Stadelman, JR., Stanley Thomas, Namperumal, Srihari Kumar, Ribeira, Jason Robert, Miller, Christopher Ryan
• Granted Patent: US 10,672,288 B2
• Time in Patent Office: Days
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09B 23/28 for medicine
• CPC Class Codes: G09B 23/285 for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas
• CPC Class Codes: G09B 23/30 Anatomical models G09B23/281 - G09B23/288 take precedence
• CPC Class Codes: G09B 25/02 of industrial processes; of machinery
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G09B 5/10 all student stations being capable of presenting the same information simultaneously G09B5/14 takes precedence
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131
• CPC Class Codes: G16H 20/10 relating to drugs or medications, e.g. for ensuring correct administration to patients
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders

Abstract

A method and apparatus for an augmented reality simulator for professional and educational training is provided. The simulator provides a training environment spanning one or more physical locations in which one or more virtual avatars representing purely virtual objects or persons or real physical objects or persons which are located at a different physical location are projected into the physical space. The avatars are interactive with other avatars and real objects or persons and update over time or in response to actions taken by other real or virtual elements, or based on predefined instructions. Sensors and devices are used to detect the locations of and actions taken by real persons or real objects and this sensed data is used to evolve the state of the simulation and avatars based on predefined instructions and programs and update the view of all participants.

Claims

1. A method comprising;

receiving, by an authoring application running on a computing device, a set of one or more values that define at least one of a physical or a physiological property of a virtual avatar;

in response to receiving the set of one more values, generating, by the authoring application running on the computing device, an instruction set for virtual avatar projection into an augmented or virtual reality environment;

wherein the instruction set includes instructions used by an augmented or virtual reality application to project the virtual avatar into real or virtual space such that the virtual avatar exhibits a behavior based, at least in part, on at least one of the physical or the physiological properties defined through the authoring application.

2. The method according to claim 1, wherein the instruction set generated by the authoring application is operable to trigger changes to the augmented or virtual reality environment over time or based upon the passage of time, such that a sequence of events occurs in the augmented or virtual reality environment as defined by the instruction set.

3. The method according to claim 2, wherein a sequence of events occurring over time and defined by an instruction set is operable to be navigated by pausing execution, reversing execution, accelerating execution, moving to any time point in the instruction set and starting execution from that point, resuming execution, or stopping execution for all participants sharing the augmented or virtual reality environment.

4. The method according to claim 1, wherein generating the instruction set comprises selecting from a set of predefined instructions, composing new instructions, or uploading instructions from an external piece of computer readable media.

5. The method according to claim 1, wherein the authoring application allows for an author to allow at least one characteristic of at least one virtual avatar to change in response to any change in the augmented or virtual reality environment.

6. The method according to claim 1, wherein instructions generated by the authoring application can be distributed to other users via an online marketplace, allowing for control of access to those instructions, charging of fees for those instruction, or collection of data regarding the usage of those instructions.

7. The method according to claim 1, wherein the instruction set generated by the authoring application causes the behavior is a physiological reaction that is exhibited responsive to interactions taken by a participant of the augmented or virtual reality environment during simulation of a medical procedure.

8. A method comprising;

detecting, using at least one sensor, an occurrence of an action within an augmented or virtual reality environment;

recording, using at least one computer device, the occurrence of an action within an augmented or virtual reality environment;

comparing a record of the action to a performance metric that identifies a pre-determined set of goal actions;

generating, based at least in part on said comparing, a performance evaluation that includes one or more measures that indicate a performance of at least one participant in the augmented or virtual reality environment.

9. The method according to claim 8, wherein the action by the at least one participant that is detected and recorded is comprised of at least one body movement, direct input of data into the at least one computer device, at least one vocalization, or the expression of specific ideas via body movement or vocalization.

10. The method according to claim 8, wherein the actions recorded within an augmented or virtual reality environment are appended to a record of the augmented or virtual reality environment session.

11. The method according to claim 10, wherein the record of the augmented or virtual reality environment session is comprised of the record of the execution of a file containing an instruction set controlling the augmented or virtual reality environment, timing data regarding when instructions were executed and events occurred, a plurality of events occurring over the course of execution of an instruction set, a plurality of data collected from a plurality of participant apparatuses, and video representing the augmented or virtual reality environment from the perspective of at least one participant or a virtual 3rd party perspective.

12. The method according to claim 11, wherein an event is comprised of any change to the augmented or virtual reality environment.

13. One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, causes operations comprising:

receiving, by an authoring application running on a computing device, a set of one or more values that define at least one of a physical or a physiological property of a virtual avatar;

in response to receiving the set of one more values, generating, by the authoring application running on the computing device, an instruction set for virtual avatar projection into an augmented or virtual reality environment;

wherein the instruction set includes instructions used by an augmented or virtual reality application to project the virtual avatar into real or virtual space such that the virtual avatar exhibits a behavior based, at least in part, on at least one of the physical or the physiological properties defined through the authoring application.

14. The one or more non-transitory computer-readable media of claim 13, wherein the instruction set generated by the authoring application is operable to trigger changes to the augmented or virtual reality environment over time or based upon the passage of time, such that a sequence of events occurs in the augmented or virtual reality environment as defined by the instruction set.

15. The one or more non-transitory computer-readable media of claim 14, wherein a sequence of events occurring over time and defined by an instruction set is operable to be navigated by pausing execution, reversing execution, accelerating execution, moving to any time point in the instruction set and starting execution from that point, resuming execution, or stopping execution for all participants sharing the augmented or virtual reality environment.

16. The one or more non-transitory computer-readable media of claim 13, wherein generating the instruction set comprises selecting from a set of predefined instructions, composing new instructions, or uploading instructions from an external piece of computer readable media.

17. The one or more non-transitory computer-readable media of claim 13, wherein the authoring application allows for an author to allow at least one characteristic of at least one virtual avatar to change in response to any change in the augmented or virtual reality environment.

18. The one or more non-transitory computer-readable media of claim 13, wherein instructions generated by the authoring application can be distributed to other users via an online marketplace, allowing for control of access to those instructions, charging of fees for those instruction, or collection of data regarding the usage of those instructions.

19. The one or more non-transitory computer-readable media of claim 13, wherein the instruction set generated by the authoring application causes the behavior is a physiological reaction that is exhibited responsive to interactions taken by a participant of the augmented or virtual reality environment during simulation of a medical procedure.