Augmented Reality

• US 20160267808A1
• Filed: 03/09/2016
• Published: 09/15/2016
• Est. Priority Date: 03/09/2015
• Status: Active Grant

• Current Assignee: Alchemy Systems, L.P. (Intertek Group PLC)
• Original Assignee: Alchemy Systems, L.P. (Intertek Group PLC)
• Inventors: Agostini, Ric, Lang, Scott, Larmond, Kimberly-Anne, Martin, Scott, McGuire, Terence Patrick, Scozzari, Salvatore
• Granted Patent: US 11,012,595 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G09B 7/08 characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying further information
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

In a method and system of training, one or more digital assets, such as two-dimensional and three-dimensional computer-generated objects, are superimposed over a live camera view to generate a simulated training scenario, referred to herein as augmented reality (“AR”) technology. By leveraging AR technology, a live simulation of real-world events, situations, and skills is generated for which an employee, student, customer, or any type of person in need of training, is being trained. A trainee is thus immersed directly into the training material. The physical environment and working conditions are integrated into the sequence of training material, and it does this live, that is, with immediate feedback from a camera'"'"'s live screen view. This technique may, by way of examples, also be described as an automated guided tour used to facilitate employee on-boarding and training, or as a guided tour through a warehouse store for its customers.

Claims

1. A training system comprising:

a mobile computing device (“

MCD”

) having a processor and a memory configured for storing an application software program executable by the processor;

a camera mounted on the MCD for capturing an image of an object and for generating image data indicative of the location and orientation of the image for storage in memory;

position sensors mounted on the MCD for generating data indicative of the location and orientation of the MCD relative to the object, and for storage of same in memory;

a display operative by the processor;

augmented reality (“

AR”

) sequence definitions stored in the memory and accessible by the program for defining steps of a lesson; and

digital assets accessible by the AR sequence definitions;

wherein the program is executable by the processor, with reference to an AR sequence definition and data input from the camera and position sensors, for overlaying an image from the camera with digital assets to thereby generate an augmented reality image and display the image on the display.

2. The system of claim 1 wherein the camera captures a live view of a trainee'"'"'s actual environment.

3. The system of claim 1 wherein the camera captures a live view of a trainee'"'"'s actual environment, incorporating elements of the actual physical environment into the view, wherein elements include at least one of objects, equipment, employees, customers, lighting levels, noise levels, and vibration.

4. The system of claim 1, wherein data assets includes at least one of 2D and 3D objects, audio clips, video clips, static images, text data, animations, and hyperlinks.

5. The system of claim 1 further comprising multiple mobile devices configured to enable a single supervisor, instructor, or customer representative to provide multiple training sessions in parallel.

6. A method of training using a mobile computing device (“

MCD”

), the method comprising steps of;

capturing an original image of an object and its environment;

recognizing the object;

detecting a trigger image;

determining the trigger location and orientation;

detecting the location and orientation of the MCD;

calling up one or more digital assets;

generating an augmented reality (“

AR”

) image by superimposing the one or more digital assets over the original image; and

displaying the AR image to a user.

7. The method of claim 6, further comprising the step of adjusting digital assets in location and orientation relative to the objects in the image.

8. The method of claim 6, wherein the original image includes a live view of a trainee'"'"'s actual environment, incorporating elements of the actual physical environment into the view, wherein elements include at least one of objects or equipment, other employees, customers, lighting levels, noise levels, and vibration.

9. The method of claim 6, wherein data assets includes at least one of 2D and 3D objects, audio clips, video clips, static images, text data, animations, and hyperlinks.

10. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated.

11. The method of claim 6, further comprising a step of prompting a user with 3D animated renderings to choose among a series of options, including at least one of a multiple-choice question that represents real-world strategies to resolve a situation presented in a lesson.

12. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated, wherein the questions include 3D-rendered objects that simulate options available to address a problem of a given situation.

13. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated, wherein correct responses to questions include 3D animated renderings and audio effects to simulate the result of one or more correct responses in a multiple-choice question, and illustrating to the trainee how one or more correct responses cause the problem to be solved or ill effects to be minimized.

14. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated, wherein incorrect responses to questions include 3D animated renderings and audio effects to simulate the result of an incorrect choice in a multiple-choice question, and illustrating to the trainee how an incorrect choice either has no effect on a problem, or causes the problem to get worse, or causes another, potentially worse problem to arise.

15. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated, wherein, when the trainee chooses an incorrect option, the application flows into a remediation loop, which remediation loop displays the relevant training material to the trainee once again, optionally in a different order, format, or style, adapting to the user'"'"'s personal learning style and preferences.

16. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated, wherein, when the trainee chooses a correct option, the application simulates the effect of eliminating the problem or reducing the problem to a manageable level, thus providing positive feedback to the trainee as a reward for having made the correct choice.

17. The method of claim 6, further comprising a step of prompting a user to answer questions that test the user'"'"'s understanding of the material based on the situation that is being simulated, wherein the trainee has available multiple correct choices, one or more of which may be selected, each of which choices can reduce or eliminate the problem, and not necessarily in the same way as other correct options.

18. A training method comprising steps of:

displaying on a mobile computing device a live view of an object and surrounding environment;

superimposing 3D animated renderings and audio effects over the live view to thereby generate a superimposed view; and

prompting, with the superimposed view, a trainee to follow a pre-defined sequence of steps required to perform a particular procedure as required for a training lesson.

19. The training method of claim 18, further comprising steps of:

reading by the trainee of a description of at least one particular step; and

interacting with objects displayed on the mobile computing device to trigger an animation or visualization of the at least one particular step.

20. The training method of claim 18, further comprising the step of guiding the trainee from one step to the next, in exactly the correct order as required by the procedure and as built into the application.

21. The training method of claim 18, further comprising the step wherein, upon completion of the entire sequence of steps, simulating the final result of the procedure as a set of 3D renderings and audio effects superimposed on the live view, the audio effects being played through the speakers incorporated into the mobile computing device, thereby providing positive feedback to the trainee.