PROCEDURAL LANGUAGE AND CONTENT GENERATION ENVIRONMENT FOR USE IN AUGMENTED REALITY/MIXED REALITY SYSTEMS TO SUPPORT LABORATORY AND RELATED OPERATIONS

• US 20190228269A1
• Filed: 01/04/2019
• Published: 07/25/2019
• Est. Priority Date: 01/04/2018
• Status: Active Grant

• Current Assignee: Lablightar Incorporated
• Original Assignee: Ias Machine, LLC
• Inventors: Brent, Roger, Tremaine, Jamie Douglas, Kellum, John Max
• Granted Patent: US 10,963,750 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 19/0069 Engineering, e.g. mechanical, electrical design computer engineering G09B19/0053
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip

Abstract

A method of operating a procedural language and content generation system that involves correlating environment objects and object movement to input controls through operation of a correlator, operating an interpreter to evaluate the correlation of the input controls and object/object movement against known libraries to generate programmatic instructions, storing the programmatic instructions as an instruction set, transforming the instruction set into executable commands through a compiler, and configuring control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from an image processor.

Claims

1. A method of operating a procedural language and content generation system comprises:

correlating environment objects and object movement detected through an image processor to input controls through operation of a correlator;

operating an interpreter to;

evaluate an input set from a protocol development user interface against a known objects library, a protocol library, and declared objects;

evaluate correlation of the input controls, the environment objects, and the object movement against the known objects library, the protocol library, and the declared objects; and

generate programmatic instructions;

storing the programmatic instructions as an instruction set within a protocol repository;

transforming the instruction set into executable commands through operation of a compiler;

configuring control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from the image processor;

operating an audio device in response to at least one executed command performed by the control logic;

operating a constructor to generate an augmented reality layer in response to the at least one executed command performed by the control logic;

overlaying the augmented reality layer over an environment image layer through operation of a combiner; and

displaying the augmented reality layer above the environment image layer in a display device.

2. The method of claim 1 further comprises:

correlating the environment image layer and the augmented reality layer of an in progress instruction set to an edit control through operation of the correlator;

operating the interpreter to;

evaluate correlation of the edit control, the environment image layer, and the augmented reality layer against the known objects library, the protocol library, the declared objects, and stored programmatic instructions for the in progress instruction set; and

generate revised programmatic instructions;

storing the revised programmatic instructions as a revised instruction set within the protocol repository;

transforming the instruction set into the executable commands through operation of the compiler;

configuring the control logic to perform the executable commands in response to receiving the detected environment objects and the detected object movement from the image processor;

operating the audio device in response to the at least one executed command performed by the control logic;

operating the constructor to generate a revised augmented reality layer in response to the at least one executed command performed by the control logic;

overlaying the revised augmented reality layer over the environment image layer through operation of the combiner; and

displaying the revised augmented reality layer above the environment image layer in the display device.

3. The method of claim 1, wherein the image processor comprises an object recognition engine and a spatiotemporal activity engine.

4. The method of claim 3, wherein the object recognition engine and the spatiotemporal activity engine utilizes at least one neural network for detecting and classifying the environment objects and tracking the object movement.

5. The method of claim 1, wherein the protocol development user interface generates a flow diagram.

6. The method of claim 1, wherein the protocol development user interface generates a three dimensional protocol workflow.

7. A computing apparatus, the computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the apparatus to;

correlate environment objects and object movement detected through an image processor to input controls through operation of a correlator;

operate an interpreter to;

evaluate an input set from a protocol development user interface against a known objects library, a protocol library, and declared objects;

evaluate correlation of the input controls, the environment objects, and the object movement against the known objects library, the protocol library, and the declared objects; and

generate programmatic instructions;

store the programmatic instructions as an instruction set within a protocol repository;

transform the instruction set into executable commands through operation of a compiler;

configure control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from the image processor;

operate an audio device in response to at least one executed command performed by the control logic;

operate a constructor to generate an augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the augmented reality layer over an environment image layer through operation of a combiner; and

display the augmented reality layer above the environment image layer in a display device.

8. The computing apparatus of claim 7 further comprises:

correlate the environment image layer and the augmented reality layer of an in progress instruction set to an edit control through operation of the correlator;

operate the interpreter to;

evaluate correlation of the edit control, the environment image layer, and the augmented reality layer against the known objects library, the protocol library, the declared objects, and stored programmatic instructions for the in progress instruction set; and

generate revised programmatic instructions;

store the revised programmatic instructions as a revised instruction set within the protocol repository;

transform the instruction set into the executable commands through operation of the compiler;

configure the control logic to perform the executable commands in response to receiving the detected environment objects and the detected object movement from the image processor;

operate the audio device in response to the at least one executed command performed by the control logic;

operate the constructor to generate a revised augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the revised augmented reality layer over the environment image layer through operation of the combiner; and

display the revised augmented reality layer above the environment image layer in the display device.

9. The computing apparatus of claim 7, wherein the image processor comprises an object recognition engine and a spatiotemporal activity engine.

10. The computing apparatus of claim 9, wherein the object recognition engine and the spatiotemporal activity engine utilizes at least one neural network for detecting and classify the environment objects and tracking the object movement.

11. The computing apparatus of claim 7, wherein the protocol development user interface generates a flow diagram.

12. The computing apparatus of claim 7, wherein the protocol development user interface generates a three dimensional protocol workflow.

13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

correlate environment objects and object movement detected through an image processor to input controls through operation of a correlator;

operate an interpreter to;

evaluate an input set from a protocol development user interface against a known objects library, a protocol library, and declared objects;

evaluate correlation of the input controls, the environment objects, and the object movement against the known objects library, the protocol library, and the declared objects; and

generate programmatic instructions;

store the programmatic instructions as an instruction set within a protocol repository;

transform the instruction set into executable commands through operation of a compiler;

configure control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from the image processor;

operate an audio device in response to at least one executed command performed by the control logic;

operate a constructor to generate an augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the augmented reality layer over an environment image layer through operation of a combiner; and

display the augmented reality layer above the environment image layer in a display device.

14. The computer-readable storage medium of claim 13 further comprises:

correlate the environment image layer and the augmented reality layer of an in progress instruction set to an edit control through operation of the correlator;

operate the interpreter to;

evaluate correlation of the edit control, the environment image layer, and the augmented reality layer against the known objects library, the protocol library, the declared objects, and stored programmatic instructions for the in progress instruction set; and

generate revised programmatic instructions;

store the revised programmatic instructions as a revised instruction set within the protocol repository;

transform the instruction set into the executable commands through operation of the compiler;

configure the control logic to perform the executable commands in response to receiving the detected environment objects and the detected object movement from the image processor;

operate the audio device in response to the at least one executed command performed by the control logic;

operate the constructor to generate a revised augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the revised augmented reality layer over the environment image layer through operation of the combiner; and

display the revised augmented reality layer above the environment image layer in the display device.

15. The computer-readable storage medium of claim 13, wherein the image processor comprises an object recognition engine and a spatiotemporal activity engine.

16. The computer-readable storage medium of claim 15, wherein the object recognition engine and the spatiotemporal activity engine utilizes at least one neural network for detecting and classify the environment objects and tracking the object movement.

17. The computer-readable storage medium of claim 13, wherein the protocol development user interface generates a flow diagram.

18. The computer-readable storage medium of claim 13, wherein the protocol development user interface generates a three dimensional protocol workflow.