Augmented and virtual reality based process oversight

• US 10,367,823 B2
• Filed: 10/19/2016
• Issued: 07/30/2019
• Est. Priority Date: 08/17/2015
• Status: Active Grant

• Current Assignee: Toronto-Dominion Bank
• Original Assignee: Toronto-Dominion Bank
• Inventors: Moretti, Christianne, Pryor, James Zachary, Hamilton, Matthew, Barnett, Jonathan K., Chan, Paul Mon-Wah, Del Vecchio, Orin, Barbon, John, Lee, John Jong-Suk
• Primary Examiner(s): Plecha, Thaddeus J
• Application Number:
  US15/297,702
• Publication Number:
  US 20170054735A1
• Time in Patent Office: 1,014 Days
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06Q 10/103 Workflow collaboration or project management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 63/107 wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals

Abstract

The present disclosure involves systems and computer implemented methods for generating a virtual or augmented reality-based oversight system of business processes at a location remote from where the business process is executed. In one example, system operations include identifying a process-based operation performed at a physical first location, where the operation is associated with a request for oversight from a remote second location. A current state of the operation at the first location is identified, and at least one input source at the first location is identified. A simulation of the current state of the operation at the first location incorporating input from the input source(s) is generated for presentation at the second location. An action performed at the first location in the current state of the operation is identified, and the second location is then transmitted a request to authorize the identified action.

Claims

1. A system at a second location, the system comprising:

a memory;

a communications module; and

at least one hardware processor interoperably coupled with the memory and the communications module, the at least one hardware processor configured to;

monitor, from the second location remote from a first location, a process-based operation performed at the first location;

receive, at the second location and via the communications module, a first signal from the first location indicating availability for oversight of the process-based operation, wherein the first signal is associated with an automatic request for oversight to the system at the second location generated by an application being executed at the first location, wherein the application is associated with the execution of the process-based operation, and wherein the application identifies a particular operation of the process-based operation determined to require oversight prior to the particular operation being performed;

receive, at the second location and via the communications module, a second signal from the first location including contextual data from at least one data source located at the first location associated with a current state of the process-based operation associated with the particular operation;

identify a user at the first location associated with the execution of the application and the process-based operation;

determine a plurality of users associated with the second system authorized to perform oversight to the particular operation being performed and associated with the identified user; and

based on a set of prioritization rules, identify a particular one of the determined plurality of users as a supervising user to which the oversight is assigned;

identify, at the second location and for execution at the first location, a set of instructions associated with the current state of the process-based operation at the first location as provided by the supervising user; and

send, via the communications module, a third signal including the set of instructions as provided by the supervising user to the first location.

2. The system of claim 1, wherein the set of instructions includes interruption instructions for the process-based operation at the first location.

3. The system of claim 1, wherein the set of instructions include feedback received from the second location.

4. The system of claim 3, wherein the feedback comprises a proposed change or modification to an action performed at the first location.

5. The system of claim 4, wherein the set of instructions cause the feedback to be presented at a display associated with the first location.

6. The system of claim 5, wherein the feedback includes at least one textual or verbal instruction.

7. The system of claim 1, wherein the set of instructions include instructions causing the current state of the process-based operation to be changed to an updated state at the first location based on the set of instructions generated at the second location.

8. The system of claim 1, wherein the set of instructions include instructions causing control of the process-based operation to be transferred to the second location.

9. The system of claim 1, wherein the first location is physically remote from the second location.

10. The system of claim 1, wherein the contextual data from at least one data source at the first location associated with the current state of the process-based operation includes at least one of

a video or image feed at the first location;

a video or image feed associated with the first location;

a video or image feed associated with the process-based operation performed at the first location;

a visualization of a display associated with the first location;

a state indication associated with at least one connected device present at the first location;

or

a state indication associated with at least one connected device involved in or associated with the process-based operation.

11. The system of claim 1, the at least one hardware processor configured to, in response to receiving the second signal via the communications module from the first location including contextual data from at least one data source at the first location associated with the current state of the process-based operation, generate a simulation of the current state of the process-based operation for presentation at the second location, wherein the generated simulation of the current state of the process-based operation is generated for display at the second location via at least one of:

an augmented reality display, wherein the augmented reality display comprises an overlay of at least a portion of the contextual data on real-world content at the second location; and

a virtual reality display, wherein the virtual reality display is presented within a virtual reality output device at the second location.

12. The system of claim 11, wherein the set of instructions associated with the current state at the first location are identified based on interactions with the simulation presented at the second location, wherein the interactions with the simulation are performed at the second location.

13. A computerized method performed by at least one processor, the method comprising:

monitoring, from a second location remote from a first location, a process-based operation performed at the first location;

receive, at the second location and via a communications module, a first signal from the first location indicating availability for oversight of the process-based operation, wherein the first signal is associated with an automatic request for oversight at the second location generated by an application being executed at the first location, wherein the application is associated with the execution of the process-based operation, and wherein the application identifies a particular operation of the process-based operation determined to require oversight prior to the particular operation being performed;

receive, at the second location and via the communications module, a second signal from the first location including contextual data from at least one data source located at the first location associated with a current state of the process-based operation associated with the particular operation;

identify a user at the first location associated with the execution of the application and the process-based operation;

determine a plurality of users associated with the second location authorized to perform oversight to the particular operation being performed and associated with the identified user; and

based on a set of prioritization rules, identify a particular one of the determined plurality of users as a supervising user to which the oversight is assigned;

identify, at the second location and for execution at the first location, a set of instructions associated with the current state of the process-based operation at the first location as provided by the supervising user; and

send, from the second location, a third signal including the set of instructions as provided by the supervising user to the first location.

14. The method of claim 13, wherein the set of instructions includes interruption instructions for the process-based operation at the first location, wherein the interruption instructions cause control of the process-based operation performed at the first location to be transferred to the second location.

15. The method of claim 13, wherein the set of instructions include feedback received from the second location, wherein the feedback comprises a proposed change or modification to an action performed at the first location and wherein the set of instructions cause the feedback to be presented at a display associated with the first location.

16. The method of claim 15, wherein the feedback includes at least one textual or verbal instruction.

17. The method of claim 13, wherein the set of instructions include instructions causing the current state of the process-based operation to be changed to an updated state at the first location based on the set of instructions generated at the second location.

18. The method of claim 13, wherein the first location is physically remote from the second location.

19. The method of claim 13, wherein the contextual data from at least one data source at the first location associated with the current state of the process-based operation includes at least one of:

a video or image feed at the first location;

a video or image feed associated with the first location;

a video or image feed associated with the process-based operation performed at the first location;

a visualization of a display associated with the first location;

a state indication associated with at least one connected device present at the first location;

or

a state indication associated with at least one connected device involved in or associated with the process-based operation;

the method further comprising, in response to receiving the signal via the communications module from the first location including contextual data from at least one data source at the first location associated with the current state of the process-based operation, generate a simulation of the current state of the process-based operation for presentation at the second location, wherein the generated simulation of the current state of the process-based operation is generated for display at the second location via at least one of;

an augmented reality display, wherein the augmented reality display comprises an overlay of at least a portion of the contextual data on real-world content at the second location; and

a virtual reality display, wherein the virtual reality display is presented within a virtual reality output device at the second location; and

wherein the set of instructions associated with the current state at the first location are identified based on interactions with the simulation presented at the second location, wherein the interactions with the simulation are performed at the second location.