Providing technical support in an augmented reality environment

• US 10,360,735 B1
• Filed: 05/30/2018
• Issued: 07/23/2019
• Est. Priority Date: 05/30/2018
• Status: Active Grant

• Current Assignee: Alvaria Cayman
• Original Assignee: Noble Systems Corporation
• Inventors: Reeder, Lewis J., Haggerty, Christopher S.
• Primary Examiner(s): Chu, David H
• Application Number:
  US15/993,033
• Time in Patent Office: 419 Days
• Field of Search: None
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Various embodiments of the invention provide methods, systems, and computer program products for providing support to a party who is using an augmented reality device and seeking to resolve an issue. One or more pieces of information are obtained through the augmented reality device and a cause of the issue is identified by utilizing one or more predictive models and the pieces of information as input to the models. Once the cause has been identified, a script is queried that includes instructions to be performed by the party to address the cause and media to execute on the augmented reality device to aid the party in performing the instructions. Accordingly, the instructions and media are communicated to the party through the augmented reality device so that the party may perform the instructions with the aid of the media to address the cause of the issue.

Claims

1. A method for providing support to a party who is using an augmented reality device and seeking to resolve an issue, the method comprising:

receiving by at least one computer processor the issue the party who is using the augmented reality device is seeking to resolve, the augmented reality device being remote from the at least one computer processor;

obtaining one or more pieces of information from the augmented reality device needed to identify a cause of the issue by the at least one computer processor, the one or more pieces of information comprising at least one of (1) an image of a real world, physical object photographed by a camera coupled to the augmented reality device, (2) a video of the real world, physical object recorded by the camera coupled to the augmented reality device, (3) a physical sound recorded through a microphone on the augmented reality device, and (4) input provided by the party on an input element on the augmented reality device;

identifying the cause of the issue by utilizing one or more predictive models and at least one of the one or more pieces of information as input to the one or more predictive models, wherein the one or more predictive models are arranged in a hierarchical program structure of possible causes to provide an order in which the one or more predictive models are utilized for identifying the cause of the issue and identifying the cause of the issue involves systematically working through the possible causes by making use of the one or more predictive models along with the at least one of the one or more pieces of information as input to the one or more predictive models to determine a likelihood of each of one or more of the possible causes being the cause of the issue;

querying a script for the identified cause of the issue, the script comprising one or more instructions to be performed by the party and one or more pieces of media to execute on the augmented reality device to aid the party in performing the one or more instructions; and

communicating the one or more instructions and the one or more pieces of media to the party through the augmented reality device.

2. The method of claim 1 comprising communicating location information to the augmented reality device, the location information identifying at least one of where to place one of the one or more instructions and one of the one or more pieces of media on a display of the augmented reality device so that the at least one of the one of the one or more instructions and the one of the one or more pieces of media are superimposed over a field of view of the party in one or more positions with respect to at least one physical object viewable in the field of view of the party.

3. The method of claim 1, wherein the one or more predictive models comprise one or more of a decision tree, a support vector machine, Bayesian network, clustering, reinforcement learning, and neural network.

4. The method of claim 1 further comprising requesting access to the augmented reality device by the at least one computer processor so that the at least one computer processor is able to control certain aspects of the augmented reality device to obtain the one or more pieces of information.

5. The method of claim 1, wherein the one or more instructions are communicated to the party one at a time and the method further comprises:

determining by the at least one computer processor whether an instruction from the one or more instructions communicated to the party was performed correctly by the party;

in response to determining the instruction was not performed correctly by the party, communicating the instruction to the party again so that the party can re-perform the instruction correctly; and

in response to determining the instruction was performed correctly by the party, communicating a next instruction from the one or more instructions to the party to perform.

6. The method of claim 1, wherein the one or more pieces of media comprise at least one of (1) an image to be displayed on a display of the augmented reality device, (2) a video to be displayed on the display of the augmented reality device, and (3) a sound to be played over a speaker coupled with the augmented reality device.

7. A non-transitory, computer-readable medium comprising computer-executable instructions for providing support to a party who is using an augmented reality device and seeking to resolve an issue, that when executed, cause at least one computer processor to:

receive the issue the party who is using the augmented reality device is seeking to resolve, the augmented reality device being remote from the at least one computer processor;

obtain one or more pieces of information from the augmented reality device needed to identify a cause of the issue, the one or more pieces of information comprising at least one of (1) an image of a real world, physical object photographed by a camera coupled to the augmented reality device, (2) a video of the real world, physical object recorded by the camera coupled to the augmented reality device, (3) a physical sound recorded through a microphone on the augmented reality device, and (4) input provided by the party on an input element on the augmented reality device;

identify the cause of the issue by utilizing one or more predictive models and at least one of the one or more pieces of information as input to the one or more predictive models, wherein the one or more predictive models are arranged in a hierarchical program structure of possible causes to provide an order in which the one or more predictive models are utilized for identifying the cause of the issue and the computer-executable instructions cause the at least one computer process to identify the cause of the issue by systematically working through the possible causes by making use of the one or more predictive models along with the at least one of the one or more pieces of information as input to the one or more predictive models to determine a likelihood of each of one or more of the possible causes being the cause of the issue;

query a script for the identified cause of the issue, the script comprising one or more instructions to be performed by the party and one or more pieces of media to execute on the augmented reality device to aid the party in performing the one or more instructions; and

communicate the one or more instructions and the one or more pieces of media to the party through the augmented reality device.

8. The non-transitory, computer-readable medium of claim 7, wherein the computer-executable instructions cause the at least one computer processor to communicate location information to the augmented reality device, the location information identifying at least one of where to place one of the one or more instructions and one of the one or more pieces of media on a display of the augmented reality device so that the at least one of the one of the one or more instructions and the one of the one or more pieces of media are superimposed over a field of view of the party in one or more positions with respect to at least one physical object viewable in the field of view of the party.

9. The non-transitory, computer-readable medium of claim 7, wherein the one or more predictive models comprise one or more of a decision tree, a support vector machine, Bayesian network, clustering, reinforcement learning, and neural network.

10. The non-transitory, computer-readable medium of claim 7, wherein the computer-executable instructions cause the at least one computer processor to request access to the augmented reality device so that the at least one computer processor is able to control certain aspects of the augmented reality device to obtain the one or more pieces of information.

11. The non-transitory, computer-readable medium of claim 7, wherein the one or more instructions are communicated to the party one at a time and the computer-executable instructions cause the at least one computer processor to:

determine whether an instruction from the one or more instructions communicated to the party was performed correctly by the party;

in response to determining the instruction was not performed correctly by the party, communicate the instruction to the party again so that the party can re-perform the instruction correctly; and

in response to determining the instruction was performed correctly by the party, communicate a next instruction from the one or more instructions to the party to perform.

12. The non-transitory, computer-readable medium of claim 7, wherein the one or more pieces of media comprise at least one of (1) an image to be displayed on a display of the augmented reality device, (2) a video to be displayed on the display of the augmented reality device, and (3) a sound to be played over a speaker coupled with the augmented reality device.

13. A system for providing support to a party who is using an augmented reality device and seeking to resolve an issue, the system comprising:

at least one computer processor configured to;

receive the issue the party who is using the augmented reality device is seeking to resolve, the augmented reality device being remote from the at least one computer processor;

obtain one or more pieces of information from the augmented reality device needed to identify a cause of the issue, the one or more pieces of information comprising at least one of (1) an image of a real world, physical object photographed by a camera coupled to the augmented reality device, (2) a video of the real world, physical object recorded by the camera coupled to the augmented reality device, (3) a physical sound recorded through a microphone on the augmented reality device, and (4) input provided by the party on an input element on the augmented reality device;

identify the cause of the issue by utilizing one or more predictive models and at least one of the one or more pieces of information as input to the one or more predictive models, wherein the one or more predictive models are arranged in a hierarchical program structure of possible causes to provide an order in which the one or more predictive models are utilized for identifying the cause of the issue and the at least one computer process is configured to identify the cause of the issue by systematically working through the possible causes by making use of the one or more predictive models along with the at least one of the one or more pieces of information as input to the one or more predictive models to determine a likelihood of each of one or more of the possible causes being the cause of the issue;

query a script for the identified cause of the issue, the script comprising one or more instructions to be performed by the party and one or more pieces of media to execute on the augmented reality device to aid the party in performing the one or more instructions; and

communicate the one or more instructions and the one or more pieces of media to the party through the augmented reality device.

14. The system of claim 13, wherein the at least one computer processor is configured to communicate location information to the augmented reality device, the location information identifying at least one of where to place one of the one or more instructions and one of the one or more pieces of media on a display of the augmented reality device so that the at least one of the one of the one or more instructions and the one of the one or more pieces of media are superimposed over a field of view of the party in one or more positions with respect to at least one physical object viewable in the field of view of the party.

15. The system of claim 13, wherein the one or more predictive models comprise one or more of a decision tree, a support vector machine, Bayesian network, clustering, reinforcement learning, and neural network.

16. The system of claim 13, wherein the at least one computer processor is configured to request access to the augmented reality device so that the at least one computer processor is able to control certain aspects of the augmented reality device to obtain the one or more pieces of information.

17. The system of claim 13, wherein the one or more instructions are communicated to the party one at a time and the at least one computer processor is configured to:

determine whether an instruction from the one or more instructions communicated to the party was performed correctly by the party;

in response to determining the instruction was not performed correctly by the party, communicate the instruction to the party again so that the party can re-perform the instruction correctly; and

in response to determining the instruction was performed correctly by the party, communicate a next instruction from the one or more instructions to the party to perform.

18. The system of claim 13, wherein the one or more pieces of media comprise at least one of (1) an image to be displayed on a display of the augmented reality device, (2) a video to be displayed on the display of the augmented reality device, and (3) a sound to be played over a speaker coupled with the augmented reality device.