Virtual behavior training using augmented reality user devices

• US 10,217,375 B2
• Filed: 12/13/2016
• Issued: 02/26/2019
• Est. Priority Date: 12/13/2016
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Waldron, Willard H., Dravneek, Victoria L., Johansen, Joseph N., Goebel, Darryl C.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/377,795
• Publication Number:
  US 20180165984A1
• Time in Patent Office: 805 Days
• Field of Search: 345633
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/606 by securing the transmission between two devices or processes
• CPC Class Codes: G06F 2221/2111 Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09G 2358/00 Arrangements for display data security
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/061 for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/0861 using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

An augmented reality system that includes an augmented reality user device. The augmented reality user device includes display for overlaying virtual objects onto tangible objects in a real scene and a memory storing scenarios for a user. The augmented reality user device includes a processor implementing a virtual assessment engine and a virtual overlay engine. The virtual assessment engine identifies a user identifier for the user and executes a scenario for the user. The virtual assessment engine identifies actions performed by the user related to the scenario. The virtual assessment engine sends a token with the user identifier and user action information to a remote server and receives information for the user from the server. The virtual overlay engine presents the information as virtual objects overlaid with the real scene.

Claims

1. An augmented reality system comprising:

an augmented reality user device for a user comprising;

a display configured to overlay virtual objects onto tangible objects in a real scene in real-time;

a memory operable to store a plurality of scenarios linked with a virtual account for the user, wherein each scenario is linked with one or more challenges that affect the virtual account;

one or more processors operably coupled to the display and the memory, and configured to implement;

a virtual assessment engine configured to;

identify a user identifier for the user in response to authenticating the user;

receive an input from the user identifying a scenario from among the plurality of scenarios;

execute the scenario for the user, wherein executing the scenario identifies one or more challenges linked with the scenario;

identify one or more user actions;

determine whether the user actions are related to a challenge linked with the scenario;

generate user action information identifying the user actions and the scenario when the user actions are related to the scenario;

generate a token comprising;

 

the user identifier, and

 

the user action information;

send the token to a remote server;

receive virtual assessment data in response to sending the token to the remote server, wherein the virtual assessment data comprises;

 

virtual account information for the user comprising a virtual account balance linked with the plurality of scenarios, and

 

account information for the user; and

a virtual overlay engine configured to present the virtual account information and the account information as virtual objects overlaid with the real scene; and

the remote server comprising an augmented reality engine configured to;

identify the account information for the user based on the user identifier;

identify virtual account information for the user based on the user identifier;

modify the virtual account information based on the user action information;

generate the virtual assessment data comprising the modified virtual account information and the account information for the user; and

send the virtual assessment data to the augmented reality user device.

2. The system of claim 1, wherein the augmented reality engine is configured to:

determine whether the user qualifies for a reward based on the modified virtual account information; and

modify the account information according to the reward when the user qualifies for the reward, and

wherein the virtual assessment data comprises the modified account information.

3. The system of claim 1, wherein the augmented reality engine is configured to:

determine whether to modify the account information for the user based on the virtual account balance of the modified virtual account information; and

modify the account information for the user when the balance of the modified virtual account information meets a predetermined threshold value, and

wherein the virtual assessment data comprises the modified account information.

4. The system of claim 1, wherein the augmented reality engine is configured to:

compare the virtual account information for the user to virtual account information for one or more other people;

compare the account information for the user to account information for one or more other people;

determine ranking information for the user based on the comparison of the virtual account information and the comparison of the account information; and

wherein the virtual assessment data comprises the ranking information.

5. The system of claim 1, wherein the scenario comprises a task identifying a series of user actions;

and

determining whether the user actions are related to the scenario comprises;

comparing the identified one or more user actions with the series of actions identified by the task; and

determining the user actions are related to the scenario when the identified one or more user actions match the series of actions identified by the task.

6. The system of claim 1, wherein:

the augmented reality user device comprises a global position sensor configured to provide the geographic location of the user; and

the user action information comprises information identifying the geographic location of the user when performing the identified one or more user actions.

7. The system of claim 1, wherein identifying the one or more user actions comprises:

performing object recognition to identify an object in the real scene; and

performing gesture recognition to identify a user action with the identified object.

8. The system of claim 1, wherein the virtual overlay engine configured to present information related to the scenario as virtual objects overlaid with the real scene.

9. An augmented reality overlaying method comprising:

identifying, by a virtual assessment engine, a user identifier for a user in response to authenticating the user;

receiving, by the virtual assessment engine, an input from the user identifying a scenario from among a plurality of scenarios, wherein;

the plurality of scenarios is linked with a virtual account for the user; and

each scenario is linked with one or more challenges that affect the virtual account;

executing, by the virtual assessment engine, the scenario for the user, wherein executing the scenario identifies one or more challenges linked with the scenario;

identifying, by the virtual assessment engine, one or more user actions;

determining, by the virtual assessment engine, whether the user actions are related to a challenge linked with the scenario;

generating, by the virtual assessment engine, user action information identifying the user actions and the scenario when the user actions are related to the scenario;

generating, by the virtual assessment engine, a token comprising the user identifier and the user action information;

sending, by the virtual assessment engine, the token to a remote server;

identifying, by an augmented reality engine of the remote server, account information for the user based on the user identifier;

identifying, by the augmented reality engine, virtual account information for the user based on the user identifier, wherein the virtual account information comprises a virtual account balance linked with the plurality of scenarios;

modifying, by the augmented reality engine, the virtual account information based on the user action information;

generating, by the augmented reality engine, virtual assessment data comprising the modified virtual account information and the account information for the user; and

sending, by the augmented reality engine, the virtual assessment data to the augmented reality user device; and

presenting, by a virtual overlay engine, the virtual account information and the account information as virtual objects overlaid with the real scene.

10. The method of claim 9, further comprising:

determining, by the augmented reality engine, whether the user qualifies for a reward based on the modified virtual account information; and

modifying, by the augmented reality engine, the account information according to the reward when the user qualifies for the reward, and

wherein the virtual assessment data comprises the modified account information.

11. The method of claim 9, further comprising:

determining, by the augmented reality engine, whether to modify the account information for the user based on the virtual account balance of the modified virtual account information; and

modifying, by the augmented reality engine, the account information for the user when the balance of the modified virtual account information meets a predetermined threshold value, and

wherein the virtual assessment data comprises the modified account information.

12. The method of claim 9, further comprising:

comparing, by the augmented reality engine, the virtual account information for the user to virtual account information for one or more other people;

comparing, by the augmented reality engine, the account information for the user to account information for one or more other people; and

determining, by the augmented reality engine, ranking information for the user based on the comparison of the virtual account information and the comparison of the account information; and

wherein the virtual assessment data comprises the ranking information.

13. The method of claim 9, wherein the scenario comprises a task identifying a series of user actions;

and

determining whether the user actions are related to the scenario comprises;

comparing the identified one or more user actions with the series of actions identified by the task; and

determining the user actions are related to the scenario when the identified one or more user actions match the series of actions identified by the task.

14. The method of claim 9, wherein the user action information comprises information identifying a geographic location of the user when performing the identified one or more user actions.

15. The method of claim 9, wherein identifying the one or more user actions comprises:

performing object recognition to identify an object in the real scene; and

performing gesture recognition to identify a user action with the identified object.

16. The method of claim 9, further comprising presenting, by the virtual overlay engine, information related to the scenario as virtual objects overlaid with the real scene.

17. An augmented reality user device comprising:

a display configured to overlay virtual objects onto tangible objects in a real scene in real-time;

a memory operable to store a plurality of scenarios linked with a virtual account for a user, wherein each scenario is linked with one or more challenges that affect the virtual account;

one or more processors operably coupled to the display and the memory, and configured to implement;

a virtual assessment engine configured to;

identify a user identifier for the user in response to authenticating the user;

receive an input from the user identifying a scenario from among the plurality of scenarios;

execute the scenario for the user, wherein executing the scenario identifies one or more challenges linked with the scenario;

identify one or more user actions;

determine whether the user actions are related to a challenge linked with the scenario;

generate user action information identifying the user actions and the scenario when the user actions are related to the scenario;

generate a token comprising;

the user identifier, and

the user action information;

send the token to a remote server;

receive virtual assessment data in response to sending the token to the remote server, wherein the virtual assessment data comprises;

virtual account information for the user comprising a virtual account balance linked with the plurality of scenarios, and

account information for the user; and

a virtual overlay engine configured to present the virtual account information and the account information as virtual objects overlaid with the real scene.

18. The device of claim 17, wherein the scenario comprises a task identifying a series of user actions;

and

determining whether the user actions are related to the scenario comprises;

comparing the identified one or more user actions with the series of actions identified by the task; and

determining the user actions are related to the scenario when the identified one or more user actions match the series of actions identified by the task.

19. The device of claim 17, wherein:

the augmented reality user device comprises a global position sensor configured to provide the geographic location of the user; and

the user action information comprises information identifying the geographic location of the user when performing the identified one or more user actions.

20. The device of claim 17, wherein identifying the one or more user actions comprises:

performing object recognition to identify an object in the real scene; and

performing gesture recognition to identify a user action with the identified object.