Changing an application state using neurological data

• US 9,864,431 B2
• Filed: 05/11/2016
• Issued: 01/09/2018
• Est. Priority Date: 05/11/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Keskin, Cem, Kim, David, Chau, Bill, Kim, Jaeyoun, Koishida, Kazuhito, Shahid, Khuram
• Primary Examiner(s): DAVIS, TONY O
• Application Number:
  US15/152,401
• Publication Number:
  US 20170329404A1
• Time in Patent Office: 608 Days
• Field of Search: 600300, 600566
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computer systems, methods, and storage media for changing the state of an application by detecting neurological user intent data associated with a particular operation of a particular application state, and changing the application state so as to enable execution of the particular operation as intended by the user. The application state is automatically changed to align with the intended operation, as determined by received neurological user intent data, so that the intended operation is performed. Some embodiments relate to a computer system creating or updating a state machine, through a training process, to change the state of an application according to detected neurological data.

Claims

1. A computer-implemented method for changing a state of an application using neurological data, the method being implemented by a computing system that includes at least one processor and one or more hardware storage devices having stored computer-executable instructions that are executable by the at least one processor to cause the computing system to perform at least the following:

operating an application that includes at least a first application state, a second application state, and an application state dependent operation, wherein invoking the application state dependent operation in response to detecting a particular user input causes a first action within the application while the application is operating in the first application state but causes a second action within the application while the application is operating in the second application state;

while the application is operating in the first application state, detecting the particular user input invoking the application state dependent operation, the particular user input including neurological user intention data indicating that the user intends the particular input to cause the second action in accordance with the second application state;

based on detecting the particular user input, automatically changing the application from the first application state to the second application state; and

causing the second action to be executed at the application.

2. The method of claim 1, further comprising performing the second operation in response to detecting the user input comprising the neurological user intention data.

3. The method of claim 1, wherein the application operates in a virtual reality or an augmented reality environment.

4. The method of claim 1, wherein the neurological user intention data of the detected user input includes electroencephalography readings that are generated from user performance of the particular gesture with an intention of causing the second operation through performance of the particular gesture.

5. The method of claim 1, further comprising:

detecting a subsequent user input while the application is operating in the second application state, after being changed from the first application state to the second application state, the detected subsequent user input comprising neurological user intention data that is associated with the particular gesture according to the first operation; and

changing the application from the second application state to the first application state in response to detecting the subsequent user input.

6. The method of claim 5, further comprising performing the first operation in response to detecting the subsequent user input.

7. The method of claim 5, wherein the neurological user intention data of the detected subsequent user input includes electroencephalography readings that are generated from user performance of the particular gesture with an intention of causing the first operation through performance of the particular gesture.

8. The method of claim 1, wherein the particular gesture is performed through one or more user movements without the accompaniment of additional input hardware.

9. The method of claim 1, wherein the neurological user intention data corresponding to the particular gesture is generated without physical performance of the particular gesture.

10. The method of claim 1, wherein the particular gesture is performed using one or more input hardware components.

11. The method of claim 1, wherein the application is a modeling application.

12. A computer-implemented method for creating and managing a state machine for at least one application, the state machine being configured to change a state of the at least one application based on neurological data received from a user, the method being implemented by a computing system that includes at least one processor and one or more hardware storage devices having stored thereon computer-executable instructions that are executable by the at least one processor to cause the computing system to perform at least the following:

associating a set of application state dependent operations to a corresponding set of application states within an application configured to include multiple application states, a particular user input causing the application to perform a particular application state dependent operation from the set of application state dependent operations based on a corresponding application state, wherein the application includes at least a first application state associated with a first application state dependent operation and a second application state associated with a second application state dependent operation;

as a result of detecting a first neurological user intention data generated by a user during performance of the particular user input, causing the application to perform a first application state dependent operation when the application is in the first application state;

as a result of detecting a second neurological user intention data generated by a user during performance of the particular user input, causing the application to perform a second application state dependent operation when the application is in the second application state; and

as a result of detecting the second neurological user intention data generated by a user during performance of the particular user input while the application is operating in the first application state, automatically changing the application from the first application state to the second application state and causing the second application state operation to be executed at the application.

13. The method of claim 12, wherein the first and second neurological user intention data are generated by a single user, and wherein the state machine is thereby tailored to the single user.

14. The method of claim 12, wherein the first neurological user intention data with the first operation in the first state and the second neurological user intention data with the second operation in the second state using machine learning techniques.

15. The method of claim 12, wherein the first and second neurological user intention data include electroencephalography readings that are generated from user performance of the particular gesture with an intention of causing the first and second operation, respectively.

16. A computer-implemented method for creating and managing a state machine for at least one application, the state machine being configured to change a state of the at least one application based on neurological data received from a user, the method being implemented by a computing system that includes at least one processor and one or more hardware storage devices having stored thereon computer-executable instructions that are executable by the at least one processor to cause the computing system to perform at least the following:

operating an application that is configured to include at least a first application state, a second application state, and an application state dependent operation, wherein invoking the application state dependent operation in response to detecting a particular user input causes a first action within the application while the application is operating in the first application state but causes a second action within the application while the application is operating in the second application state;

while the application is operating in the first application state, detecting the particular user input invoking the application state dependent operation, the particular user input including neurological user intention data indicating that the user intends the particular input to cause the second action in accordance with the second application state;

based on detecting the particular user input, automatically changing the application from the first application state to the second application state; and

causing the second action to be executed at the application.