ACTIVE TRIGGER POSES

• US 20170361213A1
• Filed: 06/16/2016
• Published: 12/21/2017
• Est. Priority Date: 06/16/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: GOSLIN, Michael P., OLSON, Blade A.
• Granted Patent: US 10,092,827 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Systems, methods and articles of manufacture for controlling electronic devices in an interactive gaming environment. Embodiments detect a first interactive game is playing within the physical environment using one or more electronic devices. User movement is monitored using at least one sensor device within the physical environment and user speech is monitored using one or more microphone sensor devices within the physical environment. Upon determining that the user movement matches a predefined series of user actions and that the user speech matches a corresponding predefined speech pattern, embodiments determine a gameplay action corresponding to both the predefined series of user actions and the predefined speech pattern and transmit an instruction, to at least one of the one or more electronic devices within the physical environment, instructing the electronic device to perform the determined gameplay action.

Claims

1. A method, comprising:

detecting a first interactive game is playing within the physical environment using one or more electronic devices;

monitoring user movement using at least one sensor device within the physical environment;

monitoring user speech using one or more microphone sensor devices within the physical environment;

upon determining that the user movement matches a predefined series of user actions and that the user speech matches a corresponding predefined speech pattern;

determining a gameplay action corresponding to both the predefined series of user actions and the predefined speech pattern; and

transmitting an instruction, to at least one of the one or more electronic devices within the physical environment, instructing the electronic device to perform the determined gameplay action.

2. The method of claim 1, wherein the at least one sensor device within the physical environment comprises one or more camera devices, and further comprising:

analyzing a plurality of frames captured by the one or more camera devices in order to identify a respective plurality of pixels representing a user in each of the plurality of frames; and

determining the user movement based on changes in the plurality of pixels across the plurality of frames.

3. The method of claim 1, wherein the at least one sensor device within the physical environment comprises one or more electromyography sensors, and further comprising:

analyzing an electromyogram collected by the one or more electromyography sensors to determine muscle activations made by the user.

4. The method of claim 3, wherein determining that the user movement matches the predefined series of user actions further comprises:

determining that the determined muscle activations made by the user match a predefined pattern of muscle activations.

5. The method of claim 1, wherein the at least one sensor device within the physical environment comprises an inertial movement unit, and further comprising:

analyzing data collected by the inertial movement unit to determine a measure of exerted force and an angular rate of the user movement.

6. The method of claim 5, wherein determining that the user movement matches the predefined series of user actions further comprises:

determining that the measure of exerted force and angular rate match a predefined pattern corresponding to the predefined series of user actions.

7. The method of claim 1, wherein the user movement comprises at least one of a twisting motion and a spinning motion, and wherein the user speech comprises at least one of a spoken phrase and a song.

8. The method of claim 1, wherein determining the gameplay action corresponding to both the predefined series of user actions and the predefined speech pattern further comprises:

determining an in-game ability having a predefined association with the predefined series of user actions and the predefined speech pattern.

9. The method of claim 8, wherein transmitting an instruction, to at least one of the one or more electronic devices within the physical environment, instructing the electronic device to perform the determined gameplay action further comprises:

transmitting an instruction to an augmented reality device within the physical environment, specifying to render a plurality of frames depicting a performance of the in-game ability.

10. The method of claim 9, further comprising:

transmitting an instruction to an interactive electronic device within the physical environment, instructing the interactive electronic device to perform one or more actions in response to the performance of the in-game ability.

11. The method of claim 10, wherein the one or more actions are determined based on a current context of a story being played in the first interactive game, wherein the first interactive game includes a plurality of distinct contexts, and wherein.

12. The method of claim 11, wherein the one or more actions comprise at least one of (i) outputting a sound effect, (ii) outputting a lighting effect using one or more illumination devices, (iii) initiating a haptic feedback mechanism.

13. The method of claim 9, further comprising:

upon receiving, by the augmented reality device, the instruction specifying to render the plurality of frames depicting the performance of the in-game ability;

determining one or more predefined virtual objects corresponding to the in-game ability;

determining a virtual position for the one or more predefined virtual objects, based on a determined position of a user action within one or more frames captured using a camera sensor of the augmented reality device;

determining a size and orientation for the one or more predefined virtual objects, relative to a size of the user action within the one or more frames; and

rendering a plurality of frames for display on a display device of the augmented reality device, in which the one or more predefined virtual objects are positioned according to the determined virtual position and in which the one or more predefined virtual objects are depicted at the determined size and orientation.

14. The method of claim 1, wherein determining the gameplay action and transmitting the instruction are not performed unless both the user movement matches the predefined series of user actions and the user speech matches the corresponding predefined speech pattern.

15. A non-transitory computer-readable medium, containing computer program code that, when executed, performs an operation comprising:

detecting a first interactive game is playing within the physical environment using one or more electronic devices;

monitoring user movement using at least one sensor device within the physical environment;

monitoring user speech using one or more microphone sensor devices within the physical environment;

upon determining that the user movement matches a predefined series of user actions and that the user speech matches a corresponding predefined speech pattern;

determining a gameplay action corresponding to both the predefined series of user actions and the predefined speech pattern; and

transmitting an instruction, to at least one of the one or more electronic devices within the physical environment, instructing the electronic device to perform the determined gameplay action.

16. The non-transitory computer-readable medium of claim 15, wherein the at least one sensor device within the physical environment comprises one or more camera devices, and further comprising:

analyzing a plurality of frames captured by the one or more camera devices in order to identify a respective plurality of pixels representing a user in each of the plurality of frames; and

determining the user movement based on changes in the plurality of pixels across the plurality of frames.

17. The non-transitory computer-readable medium of claim 15, wherein the at least one sensor device within the physical environment comprises one or more electromyography sensors, and further comprising:

analyzing an electromyogram collected by the one or more electromyography sensors to determine muscle activations made by the user, and

wherein determining that the user movement matches the predefined series of user actions further comprises;

determining that the determined muscle activations made by the user match a predefined pattern of muscle activations.

18. The non-transitory computer-readable medium of claim 15, wherein the at least one sensor device within the physical environment comprises an inertial movement unit, and further comprising:

analyzing data collected by the inertial movement unit to determine a measure of exerted force and an angular rate of the user movement, and

wherein determining that the user movement matches the predefined series of user actions further comprises;

determining that the measure of exerted force and angular rate match a predefined pattern corresponding to the predefined series of user actions.

19. The non-transitory computer-readable medium of claim 15, wherein determining the gameplay action corresponding to both the predefined series of user actions and the predefined speech pattern further comprises:

determining an in-game ability having a predefined association with the predefined series of user actions and the predefined speech pattern,

wherein transmitting an instruction, to at least one of the one or more electronic devices within the physical environment, instructing the electronic device to perform the determined gameplay action further comprises;

transmitting an instruction to an augmented reality device within the physical environment, specifying to render a plurality of frames depicting a performance of the in-game ability, and the operation further comprising;

transmitting an instruction to an interactive electronic device within the physical environment, instructing the interactive electronic device to perform one or more actions in response to the performance of the in-game ability, wherein the one or more actions are determined based on a current context of a story being played in the first interactive game, wherein the first interactive game includes a plurality of distinct contexts, and wherein the one or more actions comprise at least one of (i) outputting a sound effect, (ii) outputting a lighting effect using one or more illumination devices, (iii) initiating a haptic feedback mechanism.

20. A system, comprising:

one or more computer processors; and

a memory containing computer program code that, when executed by operation of the one or more computer processors, performs an operation comprising;

detecting a first interactive game is playing within the physical environment using one or more electronic devices;

monitoring user movement using at least one sensor device within the physical environment;

monitoring user speech using one or more microphone sensor devices within the physical environment;

upon determining that the user movement matches a predefined series of user actions and that the user speech matches a corresponding predefined speech pattern;

determining a gameplay action corresponding to both the predefined series of user actions and the predefined speech pattern; and

transmitting an instruction, to at least one of the one or more electronic devices within the physical environment, instructing the electronic device to perform the determined gameplay action.