Control system using aesthetically guided gesture recognition

• US 10,338,686 B2
• Filed: 03/31/2016
• Issued: 07/02/2019
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Robertson, James R., Scanlon, Raymond J., Honeck, Michael R., Jay, Angela, Rouse, Cory J., Irmler, Holger
• Primary Examiner(s): Lowenberger, Aaron
• Application Number:
  US15/086,561
• Publication Number:
  US 20170285757A1
• Time in Patent Office: 1,188 Days
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method for facilitating and enhancing computer-based authoring of gesture definitions that are useful in controlling a walk-around character and other systems using gesture controls. The method includes, during performance of a gesture by a performer, collecting sets of raw sensor data each corresponding to differing parameters of the performance of the gesture. The method includes displaying a graphical user interface with a graphical plot of each of the sets of raw sensor data. The method includes receiving user input identifying which of the parameters to include in a gesture definition. The method includes, for the graphical plots associated with the chosen parameters receiving user input defining a starting position, an ending position, a maximum value, and a minimum value. The gesture may involve movement of a performer'"'"'s arms, legs, hands, head, eyes, and so on in a particular manner.

Claims

1. A method for authoring gesture definitions for use in gesture-based control systems, comprising:

during performance of a gesture by a performer with a plurality of sensors, generating a plurality of sets of raw sensor data each corresponding to differing parameters of the performance of the gesture;

displaying a graphical user interface (GUI) on a display device, wherein the GUI includes graphical plots of the plurality of sets of raw sensor data;

receiving director input identifying a subset of the parameters and storing the identified subset of parameters in a gesture definition, thereby first authoring the gesture definition;

for the graphical plots of the plurality of sets of sensor data corresponding to each of the parameters in the subset of the parameters in the gesture definition, receiving director input defining at least two of a starting position, an ending position, a maximum value, and a minimum value and storing the defined at least two of the starting position, the ending position, the maximum value, and the minimum value in the memory in the gesture definition, thereby second authoring the gesture definition;

receiving additional director input modifying the definition of at least one of the starting position, the ending position, the maximum value, and the minimum value for at least one of the parameters in the subset of the parameters of the gesture definition; and

receiving director input defining for at least one of the graphical plots associated with the subset of the parameters used in the gesture definition at least one of a must pass window and a must not pass window defining an area a plot of sensor data must pass or must not pass, respectively, for gesture recognition.

2. The method of claim 1, wherein the director input defining the at least two of the starting position, the ending position, the maximum value, and the minimum value includes positioning a set of bounding boxes in the graphical plots of the plurality of sets of the raw sensor data for the subset of the parameters and sizing the bounding boxes.

3. The method of claim 1, further comprising performing training of an additional performer comprising:

receiving additional sets of raw sensor data each corresponding to differing parameters of performance of the gesture by the additional performer; and

comparing the additional sets of raw sensor data with the graphical plots of the plurality of sets of sensor data corresponding to the subset of the parameters in the gesture definition to verify at least one of the starting position, the ending position, the maximum value, and the minimum value are matched.

4. The method of claim 3, wherein the performing training of an additional performer further comprises providing visual or audio feedback to the additional performer indicating whether the performance of the gesture by the additional performer satisfied the gesture definition.

5. The method of claim 4, wherein the providing of the visual or audio feedback comprises a plotting of the additional sets of raw sensor data relative to the graphical plots of the plurality of sets of raw sensor data used to create the gesture definition.

6. The method of claim 1, further comprising defining a length of time for performing the gesture and wherein the length of time is stored with the gesture definition.

7. The method of claim 1, wherein the plurality of sets of raw sensor data include data in a number of channels ranging from 2 to 13 channels, whereby the differing parameters of the performance of the gesture number from 2 to 13.

8. A system for authoring gestures for use by a gesture-based controller, comprising:

a computer with a processor managing a display and a memory device;

a gesture monitoring assembly including a set of sensors sampling data during performance of a gesture by a performer; and

a gesture authoring tool run by the processor to;

during the performance of the gesture by the performer, collect sets of sensor data from the sensors;

display a graphical user interface (GUI) on a display device, wherein the GUI includes graphical plots of the sets of sensor data;

generate a gesture definition by receiving user input identifying a subset of the sensors to include in the gesture definition and storing the identified subset of the sensors in the memory device;

for the graphical plots of each of the sensors in the subset of the sensors included in the gesture definition, receiving user input defining at least two of a starting position, an ending position, a maximum value, and a minimum value of the sensor data and storing, in the memory device, the defined at least two of the starting position, the ending position, the maximum value, and the sensor data as part of the gesture definition;

wherein the gesture authoring tool further acts to receive user input defining for one of the graphical plots associated with the subset of the sensors used in the gesture definition at least one of a must pass window and a must not pass window defining, respectively, an area a plot of sensor data must pass for gesture recognition and an area a plot of sensor data must not pass for gesture recognition; and

wherein the gesture authoring tool receives additional user input modifying the stored definition of at least one of the starting position, the ending position, the maximum value, and the minimum value for the sensor data for at least one of the sensors in the subset of the sensors included in the gesture definition.

9. The system of claim 8, wherein the user input defining the at least two of the starting position, the ending position, the maximum value, and the minimum value includes positioning a set of bounding boxes in the graphical plots of the sets of the raw sensor data for the subset of the sensors and sizing the bounding boxes.

10. The system of claim 8, wherein the gesture authoring tool performs regression analysis including comparing the gesture definition with at least one previously generated gesture definition to determine whether the gesture monitoring assembly can accurately distinguish performances of a corresponding pair of gestures and, when not, to flag the gesture definition and the at least one previously generated gesture definition for modification.

11. A gesture authoring method, comprising:

collecting a set of data from a plurality of sensors during a data collection window;

receiving first user input defining a gesture performance time period with a start time and an end time for performance of a gesture by a performer during the collecting;

creating a first gesture definition for the gesture including the set of data from a subset of the plurality of sensors during the gesture performance time period;

receiving second user input marking in the first gesture definition, for each of the sensors in the subset of the plurality of sensors, at least two of a start position, an end position, a maximum value, and a minimum value for the set of data;

receiving third user input marking in the first gesture definition, for at least one of the sensors in the subset of the plurality of sensors, at least one of a must pass window and a must not pass window defining an area sensor data must pass or must not pass, respectively, for gesture recognition;

creating a second gesture definition for the gesture comprising a modified version of the first gesture definition including the markings from the second user input and the third user input; and

operating a gesture-based control system to generate a control signal when the gesture-based control system determines a performer has performed the gesture defined by the second gesture definition.

12. The method of claim 11, further comprising, prior to the receiving of the second user input, generating a graphical user interface including a graphical plot of the set of data from the sensor and displaying the graphical user interface on a monitor of a computer.

13. The method of claim 12, wherein the receiving of second user input providing the marking includes displaying bounding boxes for the at least two of the start position, the end position, the maximum value and the minimum value and receiving user input placing the bounding boxes relative to the graphical plot and sizing the bounding boxes.

14. The method of claim 12, further comprising receiving fourth user input modifying the marking from the second user input and, in response, creating a third gesture definition comprising a modified version of the second gesture definition.

15. The method of claim 11, further comprising using the second gesture definition for training a performer including collecting additional data from the sensors and comparing the additional data to the graphical plot to verify at least the marked at least two of the start position, the end position, the maximum value and the minimum value are matched.