Evolving universal gesture sets

• US 9,019,201 B2
• Filed: 01/08/2010
• Issued: 04/28/2015
• Est. Priority Date: 01/08/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Holmdahl, Todd Eric, LeGrow, Ian
• Primary Examiner(s): Leiby, Christopher E
• Application Number:
  US12/684,556
• Publication Number:
  US 20110169726A1
• Time in Patent Office: 1,936 Days
• Field of Search: 345/156, 348/155, 382/103, 382/278
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

In a gesture-based system, gestures may control aspects of a computing environment or application, where the gestures may be derived from a user'"'"'s position or movement in a physical space. Gesture recognition data, used to recognize gestures from captured data representative of a user'"'"'s input gestures, may be evolved based on captured data from a plurality of users. A common set or default set of gesture recognition data may be evolved by selecting a plurality of users for tracking. Captured data of the plurality of users may be processed to identify input gesture data for the plurality of users, and the gesture recognition data may be evolved based on features of the input gesture data that is common to multiple users. The evolved gesture recognition data may be implemented not only for the users tracked, but for users not tracked. An identifier may identify when the evolved gesture recognition data applies and implement the evolved gesture recognition data when the identifier is present.

Claims

1. A method for evolving a gesture set in a gesture-based system, the method comprising:

capturing data representative of a user in a physical space;

processing the captured data to identify a first motion or pose for the user, the first motion or pose for the user being processed to determine a first confidence level that the first motion or pose is the same as a computer representation of a motion or pose for a particular gesture that invokes an input command to modify a visual output of a computer, wherein the first motion or pose is processed to generate values for parameters associated with the particular gesture in a gesture filter, said first confidence level indicating a likelihood that the values of the parameters associated with the first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer; and

modifying the computer representation of the motion or pose invoking the input command based on the first motion or pose by the user such that subsequently captured data of the user or another user performing the first motion or pose is processed to determine a second confidence level that the first motion or pose is the same as the modified computer representation of the motion or pose for the particular gesture, said second confidence level indicating a likelihood that the values of the parameters associated with the subsequently captured first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the subsequently captured first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer, the second confidence level being greater than the first confidence level.

2. The method of claim 1, further comprising:

associating an identifier with the modified computer representation of the motion or pose, the identifier identifying a basis for why the first motion or pose from the user may fail to be recognized as the particular gesture and/or as a basis for implementing the modified computer representation of the motion or pose rather than computer representation of the motion or pose in evaluating subsequently captured data.

3. The method of claim 2, wherein the identifier comprises an indication of an operating system, an application, the user or another user, a feature of the user or another user, a location, a type of application, a hardware configuration, a software configuration, a culture, a geography, a demography, a linguistic characteristic, or a style.

4. The method of claim 1, wherein the user is selected by the gesture-based system by a passive request.

5. The method of claim 1, wherein the computer representation of the motion or pose is modified at a server computer of the gesture-based system, further comprising:

distributing the modified computer representation of the motion or pose from the server computer to client computers of the gesture-based system.

6. The method of claim 1, wherein processing the captured data to identify a first motion or pose for the user, the first motion or pose for the user being processed to determine the first confidence level that the motion or pose is the same as the computer representation of the motion or pose for a particular gesture that invokes an input command to modify the visual output of the computer comprises:

determining the first confidence level based on a first amount of variation between the first motion or pose for the user and the computer representation of the motion or pose; and

wherein modifying the computer representation of the motion or pose invoking the input command based on the first user motion or pose by the user such that subsequently captured data of the user or another user performing the first motion or pose is processed to determine the second confidence level that the motion or pose is the same as the modified computer representation of the motion or pose for the particular gesture comprises;

determining the second confidence level based on a second amount of variation between the first motion or pose for the user and the modified computer representation of the motion or pose, the first amount of variation being greater than the second amount of variation.

7. The method of claim 1, further comprising:

capturing second data representative of a second user in the physical space or a second physical space;

processing the second captured data to identify a second motion or pose for the second user, the second motion or pose being processed to determine a third confidence level that the second motion or pose is the same as the computer representation of the motion or pose for the particular gesture, wherein the second motion or pose is processed to generate values for parameters associated with the particular gesture in the gesture filter, said third confidence level indicating a likelihood that the values of the parameters associated with the second motion or pose correspond to the values of the parameters of the particular gesture so as to cause the second motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer; and

wherein modifying the computer representation of the motion or pose invoking the input command based on the first motion or pose by the user comprises modifying the computer representation based on the first motion or pose and the second motion or pose.

8. The method of claim 7, further comprising:

determining to modify the computer representation based on the first motion or pose and the second motion or pose in response to determining that the first user and the second user share a demographic characteristic, a language, or a culture.

9. The method of claim 7, further comprising:

determining to process third data representative of a third user using the modified computer representation of the motion or pose in response to determining that the third user shares a demographic characteristic, a language, or a culture with the first and second users.

10. The method of claim 7, further comprising:

determining to process third data representative of a third user using a computer representation of the motion or pose that differs from the modified computer representation of the motion or pose in response to determining that the third user differs in a demographic characteristic, a language, or a culture from the first and second users.

11. The method of claim 7, wherein the first user is located in the physical space, the second user is located in the second physical space, a first capture device is used to capture the data representative of the user in the physical space, and a second capture device is used to capture the second data representative of the second user in the second physical space, said first and second capture devices both being associated with said gesture-based system.

12. A computer-readable storage device for associating a gesture set to a user, bearing computer-readable instructions that, when executed on the computer, cause the computer to perform operations comprising:

capturing data representative of a user in a physical space;

processing the captured data to identify a first motion or pose for the user, the first motion or pose for the user being processed to determine a first confidence level that the first motion or pose is the same as a computer representation of a motion or pose for a particular gesture that invokes an input command to modify a visual output of a computer, including processing the first motion or pose to generate values for parameters associated with the particular gesture in a gesture filter, said first confidence level indicating a likelihood that the values of the parameters associated with the first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer; and

modifying the computer representation of the motion or pose invoking the input command based on the first motion or pose by the user such that subsequently captured data of the user or another user performing the first motion or pose is processed to determine a second confidence level that the first motion or pose is the same as the modified computer representation of the motion or pose for the particular gesture, said second confidence level indicating a likelihood that the values of the parameters associated with the subsequently captured first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the subsequently captured first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer, the second confidence level being greater than the first confidence level.

13. The computer-readable storage device of claim 12, further comprising:

correlating at least one feature of the user'"'"'s motion or pose to an identifier associated with the modified computer representation of the motion or pose, and implementing the modified computer representation of the motion or pose associated with the identifier.

14. The computer-readable storage device of claim 13, wherein the identifier identifies at least one of an operating system, an application, the user or another user, a feature of the user or another user, a location, a type of application, a hardware configuration, a software configuration, a culture, a current user, a geography, a demography, a linguistic characteristic, or a style.

15. The computer-readable storage device of claim 12, wherein processing the captured data to identify a first motion or pose for the user, the first motion or pose for the user being processed to determine the first confidence level that the motion or pose is the same as the computer representation of the motion or pose for a particular gesture that invokes an input command to modify the visual output of the computer comprises:

determining the first confidence level based on a first amount of variation between the first motion or pose for the user and the computer representation of the motion or pose; and

wherein modifying the computer representation of the motion or pose invoking the input command based on the first user motion or pose by the user such that subsequently captured data of the user or another user performing the first motion or pose is processed to determine the second confidence level that the motion or pose is the same as the modified computer representation of the motion or pose for the particular gesture comprises;

determining the second confidence level based on a second amount of variation between the first motion or pose for the user and the modified computer representation of the motion or pose, the first amount of variation being greater than the second amount of variation.

16. A system for evolving gesture recognition data, the system comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to;

capture data representative of a user in a physical space;

process the captured data to identify a first motion or pose for the user, the first motion or pose for the user being processed to determine a first confidence level that the first motion or pose is the same as a computer representation of a motion or pose for a particular gesture that invokes an input command to modify a visual output of a computer, wherein the first motion or pose is processed to generate values for parameters associated with the particular gesture in a gesture filter, said first confidence level indicating a likelihood that the values of the parameters associated with the first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer; and

modify the computer representation of the motion or pose invoking the input command based on the first motion or pose by the user such that subsequently captured data of the user or another user performing the first motion or pose is processed to determine a second confidence level that the first motion or pose is the same as the modified computer representation of the motion or pose for the particular gesture, said second confidence level indicating a likelihood that the values of the parameters associated with the subsequently captured first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the subsequently captured first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer, the second confidence level being greater than the first confidence level.

17. The system of claim 16, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

associate an identifier with the modified computer representation of the motion or pose, the identifier identifying a basis for why the first motion or pose from the user may fail to be recognized as the particular gesture and/or as a basis for implementing the modified computer representation of the motion or pose rather than the computer representation of the motion or pose in evaluating the subsequently captured data.

18. The system of claim 16, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

capture second data representative of a second user in the physical space or a second physical space;

process the second captured data to identify a second motion or pose for the second user, the second motion or pose being processed to determine a third confidence level that the second motion or pose is the same as the computer representation of the motion or pose for the particular gesture, wherein the second motion or pose is processed to generate values for parameters associated with the particular gesture in the gesture filter, said third confidence level indicating a likelihood that the values of the parameters associated with the second motion or pose correspond to the values of the parameters of the particular gesture so as to cause the second motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer; and

wherein the instructions that, when executed on the processor, cause the system at least to modify the computer representation of the motion or pose invoking the input command based on the first motion or pose by the user further cause the system at least to modify the computer representation based on the first motion or pose and the second motion or pose.

19. The system of claim 18, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

determine to modify the computer representation based on the first motion or pose and the second motion or pose in response to determining that the first user and the second user share a demographic characteristic, a language, or a culture.

20. The system of claim 18, wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:

determine to process third data representative of a third user using the modified computer representation of the motion or pose in response to determining that the third user shares a demographic characteristic, a language, or a culture with the first and second users.