Contact grouping and gesture recognition for surface computing

• US 8,427,440 B2
• Filed: 05/05/2009
• Issued: 04/23/2013
• Est. Priority Date: 05/05/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Kuo, Sun, Xueqing, Zhang, Guanggen, Huang, KeCheng
• Primary Examiner(s): Beck, Alexander S
• Assistant Examiner(s): Ngo, Tony N
• Application Number:
  US12/435,419
• Publication Number:
  US 20100283739A1
• Time in Patent Office: 1,449 Days
• Field of Search: 345173-178, 715/863
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Tools and techniques for contact grouping and gesture recognition for surface computing are provided. These tools receive, from a finger grouping subsystem, data representing velocity parameters associated with contact points, with the contact points representing physical actions taken by users in interacting with software applications presented on a surface computing system. These tools also identify gestures corresponding to the physical actions by analyzing the velocity parameters, and provide indications of the identified gestures to the software applications.

Claims

1. A computer-readable storage medium that is not a signal having stored thereon computer-executable instructions that, when loaded into at least one hardware processor and executed, cause the at least one hardware processor to:

receive at least one indication of a physical action directed by at least one user toward a surface computing system, wherein the physical action comprises a plurality of fingers of the at least one user being in physical contact with the surface computing system to interact with a software application program being presented on the surface computing system and, as a result, each of the plurality of fingers being interpreted by the surface computing system as a contact point of a plurality of contact points;

associate respective velocity parameters with the plurality of contact points;

calculate differentials between velocity parameters associated with pairs of the plurality contact points;

select a first contact point of the plurality of contact points and a second contact point of the plurality of contact points to identify a pair of contact points having a greatest differential in velocity and thereby separately identifying the first contact point as being associated with a first actual hand of the at least one user and the second contact point as being associated with a second actual hand of the at least one user, the first contact point having a first contact point velocity parameter and the second contact point having a second contact point velocity parameter;

associate the first contact point with a first virtual hand data structure that is representative of the first actual hand;

associate the second contact point with a second virtual hand data structure that is representative of the second actual hand;

assign the first contact point velocity parameter to the first virtual hand data structure;

assign the second contact point velocity parameter to the second virtual hand data structure;

associate a first remainder of the plurality of contact points with the first virtual hand data structure; and

associate a second remainder of the plurality of contact points with the second virtual hand data structure.

2. The computer-readable storage medium of claim 1, wherein the plurality of fingers comprises different fingers on different hands.

3. The computer-readable storage medium of claim 1, wherein the instructions to associate respective velocity parameters include instructions to associate parameters that indicate a speed and direction associated with the physical action, with respect to the surface computing system.

4. The computer-readable storage medium of claim 1, wherein the instructions to associate a first remainder of the plurality of contact points include instructions to select one of the plurality of contact points other than the first and second contact points, further comprising instructions to calculate velocity differentials between the selected other one of the plurality of contact points and the first and second virtual hand data structures, and further comprising instructions to associate the selected other one of the plurality of contact points with the first virtual hand data structure or the second virtual hand data structure based upon with which of the first virtual hand data structure and the second virtual hand data structure the selected other one of the plurality of contact points has a smallest velocity differential and thereby identifying the selected other one of the plurality of contact points as being associated with the first actual hand of the at least one user or the second actual hand of the at least one user.

5. The computer-readable storage medium of claim 1, further comprising instructions to recalculate the velocity parameters associated with the first and second virtual hand data structures in response to associating the first and second remainders of the plurality of contact points with the first and second virtual hand data structures.

6. The computer-readable storage medium of claim 5, wherein the instructions to recalculate the velocity parameters of the first and second data structures include instructions to calculate a first virtual hand velocity parameter of the first virtual hand data structure by averaging the velocity parameters of the contact points assigned to the first virtual hand data structure, and include instructions to calculate a second virtual hand velocity parameter of the second virtual hand data structure by averaging the velocity parameters of the contact points assigned to second first data structure.

7. The computer-readable storage medium of claim 6, further comprising instructions to reassign the plurality of contact points to the first and second virtual hand data structures in response to recalculating the velocity parameters associated with the first and second virtual hand data structures.

8. The computer-readable storage medium of claim 7, wherein the instructions to reassign the plurality of contact points include instructions to calculate respective velocity differentials between the plurality of contact points and the first and second virtual hand data structures, and further comprising instructions to reassign the plurality of contact points to the first or second virtual hand data structures to minimize the velocity differential therebetween.

9. The computer-readable storage medium of claim 8, further comprising instructions to discard a contact point reassigned to the first virtual hand data structure, wherein the velocity parameter of the discarded contact point deviates a maximum value as compared to other contact points reassigned to the first virtual hand data structure, and further comprising instructions to discard a further contact point reassigned to the second virtual hand data structure, wherein the velocity parameter of the further discarded contact point deviates a maximum value as compared to other contact points reassigned to the second virtual hand data structure.

10. A surface computing system comprising:

at least one instance of processing hardware;

at least one bus system in communication with the processing hardware;

at least one computer-readable storage medium that is not a sign in communication with the processing hardware via the bus system, comprising storage having computer-executable instructions stored thereon that, when loaded into the processing hardware, transform the processing hardware to

receive at least one indication of a physical action directed by at least one user toward the surface computing system, wherein the physical action is represented by a plurality of contact points that correspond to a plurality of fingers associated with the user;

associate respective velocity and position parameters with the plurality of contact points;

select two of the plurality of contact points having a maximum velocity differential therebetween;

associate a first one of the selected contact points with a first virtual hand data structure;

associate a second one of the selected contact points with a second virtual hand data structure;

assign the velocity and position parameters of the first selected contact point to the first virtual hand structure;

assign the velocity and position parameters of the second selected contact point to the second virtual hand structure; and

calculate a gesture represented by the physical action, based on relative movement between the first and second virtual hand structures as indicated by analysis of the first and second contact points.

11. The surface computing system of claim 10, wherein the instructions to calculate a gesture include instructions to infer that the physical action represents a pan command, zoom command, or rotate command issued by the user to the software application.

12. The surface computing system of claim 10, wherein the instructions to calculate a gesture include instructions to analyze movement represented in the first virtual hand data structure relative to movement represented in the second virtual hand data structure.

13. A computer-implemented method comprising performing computer-implemented operations for:

receiving at least one indication of a physical action directed by at least one user toward a surface computing system, wherein the physical action comprises a plurality of fingers of the at least one user being in physical contact with the surface computing system to interact with a software application program being presented on the surface computing system and, as a result, each of the plurality of fingers being interpreted by the surface computing system as a contact point of a plurality of contact points;

associating respective velocity parameters with the plurality of contact points;

calculating differentials between velocity parameters associated with pairs of the plurality contact points;

selecting a first contact point of the plurality of contact points and a second contact point of the plurality of contact points to identify a pair of contact points having a greatest differential in velocity and thereby separately identifying the first contact point as being associated with a first actual hand of the at least one user and the second contact point as being associated with a second actual hand of the at least one user, the first contact point having a first contact point velocity parameter and the second contact point having a second contact point velocity parameter;

associating the first contact point with a first virtual hand data structure that is representative of the first actual hand;

associating the second contact point with a second virtual hand data structure that is representative of the second actual hand;

assigning the first contact point velocity parameter to the first virtual hand data structure;

assigning the second contact point velocity parameter to the second virtual hand data structure;

associating a first remainder of the plurality of contact points with the first virtual hand data structure; and

associating a second remainder of the plurality of contact points with the second virtual hand data structure.

14. The computer-implemented method of claim 13, wherein associating the respective velocity parameters with the plurality of contact points comprises associating parameters that indicate a speed and direction associated with the physical action, with respect to the surface computing system.

15. The computer-implemented method of claim 13, wherein associating the first remainder of the plurality of contact points with the first virtual hand data structure comprises selecting one of the plurality of contact points other than the first and second contact points, and further comprising:

calculating velocity differentials between the selected other one of the plurality of contact points and the first and second virtual hand data structures, and

associating the selected other one of the plurality of contact points with the first virtual hand data structure or the second virtual hand data structure based upon with which of the first virtual hand data structure and the second virtual hand data structure the selected other one of the plurality of contact points has a smallest velocity differential and thereby identifying the selected other one of the plurality of contact points as being associated with the first actual hand of the at least one user or the second actual hand of the at least one user.

16. The computer-implemented method of claim 13, further comprising recalculating the velocity parameters associated with the first and second virtual hand data structures in response to associating the first and second remainders of the plurality of contact points with the first and second virtual hand data structures.

17. The computer-implemented method of claim 16, wherein recalculating the velocity parameters associated with the first and second virtual hand data structures comprises calculating a first virtual hand velocity parameter of the first virtual hand data structure by averaging the velocity parameters of the contact points assigned to the first virtual hand data structure, and calculating a second virtual hand velocity parameter of the second virtual hand data structure by averaging the velocity parameters of the contact points assigned to second first data structure.

18. The computer-implemented method of claim 17, further comprising reassigning the plurality of contact points to the first and second virtual hand data structures in response to recalculating the velocity parameters associated with the first and second virtual hand data structures.

19. The computer-implemented method of claim 18, wherein reassigning the plurality of contact points to the first and second virtual hand data structures comprises calculating respective velocity differentials between the plurality of contact points and the first and second virtual hand data structures, and reassigning the plurality of contact points to the first or second virtual hand data structures to minimize the velocity differential therebetween.

20. The computer-implemented method of claim 19, further comprising:

discarding a contact point reassigned to the first virtual hand data structure, wherein the velocity parameter of the discarded contact point deviates a maximum value as compared to other contact points reassigned to the first virtual hand data structure; and

discarding a further contact point reassigned to the second virtual hand data structure, wherein the velocity parameter of the further discarded contact point deviates a maximum value as compared to other contact points reassigned to the second virtual hand data structure.