Mapping of physical controls for surface computing

• US 8,264,455 B2
• Filed: 02/03/2009
• Issued: 09/11/2012
• Est. Priority Date: 02/03/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Fiebrink, Rebecca, Morris, Dan, Morris, Meredith
• Primary Examiner(s): Boddie, William
• Assistant Examiner(s): THOMPSON, CHRISTOPHER D
• Application Number:
  US12/365,145
• Publication Number:
  US 20100194677A1
• Time in Patent Office: 1,316 Days
• Field of Search: None
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/04809 Textured surface identifying touch areas, e.g. overlay structure for a virtual keyboard
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Physical controls on a physical controller device (PCD) are dynamically mapped to application controls for an application being executed on a computer having a touch-sensitive display surface. The computer identifies a PCD which has been placed by a user on the display surface and displays a mapping aura for the PCD. When the user touches an activate direct-touch button displayed within the mapping aura, the computer activates a mapping procedure for the PCD and displays a highlighted direct-touch button over each application control which is available to be mapped to the physical controls on the PCD. When the user selects a particular application control which is available to be mapped by touching the highlighted button residing over the control, the computer creates a dynamic mapping between the selected application control and a user-selected physical control on the PCD.

Claims

1. A computer-implemented process for dynamically mapping physical controls on a repositionable physical controller device to application controls for an application being executed on a computer comprising a shared touch-sensitive display surface, comprising using the computer to perform the following process actions:

upon a user placing a physical controller device (PCD) comprising one or more physical controls on the display surface, recognizing and identifying the PCD and displaying a mapping aura for the PCD;

upon the user touching an activate direct-touch button which is displayed within the mapping aura,

activating a mapping procedure for the PCD, and

displaying a highlighted direct-touch button over each application control displayed on the display surface which is available to be mapped to the physical controls on the PCD; and

upon the user selecting a particular application control which is available to be mapped by touching the highlighted direct-touch button residing over said particular application control, creating a dynamic mapping between the selected application control and a user-selected physical control on the PCD.

2. The process of claim 1, wherein,

the display surface comprises a central editor sector within which the application, application controls and highlighted direct-touch buttons are displayed, and wherein a plurality of direct-touch widgets are displayed, each widget corresponding to a particular application control,

the mapping aura comprises,

a clipboard area comprising a plurality of virtual clipboard slots, and

one or more continuous virtual slots, each corresponding to a different continuously adjustable physical control on the PCD,

the physical controls on the PCD comprise one or more continuously adjustable physical controls, and wherein

whenever the selected application control is continuously adjustable, the process action of creating a dynamic mapping between the selected application control and a user-selected physical control on the PCD comprises the actions of:

displaying a copy of the direct-touch widget which corresponds to the selected application control within a particular virtual clipboard slot;

upon the user touching the particular virtual clipboard slot,

displaying a highlight over each of the continuous virtual slots; and

upon the user touching a highlighted continuous virtual slot,

clearing the contents of the particular virtual clipboard slot,

displaying a copy of the direct-touch widget which corresponds to the selected application control within the touched continuous virtual slot, and

dynamically mapping the selected application control to the particular continuously adjustable physical control which corresponds to the touched continuous virtual slot, wherein said mapping allows the user to dynamically choose to interact with the selected application control either via the particular continuously adjustable physical control, the direct-touch widget displayed within the central editor sector which corresponds to the selected application control, or the copy of the direct-touch widget displayed within the touched continuous virtual slot.

3. The process of claim 2, wherein each continuously adjustable physical control comprises either a rotatable knob or a slider.

4. The process of claim 1, wherein,

the display surface comprises a central editor sector within which the application, application controls and highlighted direct-touch buttons are displayed, and wherein a plurality of direct-touch widgets are displayed, each widget corresponding to a particular application control,

the mapping aura comprises,

a clipboard area comprising a plurality of virtual clipboard slots, and

one or more discrete virtual slots, each corresponding to a different discretely adjustable physical control on the PCD,

the physical controls on the PCD comprise one or more discretely adjustable physical controls, and wherein

whenever the selected application control is discretely adjustable, the process action of creating a dynamic mapping between the selected application control and a user-selected physical control on the PCD comprises the actions of:

displaying a copy of the direct-touch widget which corresponds to the selected application control within a particular virtual clipboard slot;

upon the user touching the particular virtual clipboard slot, displaying a highlight over each of the discrete virtual slots; and

upon the user touching a highlighted discrete virtual slot,

clearing the contents of the particular virtual clipboard slot,

displaying a copy of the direct-touch widget which corresponds to the selected application control within the touched discrete virtual slot, and

dynamically mapping the selected application control to the particular discretely adjustable physical control which corresponds to the touched discrete virtual slot, wherein said mapping allows the user to dynamically choose to interact with the selected application control either via the particular discretely adjustable physical control, the direct-touch widget displayed within the central editor sector which corresponds to the selected application control, or the copy of the direct-touch widget displayed within the touched discrete virtual slot.

5. The process of claim 4, wherein each discretely adjustable physical control comprises either a depressible button or a switch.

6. The process of claim 1, wherein,

the display surface comprises a central editor sector within which the application, application controls and highlighted direct-touch buttons are displayed, and wherein a plurality of direct-touch widgets are displayed, each widget corresponding to a particular application control,

the physical controls on the PCD comprise one or more continuously adjustable physical controls, and

the mapping aura comprises one or more continuous virtual slots, each corresponding to a different continuously adjustable physical control on the PCD, and wherein,

whenever the selected application control is continuously adjustable, the process action of creating a dynamic mapping between the selected application control and a user-selected physical control on the PCD comprises the actions of:

upon the user dragging a direct-touch widget associated with the selected application control into a chosen continuous virtual slot,

displaying a copy of the direct-touch widget which corresponds to the selected application control within the chosen continuous virtual slot, and

dynamically mapping the selected application control to the particular continuously adjustable physical control which corresponds to the chosen continuous virtual slot, wherein said mapping allows the user to dynamically choose to interact with the selected application control either via the particular continuously adjustable physical control, the direct-touch widget displayed within the central editor sector which corresponds to the selected application control, or the copy of the direct-touch widget displayed within the desired continuous virtual slot.

7. The process of claim 1, wherein,

the display surface comprises a central editor sector within which the application, application controls and highlighted direct-touch buttons are displayed, and wherein a plurality of direct-touch widgets are displayed, each widget corresponding to a particular application control,

the physical controls on the PCD comprise one or more discretely adjustable physical controls, and

the mapping aura comprises one or more discrete virtual slots, each corresponding to a different discretely adjustable physical control on the PCD, and wherein

whenever the selected application control is discretely adjustable, the process action of creating a dynamic mapping between the selected application control and a user-selected physical control on the PCD comprises the actions of:

upon the user dragging a direct-touch widget associated with the selected application control into a chosen discrete virtual slot,

displaying a copy of the direct-touch widget which corresponds to the selected application control within the chosen discrete virtual slot, and

dynamically mapping the selected application control to the particular discretely adjustable physical control which corresponds to the chosen discrete virtual slot, wherein said mapping allows the user to dynamically choose to interact with the selected application control either via the particular discretely adjustable physical control, the direct-touch widget displayed within the central editor sector which corresponds to the selected application control, or the copy of the direct-touch widget displayed within the desired discrete virtual slot.

8. The process of claim 1, further comprising using the computer to perform the actions of:

upon the user again touching the activate direct-touch button,

deactivating the mapping procedure for the PCD, and

returning the application to a normal operating mode.

9. The process of claim 8, wherein,

the process action of activating a mapping procedure for the PCD comprises an action of displaying a highlighted halo border around the mapping aura, and

the process action of deactivating the mapping procedure for the PCD comprises an action of removing said highlighted halo border.

10. The process of claim 1, wherein the PCD comprises a tag which is located on the bottom of the first PCD and which uniquely identifies the PCD, and wherein the process action of recognizing and identifying the PCD comprises an action of deciphering the tag to uniquely identify the first PCD.

11. The process of claim 1, further comprising using the computer to perform the actions of:

upon a second user placing a second PCD on the display surface, automatically recognizing and identifying the second PCD and displaying a second mapping aura for the second PCD; and

upon the second user touching an activate direct-touch button which is displayed within the second mapping aura,

deactivating the mapping procedure for the first PCD, and

activating a mapping procedure for the second PCD.

12. The process of claim 11, wherein,

the process action of activating a mapping procedure for the first PCD comprises an action of displaying a first highlighted halo border around the first mapping aura,

the process action of deactivating the mapping procedure for the first PCD comprises an action of removing said first highlighted halo border, and

the process action of activating a mapping procedure for the second PCD comprises the actions of,

displaying a second highlighted halo border around the second mapping aura, and

transferring a current mapping state for the first PCD to the second mapping aura.

13. A system for dynamically mapping physical controls to application controls for an application, comprising:

a plurality of repositionable physical controller devices (PCDs), each PCD comprising one or more physical controls;

a computer comprising a touch-sensitive display surface which is shared by a plurality of collaborating users, wherein the application is executed on the computer, and the application and application controls are displayed on the display surface; and

a computer program having program modules executable by the computer, wherein the computing device is directed by the program modules of the computer program to,

recognize and identify each PCD placed by the users on the display surface, and

create a dynamic mapping between a user-selected application control and a user-selected physical control on a user-selected PCD.

14. The system of claim 13, wherein the computer program further comprises a program module for displaying a mapping aura for each PCD on the display surface, and whenever a user moves their PCD to a new location on the display surface, the mapping aura displayed for their PCD is moved accordingly.

15. The system of claim 14, wherein the program module for displaying the mapping aura comprises displaying a save direct-touch button in the aura, and wherein the computer program further comprises program modules for:

whenever a user touches the save button within the mapping aura for their PCD,

saving a current mapping state for their PCD on the computer,

appending the current mapping state to a mapping history list which is also saved on the computer, said list comprising a chronological listing of each of the saved mapping states for their PCD,

displaying the list within a scrollable sector adjacent to the mapping aura for their PCD, and

upon the user selecting a mapping state from the list, implementing the selected mapping state on their PCD.

16. The system of claim 15, wherein, each PCD further comprises a memory and a communications link with the computer, and whenever a user touches the save button within the mapping aura for a PCD, said mapping history list is transferred to the PCD from the computer via the communications link and saved in the PCD'"'"'s memory.

17. The system of claim 15, wherein the computer program further comprises program modules for:

whenever a first user drags the displayed mapping history list for their PCD into a mapping aura displayed for the PCD of a second user,

docking the dragged history list to the second mapping aura, and

highlighting the dragged history list and the second mapping aura to visually indicate the docking.

18. The system of claim 14, wherein each mapping aura comprises a delete direct-touch button, and wherein the computer program further comprises a program module for, whenever a user touches the delete button within the mapping aura for their PCD, deleting a particular mapping which was previously created between an application control and a physical control on their PCD.

19. The system of claim 14, wherein each mapping aura comprises a query direct-touch button, and wherein the computer program further comprises a program module for, whenever a user touches the query button within the mapping aura for their PCD, and the user then touches a particular virtual slot of the mapping aura for their PCD, flashing each direct-touch widget displayed on the display surface which corresponds to the application control mapped to the physical control on the PCD associated with the touched particular virtual slot, in a highlighted color, for a prescribed period of time.

20. A computer-implemented process for dynamically mapping physical controls on a repositionable physical controller device to application controls for an application being executed on a computer comprising a shared touch-sensitive display surface, comprising using the computer to perform the following process actions:

upon a user placing a physical controller device (PCD) on the display surface, recognizing and identifying the PCD and displaying a mapping aura for the PCD, wherein,

the PCD comprises one or more continuously adjustable physical controls and one or more discretely adjustable physical controls, and

the mapping aura comprises,

one or more continuous virtual slots, each corresponding to a different continuously adjustable physical control, and

one or more discrete virtual slots, each corresponding to a different discretely adjustable physical control;

upon the user touching an activate direct-touch button which is displayed within the mapping aura, displaying a highlighted direct-touch widget adjacent to each application control displayed on the display surface which is available to be mapped to the physical controls on the PCD;

upon the user selecting a particular application control which is available to be mapped and which is continuously adjustable, by touching the highlighted direct-touch widget displayed adjacent to said particular application control, and upon the user dragging the touched direct-touch widget into a chosen continuous virtual slot displayed in the mapping aura associated with the user'"'"'s PCD,

displaying a copy of the direct-touch widget which corresponds to the selected application control within the chosen continuous virtual slot, and

dynamically mapping the selected application control to the particular continuously adjustable physical control which corresponds to the chosen continuous virtual slot, wherein said mapping allows the user to dynamically choose to interact with the selected application control either via the particular continuously adjustable physical control, the direct-touch widget displayed adjacent to the selected application control, or the copy of the direct-touch widget displayed within the desired continuous virtual slot; and

upon the user selecting a particular application control which is available to be mapped and which is discretely adjustable, by touching the highlighted direct-touch widget displayed adjacent to said particular application control, and upon the user dragging the touched direct-touch widget into a chosen discrete virtual slot displayed in the mapping aura associated with the user'"'"'s PCD,

displaying a copy of the direct-touch widget which corresponds to the selected application control within the chosen discrete virtual slot, and

dynamically mapping the selected application control to the particular discretely adjustable physical control which corresponds to the chosen discrete virtual slot, wherein said mapping allows the user to dynamically choose to interact with the selected application control either via the particular discretely adjustable physical control, the direct-touch widget displayed adjacent to the selected application control, or the copy of the direct-touch widget displayed within the desired discrete virtual slot.