Mode sensitive processing of touch data

• US 20080309624A1
• Filed: 06/13/2007
• Published: 12/18/2008
• Est. Priority Date: 06/13/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hotelling, Steve Porter
• Granted Patent: US 9,052,817 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Embodiments of the present invention provide for a hardware or software utility layer operating at a multi-touch enabled device that can perform application aware processing of touch data. More specifically, various applications executing at the device can send to the utility layer definitions of the types of touch data they require from the multi-touch enabled display. The utility layer can then process incoming touch data in relation to these definitions and send back to the applications result data in a format requested by the applications. Thus, the computational load associated with processing of touch data can be decreased. Also, in certain cases, applications can obtain more accurate data than available in prior systems.

Claims

1. A method for operating a multi-touch enabled device, comprising:

generating by one or more applications executing at the multi-touch enabled device one or more control instances;

sending the control instances to a multi-touch utility;

receiving raster touch data by the multi-touch utility;

processing the raster touch data in view of the control instances by the multi-touch utility to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

sending to the one or more applications the one or more sets of results.

2. The method of claim 1, further comprising:

displaying one or more interface images at a display of the multi-touch enabled device, each interface image being associated with a control instance; and

allowing a user to touch one or more of the interface images,

wherein the raster touch data includes data indicating the user'"'"'s touch of the one or more interface images, and the processing the raster touch data includes;

obtaining the user'"'"'s touch data from the raster touch data; and

processing it in a manner associated with the control instance in order to obtain a set of results.

3. The method of claim 1, wherein the one or more control instances define the format of the sets of results.

4. The method of claim 1, wherein the sets of results do not include raster touch data, and at least one of the one or more applications does not process raster touch data.

5. The method of claim 1, wherein the processing of raster touch data includes processing current raster touch data as well as processing historical raster touch data.

6. The method of claim 5, wherein processing raster touch data includes comparing the current raster touch data with historical raster touch data to obtain incremental results.

7. The method of claim 5, further including:

determining by the multi-touch utility based on the control instances, which received raster touch data may be needed in the future as historical raster touch data;

saving the received raster touch data that may be needed in the future; and

discarding all other received raster touch data.

8. The method of claim 1, wherein:

processing the raster touch data further includes deriving intermediate touch data, the intermediate touch data being in a form other than a raster form;

the method further including;

saving the intermediate touch data; and

using the intermediate touch data in the future to obtain incremental results.

9. The method of claim 1, wherein the raster touch data includes a plurality of binary values, each binary value indicating whether a particular pixel is being touched.

10. The method of claim 1, wherein the raster touch data includes a plurality of values each value indicating a strength or pressure of touching of a particular pixel.

11. The method of claim 1, wherein each control instance defines a relevant region of the display, and wherein the results associated with each control instance are derived from touch data from the relevant region of the display for the control instance.

12. The method of claim 11, wherein:

one or more control instances include a level parameter, indicating a virtual level of each control instance;

one or more control instances are defined as being opaque; and

processing the raster touch data further includes for each opaque instance removing all touch events for areas of instances at a lower level of the opaque instance which are overlaid by the area of the opaque instance.

13. The method of claim 12, wherein one or more control instances are defined as being transparent and during processing of the touch data instances being overlaid by the transparent instances are not affected.

14. The method of claim 1, wherein each control instance is associated with a control type of one or more control types.

15. The method of claim 1 wherein one of the control instances is a button control instance and the results associated with said button control instance indicate whether the user is pressing a button appearing on the display and associated with said button control instance.

16. The method of claim 1 wherein one of the control instances is a slider control instance and the results associated with said slider control instance indicate incremental changes of the position of a user'"'"'s finger or fingers along a slider appearing on the display and associated with the slider control instance.

17. The method of claim 1 wherein one of the control instances is a knob control instance and the results associated with said knob control instance indicate incremental changes of the rotational orientation of a user'"'"'s finger or fingers along a knob appearing on the display and associated with the knob control instance.

18. The method of claim 1 wherein one of the control instances is a navigational surface control instance and the results associated with said navigational surface control instance indicate incremental changes of the relative positions of several fingers along a navigational surface appearing on the display and associated with the navigational surface control instance.

19. A computer readable medium comprising a multi-touch utility software configured to be executed at a processor of a multi-touch enabled device, the processor also executing one or more applications, the software being configured to cause the processor to:

receive one or more control instances from the applications;

receive raster touch data;

process the raster touch data in view of the control instances to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

20. The computer readable medium of claim 19, wherein the software is further configured to cause the processor to:

display one or more interface images at a display of the multi-touch enabled device, each interface image being associated with a control instance, a user being allowed to touch one or more of the interface images,

wherein the raster touch data includes data indicating the user'"'"'s touch of the one or more interface images, and the processing the raster touch data includes;

obtaining the user'"'"'s touch data from the raster touch data; and

processing it in a manner associated with the control instance in order to obtain a set of results.

21. The computer readable medium of claim 19, wherein the one or more control instances define the format of the sets of results.

22. The computer readable medium of claim 19, wherein the sets of results do not include raster touch data and at least one of the one or more applications does not process raster touch data.

23. The computer readable medium of claim 19, wherein the processing of raster touch data includes processing current raster touch data as well as processing historical raster touch data.

24. The computer readable medium of claim 23, wherein processing raster touch data includes comparing the current raster touch data with historical raster touch data to obtain incremental results.

25. The computer readable medium of claim 23, wherein the software is configured to further cause the processor to:

determine based on the control instances, which received raster touch data may be needed in the future as historical raster touch data;

save the received raster touch data that may be needed in the future; and

discard all other received raster touch data.

26. The computer readable medium of claim 19, wherein:

processing the raster touch data further includes deriving intermediate touch data, the intermediate touch data being in a form other than a raster form;

wherein the software is further configured to cause the processor to;

save the intermediate touch data; and

use the intermediate touch data in the future to obtain incremental results.

27. The computer readable medium of claim 19, wherein the raster touch data includes a plurality of binary values, each binary value indicating whether a particular pixel is being touched.

28. The computer readable medium of claim 19, wherein the raster touch data includes a plurality of values each value indicating a strength or pressure of touching of a particular pixel.

29. The computer readable medium of claim 19, wherein each control instance defines a relevant region of the display, and wherein the results associated with each control instance are derived from touch data from the relevant region of the display for the control instance.

30. The computer readable medium of claim 29, wherein:

one or more control instances include a level parameter, indicating a virtual level of each control instance;

one or more control instances are defined as being opaque; and

processing the raster touch data further includes for each opaque instance removing all touch events for areas of instances at a lower level of the opaque instance which are overlaid by the area of the opaque instance.

31. The computer readable medium of claim 30, wherein one or more control instances are defined as being transparent and during processing of the touch data instances being overlaid by the transparent instances are not affected.

32. The computer readable medium of claim 19, wherein each control instance is associated with a control type of one or more control types.

33. The computer readable medium of claim 19 wherein one of the control instances is a button control instance and the results associated with said button control instance indicate whether the user is pressing a button appearing on the display and associated with said button control instance.

34. The computer readable medium of claim 19 wherein one of the control instances is a slider control instance and the results associated with said slider control instance indicate incremental changes of the position of a user'"'"'s finger or fingers along a slider appearing on the display and associated with the slider control instance.

35. The computer readable medium of claim 19 wherein one of the control instances is a knob control instance and the results associated with said knob control instance indicate incremental changes of the rotational orientation of a user'"'"'s finger or fingers along a knob appearing on the display and associated with the knob control instance.

36. The computer readable medium of claim 19 wherein one of the control instances is a navigational surface control instance and the results associated with said navigational surface control instance indicate incremental changes of the relative positions of several fingers along a navigational surface appearing on the display and associated with the navigational surface control instance.

37. A multi-touch utility circuit included in a multi-touch enabled device, the multi-touch enabled device also including a processor, the processor executing one or more applications, the multi-touch utility circuit configured to cause the processor to:

receive one or more control instances from the applications;

receive raster touch data;

process the raster touch data in view of the control instances to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

38. A multi-touch enabled device comprising:

a display having a multi touch panel integrated therein;

a memory;

a processor;

a multi-touch utility stored in the memory and executed by the processor; and

one or more applications stored in the memory, executed by the processor and configured to cause the processor to generate one or more control instances and send them to the multi-touch utility,

wherein the multi-touch utility is further configured to cause the processor to;

receive one or more control instances from the applications;

receive raster touch data;

process the raster touch data in view of the control instances to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

39. The multi-touch enabled device of claim 38, wherein the multi-touch utility is further configured to cause the processor to:

display one or more interface images at the display, each interface image being associated with a control instance, a user being allowed to touch one or more of the interface images,

wherein the raster touch data includes data indicating the user'"'"'s touch of the one or more interface images, and the processing the raster touch data includes;

obtaining the user'"'"'s touch data from the raster touch data; and

processing it in a manner associated with the control instance in order to obtain a set of results.

40. The multi-touch enabled device of claim 38, wherein the one or more control instances define the format of the sets of results.

41. The multi-touch enabled device of claim 38, wherein the sets of results do not include raster touch data and at least one of the one or more applications does not process raster touch data.

42. The multi-touch enabled device of claim 38, wherein the processing of raster touch data includes processing current raster touch data as well as processing historical raster touch data.

43. The multi-touch enabled device of claim 42, wherein processing raster touch data includes comparing the current raster touch data with historical raster touch data to obtain incremental results.

44. The multi-touch enabled device of claim 42, wherein the multi-touch utility is configured to further cause the processor to:

determine based on the control instances, which received raster touch data may be needed in the future as historical raster touch data;

save the received raster touch data that may be needed in the future; and

discard all other received raster touch data.

45. The multi-touch enabled device of claim 38, wherein:

processing the raster touch data further includes deriving intermediate touch data, the intermediate touch data being in a form other than a raster form;

wherein the multi-touch utility is further configured to cause the processor to;

save the intermediate touch data; and

use the intermediate touch data in the future to obtain incremental results.

46. The multi-touch enabled device of claim 38, wherein the raster touch data includes a plurality of binary values, each binary value indicating whether a particular pixel is being touched.

47. The multi-touch enabled device of claim 38, wherein the raster touch data includes a plurality of values each value indicating a strength or pressure of touching of a particular pixel.

48. The multi-touch enabled device of claim 38, wherein each control instance defines a relevant region of the display, and wherein the results associated with each control instance are derived from touch data from the relevant region of the display for the control instance.

49. The multi-touch enabled device of claim 48, wherein:

one or more control instances include a level parameter, indicating a virtual level of each control instance;

one or more control instances are defined as being opaque; and

processing the raster touch data further includes for each opaque instance removing all touch events for areas of instances at a lower level of the opaque instance which are overlaid by the area of the opaque instance.

50. The multi-touch enabled device of claim 49, wherein one or more control instances are defined as being transparent and during processing of the touch data instances being overlaid by the transparent instances are not affected.

51. The multi-touch enabled device of claim 38, wherein each control instance is associated with a control type of one or more, control types.

52. The multi-touch enabled device of claim 38 wherein one of the control instances is a button control instance and the results associated with said button control instance indicate whether the user is pressing a button appearing on the display and associated with said button control instance.

53. The multi-touch enabled device of claim 38 wherein one of the control instances is a slider control instance and the results associated with said slider control instance indicate incremental changes of the position of a user'"'"'s finger or fingers along a slider appearing on the display and associated with the slider control instance.

54. The multi-touch enabled device of claim 38 wherein one of the control instances is a knob control instance and the results associated with said knob control instance indicate incremental changes of the rotational orientation of a user'"'"'s finger or fingers along a knob appearing on the display and associated with the knob control instance.

55. The multi-touch enabled device of claim 38 wherein one of the control instances is a navigational surface control instance and the results associated with said navigational surface control instance indicate incremental changes of the relative positions of several fingers along a navigational surface appearing on the display and associated with the navigational surface control instance.

56. The multi-touch enabled device of claim 38, wherein the multi-touch enabled device is a mobile telephone.

57. The multi-touch enabled device of claim 38, wherein the multi-touch enabled device is an audio player.

58. A mobile telephone comprising:

a display having a multi touch panel integrated therein;

a memory;

a processor;

a multi-touch utility stored in the memory and executed by the processor; and

one or more applications stored in the memory, executed by the processor and configured to cause the processor to generate one or more control instances and send them to the multi-touch utility,

wherein the multi-touch utility is further configured to cause the processor to;

receive one or more control instances from the applications;

receive raster touch data;

process the raster touch data in view of the control instances to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

59. An audio player comprising:

a display having a multi touch panel integrated therein;

a memory;

a processor;

a multi-touch utility stored in the memory and executed by the processor; and

one or more applications stored in the memory, executed by the processor and configured to cause the processor to generate one or more control instances and send them to the multi-touch utility,

wherein the multi-touch utility is further configured to cause the processor to;

receive one or more control instances from the applications;

receive raster touch data;

process the raster touch data in view of the control instances to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

60. A computer readable medium comprising an SDK, the SDK being configured to be executed at a software development computer and to cause the processor of the software development computer to:

allow a user to create an application for a multi-touch enabled device;

generate code defining one or more control instances and place said code within the application, each control instance defining general rules for processing multi touch data associated with a respective visual interface element;

wherein the application is configured to be executed at a processor of a multi-touch enabled device and to cause the processor of the multi-touch enabled device to;

send the one or more control instances to a multi-touch utility; and

receive from the multi touch utility one or more sets of results, each set of results being associated with a respective control instance and indicating interaction between the user and the visual control element associated with said control instance.

61. A multi-touch enabled device comprising:

a multi-touch enabled display;

one or more applications; and

a multi-touch utility,

wherein;

the one or more applications are configured to send control instances to the multi-touch utility, each control instance defining a user interface element; and

the multi-touch utility is configured to display the user interface elements defined by the control instances at the display, to process touch events resulting from a user interacting with the interface elements at the display and to send results indicative of the user'"'"'s interaction with the interface elements and associated with respective control instances to the one or more applications.

62. The multi-touch enabled device of claim 61, wherein each control instance further comprises data defining methods for processing in coming touch data for that instance.

63. A method for operating a multi-touch enabled device comprising a multi-touch enabled display, one or more applications and a multi-touch utility, the method comprising:

sending one or more control instances by the one or more applications to the multi-touch utility, each control instance defining a user interface element;

displaying the user interface elements defined by the control instances at the display;

allowing a user to touch one or more of the user interface elements;

processing touch events resulting from the user touching the interface elements; and

sending results indicative of the user'"'"'s touching of the interface elements and associated with respective control instances to the one or more applications.

64. The method of claim 63, wherein each control instance further comprises data defining methods for processing in coming touch data for that instance.