Mode sensitive processing of touch data

• US 9,052,817 B2
• Filed: 06/13/2007
• Issued: 06/09/2015
• Est. Priority Date: 06/13/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hotelling, Steve Porter
• Primary Examiner(s): Moon, Seokyun
• Application Number:
  US11/818,334
• Publication Number:
  US 20080309624A1
• Time in Patent Office: 2,918 Days
• Field of Search: 345/173
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Embodiments of the present invention provide for a hardware or software utility layer operating at a multi-touch enabled device that can perform application aware processing of touch data. More specifically, various applications executing at the device can send to the utility layer definitions of the types of touch data they require from the multi-touch enabled display. The utility layer can then process incoming touch data in relation to these definitions and send back to the applications result data in a format requested by the applications. Thus, the computational load associated with processing of touch data can be decreased. Also, in certain cases, applications can obtain more accurate data than available in prior systems.

Claims

1. A method for operating a touch enabled device, comprising:

generating by one or more applications executing at the touch enabled device one or more control instances, the control instances defining the types of user operable controls required by the applications;

sending the control instances to a touch utility;

receiving touch data by the touch utility from a touch panel;

instructing a hardware abstraction layer of the touch panel by the touch utility to process only the touch data required to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

sending to the one or more applications the one or more sets of results.

2. The method of claim 1, further comprising:

displaying one or more interface images at a display of the touch enabled device, each interface image being associated with a control instance; and

detecting one or more touches at one or more of the interface images,

wherein processing the touch data includes;

obtaining the touch data from raster data; and

processing the touch data in a manner associated with the control instance in order to obtain the one or more sets of results.

3. The method of claim 1, wherein the one or more control instances define the format of the sets of results.

4. The method of claim 1, wherein the processing of touch data includes processing current touch data as well as processing historical touch data.

5. The method of claim 4, wherein processing touch data includes comparing the current touch data with historical touch data to obtain incremental results.

6. The method of claim 4, further including:

determining by the touch utility based on the control instances, which received touch data may be needed in the future as historical touch data;

saving the received touch data that may be needed in the future; and

discarding all other received touch data.

7. The method of claim 1, wherein:

processing the touch data further includes deriving intermediate touch data, the intermediate touch data being in a form other than raster data;

the method further including;

saving the intermediate touch data; and

using the intermediate touch data to obtain incremental results.

8. The method of claim 1, wherein the touch data includes a plurality of binary values, each binary value indicating whether a particular pixel is being touched.

9. The method of claim 1, wherein the touch data includes a plurality of values, each value indicating a strength or pressure of touching of a particular pixel.

10. The method of claim 1, wherein each control instance defines a relevant region of the display, and wherein the results associated with each control instance are derived from touch data from the relevant region of the display for the control instance.

11. The method of claim 10, wherein:

one or more control instances include a level parameter, indicating a virtual level of each control instance;

one or more control instances are defined as being opaque; and

processing the touch data further includes, for each opaque instance, removing all touch events for areas of instances at a lower level of the opaque instance which are overlaid by the area of the opaque instance.

12. The method of claim 11, wherein one or more control instances are defined as being transparent and during processing of the touch data instances being overlaid by the transparent instances are not affected.

13. The method of claim 1, wherein each control instance is associated with a control type of one or more control types.

14. The method of claim 1, wherein one of the control instances is a button control instance and the results associated with the button control instance indicate whether a button appearing on the display and associated with the button control instance is being pressed.

15. The method of claim 1, wherein one of the control instances is a slider control instance and the results associated with the slider control instance indicate incremental changes of the position of an object or objects along a slider appearing on the display and associated with the slider control instance.

16. The method of claim 1, wherein one of the control instances is a knob control instance and the results associated with the knob control instance indicate incremental changes of the rotational orientation of an object or objects along a knob appearing on the display and associated with the knob control instance.

17. The method of claim 1, wherein one of the control instances is a navigational surface control instance and the results associated with the navigational surface control instance indicate incremental changes of the relative positions of several objects along a navigational surface appearing on the display and associated with the navigational surface control instance.

18. A non-transitory computer readable medium comprising touch utility software configured to be executed at a processor of a touch enabled device, the processor also executing one or more applications, the touch utility software being configured to:

receive one or more control instances from the applications, the control instances defining the types of user operable controls required by the applications;

receive touch data from a touch panel;

instructing a hardware abstraction layer of the touch panel by the touch utility to process only the touch data required to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

19. The non-transitory computer readable medium of claim 18, wherein the software is further configured to cause the processor to:

display one or more interface images at a display of the touch enabled device, each interface image being associated with a control instance, a user being allowed to touch one or more of the interface images,

wherein the touch data includes data indicating one or more touches at the one or more interface images, and processing the touch data includes;

obtaining the touch data from raster data; and

processing the touch data in a manner associated with the control instance in order to obtain the one or more sets of results.

20. The non-transitory computer readable medium of claim 18, wherein the one or more control instances define the format of the sets of results.

21. The non-transitory computer readable medium of claim 18, wherein the processing of touch data includes processing current touch data as well as processing historical touch data.

22. The non-transitory computer readable medium of claim 21, wherein processing touch data includes comparing the current touch data with historical touch data to obtain incremental results.

23. The non-transitory computer readable medium of claim 21, wherein the software is configured to further cause the processor to:

determine, based on the control instances, which received touch data may be needed as historical touch data;

save the received touch data that may be needed as historical touch data; and

discard all other received touch data.

24. The non-transitory computer readable medium of claim 18, wherein:

processing the touch data further includes deriving intermediate touch data, the intermediate touch data being in a form other than raster data;

wherein the software is further configured to cause the processor to;

save the intermediate touch data; and

use the intermediate touch data to obtain incremental results.

25. The non-transitory computer readable medium of claim 18, wherein the touch data includes a plurality of binary values, each binary value indicating whether a particular pixel is being touched.

26. The non-transitory computer readable medium of claim 18, wherein the touch data includes a plurality of values, each value indicating a strength or pressure of touching of a particular pixel.

27. The non-transitory computer readable medium of claim 18, wherein each control instance defines a relevant region of the display, and wherein the results associated with each control instance are derived from touch data from the relevant region of the display for the control instance.

28. The non-transitory computer readable medium of claim 27, wherein:

one or more control instances include a level parameter, indicating a virtual level of each control instance;

one or more control instances are defined as being opaque; and

processing the touch data further includes, for each opaque instance, removing all touch events for areas of instances at a lower level of the opaque instance which are overlaid by the area of the opaque instance.

29. The non-transitory computer readable medium of claim 28, wherein one or more control instances are defined as being transparent, and during processing of the touch data, instances being overlaid by the transparent instances are not affected.

30. The non-transitory computer readable medium of claim 18, wherein each control instance is associated with a control type of one or more control types.

31. The non-transitory computer readable medium of claim 18, wherein one of the control instances is a button control instance and the results associated with the button control instance indicate whether a button appearing on the display and associated with the button control instance is being pressed.

32. The non-transitory computer readable medium of claim 18, wherein one of the control instances is a slider control instance and the results associated with the slider control instance indicate incremental changes of the position of an object or objects along a slider appearing on the display and associated with the slider control instance.

33. The non-transitory computer readable medium of claim 18, wherein one of the control instances is a knob control instance and the results associated with the knob control instance indicate incremental changes of the rotational orientation of an object or objects along a knob appearing on the display and associated with the knob control instance.

34. The non-transitory computer readable medium of claim 18, wherein one of the control instances is a navigational surface control instance and the results associated with the navigational surface control instance indicate incremental changes of the relative positions of several objects along a navigational surface appearing on the display and associated with the navigational surface control instance.

35. A touch utility circuit included in a touch enabled device, the touch enabled device also including a processor, the processor executing one or more applications, the touch utility circuit configured to:

receive one or more control instances from the applications, the control instances defining the types of user operable controls required by the applications;

receive touch data from a touch panel;

instructing a hardware abstraction layer of the touch panel to process only the touch data required to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

36. A touch enabled device comprising:

a display having a touch panel integrated therein;

a memory;

a processor;

a touch utility stored in the memory and executed by the processor; and

one or more applications stored in the memory, executed by the processor and configured to cause the processor to generate one or more control instances and send them to the touch utility,

wherein the touch utility is further configured to;

receive one or more control instances from the applications, the control instances defining the types of user operable controls required by the applications;

receive touch data from the touch panel;

instructing a hardware abstraction layer of the touch panel to process only the touch data required to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

37. The touch enabled device of claim 36, wherein the touch utility is further configured to cause the processor to:

display one or more interface images at the display, each interface image being associated with a control instance, and detecting one or more touches at one or more of the interface images,

wherein processing the touch data includes;

obtaining the touch data from raster data; and

processing the touch data in a manner associated with the control instance in order to obtain the one or more sets of results.

38. The touch enabled device of claim 36, wherein the one or more control instances define the format of the sets of results.

39. The touch enabled device of claim 36, wherein the processing of touch data includes processing current touch data as well as processing historical touch data.

40. The touch enabled device of claim 39, wherein processing touch data includes comparing the current touch data with historical touch data to obtain incremental results.

41. The touch enabled device of claim 39, wherein the touch utility is configured to further cause the processor to:

determine, based on the control instances, which received touch data may be needed as historical touch data;

save the received touch data that may be needed as historical touch data; and

discard all other received touch data.

42. The touch enabled device of claim 36, wherein:

processing the touch data further includes deriving intermediate touch data, the intermediate touch data being in a form other than raster data;

wherein the touch utility is further configured to cause the processor to;

save the intermediate touch data; and

use the intermediate touch data to obtain incremental results.

43. The touch enabled device of claim 36, wherein the touch data includes a plurality of binary values, each binary value indicating whether a particular pixel is being touched.

44. The touch enabled device of claim 36, wherein the touch data includes a plurality of values, each value indicating a strength or pressure of touching of a particular pixel.

45. The touch enabled device of claim 36, wherein each control instance defines a relevant region of the display, and wherein the results associated with each control instance are derived from touch data from the relevant region of the display for the control instance.

46. The touch enabled device of claim 45, wherein:

one or more control instances include a level parameter, indicating a virtual level of each control instance;

one or more control instances are defined as being opaque; and

processing the touch data further includes for each opaque instance removing all touch events for areas of instances at a lower level of the opaque instance which are overlaid by the area of the opaque instance.

47. The touch enabled device of claim 46, wherein one or more control instances are defined as being transparent and during processing of the touch data instances being overlaid by the transparent instances are not affected.

48. The touch enabled device of claim 36, wherein each control instance is associated with a control type of one or more control types.

49. The touch enabled device of claim 36, wherein one of the control instances is a button control instance and the results associated with the button control instance indicate whether a button appearing on the display and associated with the button control instance is being pressed.

50. The touch enabled device of claim 36, wherein one of the control instances is a slider control instance and the results associated with the slider control instance indicate incremental changes of the position of an object or objects along a slider appearing on the display and associated with the slider control instance.

51. The touch enabled device of claim 36, wherein one of the control instances is a knob control instance and the results associated with the knob control instance indicate incremental changes of the rotational orientation of an object or objects along a knob appearing on the display and associated with the knob control instance.

52. The touch enabled device of claim 36, wherein one of the control instances is a navigational surface control instance and the results associated with the navigational surface control instance indicate incremental changes of the relative positions of several objects along a navigational surface appearing on the display and associated with the navigational surface control instance.

53. The touch enabled device of claim 36, wherein the touch enabled device is a mobile telephone.

54. The touch enabled device of claim 36, wherein the touch enabled device is an audio player.

55. A mobile telephone comprising:

a display having a touch panel integrated therein;

a memory;

a processor;

a touch utility stored in the memory and executed by the processor; and

one or more applications stored in the memory, executed by the processor and configured to cause the processor to generate one or more control instances and send them to the touch utility,

wherein the touch utility is further configured to;

receive one or more control instances from the applications, the control instances defining the types of user operable controls required by the applications;

receive touch data from the touch panel;

instructing a hardware abstraction layer of the touch panel by the touch utility to process only the touch data required to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

56. An audio player comprising:

a display having a touch panel integrated therein;

a memory;

a processor;

a touch utility stored in the memory and executed by the processor; and

one or more applications stored in the memory, executed by the processor and configured to cause the processor to generate one or more control instances and send them to the touch utility,

wherein the touch utility is further configured to;

receive one or more control instances from the applications, the control instances defining the types of user operable controls required by the applications;

receive touch data from the touch panel;

determine if any of the raster touch data is relevant to any of the control instances;

instructing a hardware abstraction layer of the touch panel by the touch utility to process only the touch data required to obtain one or more sets of results, wherein each set of results is associated with a respective control instance; and

send to the one or more applications the one or more sets of results.

57. A non-transitory computer readable medium comprising an SDK, the SDK being configured to be executed at a software development computer and to cause the processor of the software development computer to:

allow creation of an application for a touch enabled device;

generate code defining one or more control instances and place said code within the application, the control instances defining the types of user operable controls required by the application;

wherein the application is configured to be executed at a processor of a touch enabled device to

send the one or more control instances to a touch utility; and

receive from the touch utility one or more sets of results, the results, each set of results being associated with a respective control instance and indicating interaction with the visual interface element associated with said control instance,

wherein each set of results is obtained by a touch utility instructing a hardware abstraction layer of a touch panel to process processing only raster touch data required by at least one of the control instances.

58. A touch enabled device comprising:

a touch enabled display;

one or more applications; and

a touch utility,

wherein the one or more applications are configured to send control instances to the touch utility, the control instances defining the types of user operable controls required by the application;

wherein the touch utility is configured to display the user interface elements defined by the control instances at the display, to process touch events resulting from interaction with the interface elements at the display, the touch events captured in raster touch data, and to send results indicative of the interaction with the interface elements and associated with respective control instances to the one or more applications, and

wherein the results are obtained by instructing a hardware abstraction layer of the touch enabled display to process only the touch data required by at least one of the control instances.

59. The touch enabled device of claim 58, wherein each control instance further comprises data defining methods for processing in coming touch data for that instance.

60. A method for operating a touch enabled device comprising a touch enabled display, one or more applications and a touch utility, the method comprising:

sending one or more control instances by the one or more applications to the touch utility, the control instances defining the types of user operable controls required by the application, each control instance defining a user interface element;

displaying the user interface elements defined by the control instances at the display;

detecting a touch at one or more of the user interface elements;

processing touch data resulting from the user touching the interface elements;

generating results by instructing a hardware abstraction layer of the touch enabled display to process only the touch data required by at least one of the control instances; and

sending the results indicative of the detected touching of the interface elements and associated with respective control instances to the one or more applications.

61. The method of claim 60, wherein each control instance further comprises data defining methods for processing in coming touch data for that instance.