Hybrid systems and methods for low-latency user input processing and feedback

• US 9,927,959 B2
• Filed: 10/04/2013
• Issued: 03/27/2018
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Tactual Labs Co.
• Original Assignee: Tactual Labs Co.
• Inventors: Wigdor, Daniel, Sanders, Steven Leonard, Jota Costa, Ricardo Jorge, Forlines, Clifton
• Primary Examiner(s): Kawsar, Abdullah Al
• Assistant Examiner(s): Luu, David V
• Application Number:
  US14/046,823
• Publication Number:
  US 20140143692A1
• Time in Patent Office: 1,635 Days
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining

Abstract

A system for processing user input includes an input device, an input processing unit, a high-latency subsystem, a low-latency subsystem, input processing unit software for generating signals in response to user inputs, and an output device. The low-latency subsystem receives the signals and generates low-latency output and the high-latency subsystem processes the signals and generates high-latency output.

Claims

1. A system for processing user input, comprising:

display comprising a visualization of a user interface;

user input device comprising at least one selected from the group consisting of touch input and pen input, the user input device configured to output signals corresponding to user input;

input processor operatively connected to the user input device, the input processor configured to receive the signals corresponding to user input and to emit signals corresponding to the received signals, the input processor being configured to emit at least some of the emitted signals to a first graphical processor and output system and to simultaneously emit the at least some of the emitted signals to a second graphical processor and output system;

the first graphical processor and output system comprising application software running in a software stack, the first graphical processing and output system configured to receive the at least some of the emitted signals and to cause an application logic response that can affect the visualization of the user interface on the display and to output the application logic response for use by the display in response to the at least some of the emitted signals;

the second graphical processor and output system implemented at least partially in hardware, the second graphical processor and output system configured to receive the at least some of the emitted signals and to cause a basic system response that is not a response from the application software that can affect the visualization of the user interface on the display and to output the generated basic system response for use by the display in response to the at least some of the emitted signals;

wherein the at least one basic system response is dependent upon the state information reflecting at least one state of the first graphical processor and output system;

wherein, with respect to the response to the at least some of the emitted signals, the second graphical processor and output system is configured to output the generated basic system response with low latency relative to the output of the application logic response by the first graphical processor and output system; and

the display being configured to display at least some of the basic system response and at least some of the application logic response on the visualization of the user interface, wherein basic system response visualization and the application logic response visualization are combined on the display by at least one of;

logically combining the basic system response and the application logic response visualizations in a display buffer, overlapping two transparent layers each capable of displaying one of the application logic response and basic system response visualizations, interlacing pixels of the display such that some are filled from the basic system response visualization and some are filled from the application logic response visualization, or interlacing the frames of the display such that some are filled from the basic system response visualization and some are filled from the application logic response visualization.

2. The system for processing user input according to claim 1, wherein the basic system response is generated by associating data corresponding to a defined response with one or more of the at least some of the emitted signals.

3. The system for processing user input according to claim 1, wherein the application logic response is generated by rendering a response to one or more of the at least some of the emitted signals.

4. The system for processing user input according to claim 1, wherein the application logic response and the basic system response have substantially equal fidelity.

5. The system for processing user input according to claim 1, wherein the application logic response and the basic system response have unequal fidelity.

6. The system for processing user input according to claim 5, wherein at least some of the application logic response is of higher fidelity than the corresponding basic system response.

7. The system for processing user input according to claim 1, wherein the second graphical processor and output system is implemented, at least in part, in at least one selected from the group of field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), a microcontroller, a graphics processing unit (GPU), or a system on a chip (SoC).

8. The system for processing user input according to claim 1, wherein the second graphical processor and output system is implemented in logic integrated into an existing component of a computing system.

9. The system for processing user input according to claim 8, wherein the existing component of the computing system comprises at least one selected from the group of a graphics processing unit (GPU), input device controller, central processing unit (CPU), microcontroller, or system on a chip (SoC).

10. The system for processing user input according to claim 1, wherein the second graphical processor and output system comprises user interface element logic.

11. The system for processing user input according to claim 10, further comprising:

UI elements data containing data regarding the visual hierarchy and properties of visible UI elements; and

wherein the second graphical processor and output system is adapted to perform hit testing based on information in the UI elements data.

12. The system for processing user input according to claim 1, wherein the emitted signals are sent in a stream to the first and second graphical processor and output system, and the second graphical processor and output system is configured to delete one or more of the emitted signals from the stream, preventing such deleted signals from being part of the at least some of the emitted signals corresponding to the received data that are received by the first graphical processor and output system.

13. The system for processing user input according to claim 1, wherein the at least some of the emitted signals corresponding to the application logic response for use by the display differ from the at least some of the emitted signals corresponding to the basic system response.

14. The system for processing user input according to claim 1, wherein the second graphical processor and output system outputs the generated basic system response for use by the display within 10 ms of receiving a corresponding emitted signal.

15. The system for processing user input according to claim 1, wherein the second graphical processor and output system outputs the generated basic system response for use by the display within 1 ms of receiving a corresponding emitted signal.

16. The system for processing user input according to claim 1, wherein a difference between at least some of the basic system response that can affect the visualization of the user interface and corresponding application logic response that can affect the visualization of the user interface is perceptible to a user.

17. The system for processing user input according to claim 1, wherein at least some of the basic system response that can affect the visualization of the user interface and at least some of the corresponding application logic response that can affect the visualization of the user interface is blended so that it is not perceptible by a user.

18. The system for processing user input according to claim 1, configured to provide a low-latency system response in a form that is dictated by application logic.

19. The system for processing user input according to claim 1, wherein the form of low-latency system response comprises a response or set of responses selected from the group consisting of:

an auditory response, a visual response, and a haptic response.

20. The system for processing user input according to claim 1, configured to provide a low-latency system response in a form that is dictated by system logic.

21. The system for processing user input according to claim 1, wherein the user input device comprises a direct-touch input device employing at least one of the technologies selected from the group consisting of:

resistive, direct illumination, frustrated total-internal reflection, diffuse illumination, projected capacitive, capacitive coupling, acoustic wave, and sensor-in-pixel.

22. The system for processing user input according to claim 1, wherein the second graphical processor and output system'"'"'s response is independent of the first graphical processor and output system.

23. The system for processing user input according to claim 1, wherein the second graphical processor and output system is configured to process the events with low fidelity with respect to a fidelity of the first graphical processor and output system.

24. A system for processing user input, comprising:

display comprising a visualization of a user interface;

user input device configured to output signals corresponding to user input;

input processor operatively connected to the user input device, the input processor configured to receive the signals corresponding to user input and to emit a stream of signals corresponding to at least some of the received signals, the input processor being configured to emit the stream of signals to a first graphical processor and output system and to simultaneously emit the stream of signals to a second graphical processor and output system;

the first graphical processor and output system comprising application software configured to receive the stream of signals and to generate application logic data that can affect the visualization of the user interface on the display and to output the application logic data for use by the display in response to at least some of the stream of signals received by the first graphical processor and output system;

the second graphical processor and output system configured to receive the stream of signals and to generate basic data that is not a response from the application software that can affect the visualization of the user interface on the display and to output the generated basic data for use by the display in response to the at least some of the stream of signals received by the second graphical processor and output system, the second graphical processor and output system being configured to output the generated basic data with low latency relative to the output of the application logic data by the first output graphical processor and output system;

wherein the basic data is dependent upon the state information reflecting at least one state of the first graphical processor and output system, and;

the display being configured to merge at least some of the basic data and at least some of the application logic data onto the visualization of the user interface, by at least one of;

logically combining the basic data and the at least some of the application logic data visualizations in a display buffer, overlapping two transparent layers each capable of displaying one of the basic and application logic visualizations, interlacing pixels of the display such that some are filled from the basic data visualizations and some are filled from the application logic data visualizations, or interlacing the frames of the display such that some are filled from the basic data visualizations and some are filled from the application logic visualizations.

25. The system for processing user input claimed in claim 24, wherein the application logic data is generated by rendering.

26. The system for processing user input according to claim 24, wherein the application logic data and the basic system data have substantially equal fidelity.

27. The system for processing user input according to claim 24, wherein the application logic data and the basic system data have unequal fidelity.

28. The system for processing user input according to claim 27, wherein at least some of the application logic data is of higher fidelity than the corresponding basic system data.

29. The system for processing user input according to claim 24, wherein the second graphical processor and output system is implemented, at least in part, in at least one selected from the group of field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), a microcontroller, a graphics processing unit (GPU), or a system on a chip (SoC).

30. The system for processing user input according to claim 24, wherein the second graphical processor and output system is implemented in logic integrated into an existing component of a computing system.

31. The system for processing user input according to claim 29, wherein the existing component of the computing system comprises at least one selected from the group of a graphics processing unit (GPU), input device controller, central processing unit (CPU), microcontroller, or system on a chip (SoC).

32. The system for processing user input according to claim 24, wherein the first graphical processor and output system includes state information reflecting at least one state of the first graphical processor and output system, and the second graphical processor and output system'"'"'s response is dependent upon the state information.

33. The system for processing user input according to claim 24, wherein the second graphical processor and output system comprises user interface element logic.

34. The system for processing user input according to claim 33, further comprising:

UI elements data containing data regarding the visual hierarchy and properties of visible UI elements; and

wherein the second graphical processing and output system is configured to perform hit testing based on information in the UI elements data.

35. The system for processing user input according to claim 24, wherein the second graphical processor and output system is configured to delete one or more of the signals from the stream, preventing such deleted signals from being part of the at least a portion of the stream of signals corresponding to at least some of the received data that are received by the first graphical processor and output system.

36. The system for processing user input according to claim 24, wherein the at least a portion of the stream of signals corresponding to the application logic data differ from the at least a portion of signals corresponding to the basic data.

37. The system for processing user input according to claim 24, wherein a difference between at least some of the basic data that can affect the visualization of the user interface and corresponding application data that can affect the visualization of the user interface is perceptible to a user.

38. The system for processing user input according to claim 24, wherein at least some of the basic data that can affect the visualization of the user interface and at least some of the corresponding application data that can affect the visualization of the user interface is blended so that it is not perceptible by a user.

39. The system for processing user input according to claim 24, configured to provide a low-latency system response in a form that is dictated by application logic.

40. The system for processing user input according to claim 39, wherein the form of low-latency system response comprises a response or set of responses selected from the group consisting of:

an auditory response, a visual response, and a haptic response.

41. The system for processing user input according to claim 24, configured to provide a low-latency system response in a form that is dictated by system logic.

42. The system for processing user input according to claim 24, wherein the user input device comprises a direct-touch input device employing at least one of the technologies selected from the group consisting of:

resistive, direct illumination, frustrated total-internal reflection, diffuse illumination, projected capacitive, capacitive coupling, acoustic wave, and sensor-in-pixel.

43. A system for processing user input, comprising:

input device;

input processor;

high-latency subsystem processor comprising application software;

low-latency subsystem processor implemented at least partially in hardware;

input processor for generating signals in response to user inputs; and

,

output device;

wherein the low-latency subsystem processor receives the signals and generates at least one basic output that is not a response from the application software with low latency relative to the high-latency subsystem processor and the high-latency subsystem processor processes at least some of the signals and causes generation of at least one application logic output with high latency relative to the low-latency subsystem processor;

wherein the basic output is dependent upon the state information reflecting at least one state of the high-latency subsystem processor;

wherein the input device, the input processor, the high-latency subsystem processor, the low-latency subsystem processor, the input processor software and the output device being arranged such that;

the low-latency processor subsystem operates independently of and parallel to the high-latency processing subsystem processor; and

at least some of the user input signals are processed by both the low-latency subsystem processor and the high-latency subsystem processor, the low-latency subsystem processor outputting to the output device a response to a given user input signal with low-latency and the high-latency subsystem processor causing output to the output device a response to the given user input signal with high-latency; and

the output device being configured to merge at least some of the basic output and at least some of the application logic output by at least one of;

logically combining the basic output and the at least some of the application logic output visualizations in a display buffer, overlapping two transparent layers each capable of displaying one of the basic output visualization and application logic output visualization, interlacing pixels of the display such that some are filled from the basic output visualization and some are filled from the application logic visualization, or interlacing the frames of the display such that some are filled from the basic output visualization and some are filled from the application logic output visualization.

44. The system for processing user input claimed in claim 43, wherein application logic output caused by the high-latency subsystem processor is generated by rendering.

45. The system for processing user input according to claim 43, wherein application logic output caused by the high-latency subsystem processor and basic output caused by the low-latency subsystem processor have substantially equal fidelity.

46. The system for processing user input according to claim 43, wherein application logic output caused by the high-latency subsystem processor and basic output caused by the low-latency subsystem processor have unequal fidelity.

47. The system for processing user input according to claim 43, wherein the high-latency subsystem processor generates application logic output and the low-latency subsystem processor generates basic output, and wherein at least some of the application logic output is of higher fidelity than the corresponding basic output.

48. The system for processing user input according to claim 43, wherein the low-latency subsystem processor is implemented, at least in part, in at least one selected from the group of field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), a microcontroller, a graphics processing unit (GPU), or a system on a chip (SoC).

49. The system for processing user input according to claim 43, wherein the low-latency subsystem processor is implemented in logic integrated into an existing component of a computing system.

50. The system for processing user input according to claim 49, wherein the existing component of the computing system comprises at least one selected from the group of a graphics processing unit (GPU), input device controller, central processing unit (CPU), microcontroller, or system on a chip (SoC).

51. The system for processing user input according to claim 43, wherein the low-latency subsystem processor comprises user interface element logic.

52. The system for processing user input according to claim 51, further comprising:

UI elements data containing data regarding the visual hierarchy and properties of visible UI elements; and

wherein the low-latency subsystem processor is configured to perform hit testing based on information in the UI elements data.

53. The system for processing user input according to claim 43, wherein the low-latency subsystem processor is configured to delete one or more of the signals from the stream, preventing such deleted signals from being part of the at least a portion of the stream of signals corresponding to at least some of the received data that are received by the high-latency subsystem processor.

54. The system for processing user input according to claim 43, wherein the at least a portion of the stream of signals corresponding to the application logic output differ from the at least a portion of signals corresponding to the basic output.

55. The system for processing user input according to claim 43, wherein a difference between at least some of the basic output that can affect the visualization of the user interface and corresponding application logic output that can affect the visualization of the user interface is perceptible to a user.

56. The system for processing user input according to claim 43, wherein at least some of the basic system output that can affect the visualization of the user interface and at least some of the corresponding application logic output that can affect the visualization of the user interface is blended so that it is not perceptible by a user.

57. The system for processing user input according to claim 43, configured to provide a low-latency system response in a form that is dictated by application logic.

58. The system for processing user input according to claim 57, wherein the form of low-latency system response comprises a response or set of responses selected from the group consisting of:

an auditory response, a visual response, and a haptic response.

59. The system for processing user input according to claim 43, configured to provide a low-latency system response in a form that is dictated by system logic.

60. The system for processing user input according to claim 43, wherein the user input device comprises a direct-touch input device employing at least one of the technologies selected from the group consisting of:

resistive, direct illumination, frustrated total-internal reflection, diffuse illumination, projected capacitive, capacitive coupling, acoustic wave, and sensor-in-pixel.

61. A system for processing user input, comprising:

input device;

input processor;

high-latency subsystem processor comprising application software running in a software stack;

low-latency subsystem processor;

input processor configured to generate signals in response to user inputs; and

,

output device;

wherein the low-latency subsystem processor processes the signals with low latency relative to the high-latency subsystem processor and the high-latency subsystem processor processes the user input events with high latency relative to the low-latency subsystem processor;

wherein the low-latency subsystem processor receives the signals and generates at least one basic output that is not a response from the application software with low latency relative to the high-latency subsystem processor and the high-latency subsystem processor processes at least some of the signals and generates at least one application logic output with high latency relative to the low-latency subsystem processor;

wherein the basic output is dependent upon the state information reflecting at least one state of the high-latency subsystem processor;

wherein the input device, the input processor, the high-latency subsystem processor, the low-latency subsystem processor, the input processor software and the output device being arranged such that;

the low-latency subsystem processor operates independently of and parallel to the high-latency subsystem processor; and

at least some of the user input signals are processed by both the low-latency subsystem processor and the high-latency subsystem processor, the low-latency subsystem processor outputting to the output device a response to a given user input signal with low-latency subsystem processor and the high-latency subsystem processor outputting to the output device a response to the given user input signal with high-latency; and

the output device being configured to merge at least some of the basic output and at least some of the application logic output by at least one of;

logically combining the basic output and the at least some of the application logic output visualizations in a display buffer, overlapping two transparent layers each capable of displaying one of a basic output visualization and an application logic output visualization, interlacing pixels of the display such that some are filled from the basic output visualization and some are filled from the application logic output visualization, or interlacing the frames of the display such that some are filled from the basic output visualization and some are filled from the application logic output visualization.

62. The system for processing user input claimed in claim 61, wherein application logic output generated by the high-latency subsystem processor is generated by rendering.

63. The system for processing user input according to claim 61, wherein application logic output generated by the high-latency subsystem processor and basic output generated by the low-latency subsystem processor have substantially equal fidelity.

64. The system for processing user input according to claim 61, wherein application logic output generated by the high-latency subsystem processor and basic output generated by the low-latency subsystem processor have unequal fidelity.

65. The system for processing user input according to claim 61, wherein the high-latency subsystem processor generates application logic output and the low-latency subsystem processor generates basic output, and wherein at least some of the application logic output is of higher fidelity than the corresponding basic output.

66. The system for processing user input according to claim 61, wherein the low-latency subsystem processor is implemented, at least in part, in at least one selected from the group of field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), a microcontroller, a graphics processing unit (GPU), or a system on a chip (SoC).

67. The system for processing user input according to claim 61, wherein the low-latency subsystem processor is implemented in logic integrated into an existing component of a computing system.

68. The system for processing user input according to claim 67, wherein the existing component of the computing system comprises at least one selected from the group of a graphics processing unit (GPU), input device controller, central processing unit (CPU), microcontroller, or system on a chip (SoC).

69. The system for processing user input according to claim 61, wherein the low-latency subsystem processor comprises user interface element logic.

70. The system for processing user input according to claim 69, further comprising:

UI elements data containing data regarding the visual hierarchy and properties of visible UI elements; and

wherein the low-latency subsystem is configured to perform hit testing based on information in the UI elements data.

71. The system for processing user input according to claim 61, wherein the low-latency subsystem processor is configured to delete one or more of the signals from the stream, preventing such deleted signals from being part of the at least a portion of the stream of signals corresponding to at least some of the received data that are received by the high-latency subsystem processor.

72. The system for processing user input according to claim 61, wherein the at least a portion of the stream of signals corresponding to the application logic output differ from the at least a portion of signals corresponding to the basic output.

73. The system for processing user input according to claim 61, wherein a difference between at least some of the basic output that can affect the visualization of the user interface and corresponding application logic output that can affect the visualization of the user interface is perceptible to a user.

74. The system for processing user input according to claim 61, wherein at least some of the basic output that can affect the visualization of the user interface and at least some of the corresponding application logic output that can affect the visualization of the user interface is blended so that it is not perceptible by a user.

75. The system for processing user input according to claim 61, configured to provide a low-latency system response in a form that is dictated by application logic.

76. The system for processing user input according to claim 75, wherein the form of low-latency system response comprises a response or set of responses selected from the group consisting of:

an auditory response, a visual response, and a haptic response.

77. The system for processing user input according to claim 61, configured to provide a low-latency system response in a form that is dictated by system logic.

78. The system for processing user input according to claim 61, wherein the user input device comprises a direct-touch input device employing at least one of the technologies selected from the group consisting of:

resistive, direct illumination, frustrated total-internal reflection, diffuse illumination, projected capacitive, capacitive coupling, acoustic wave, and sensor-in-pixel.

79. A system for processing user input, comprising:

input subsystem processor including an input device, the subsystem configured to output signals in response to a plurality of user inputs to the input device, the signals comprising information concerning the input;

low-latency subsystem processor configured to receive the signals and to generate a basic response that is not a response from an application software in response to at least two of the signals, and to send at least one of the at least two signals to a high-latency subsystem processor;

the high-latency subsystem processor comprising application software configured to receive the at least one of the at least two signals, and to cause an application logic response in response to one or more of the at least one of the at least two signals;

output device operatively connected to the low-latency subsystem processor and the high-latency subsystem processor, the output device configured to blend the basic response and the application logic response;

wherein the at least one basic response is dependent upon the state information reflecting at least one state of the high-latency subsystem processor, and;

wherein the low-latency subsystem processor causes the basic response with low latency relative to the high-latency subsystem processor and the high-latency subsystem processor causes the application logic response with high latency relative to the low-latency subsystem processor; and

an output device being configured to merge at least some of the basic response and at least some of the application logic response by at least one of;

logically combining the basic response and the at least some of the application logic response visualizations in a display buffer, overlapping two transparent layers each capable of displaying one of the application logic response visualization and basic response visualization, interlacing pixels of the display such that some are filled from the basic response visualization and some are filled from the application response visualization, or interlacing the frames of the display such that some are filled from the basic response visualization and some are filled from the application logic response visualization.

80. The system for processing user input claimed in claim 79, wherein application logic response caused by the high-latency subsystem processor is generated by rendering.

81. The system for processing user input according to claim 79, wherein application logic response caused by the high-latency subsystem processor and basic response caused by the low-latency subsystem processor have substantially equal fidelity.

82. The system for processing user input according to claim 79, wherein application logic response caused by the high-latency subsystem processor and basic response caused by the low-latency subsystem processor have unequal fidelity.

83. The system for processing user input according to claim 79, wherein the high-latency subsystem processor generates application logic response and the low-latency subsystem processor generates basic response, and wherein at least some of the application logic response is of higher fidelity than the corresponding basic response.

84. The system for processing user input according to claim 79, wherein the low-latency subsystem processor is implemented, at least in part, in at least one selected from the group of field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), a microcontroller, a graphics processing unit (GPU), or a system on a chip (SoC).

85. The system for processing user input according to claim 79, wherein low-latency subsystem processor is implemented in logic integrated into an existing component of a computing system.

86. The system for processing user input according to claim 79, wherein the existing component of the computing system comprises at least one selected from the group of a graphics processing unit (GPU), input device controller, central processing unit (CPU), microcontroller, or system on a chip (SoC).

87. The system for processing user input according to claim 79, wherein the high-latency subsystem processor includes state information reflecting at least one state of the high-latency subsystem processor, and the low-latency subsystem processor'"'"'s response is dependent upon the state information.

88. The system for processing user input according to claim 79, wherein the low-latency subsystem processor comprises user interface element logic.

89. The system for processing user input according to claim 88, further comprising:

UI elements data containing data regarding the visual hierarchy and properties of visible UI elements; and

wherein the low-latency subsystem processor is configured to perform hit testing based on information in the UI elements data.

90. The system for processing user input according to claim 79, wherein the low-latency subsystem processor is configured to delete one or more of the signals from the stream, preventing such deleted signals from being part of the at least a portion of the stream of signals corresponding to at least some of the received data that are received by the low-latency subsystem processor.

91. The system for processing user input according to claim 79, wherein the at least a portion of the stream of signals corresponding to the application logic response differ from the at least a portion of signals corresponding to the basic response.

92. The system for processing user input according to claim 79, wherein a difference between at least some of the basic response that can affect the visualization of the user interface and corresponding application logic response that can affect the visualization of the user interface is perceptible to a user.

93. The system for processing user input according to claim 79, wherein at least some of the basic response that can affect the visualization of the user interface and at least some of the corresponding application logic response that can affect the visualization of the user interface is blended so that it is not perceptible by a user.

94. The system for processing user input according to claim 79, configured to provide a low-latency system response in a form that is dictated by application logic.

95. The system for processing user input according to claim 94, wherein the form of low-latency system response comprises a response or set of responses selected from the group consisting of:

an auditory response, a visual response, and a haptic response.

96. The system for processing user input according to claim 79, configured to provide a low-latency system response in a form that is dictated by system logic.

97. The system for processing user input according to claim 79, wherein the user input device comprises a direct-touch input device employing at least one of the technologies selected from the group consisting of:

resistive, direct illumination, frustrated total-internal reflection, diffuse illumination, projected capacitive, capacitive coupling, acoustic wave, and sensor-in-pixel.

98. A system for processing user input with low latency, comprising:

an input device;

output device;

an input processor;

input processor software configured to generate signals in response to user inputs;

a high-latency response subsystem processor comprising application software configured to cause an application logic response to at least some of the signals;

a low-latency response subsystem processor implemented at least partially in hardware, the low-latency response subsystem processor configured to cause a basic response that is not a response from the application software to at least some of the signals, the low-latency response subsystem processor comprising one or more parameters affecting the low-latency response;

wherein the basic response is not of higher latency than the application logic response, and the application logic response is not of lower latency than the basic response with respect to common signals;

wherein the basic response is dependent upon the state information reflecting at least one state of the high-latency response subsystem processor;

an output device being configured to merge at least some of the basic response and at least some of the application logic response by at least one of;

logically combining the basic response and the at least some of the application logic response visualizations in a display buffer, overlapping two transparent layers each capable of displaying one of the basic response visualization and the application logic response visualization, interlacing pixels of the display such that some are filled from the basic response visualization and some are filled from the application logic visualization, or interlacing the frames of the display such that some are filled from the basic response visualization and some are filled from the application response visualization;

wherein the input device, output device, the input processor, input processor software, the high-latency response subsystem processor and the low-latency response subsystem processor are arranged such that;

the user input signals are processed in parallel by said low-latency subsystem processor and said high-latency subsystem processor;

the output device outputs the basic response and the application logic response; and

wherein application software can alter the one or more parameters affecting the basic response.

99. The system for processing user input claimed in claim 98, wherein application logic response caused to be output by the high-latency subsystem processor is generated by rendering.

100. The system for processing user input according to claim 98, wherein application logic response caused to be output by the high-latency subsystem processor and basic response generated by the low-latency subsystem processor have substantially equal fidelity.

101. The system for processing user input according to claim 98, wherein application logic response caused to be output by the high-latency subsystem processor and basic response generated by the low-latency subsystem processor have unequal fidelity.

102. The system for processing user input according to claim 98, wherein the high-latency subsystem processor causes the application logic response to be output and the low-latency subsystem processor causes the basic response to be output, and wherein at least some of the application logic response is of higher fidelity than the corresponding basic response.

103. The system for processing user input according to claim 98, wherein the low-latency subsystem processor is implemented, at least in part, in at least one selected from the group of field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), a microcontroller, a graphics processing unit (GPU), or a system on a chip (SoC).

104. The system for processing user input according to claim 98, wherein the low-latency subsystem processor is implemented in logic integrated into an existing component of a computing system.

105. The system for processing user input according to claim 98, wherein the existing component of the computing system comprises at least one selected from the group of a graphics processing unit (GPU), input device controller, central processing unit (CPU), microcontroller, or system on a chip (SoC).

106. The system for processing user input according to claim 98, wherein the high-latency subsystem processor includes state information reflecting at least one state of the high-latency subsystem processor, and the low-latency subsystem processor'"'"'s response is dependent upon the state information.

107. The system for processing user input according to claim 98, wherein the low-latency subsystem processor comprises user interface element logic.

108. The system for processing user input according to claim 107, further comprising:

UI elements data containing data regarding the visual hierarchy and properties of visible UI elements; and

wherein the low-latency subsystem processor is configured to perform hit testing based on information in the UI elements data.

109. The system for processing user input according to claim 98, wherein the low-latency subsystem processor is configured to delete one or more of the signals from the stream, preventing such deleted signals from being part of the at least a portion of the stream of signals corresponding to at least some of the received data that are received by the high-latency subsystem processor.

110. The system for processing user input according to claim 98, wherein the at least a portion of the stream of signals corresponding to the application logic response differ from the at least a portion of signals corresponding to the basic response.

111. The system for processing user input according to claim 98, wherein a difference between at least some of the basic response that can affect the visualization of the user interface and corresponding application logic response that can affect the visualization of the user interface is perceptible to a user.

112. The system for processing user input according to claim 98, wherein at least some of the basic response that can affect the visualization of the user interface and at least some of the corresponding application logic response that can affect the visualization of the user interface is blended so that it is not perceptible by a user.

113. The system for processing user input according to claim 98, configured to provide a low-latency system response in a form that is dictated by application logic.

114. The system for processing user input according to claim 113, wherein the form of low-latency system response comprises a response or set of responses selected from the group consisting of:

an auditory response, a visual response, and a haptic response.

115. The system for processing user input according to claim 98, configured to provide a low-latency system response in a form that is dictated by system logic.

116. The system for processing user input according to claim 98, wherein the user input device comprises a direct-touch input device employing at least one of the technologies selected from the group consisting of:

resistive, direct illumination, frustrated total-internal reflection, diffuse illumination, projected capacitive, capacitive coupling, acoustic wave, and sensor-in-pixel.