Computer system interface surface
First Claim
1. A method of enabling user interaction with computer software running in a computer system, the method including the steps of:
- (a) providing an interface surface containing;
human-readable information relating to the computer software; and
coded data indicative of at least one interactive element relating to the computer software, wherein the information and the coded data are printed substantially simultaneously onto a substrate; and
in the computer system;
(b) receiving indicating data from a sensing device, the indicating data being generated using coded data sensed in response to placing of the sensing device in an operative position relative to the interface surface and being indicative of;
(i) the at least one interactive element; and
(ii) movement of the sensing device relative to the interface surface, (c) determining movement of the sensing device to the interface surface from the indicating data;
(d) identifying the at least one interactive element from the indicating data; and
(e) operating the computer software at least partly in reliance on the movement data, and in accordance with instructions associated with the at least one interactive element.
1 Assignment
0 Petitions
Accused Products
Abstract
A method and system for enabling user interaction with computer software running in a computer system. The user interaction is via an interface, such as a sheet of paper bearing information relating to the computer software and coded data indicative of an interactive element relating to the computer software. A sensing device is placed in an operative position relative to the interface surface, and senses indicating data indicative of the interactive element. The sensing device also generates movement data indicative of the sensing device'"'"'s movement relative to the interface surface. The indicating and movement data from the sensing device are received in a computer, and the interactive element identified. The computer software is then operated at least partly in reliance on the movement data, and in accordance with instructions associated with the interactive element.
-
Citations
184 Claims
-
1. A method of enabling user interaction with computer software running in a computer system, the method including the steps of:
-
(a) providing an interface surface containing;
human-readable information relating to the computer software; and
coded data indicative of at least one interactive element relating to the computer software, wherein the information and the coded data are printed substantially simultaneously onto a substrate; and
in the computer system;
(b) receiving indicating data from a sensing device, the indicating data being generated using coded data sensed in response to placing of the sensing device in an operative position relative to the interface surface and being indicative of;
(i) the at least one interactive element; and
(ii) movement of the sensing device relative to the interface surface, (c) determining movement of the sensing device to the interface surface from the indicating data;
(d) identifying the at least one interactive element from the indicating data; and
(e) operating the computer software at least partly in reliance on the movement data, and in accordance with instructions associated with the at least one interactive element. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92)
CMY;
CMYK;
CMYRGB; and
spot colour.
-
-
67. A method according to claim 28, wherein at least a plurality of the tags are disposed stochastically upon the surface.
-
68. A method according to claim 28, wherein the tags are disposed in a regular array on the surface.
-
69. A method according to claim 68, wherein the array is triangular.
-
70. A method according to claim 68, wherein the array is rectangular.
-
71. A method according to claim 68, wherein the tags are tiled over the surface.
-
72. A method according to claim 28, further including additional non-tag information disposed on the surface.
-
73. A method according to claim 1, wherein the region is identified with sufficient precision to distinguish the region from 1.5×
- 1014 other regions.
-
74. A method according to claim 1, wherein any 10 millimeter diameter subregion of the region includes sufficient information to identify the region.
-
75. A method according to claim 74, wherein any 10 millimeter subregion of the region includes sufficient information to identify at least one point of the region.
-
76. A method according to claim 1, wherein the sensing device includes at least one acceleration measuring device for measuring acceleration of the sensing device as it is used to draw the user input onto the surface, the movement data being generated by periodically sampling the acceleration of the sensing device as it is used to draw the user input onto the surface.
-
77. A method according to claim 76, further including the step of generating movement data in the form of a locus of the sensing means in relation to the surface, the locus being determined by ascertaining relative displacement of the sensing means due to its velocity within each sampling period.
-
78. A method according to claim 77, wherein the accelerometers are configured to measure at least two orthogonal components of acceleration.
-
79. A method according to claim 1, including the step of providing the user with printed information including position elements, the position elements being disposed on a surface, the sensing device being configured to periodically sense position elements as it is used to draw the user input onto the surface, the method including the step of generating the movement data in the form of a locus of the sensing means in relation to the surface by ascertaining relative displacement of the sensing means with respect to at least one of the position elements.
-
80. A method according to claim 79, wherein the position elements are disposed on the surface as a regular array of dots, lines or other formations.
-
81. A method according to claim 79, wherein the position elements are disposed on the surface stochastically.
-
82. A method according to claim 1, wherein the movement data is generated by ascertaining a locus of the sensing device in relation to the surface by ascertaining relative movement of one or more motion sensing elements rotatably mounted to the sensing device for contact with the surface while the sensing device is used to draw the user input thereon.
-
83. A method according to claim 82, wherein the motion sensing elements include one or more rollerballs mounted for rotation within a constraining housing disposed substantially within the sensing device.
-
84. A method according to claim 83, wherein components of rotation of the rollerball, due to movement of the sensing device when drawing the user input onto the surface, are periodically measured.
-
85. A method according to claim 84, wherein the components of rotation of the rollerball due to movement of the sensing device by the user when drawing the user input onto the surface are measured by means of:
-
rollers disposed within the constraining housing for rotation, the rollers being configured to be driven by contact with the rotating rollerball;
oroptical sensing of rotation of the rollerball with respect to the constraining housing.
-
-
86. A method according to claim 1, wherein the movement includes at least one stroke.
-
87. A method according to claim 85, wherein a stroke is initialised when the pen is placed in an operative position relative to the interface surface and is completed when the pen is removed from the operative position.
-
88. A method according to claim 1, wherein the indicating data is indicative of a sequence of pen positions, the pen positions being determined by sensing of the coded data.
-
89. A method according to claim 88, wherein the indicating data is further indicative of the relative time at which each pen position is determined.
-
90. A method according to claim 1, wherein the indicating data is indicative of a number of strokes, each stroke being represented by a respective sequence of pen positions.
-
91. A method according to claim 90, wherein each stroke is further represented by an identifier indicative of an identity of the interface surface.
-
92. A method according to claim 90, the number of strokes representing handwritten text.
-
93. A system for enabling user interaction with computer software via:
-
an interface surface including;
human-readable information relating to the computer software; and
coded data indicative of at least one interactive element relating to the computer software, the information and coded data having been printed substantially simultaneously onto a substrate to form the interface surface; and
a sensing device that generates, using coded data sensed in response to placing the sensing device in an operative position relative to the interface surface, indicating data indicative of the at least one interactive element and movement of the sensing device relative to the interface surface;
the system including a computer system running the computer software, wherein the computer system;
(a) receives the indicating data from the sensing device; (b) determines movement of the sensing device relative to the interface surface using the indicating data; (c) identifies the at least one interactive element from the indicating data; and (d) operates the computer software at least partly in reliance on the movement, and in accordance with instructions associated with the at least one interactive element. - View Dependent Claims (94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184)
identify that the user has entered a hand-drawn mark by means of the sensing device; and
effect an operation associated with the checkbox field.
-
-
99. A system according to claim 98, wherein the computer system is configured to associate a true value with the checkbox field.
-
100. A system according to 99, wherein the computer system is configured to send data to the computer software indicative of at least the checkbox field.
-
101. A system according to claim 93, wherein the interactive element is a text field relating to the computer software, the computer system being configured to:
identify that the user has entered handwritten text data by means of the sensing device; and
effect an operation associated with the text field.
-
102. A system according to claim 101, the computer system being configured to convert the handwritten text data to computer text.
-
103. A system according to 102, the computer system being configured to associate the computer text with the text field.
-
104. A system according to claim 102, the computer system being configured to send data to the computer software indicative of at least the text field.
-
105. A system according to claim 93, wherein the interactive element is a signature field relating to the computer software, the computer system being configured to:
-
identify that the user has entered a handwritten signature by means of the sensing device; and
effect an operation associated with the signature field.
-
-
106. A system according to claim 105, the computer system being configured to send, to the computer software, data indicative of at least the signature field.
-
107. A system according to claim 105, the computer system being configured to verify that the signature is that of the user.
-
108. A system according to claim 107, the computer system being configured to use a signature key associated with the user to generate a digital signature of at least data indicative of a name and/or value of at last one field related to the computer software.
-
109. A system according to claim 108, the computer system being configured to associate the digital signature with the signature field.
-
110. A system according to claim 93, wherein the interactive element is a drawing field related to the computer software, the computer system being configured to:
-
identify that the user has entered a hand-drawn picture by means of the sensing device; and
effect an operation associated with the drawing field.
-
-
111. A system according to claim 110, the computer system being configured to send, to the computer software, data indicative of at least the drawing field.
-
112. A system according to any claim 93, further including the sensing device, wherein the sensing device includes a marking nib.
-
113. A system according to claim 93, further including the sensing device, wherein the sensing device contains identifying data indicative of an identity of the user.
-
114. A system according to claim 113, the computer system being configured to monitor the identifying data when the sensing device is in use.
-
115. A system according to 93, the computer system including a printer to print the information onto the interface surface on demand prior to use of the sensing device to sense the indicating data.
-
116. A system according to claim 115, wherein the printer includes a binding mechanism for binding multiple interface surfaces, defined by multiple corresponding pages, into a bound document.
-
117. A system according to claim 93, wherein the coded data is substantially invisible to an unaided human eye.
-
118. A system according to claim 93, the computer system being configured to retain a retrievable record of each interface surface printed, the interface surface being retrievable using the identity contained in its associated coded data.
-
119. A system according to claim 93, the information being sufficiently detailed in relation to the computer software that a user can interact with the computer system without the need for a separate display device.
-
120. A system according to claim 93, the system being configured to distribute a plurality of the interface surfaces using a mixture of multicast and pointcast communications protocols.
-
121. A system according to claim 93, wherein the coded data includes at least one tag, each tag being indicative of the interactive element.
-
122. A system according to claim 121, wherein each of the tags include identity data defining the at least one interactive element.
-
123. A system according to claim 121, wherein the surface is defined by a substrate.
-
124. A system according to claim 123, wherein the substrate is laminar.
-
125. A system according to claim 121, wherein the tags are disposed at predetermined positions on the surface.
-
126. A system according to claim 125, wherein the tags are disposed on the surface within a tessellated pattern comprising a plurality of tiles, each of the tiles containing a plurality of the tags.
-
127. A system according to claim 126, wherein the tiles interlock with each other to substantially cover the surface.
-
128. A system according to claim 127, wherein the tiles are all of a similar shape.
-
129. A system according to claim 128, wherein the tiles are triangular, square, rectangular or hexagonal.
-
130. A system according to claim 126, wherein the tags are disposed stochastically within each of the tiles.
-
131. A system according to claim 121, wherein each of the tags includes at least one common feature in addition to the identity data.
-
132. A system according to claim 131, wherein one or more common features are configured to assist finding and/or recognition of the tags by associated tag reading apparatus.
-
133. A system according to claim 131, wherein the one or more common features are represented format incorporating redundancy of information.
-
134. A system according to claim 133, wherein the at least one common feature is rotationally symmetric so as to be rotationally invariant.
-
135. A system according to claim 133, wherein the at least one common feature is ring-shaped.
-
136. A system according to claim 121, wherein each of the tags includes at least one orientation feature for enabling a rotational orientation of the tag being read to be ascertained.
-
137. A system according to claim 136, wherein the at least one orientation feature is represented in a format incorporating redundancy of information.
-
138. A system according to claim 137, wherein the at least one orientation feature is rotationally asymmetric.
-
139. A system according to claim 137, wherein the at least one orientation feature is skewed along its major axis.
-
140. A system according to claim 121, wherein each of the tags includes at least one perspective feature for enabling a perspective distortion of the tag being read to be ascertained.
-
141. A system according to claim 140, wherein the at least one perspective feature includes at least four sub-features, the relative positions of which define a non-degenerate quadrilateral.
-
142. A system according to claim 121, wherein each tag includes a plurality of tag elements, the first and second identity data each being defined by a plurality of the elements.
-
143. A system according to claim 142, wherein the tag elements are disposed in one or more arcuate bands around a central region of each tag.
-
144. A system according to claim 143, wherein there are a plurality of the arcuate bands disposed concentrically with respect to each other.
-
145. A system according to claim 142, wherein each element takes the form of a dot having a plurality of possible values.
-
146. A system according to claim 145, wherein the number of possible values is two.
-
147. A system according to claim 145, wherein when representing one of the possible values, the tag elements absorb, reflect or fluoresce electromagnetic radiation of a predetermined wavelength or range of wavelengths to a predetermined greater or lesser extent than the surface.
-
148. A system according to claim 145, wherein the possible values of the tag elements are defined by different relative absorption, reflection or fluorescence of electromagnetic radiation of a predetermined wavelength or range of wavelengths.
-
149. A system according to claim 145, wherein the tags are not substantially visible to an average unaided human eye under daylight or ambient lighting conditions.
-
150. A system according to claim 145, wherein the tags are slightly visible to an average unaided human eye under daylight or ambient lighting conditions.
-
151. A system according to claim 145, wherein the tags are visible to an average unaided human eye under daylight or ambient lighting conditions.
-
152. A system according to claim 121, wherein the first identity data is represented in a format incorporating redundancy of information.
-
153. A system according to claim 121, wherein the second identity data is represented in a format incorporating redundancy of information.
-
154. A system according to claim 121, wherein the tags are printed onto the surface by means of a printer.
-
155. A system according to claim 154, wherein the printer is an ink printer.
-
156. A system according to claim 155, wherein the tags are printed using ink that is absorbent or reflective in the ultraviolet spectrum or the infrared spectrum.
-
157. A system according to claim 154, wherein the printer also prints additional information onto the surface.
-
158. A system according to claim 157, wherein the additional information is printed onto the surface using colored or monochrome inks.
-
159. A system according to claim 158, wherein the additional information is printed onto the surface using one of the following combinations of colored inks:
-
CMY;
CMYK;
CMYRGB; and
spot colour.
-
-
160. A system according to claim 121, wherein at least a plurality of the tags are disposed stochastically upon the surface.
-
161. A system according to claim 121, wherein the tags are disposed in a regular array on the surface.
-
162. A system according to claim 161, wherein the array is triangular.
-
163. A system according to claim 161, wherein the array is rectangular.
-
164. A system according to claim 161, wherein the tags are tiled over the surface.
-
165. A system according to claim 121, further including additional non-tag information disposed on the surface.
-
166. A system according to claim 93, wherein the region is identified with sufficient precision to distinguish the region from 1.5×
- 1014 other regions.
-
167. A system according to claim 93, wherein any 10 millimeter diameter subregion of the region includes sufficient information to identify the region.
-
168. A system according to claim 167, wherein any 10 millimeter subregion of the region includes sufficient information to identify at least one point of the region.
-
169. A system according to claim 93, wherein the sensing device includes at least one acceleration measuring device for measuring acceleration of the sensing device as it is used to draw the user input onto the surface, wherein step (b) includes the substep of periodically sampling the acceleration of the sensing device as it is used to draw the user input onto the surface, and step (d) includes the substep of:
(i) ascertaining a locus of the sensing means in relation to the surface by ascertaining relative displacement of the sensing device due to its velocity within each sampling period.
-
170. A system according to claim 169, wherein the accelerometers are configured to measure at least two orthogonal components of acceleration.
-
171. A system according to claim 93, the printed information including position elements, the position elements being disposed on a surface, wherein step (b) includes the substep of periodically sensing the position elements with the sensing device as it is used to draw the user input onto the surface, and step (d) includes the substep of:
(i) ascertaining a locus of the sensing means in relation to the surface by ascertaining relative displacement of the sensing means with respect to the position elements during periods between reading thereof.
-
172. A system according to claim 171, wherein the position elements are disposed on the surface as a regular array of dots, lines or other formations.
-
173. A system according to claim 171, wherein the position elements are disposed on the surface stochastically.
-
174. A system according to claim 93, wherein step (d) includes the substep of:
(i) ascertaining a locus of the sensing means in relation to the surface by ascertaining relative movement of one or more motion sensing elements rotatably mounted to the sensing device for contact with the surface while the sensing device is used to draw the user input thereon.
-
175. A system according to claim 174, wherein the motion sensing elements include one or more rollerballs mounted for free rotation within a constraining housing disposed substantially within the sensing device.
-
176. A system according to claim 175, wherein substep (d)(i) includes the substep of periodically measuring components of rotation of the rollerball due to movement of the sensing device by the user when drawing the user input onto the surface.
-
177. A system according to claim 176, wherein the components of rotation of the rollerball due to movement of the sensing device by the user when drawing the user input onto the surface are measured by means of:
-
rollers disposed within the constraining housing for rotation driven by contact with the rotating rollerball;
oroptical sensing of rotation of the rollerball with respect to the constraining housing.
-
-
178. A system according to claim 93, wherein the movement includes at least one stroke.
-
179. A system according to claim 93, wherein a stroke is initialised when the pen is placed in an operative position relative to the interface surface and is completed when the pen is removed from the operative position.
-
180. A system according to claim 93, wherein the indicating data is indicative of a sequence of pen positions, the pen positions being determined by sensing of the coded data.
-
181. A system according to claim 180, wherein the indicating data is further indicative of the relative time at which each pen position is determined.
-
182. A system according to claim 93, wherein the indicating data is indicative of a number of strokes, each stroke being represented by a respective sequence of pen positions.
-
183. A system according to claim 182, wherein each stroke is further represented by an identifier indicative of an identity of the interface surface.
-
184. A system according to claim 182, the number of strokes representing handwritten text.
Specification