Tracking motion of a writing instrument

• US 20050073508A1
• Filed: 07/17/2003
• Published: 04/07/2005
• Est. Priority Date: 08/18/1998
• Status: Active Grant

• Current Assignee: Candledragon, Inc.
• Original Assignee: Digital Ink Incorporated
• Inventors: Schiller, Ilya, Goldman, Andrew M., Funk, Ethan A., Liberman, Sergey, Pittel, Arkady, Shleppi, Garry, Selitsky, Simon, Subach, Vladimir V., Stein, Mario A., Reznik, Leonid
• Granted Patent: US 7,268,774 B2
• Time in Patent Office: Days
• US Class Current: 345/175
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06F 1/3259 Power saving in cursor control device, e.g. mouse, joystick, trackball
• CPC Class Codes: G06F 3/03542 Light pens for emitting or receiving light
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0386 for light pen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Motion of a writing instrument is tracked from sensors located in the vicinity. The signals generated from the sensors are processed and used in a wide variety of ways.

Claims

1. A method comprising

conveying light from a moving light source on the writing instrument as an indication of the location and path of the writing instrument on a two dimensional writing surface,

sensing the light at two or more sensors and generating a sequence of signals representative of the sensed light, and

applying a technique to increase the stability of subpixel reading.

2. The method of claim 1 in which the technique is based on optics.

3. The method of claim 1 in which the optics are configured to enhance the uniformity of signal response of the sensors.

4. The method of claim 3 in which the lens comprises a spherical lens.

5. The method of claim 3 in which the lens comprises an aspheric lens.

6. The method of claim 3 in which the sensors comprise arrays of sensitive pixel elements.

7. The method of claim 3 in which the sensors comprise analog sensors.

8. The method of claim 1 in which the technique is based on algorithmic processing of the generated signals.

9. The method of claim 8 in which the algorithmic processing comprises mapping the signal response of the sensors based on parameters associated with the writing instrument.

10. The method of claim 8 in which the technique is also based on optics.

11. The method of claim 8 in which the sensors comprise arrays of sensitive pixel elements.

12. The method of claim 1 in which the technique is implemented in digital hardware.

13. The method of claim 1 in which the technique is implemented in analog circuitry.

14. The method of claim 1 in which the technique comprises an algorithmic technique that also reduces the effect of variations of the light intensity based on other than dimensional effects.

15. The method of claim 1 in which

the sensors comprise pixel arrays,

the signals are grouped in frames, and

the signal processing technique comprises processing of multiple frames to cancel noise.

16. The method of claim 1 in which the light conveyed from the moving writing instrument is modulated at a frequency related to the rate at which the signals are generated by the sensors.

17. The method of claim 1 in which

the light conveyed from the writing instrument is modulated at a frequency, and

the sensor signals are chopped at the frequency of modulation.

18. The method of claim 17 in which opposite gains are applied to each of the chopped signals depending on the on or off state of the light conveyed from the writing instrument that corresponds to the signals.

19. The method of claim 17 in which the frame rate is varied.

20. The method of claim 18 in which the chopped signals are integrated over time.

21. The method of claim 1 in which

the light conveyed from the writing instrument includes a strong short pulse imposed on the modulation frequency, and

a phase lock loop determines the modulation frequency from the sensor signals, and

the sensor signal is sampled at the times triggered by the phase lock loop during the duration of the strong short pulse.

22. The method of claim 1 in which characteristics of the conveyed light are used for synchronization between the writing instrument and the sensors.

23. The method of claim 1 in which the conveyed light includes periods of lower frequency modulation and bursts of higher frequency modulation, and the sensor signal associated with the higher frequency bursts is used to lock onto a modulation clock.

24. A method comprising

conveying light from a moving writing instrument in a time-changing pattern of directions,

sensing the light at two or more sensors located at two or more different locations spaced from the writing instrument, and

determining the location of the writing instrument by detecting a phase difference between signals measured at the two or more sensors.

25. The method of claim 24 in which the time-changing pattern of directions includes a rotating pattern with respect to an X-Y plane on which the writing instrument is moving.

26. The method of claim 25 in which the signal radiated in the positive X direction is in phase quadrature to the signal radiated in the Y direction.

27. Apparatus comprising

sensors configured to receive light from a writing instrument moving across an X-Y writing surface, and featuring instability in subpixel reading. optics configured to enhance optical power of the light received from the writing instrument.

28. The apparatus of claim 27 in which the optics comprise a ball lens or an aspherical lens.

29. The apparatus of claim 27 in which the optics include a single spherical lens and the lens and the corresponding sensor are together configured to enhance the optical power of light received at large angles or longer distances or at disadvantageous positions of the writing instrument.

30. The apparatus of claim 27 in which the optics include a special lens configured to enhance optical power of the light received from a location on the X-Y surface that is beyond a predetermined position.

31. The apparatus of claim 27 in which the optics include two cylindrical lenses, one nearer the sensor to project light horizontally onto sensor, and the other positioned to collect light in the Z-axis dimension, the other lens having a body that is bent around the first lens.

32. The apparatus of claim 27 also including algorithmic processes that enhance the immunity of the signals to variations in the intensity of the received light caused by distance from or tilt of the writing instrument.

33. The apparatus of claim 32 in which the processes determine the integral power of the overall signal distribution on the sensor and calculate a subpixel position based on half of the integral power position.

34. The apparatus of claim 32 in which the processes use a polynomial approximation on the signal distribution and calculate a subpixel position as a position of approximated maximum.

35. The apparatus of claim 34 also including a calibration procedure to produce parameters to be used in combination with data from the sensors.

36. The apparatus of claim 35 in which the calibration parameters correct for non linearity of our sensors, and the algorithmic processes use a quasi triangulation technique to determine a position of the writing instrument.

37. The apparatus of claim 36 in which the calibration parameters correct for non linearity of our sensors and the algorithmic processes determine the position of the writing instrument using polynomial series, when coefficients in these polynomials are determined during the calibration procedure.

38. A method comprising

receiving light from a moving writing instrument at an array of sensing elements of a sensor,

reading the sensing elements in sequence to generate a sequence of signals indicative of light sensed by the elements of the array, and

resetting each elements after it is read and before at least some of the other elements in the array are read.

39. The method of claim 38 in which the array comprises a CMOS or CCD position sensor.

40. The method of claim 38 in which each of the elements is reset before the next element in the sequence is read.

41. The method of claim 38 in which all of the elements are read before all of the elements are reset.

42. A method comprising

conveying light from a moving hand-held instrument,

sensing the light at two or more sensors, each of the two or more sensors comprising a two-dimensional array of sensing elements,

generating signals representing the two-dimensional locations on the arrays of light that is sensed, and

determining a sequence of three-dimensional positions of the moving writing instrument based on the signals.

43. Apparatus comprising

a writing instrument system that can track three-dimensional motion of the writing instrument.

44. The apparatus of claim 43 in which the system includes sensors having 3 linear arrays.

45. The apparatus of claim 43 in which the system includes sensors that are a two-dimensional array or a one-dimensional array.

46. Apparatus comprising

a writing instrument including

an elongated housing configured to be hand-held,

a light source in the housing, and

a lens in the housing configured to receive light from the light source and convey the light through a free-air path to optical sensors spaced from the writing instrument, the lens being configured to enable light to be directed parallel to the writing surface no matter what the orientation or position of the writing instrument to the writing surface.

48. The apparatus of claim 46 in which the lens is configured to internally reflect and concentrate the light and to emit it by reflection from a reflective external surface of the lens.

49. The apparatus of claim 46 in which the lens comprises a cylindrical body having an upper surface that receives the light and a lower annular surface that reflects the light toward the optical sensors.

50. The apparatus of claim 46 in which the light source emits light in a direction toward a writing end of the writing instrument.

51. The apparatus of claim 48 in which the reflective external surface comprises a conical surface oriented at a certain degree angle to a longitudinal axis of the writing instrument.

112. The apparatus of claim 46 in which a light guide delivers light to the tip of the pen and conveys it outwardly in a disk-like pattern.

47. Apparatus comprising

a writing instrument including

an elongated housing configured to be hand-held, and

a light source in the housing, the light source comprising a an array of light sources arranged around an axis of the writing instrument and configured to emit light in a direction normal to the axis.

52. Apparatus comprising

a writing instrument including

an elongated housing configured to be hand-held, and

a light source in the housing, the light source being arranged to emit light in a direction parallel to a longitudinal axis of the writing instrument.

53. The apparatus of claim 52 in which the light source comprises one or more LEDs.

54. The apparatus of claim 52 in which the light source comprises a ring of light sources.

55. Apparatus comprising

a writing instrument,

a light-source in the writing instrument configured to convey light to sensors spaced from the writing instrument, and

a device configured to turn the light source on and off in response to a user applying pressure from the writing instrument to a writing surface, the switch being configured so that an amount of pressure required to trigger the switch is not so large as to disrupt normal writing motion of the writing instrument on the writing surface.

56. The apparatus of claim 55 in which the writing instrument includes a ballpoint cartridge having a writing point and the device is positioned at the opposite end of the cartridge from the writing point.

57. The apparatus of claim 56 in which the device could comprise a switch.

58. The apparatus of claim 56 in which the device could comprise a pressure sensor.

59. A method comprising

sending light from a moving writing instrument, the light being indicative of a position and path of the writing instrument, and

directly sensing, at one or more sensors spaced from the writing instrument, angles from which light is received from the writing instrument.

60. The method of claim 59 in which the angles are directly sensed by an array of sensitive elements of a sensing device.

61. The method of claim 60 in which the sensing device comprises a CMOS or CCD device.

62. The method of claim 60 in which the sensing-device-comprises a PSD.

63. Apparatus comprising

a writing instrument, and

a holder having a receptacle for receiving at least a portion of the writing instrument for storage of the writing instrument,

the writing instrument and the holder containing respective elements that enable wireless transmission of signals associated with motion of the writing instrument and tracking of the writing motion based on the signals.

64. The apparatus of claim 63 in which the holder comprises a pen cap.

65. The apparatus of claim 63 in which the holder includes a clip configured to attach the holder to a stack of pages or to a notebook.

66. The apparatus of claim 63 in which the holder includes at least two light sensors and a processor that processes signals from the light sensors to determine a sequence of positions of the writing instrument.

67. The apparatus of claim 63 in which the holder includes a receptacle for holding the writing instrument and enabling recharging of batteries in the writing instrument.

68. Apparatus comprising

a writing instrument,

an element that enables wireless transmission of a signal associated with motion of the writing instrument and tracking of the writing motion based on the signal,

the element being built into a cell phone, a PDA, a webpad, or a clipboard.

69. Apparatus comprising

two optical sensors separated by a known distance and arranged to

receive light from a source associated with a writing instrument,

determine directions from which the light is received relative to a known direction,

provide signals representing the directions for use in determining a sequence of locations of the writing instrument,

at least one of the two sensors comprising a CMOS of CCD array.

70. The apparatus of claim 69 in which the CMOS or CCD array comprises a linear array of sensor elements.

71. The apparatus of claim 69 in which the CMOS or CCD array comprises a two-dimensional array of sensor elements.

72. Apparatus comprising

a holder for a writing instrument,

the holder having elements used in wireless transmission of signals associated with motion of a writing instrument in the vicinity of the holder,

the holder having a mechanism for attaching the holder to a writing substrate in an orientation that enables the elements to be used in conjunction with the wireless transmission.

73. The apparatus of claim 72 in which the clipping mechanism includes a switch to activate functions of a processor in the holder when the clipping mechanism is manipulated.

74. The apparatus of claim 72 in which one of the functions comprises a new page function.

75. Apparatus comprising

a holder for a writing instrument,

the writing instrument including electronic circuitry configured to be used in conjunction with tracking writing motion of the writing instrument, the writing instrument including a rechargeable battery connected to power the electronic circuitry,

the holder including a receptacle for the writing instrument and a recharging circuit connected to recharge the battery when the writing instrument is in the receptacle.

76. Apparatus comprising

a CMOS sensor adapted to receive light associated with motion of a writing instrument and to provide signals indicative of an angle of receipt of the light with respect to a known direction, and

a lens aligned to direct the received light to the CMOS array.

77. The apparatus of claim 76 in which the lens comprises a half-ball lens.

78. The apparatus of claim 76 in which the lens comprises an aspherical lens.

79. The apparatus of claim 76 in which the lens is optimized for collection of light from an area in which the motion of the writing instrument occurs.

80. The apparatus of claim 76 in which the lens comprises a flat field lens.

81. The apparatus of claim 76 in which the lens comprises a Fresnel lens.

82. The apparatus of claim 76 in which the lens system is configured to collect light in a dimension normal to a plane of motion of the writing instrument and to project the light onto the sensor in a direction parallel to the plane of motion.

83. A method comprising

positioning a writing instrument at a succession of positions on a writing surface,

generating signals at sensors from light received from the writing instruments at the succession of positions, and

determining calibration parameters for the writing instrument for use in calibrating a process that determines the positions of the writing instrument as it is being moved.

84. The method of claim 83 in which the calibration parameters comprise coefficients used in polynomial series that are part of the position determining process.

85. The method of claim 83 in which the positions do not lie on a regular rectangular grid.

86. The method of claim 83 in which (I need help in reciting the pseudo geometrical concept.)

87. A method comprising

identifying locations on a writing surface that correspond to input elements to be entered into an electronic device, the writing surface being non-electronic and separate from the electronic device,

using a writing instrument to point to selected ones of the identified locations corresponding to input elements to be entered, and

sensing the locations at which the writing instrument is pointing and entering the corresponding data into the electronic device.

88. The method of claim 87 in which the writing surface includes a sheet of paper.

89. The method of claim 87 in which the input elements comprise characters of language.

90. The method of claim 87 in which the input elements comprise commands.

91. The method of claim 87 in which the input elements are printed on the writing surface.

92. A method comprising

moving a writing instrument across a non-electronic writing surface to indicate a path, and

remotely sensing the path use in entering the path into an electronic device that is separate from the writing surface.

93. A method comprising

modulating light that is conveyed from a moving writing instrument to light sensors spaced from the writing instrument at a predetermined frequency, and

using a phase locked loop associated with the sensors to lock onto the phase of the modulated light.

94. Apparatus comprising

circuitry for tracking writing motion of a writing instrument using wireless transmission of signals between the writing instrument and a stationary element, the stationary element including a main processor and a separate preprocessor,

the preprocessor being connected to perform at least data capture with respect to motion of the writing instrument,

the main processor being connected to perform at least data communication with respect to the tracking.

95. The apparatus of claim 94 in which the preprocessor is also connected to perform user interface functions and sub-pixel data storage.

96. The apparatus of claim 94 in which the main processor is also connected to perform background cancellation and sub-pixel calculation.

97. The apparatus of claim 94 in which the main processor is also connected to perform conversion of sub-pixel data into paper coordinates.

98. The apparatus of claim 95 in which data storage is done in the form of paper coordinates.

99. Apparatus comprising

a writing instrument and a sensor,

the writing instrument including a reflective element configured to reflect light received from outside of the writing instrument to the sensor for use in tracking motion of the writing instrument.

100. The apparatus of claim 99 also including a mechanism to enable the reflective element to reflect the light to the sensor when the writing instrument is being used for writing and to disable the reflective element from reflecting light to the sensor when the writing instrument is not being used for writing.

101. A method comprising

receiving light from a moving writing instrument at a light sensor having an array of sensitive pixel elements,

determining the location in the array at which the maximum intensity of light has been received from the writing instrument, the location being determined with sub-pixel accuracy.

102. The method of claim 101 in which the sub-pixel location is determined by

determining the integer pixel location that is closest to the subpixel location, and

finding a fractional center of gravity of a subarray that is centered on the integer pixel location.

103. A method comprising

indicating locations on a non-electronic surface that correspond to inputs to an electronic device,

detecting the locations and inputting them into the electronic device.

104. Apparatus comprising

a sensor configured to detect light from a moving writing instrument, and

a clip for clipping paper on which the writing instrument is to be moved to the sensor.

105. The apparatus of claim 104 in which the mechanism comprises part of a clipboard or a notebook.

106. The apparatus of claim 104 in which the clip includes a mechanism for enabling a user to cause the clip to grip or to release the paper.

107. The apparatus of claim 106 in in which the mechanism comprises an activation button and a spring.

108. The apparatus of claim 107 in which the mechanism includes a lever operated by the button.

109. The apparatus of claim 108 in which the lever is configured to rotate in response to the button.

110. The apparatus of claim 107 in which the button is configured to be pushed.

111. The apparatus of claim 107 in which the button is configured to be pulled.