System and method for handwriting matching using edit distance computation in a systolic array processor

• US 5,757,959 A
• Filed: 04/05/1995
• Issued: 05/26/1998
• Est. Priority Date: 04/05/1995
• Status: Expired due to Term

• Current Assignee: Panasonic Corporation Of North America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Technologies, Inc. (Panasonic Holdings Corporation)
• Inventors: Lopresti, Daniel P.
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US08/417,100
• Time in Patent Office: 1,147 Days
• Field of Search: 382/186, 382/187, 382/189, 382/315, 395/800
• US Class Current: 382/186
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 30/2272 with lexical matching

Abstract

Apparatus and a method for comparing an electronic handwritten pattern to a stored string are provided. The string includes a group of portions, each having at least one stroke. Movement of a stylus forms the pattern, and a sequence of strokes is generated. Each stroke represents a stylus movement within a predetermined alphabet. The sequence of strokes has a plurality of portions. A linear systolic array processor determines an edit distance between the string and the pattern. The processor compares a first portion of the string to a first portion of the pattern. A plurality of edit distance components are generated based on the comparison. Each component corresponds to a different set of operations that transforms the first portion of the stored string into the first portion of the pattern. The components are calculated based on a further comparison between additional portions of the stored string and the pattern. The component which has a minimum value is selected. The comparison is performed between each respective portion of the pattern and the corresponding portion of the stored string. The total edit distance is based on the component selected during a last comparison between a last portion of the stored string and a last portion of the pattern.

Claims

1. Apparatus for comparing an electronic handwritten pattern to a stored string representing a handwritten document comprising at least one of the group consisting of a paragraph, a page, a sentence, text and graphics, the stored string including a group of portions, each portion having at least one stroke, the apparatus comprising:

means for monitoring movement of a stylus that forms the electronic handwritten pattern, the handwritten pattern being shorter than the stored string, the monitoring means generating therefrom an input handwritten sequence of strokes, each stroke representing one of a plurality of predetermined stylus movements within a predetermined alphabet, the input handwritten sequence of strokes having a plurality of portions;

a linear systolic array processor for determining an edit distance between a substring of the stored string and the input handwritten sequence of strokes, thereby to search the stored string of handwritten strokes for an occurrence of the input handwritten sequence of strokes, the substring including a subset of the group of portions, the linear systolic array processor including;

means for comparing a first one of the group of portions of the stored string to a first one of the plurality of portions of the input handwritten sequence of strokes to form a comparison, and for generating a plurality of edit distance components based on the comparison, each edit distance component corresponding to a respectively different set of operations that transforms the first portion of the stored string into the first portion of the pattern, wherein at least one of the plurality of edit distance components is calculated based on a further comparison between a second portion of the stored string and a second portion of the handwritten sequence of strokes,

wherein the means for comparing generates the plurality of edit distance components including a group of initial edit components related to the stored string which are set to 0,

means for selecting the edit distance component which has a minimum value among the plurality of edit distance components,

means for determining the edit distance based on a plurality of comparisons, wherein the plurality of comparisons includes the comparison and the further comparison, and

means for determining which substring of the stored string differs from the input handwritten sequence of strokes by the smallest edit distance, thereby locating the one substring within the stored string that most closely matches the input handwritten sequence of strokes that is shorter than the stored string.

2. Apparatus according to claim 1, wherein each stroke in the sequence of strokes is represented by a respective feature vector that defines an orientation, a velocity and an acceleration.

3. Apparatus according to claim 1, wherein the linear systolic array processor comprises a plurality of processing elements connected in a line, such that the strokes in the sequence of strokes are successively transmitted across the line of processing elements in a first direction, while the corresponding strokes within the stored string are successively transmitted across the line of processing elements in a second direction opposite the first direction.

4. Apparatus according to claim 3, wherein each respective processing element comprises:

means for receiving a stroke within the sequence of strokes from a previous one of the plurality of processing elements;

means for receiving a stroke within the stored string from a next one of the plurality of processing elements; and

means for comparing the stroke within the sequence to the stroke within the stored string to produce a comparison and for adjusting the respective edit distance for the stored string based on the comparison.

5. Apparatus according to claim 4, wherein the respective comparing means in each processing element comprises:

means for determining a first edit distance component based on a given stroke being present at the specific position within the sequence of strokes but absent from the specific position within the stored string;

means for determining a second edit distance component based on the given stroke being present at the specific position within the stored string, but a different stroke being substituted therefor at the specific position within the sequence of strokes;

means for determining a third edit distance component based on the given stroke being present at a specific position within the stored string but being absent from the specific position within the sequence of strokes;

means for selecting a minimum component from the group consisting of the first, second and third edit distance components.

6. Apparatus according to claim 5, wherein each comparing means further comprises:

means for transmitting the minimum component to a previous one of the plurality of processing elements and to a next one of the plurality of processing elements.

7. Apparatus according to claim 4, wherein each respective processing element includes:

means for transmitting a stroke within the sequence of strokes to the next one of the plurality of processing elements after the comparing means executes the comparison; and

means for transmitting a stroke within the stored string to the previous one of the plurality of processing elements after the comparing means executes the comparison.

8. Apparatus according to claim 1, wherein the linear systolic array processor comprises a plurality of processing elements connected in a line, such that each one of the plurality of processing elements includes means for storing a respective one of the strokes in the sequence of strokes, and the corresponding strokes within the stored string are successively transmitted across the line of processing elements.

9. Apparatus according to claim 8, wherein the strokes within the stored string are transmitted across the line of processing elements in a further sequence opposite the sequence of strokes of the handwritten pattern.

10. Apparatus according to claim 8, wherein each respective one of the processing elements comprises:

means for receiving a stroke within the stored string from a previous one of the plurality of processing elements; and

means for comparing the respective stroke from the sequence stored within the one processing element to the stroke from the stored string, and for adjusting the respective edit distance for the one stored string based on the comparison.

11. Apparatus according to claim 10, wherein each respective processing element includes:

means for transmitting a stroke within the sequence of strokes to a next one of the plurality of processing elements after the comparing means executes the comparison.

12. Apparatus according to claim 10, wherein the respective comparing means in each processing element comprises:

deletion evaluation means for determining a first edit distance component based on a first stroke being present at the specific position within the sequence of strokes but being absent from the specific position within the stored string;

1-for-1-substitution evaluation means for determining a second edit distance component based on the first stroke being present at the specific position within the stored string, but a second stroke being substituted therefor at the specific position within the sequence of strokes;

insertion evaluation means for determining a third edit distance component based on the first stroke being present at a specific position within the stored string but being absent from the specific position within the sequence of strokes;

means for selecting a minimum component from the group which includes the first, second and third edit distance components; and

means for adding the minimum component to a previously generated edit distance component for the stored string during the further comparison.

13. Apparatus according to claim 12, wherein each processing element further comprises:

means for transmitting the minimum component to a next one of the plurality of processing elements.

14. Apparatus according to claim 13, wherein the minimum component transmitted by each processing element to the next one of the plurality of processing elements is used as an input edit distance component by the next processing element, and each processing element further comprises:

means for receiving the input edit distance component from the previous processing element;

means for storing the input edit distance component for a period of time during which the processing element processes the first stroke, and for transmitting the input edit distance component to the 1-for-1-substitution evaluation means after the period of time,

wherein the 1-for-1-substitution evaluation means uses the input edit distance component to compute the second edit distance component.

15. Apparatus according to claim 14, wherein:

the comparing means further comprises 1-for-2-substitution evaluation means for determining a 1-for-2-substitution edit distance component based on two strokes being present at the specific position within the stored string, but a single stroke being substituted therefor at the specific position within the sequence of strokes;

the group from which the minimum component is selected further includes the 1-for-2-substitution edit distance component;

the storing means includes;

(a) means for storing the input edit distance component for a further period of time during which the processing element processes a further stroke which follows the first stroke, and

(b) means for transmitting the input edit distance component to the 1-for-2-substitution evaluation means after the further period of time;

wherein the 1-for-2-substitution evaluation means uses the input edit distance component to compute the 1-for-2-substitution edit distance component.

16. Apparatus according to claim 12, wherein the plurality of processing elements are ordinally numbered first through N^th, where N is an integer, and each processing element further comprises:

means for transmitting the minimum component to an i+2^th one of the plurality of processing elements positioned after the next one of the plurality of processing elements, where i is an integer greater than two and less than N-1; and

means for receiving an input edit distance component from an i-2^th one of the plurality of processing elements positioned before the previous one of the plurality of processing elements.

17. Apparatus according to claim 16, wherein:

the comparing means further comprises 2-for-1-substitution evaluation means for determining a 2-for-1 substitution edit distance component based on a single stroke being present at the specific position within the stored string, but two strokes being substituted therefor at the specific position within the sequence of strokes;

the group from which the minimum component is selected further includes the 2-for-1 substitution edit distance component; and

the apparatus further comprises;

(a) means for storing the input edit distance component received from the i-2^th processing element for a period of time during which the i^th processing element processes one stroke of the stored string;

(b) means for transmitting the input edit distance component received from the i-2^th processing element to the 2-for-1-substitution evaluation means after the period of time;

wherein the 2-for-1-substitution evaluation means uses the input edit distance component received from the i-2^th processing element to compute the 2-for-1 substitution edit distance component.

18. Apparatus according to claim 17, wherein:

the comparing means further comprises 2-for-2-substitution evaluation means for determining a 2-for-2-substitution edit distance component based on two strokes being present at the specific position within the stored string, but two further strokes being substituted therefor at the specific position within the sequence of strokes;

the group from which the minimum component is selected further includes the 2-for-2-substitution edit distance component; and

the apparatus further comprises;

(a) means for storing the input edit distance component received from the i-2^th processing element for a further period of time during which the processing element processes a further stroke which follows the one stroke; and

(b) means for transmitting the input edit distance component received from the i-2^th processing element to the 2-for-2-substitution evaluation means after the further period of time;

wherein the 2-for-2-substitution evaluation means uses the input edit distance component received from the i-2^th processing element to compute the 2-for-2-substitution edit distance component.

19. Apparatus according to claim 12, wherein each of the plurality of processing elements includes means for adjusting the edit distance when the sequence of strokes is identified as being different from the stored string because a subset of the sequence of strokes representing a single symbol in the stored string is identified as being two symbols other than the single symbol.

20. Apparatus according to claim 12, wherein each of the plurality of processing elements includes means for adjusting the edit distance when the sequence of strokes is identified as being different from the stored string because a subset of the sequence of strokes representing two symbols in the stored string is identified as being a single symbol other than either of the two symbols.

21. Apparatus according to claim 12, wherein each of the plurality of processing elements includes means for adjusting the edit distance when the sequence of strokes is identified as being different from the stored string because a subset of the sequence of strokes representing two given symbols in the stored string is identified as representing two predetermined symbols other than the two given symbols.

25. Apparatus according to claim 1, wherein the stored string represents a database of handwritten information, and the systolic array processor determines a subset of the database which most closely matches the sequence of strokes.

22. Apparatus for comparing an electronic handwritten pattern to a stored string representing a handwritten document comprising at least one of the group consisting of a paragraph, a page, a sentence, text and graphics, the stored string including a group of strokes, comprising:

means for monitoring movement of a stylus that forms the electronic handwritten pattern, the handwritten pattern being shorter than the stored string, the monitoring means generating therefrom an input handwritten sequence of strokes, each stroke representing one of a plurality of predetermined stylus movements within a predetermined alphabet;

a linear systolic array processor for determining an edit distance between a substring of the stored string and the sequence of strokes, the substring including a subset of the group of strokes, the linear systolic array processor comprising a plurality of processing elements connected in a line, such that each one of the plurality of processing elements includes means for storing a respective one of the strokes in the input handwritten sequence of strokes, and the group of strokes within the stored string are successively transmitted across the line of processing elements, in order to search the stored string of handwritten strokes for an occurrence of the input handwritten sequence of strokes, each of the plurality of processing elements including;

means for receiving first and second input edit distance components from first and second previous ones of the plurality of processing elements, and for storing the first and second input edit distance components for a period of time during which the processing element processes two of the strokes within the input handwritten sequence of strokes,

means for comparing a portion of the stored string to a portion of the pattern to form a comparison, and for generating a plurality of edit sequence components based on (1) the comparison and (2) the first and second input edit distance components, each edit sequence component identifying a respectively different set of operations that transforms the portion of the stored string into the portion of the pattern, each edit sequence component having a respective edit distance components,

wherein the means for comparing generates the plurality of edit distance components including a group of initial edit components related to the stored string which are set to 0,

means for selecting the edit sequence component for which the edit distance component has a minimum value; and

means for determining which substring of the stored string differs from the input handwritten sequence of strokes by the smallest edit distance, thereby locating the one substring within the stored string that most closely matches the input handwritten sequence of strokes that is shorter than the stored string.

23. Apparatus for comparing an electronic handwritten pattern to a stored string representing a handwritten document comprising at least one of the group consisting of a paragraph, a page, a sentence, text and graphics, the stored string including a group of strokes, comprising:

means for monitoring movement of a stylus that forms the electronic handwritten pattern, and for generating therefrom an input handwritten sequence of strokes, each stroke representing one of a plurality of predetermined stylus movements within a predetermined alphabet;

a linear systolic array processor for determining an edit distance between the stored string and the input handwritten sequence of strokes, thereby to search the stored string of handwritten strokes for an occurrence of the input handwritten sequence of strokes, the linear systolic array processor comprising a plurality of processing elements connected in a line, such that each one of the plurality of processing elements includes means for storing a respective one of the strokes in the input handwritten sequence of strokes, and the corresponding strokes within the stored string are successively transmitted across the line of processing elements, each of the plurality of processing elements including;

(a) means for determining a first edit distance component based on a given stroke being present at the specific position within the input handwritten sequence of strokes but being absent from the specific position within the stored string;

(b) means for determining a second edit distance component based on the given stroke being present at the specific position within the stored string, but a different stroke being substituted therefor at the specific position within the input handwritten sequence of strokes;

(c) means for determining a third edit distance component based on two strokes being present at the specific position within the stored string, but a single stroke being substituted therefor at the specific position within the input handwritten sequence of strokes;

(d) means for determining a fourth edit distance component based on a single stroke being present at the specific position within the stored string, but two strokes being substituted therefor at the specific position within the input handwritten sequence of strokes;

(e) means for determining a fifth edit distance component based on two strokes being present at the specific position within the stored string, but two further strokes being substituted therefor at the specific position within the input handwritten sequence of strokes;

(f) means for determining a sixth edit distance component based on the given stroke being present at a specific position within the stored string but being absent from the specific position within the input handwritten sequence of strokes;

(g) means for selecting a minimum component from the group consisting of the first through sixth edit distance components,

wherein the systolic array processor generates the first through sixth edit distance components to include a group of initial edit components related to the stored string which are set to 0.

24. A method for comparing an electronic handwritten pattern to a stored string representing a handwritten document comprising at least one of the group consisting of a paragraph, a page, a sentence, text and graphics, the stored string including a group of strokes, comprising the steps of:

(1) monitoring movement of a stylus that forms the electronic handwritten pattern, the handwritten pattern being shorter than the stored string, and generating therefrom an input handwritten sequence of strokes, each stroke representing one of a plurality of predetermined stylus movements within a predetermined alphabet;

(2) transmitting the group of strokes and the input handwritten sequence of strokes to a linear systolic array processor comprising a plurality of cascade connected processing elements,

(3) storing a respective one of the strokes in the input handwritten sequence of strokes within each one of the plurality of processing elements

(4) transmitting the group of strokes within the stored string successively across the cascade connected processing elements, each of the plurality of processing elements performing the steps of;

(a) receiving first and second input edit distance components from first and second previous ones of the plurality of processing elements,

(b) storing the first and second input edit distance components for a period of time during which the processing element processes two of the strokes within the input handwritten sequence of strokes,

(c) comparing a portion of the stored string to a portion of the pattern to form a comparison,

(d) generating a plurality of edit sequence components based on (1) the comparison and (2) the first and second input edit distance components, each edit sequence component identifying a respectively different set of operations that transforms the portion of the stored string into the portion of the pattern, each edit sequence component having a respective edit distance component, and

(e) selecting the edit sequence component for which the edit distance component has a minimum value;

(5) determining an edit distance between a subset of the stored string and the input handwritten sequence of strokes, the substring including a subset of the group of strokes, based on the edit sequence component selected by a last one of the plurality of processing elements;

including a step of generating the edit distance to include an initial edit component related to the stored string which is set to 0, and

(6) determining which substring of the stored string differs from the input handwritten sequence of strokes by the smallest edit distance, thereby locating the one substring within the stored string that most closely matches the input handwritten sequence of strokes that is shorter than the stored string, thereby to search the stored string of handwritten strokes for an occurrence of the sequence of strokes.