Method for producing a binary tree, pattern recognition and binary vector classification method using binary trees, and system for classifying binary vectors

• US 5,263,124 A
• Filed: 02/27/1991
• Issued: 11/16/1993
• Est. Priority Date: 02/27/1991
• Status: Expired due to Term

• Current Assignee: Neuralsystems Corporation
• Original Assignee: Neuralsystems Corporation
• Inventors: Chittenden, Constance T., Weaver, Charles S.
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US07/661,330
• Time in Patent Office: 993 Days
• Field of Search: 364/513, 382/14, 382/15, 382/30, 382/34, 382/36, 382/37, 382/39, 382/41, 382/49, 382/56
• US Class Current: 382/159
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers

Abstract

A binary tree and method of producing a binary tree are shown, together with artificial neural networks which include processing units of binary trees. The binary tree-producing method includes obtaining a set of binary training pattern vectors some of which are associated with a first pattern to be recognized, and the remainder of which are not associated with the first pattern. Those associated with the first pattern and the remainder are identified as category 1 and category 0 vectors, respectively. The set of vectors is used to generate a binary tree in computer memory, which tree includes a sequence of binary doublets each of which represents a tree node. One of four branch conditions is identified by each doublet including no branches, branch only left, branch only right or branch both left and right. The sequence of binary doublets is used to classify binary vectors. A hardware version of the tree may be implemented which includes a plurality of AND gates (1L, 1R, 2L, 2R, 3L and 5L) interconnected in an N-level binary tree (FIG. 3) to which N binary inputs (X₁, X₂ and X₃) are connected to separate levels of the tree. Leaf nodes of the AND gate binary tree are connected to an OR gate (20), and a start signal (S) is supplied to the root node (1) of the tree.

Claims

1. In a method for producing a binary tree for pattern recognition use, or the like, steps comprising

a) obtaining a set of binary training pattern vectors some of which are associated with a first pattern and the remainder of which are not associated with said first pattern, those training pattern vectors associated with the first pattern being identified as category 1 training pattern vectors and the remainder of the training pattern vectors which are not associated with said first pattern being identified as category 0 training pattern vectors, each said binary training pattern vector including the same number, N, of vector elements, and

b) using the set of binary training pattern vectors as inputs to a computer, generating a binary tree having at least N levels and storing the tree in computer memory, which binary tree comprises a sequence of binary two-bit doublets each of which doublets comprises a tree node of one of four types including "no branch", "branch only left", "branch only right" and "branch both left and right" node.

2. In a method as defined in claim 1 wherein step b) includes

using only category binary training pattern vectors from the set of vectors as computer inputs, generating an unpruned binary tree and storing the same in computer memory,

using only category 0 binary training pattern vectors from the set of vectors as computer inputs, traversing the unpruned binary tree and identifying nodes traversed by said category 0 vectors as activated nodes, and nodes not traversed by said category 0 vectors as unactivated nodes,

removing from the unpruned binary tree unactivated nodes which are connected to a next higher level unactivated node, and

converting any remaining unactivated "branch both left and right" nodes to "no branch" nodes.

3. In a method as defined in claim 2 including,

during generation of the unpruned binary tree, counting the number of times that category 1 binary training pattern vectors traverse each node of the tree,

while traversing the unpruned binary tree by category 0 vectors, counting the number of times each node is traversed,

changing nodes as identified as activated to unactivated when the ratio of category 0 to category 1 traverses is below a threshold value.

4. In a method as defined in claim 2 wherein the set of binary training pattern vectors is supplied only once to the computer for generation of a pruned binary tree.

5. In a method as defined in claim 1 wherein the category 1 and category 0 binary training pattern vectors are randomly ordered, and wherein

step b) comprises repeatedly supplying the set of binary training pattern vectors as inputs to the computer for generating a binary tree which requires no subsequent pruning.

6. In a method as defined in claim 1 wherein the binary doublets for each level of the binary tree are stored in contiguous memory addresses without spacing or separators between doublets.

7. In a method as defined in claim 1 including supplying vectors to be classified to a computer with the sequence of binary doublets stored in computer memory and distinguishing between category 1 and category 0 vectors supplied thereto.

11. In a method as defined in claim 1 wherein step a) includes optically reading a plurality of input characters to be recognized to form binary vectors, some of which vectors comprise category 1 vectors and the other of which comprise category 0 vectors.

12. In a method as defined in claim 1 wherein step a) includes obtaining the set of binary training pattern vectors from the inputs and associated output of a processing element of a trained artificial neural network in response to different inputs supplied to the neural network, the inputs and output of which processing element are in binary or bi-polar form, the inputs to the processing element identifying the pattern vectors and the associated outputs identify the vector category, the resultant binary tree being useable as a substitution for said processing element of the trained artificial neural network.

8. In a method as defined in claim including reordering vector elements of all of the binary training vectors before generating a binary tree for reducing the number of nodes in the generated binary tree.

9. In a method as defined in claim 8 including,

determining the total number of 1s and total number of 0s at each vector element for the set of binary training pattern vectors, and

using the above-determined totals, reordering the vector elements of all of the binary training pattern vectors in descending order of one of the totals of 1s and 0s before generating a binary tree.

10. In a method as defined in claim 8 wherein the reordering step includes reordering the vector elements of all binary training pattern vectors in descending order of one of the total of 1s and total of 0s at each vector element for the set of binary training pattern vectors.

13. In a pattern recognition method, or the like, employing a set of training pattern vectors which set includes a plurality of groups of vectors, each of which groups is associated with a pattern to be recognized, steps during a training mode including,

using training pattern vectors from every group thereof, producing one first binary tree for each pattern to be recognized, each said first binary tree being adapted for recognizing a different one of said patterns to be recognized, and

using training pattern vectors from every group thereof, producing an associated second binary tree for each said first binary tree, each said second binary tree being adapted for recognizing every pattern except the pattern recognized by the associated first binary tree, each said binary tree having N input levels where N is a whole number.

15. In a method as defined in claim 13 which includes reordering elements of the training pattern vectors before producing said binary trees for reducing the number of nodes included in said trees.

16. In a method as defined in claim 15 wherein training pattern vectors are in binary form and each includes N vector elements.

17. In a method as defined in claim 16 including operating mode steps for recognizing binary pattern vectors having N vector elements, steps comprising

supplying the N vector elements of the binary pattern vector to be recognized to the N input levels of each of the first and second binary trees, and

in the presence of an output from one of the first binary trees, classifying the vector in accordance with the pattern recognized by said binary tree.

18. In a method as defined in claim 17 including

in the absence of an output from one of said first binary trees, changing the vector element of the binary pattern vector to be recognized at the binary tree level at which the path traversed by said vector terminates,

then, in the presence of an output from one of the first binary trees, classifying the vector in accordance with the pattern recognized by said binary tree.

14. In a pattern recognition method, or the like, employing a set of training pattern vectors which set includes a plurality of groups of vectors, each of which groups is associated with a pattern to be recognized, steps during a training mode including,

using training pattern vectors from every group thereof, producing one first binary tree for each pattern to be recognized, each said first binary tree being adapted for recognizing a different one of said patterns to be recognized, and

using training pattern vectors from every group thereof, producing an associated second binary tree for each said first binary tree, each said second binary tree being adapted for recognizing every pattern except the pattern recognized by the associated first binary tree, each said binary tree having N input levels where N is a whole number,

the steps of producing first and second binary trees includes generating in computer memory a sequence of binary two-bit doublets for each of said trees, each of which doublets comprises a tree node of one of four types including "no branch", "branch only left", "branch only right" and "branch both left and right".

19. In a pattern recognition method, or the like, employing a set of binary training pattern vectors, a vector element reordering method including,

a) obtaining a measure of variability for each vector element of at least some binary training pattern vectors included in said set, and

b) reordering the vector elements of the binary training pattern vectors dependent upon the measure of variability obtained in step a).

20. In a method as defined in claim 19 wherein step b) includes reordering the vector elements in order of increasing variability.

21. In a method as defined in claim 19 wherein step a) comprises

c) locating clusters of vectors in the set of binary training pattern vectors,

d) obtaining a measure of entropy of each vector element for each cluster, and

e) using the measures of entropy obtained in step d), obtaining a measure of entropy of each vector element over all the clusters,

and wherein step b) comprises reordering vector elements of the set dependent upon the entropy measurements obtained in step e).

22. In a method as defined in claim 21 wherein step b) includes reordering vector elements in ascending order of the measure of entropy obtained in step e).

23. In a method as defined in claim 21 wherein step d) includes for each vector element of each cluster, calculating the probability that the element equals 1,

for each vector element of each cluster, calculating the probability that the element equals 0, and

employing the calculated probabilities to obtain said measure of entropy of each vector element for each cluster.

24. In a pattern recognition method, or the like, employing a set of training pattern vectors which set includes a group of vectors associated with a pattern category to be recognized, the remainder of which vectors are not associated with said pattern category,

steps during a training mode including,

a. using said set of training pattern vectors, producing a first binary tree for recognizing pattern category vectors,

b. using said set of training pattern vectors, producing a second binary tree adapted for recognizing vectors not included in said group associated with the pattern category,

steps during pattern vector classification mode including,

c. supplying a vector to be classified to said first and second binary trees,

d. in the presence of a 1 output from said first binary tree, identifying the vector as belonging to the pattern category, and in the presence of a 1 output from said second binary tree, identifying the vector as not belonging to the pattern category,

e. in the presence of a 0 output from both of the first and second binary trees, changing the vector element of the vector at the tree level at which the path in the trees traversed by the vector terminates, and

f. repeating steps d and e until the vector is identified as either belonging to the pattern category or not belonging to the pattern category.

25. A method of classifying binary vectors by use of a binary tree which tree comprises a sequence of binary two-bit doublets each of which doublets comprises a node of one of four types including no branch, branch only left, branch only right, and branch both left and right, and wherein the binary vector specifies a path to be followed in the binary tree, binary vector elements 0 and 1 identifying branch left and branch right paths, respectively, said method comprising:

sequentially shifting binary doublets from the binary tree starting at the root node of the tree,

identifying those binary tree doublets in the path specified by the vector to be classified,

if the identified binary tree doublet identifies a no branch node, classifying the vector as one to be identified by the binary tree, and

if the identified binary tree doublet identifies a node which does not include a branch in the direction indicated by the associated vector element, then classifying the vector as one which is not identified by the binary tree.

26. A method as defined in claim 25 including,

if the identified binary tree doublet at the tree level equal to the number of elements in the binary vector identifies a node which includes a branch in the direction indicated by the associated vector element, then classifying the vector as one to be identified by the binary tree.

27. A system for classifying binary vectors which produces a first signal when the binary vector is identified as belonging to a class to be identified and produces a second signal when the binary vector is one which is identified as not belonging to said class, said system comprising,

tree memory means containing a binary tree comprising a sequence of binary two-bit doublets each of which doublets comprises a node of one of four types including no branch, branch only left, branch only right, and branch both left and right,

input memory means containing a binary vector which specifies a path to be followed in the binary tree,

logic means,

means for supplying binary doublets from said tree memory means and vector elements from said input memory means to said logic means,

said logic means in response to binary doublets and vector elements supplied thereto producing a first signal when the binary vector supplied thereto from said input memory means belongs to the class to be identified and producing a second signal when the binary vector does not belong to said class.

28. A system as defined in claim 27 wherein said logic means controls the supply of binary doublets from said tree memory means and vector elements from said input memory means to said logic means.