High efficiency learning network

• US 5,473,730 A
• Filed: 11/09/1993
• Issued: 12/05/1995
• Est. Priority Date: 11/09/1993
• Status: Expired due to Fees

• Current Assignee: AT&T IPM Corp. (AT&T, Inc.)
• Original Assignee: AT&T IPM Corp. (AT&T, Inc.)
• Inventors: Simard, Patrice Y.
• Primary Examiner(s): Davis, George B.
• Application Number:
  US08/150,677
• Time in Patent Office: 756 Days
• Field of Search: 395/23, 395/24, 395/27, 395/22
• US Class Current: 706/20
• CPC Class Codes: G06N 3/063 using electronic means

Abstract

Nodal outputs are discretized to values of S2ⁿ where n is an integer and S is equal to +1 or -1. During forward propagation, this offers the advantage of forming a product of a nodal output and a weight using a simple shift operation rather than a multiply operation. Replacing multiply operations with shift operations through out a neural network improves response times and permits building larger networks that have broader applicability. Training is also improved by increasing the efficiency of backward propagation. The multiplications involved in backward propagation are reduced To shift operations by discretizing the errors associated with each node so that they are represented as 2ⁿ where n is an integer and S is equal to +1 or -1.

Claims

1. A computational network that produces a plurality of network outputs in response to a plurality of network inputs, comprising:

a plurality of first layer computational nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero integer power of 2; and

;

a plurality of second layer computational nodes forming a second layer of the computational network, each of said plurality of second layer computational nodes receiving a plurality of inputs to produce a second layer output, each of said plurality of inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being used to produce at least one of a plurality of network outputs.

2. The computational network of claim 1, wherein said first layer output is in the form of S2ⁿ where n is a nonzero integer and S is ±

1.

3. The computational network of claim 2, further comprising product means for producing said product by shifting said weight value n times.

4. The computational network of claim 2, further comprising product means for producing said product by summing n and an exponent portion of said weight value.

5. The computational network of claim 1, wherein said plurality of network inputs are discretized to an integer power of 2.

6. A computational network that produces a plurality of network outputs in response to a plurality of network inputs, comprising:

a plurality of first layer computational nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one first layer input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero integer power of 2; and

;

a plurality of second layer computational nodes forming a second layer of the computational network, each of said plurality of second layer computational nodes receiving a plurality of second layer inputs to produce a second layer output, each of said plurality of second layer inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being produced by using a sum of said plurality of inputs as an operand of a first activation function, said second layer output being used to produce at least one of a plurality of network outputs.

7. The computational network of claim 6, wherein said first layer output is in the form of S2ⁿ where n is a nonzero integer and S is ±

1.

8. The computational network of claim 7, further comprising product means for producing said product by shifting said weight value n times.

9. The computational network of claim 8, wherein said product means shifts said weight value in a first direction for positive values of n and in a second direction for negative values of n.

10. The computational network of claim 7, further comprising product means for producing said product by summing n and an exponent portion of said weight value.

11. The computational network of claim 6, wherein said plurality of network inputs are discretized to an integer power of 2.

12. The computational network of claim 6, wherein each of said first layer computational nodes produces said first layer output by using a sum of a plurality of first layer inputs as an operand of a second activation function.

13. The computational network of claim 12, wherein said first layer output is in the form of S2ⁿ where n is a nonzero integer and S is ±

1.

14. The computational network of claim 13, further comprising product means for producing said product by shifting said weight value n times.

15. The computational network of claim 14, wherein said product means shifts said weight value in a first direction for positive values of n and in a second direction for negative values of n.

16. The computational network of claim 13, further comprising product means for producing said product by summing n and an exponent portion of said weight value.

17. The computational network of claim 12, wherein said plurality of network inputs are discretized to an integer power of 2.

18. The computational network of claim 12, wherein said first and second activation functions are the same.

19. A method of recognizing a user'"'"'s input, comprising the steps of:

forming a first plurality of values representative of the user'"'"'s input, said first plurality of values being discretized to a nonzero integer power of 2;

producing a first plurality of products using said first plurality of values and a first plurality of weights;

producing a second plurality of products using said first plurality of values and a second plurality of weights;

using a first computational node belonging to a plurality of first layer computational nodes to produce a first output by forming a sum of said first plurality of products and by using said sum of said first plurality of products as an operand of a first activation function;

using a second computational node belonging to said plurality of first layer computational nodes to produce a second output by forming a sum of said second plurality of products and by using said sum of said second plurality of products as an operand of a second activation function;

forming a first product using said first output and a first weight value;

forming a second product using said second output and a second weight value;

using a second layer computational node belonging to a plurality of second layer computational nodes to produce a second layer output by forming a sum of said first and second products and by using said sum of said first and second products as an operand of a third activation function; and

using said second layer output to produce at least one of a plurality of network outputs that identify the user'"'"'s input.

20. The method of claim 19, wherein said user'"'"'s input is an utterance.

21. The method of claim 19, wherein said user'"'"'s input is a handwritten symbol.

22. The method of claim 19, further comprising adjusting said first plurality of weights using an error value discretized to an integer power of 2.

23. The method of claim 22, further comprising adjusting said first plurality of weights using a learning rate discretized to an integer power of 2.

24. The method of claim 22 wherein said error value is in the form S2ⁿ where n is an integer and S is ±

1.

25. The method of claim 22, wherein said step of adjusting said first plurality of weights comprises forming a product of said error value and a derivative value discretized to a power of 2, said discretized derivative value obtained by using said sum of said first plurality of products as an argument of a derivative function, said derivative function being a derivative of an approximation of said first activation function.

26. The method of claim 25, wherein said derivative value is in the form S2ⁿ where n is an integer and S is ±

1.

27. The method of claim 25, wherein said derivative value is discretized to a value of 1 or 0.

28. The method of claim 25, further comprising adjusting said first plurality of weights using a learning rate discretized to an integer power of 2.

29. The method of claim 19, further comprising adjusting said first plurality of weights using a derivative value discretized to a power of 2, said discretized derivative value obtained by using said sum of said first plurality of products as an argument of a derivative function, said derivative function being a derivative of an approximation of said first activation function.

30. The method of claim 29, wherein said derivative value is in the form S2ⁿ where n is an integer and S is ±

1.

31. The method of claim 29, further comprising adjusting said first plurality of weights using a learning rate discretized to a power of 2.