Belief propagation in a hierarchical temporal memory based system

• US 7,899,775 B2
• Filed: 01/11/2007
• Issued: 03/01/2011
• Est. Priority Date: 02/10/2006
• Status: Active Grant

• Current Assignee: Numenta, Inc.
• Original Assignee: Numenta, Inc.
• Inventors: Hawkins, Jeffrey, George, Dileep
• Primary Examiner(s): Holmes; Michael B.
• Assistant Examiner(s): Kennedy; Adrian L
• Application Number:
  US11/622,458
• Publication Number:
  US 20070192271A1
• Time in Patent Office: 1,510 Days
• Field of Search: 706/52, 706/46, 706/20
• US Class Current: 706/52
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A hierarchy of computing modules is configured to (i) learn a cause of input data sensed over space and time, and (ii) determine a cause of novel sensed input data dependent on the learned cause. The hierarchy has a first level of computing modules and a second level of at least one computing module, wherein a computing module in the first level is configured to output to the computing module in the second level a first set of values representing probabilities of possible causes of input data received by the system.

Claims

1. A computer-implemented system, comprising:

a hierarchy of computing modules configured to receive first input data to learn spatial patterns and temporal sequences in the first input data in a learning stage, the hierarchy in an inference stage subsequent to the learning stage further configured to receive second input data and generate output information about the second input data, the hierarchy comprising;

a plurality of first computing modules at a first level of the hierarchy, each of the first computing modules configured to receive a part of the second input data and comprising;

a coincidence detector configured to generate coincidence information about one or more coincidences detected in the part of the second input data; and

a sequence learner configured to generate first information about temporal sequences of coincidences responsive to receiving the coincidence information, and

at least one second computing module at a second level of the hierarchy, the at least one second computing module configured to generate second information about spatial patterns and temporal sequences in the second input data based on the first information.

2. The system of claim 1, wherein the coincidence detector is configured to perform spatial pooling on the received part of the second input data.

3. The system of claim 1, wherein the coincidence information is generated at the coincidence detector based on spatial patterns learned by the first computing modules in the learning stage.

4. The system of claim 1, wherein the at least one second computing module in the second level is configured to generate the second information dependent on spatial patterns and temporal sequences learned in the learning stage.

5. The system of claim 1, wherein the sequence learner is further configured to generate the first information dependent on a conditional probability table.

6. The system of claim 1, wherein at least one of the first computing modules in the first level and the at least one second computing module in the second level is implemented at least in part in at least one of software and hardware.

7. The system of claim 1, wherein an input range of the second computing module in the second level is greater than an input range of the each of the first computing modules in the first level.

8. A computer-implemented method of recognizing an object, comprising:

receiving a first input data at a hierarchy of computing modules, the hierarchy comprising a plurality of first computing modules at a first level of the hierarchy and at least one second computing module at a second level of the hierarchy in a learning stage;

learning spatial patterns and temporal sequences in the first input data by the hierarchy of computing modules in the learning stage;

receiving second input data at the hierarchy of computing modules in an inference stage subsequent to the learning stage;

generating coincidence information at a coincidence detector of each of the first computing modules responsive to receiving a part of the first input;

generating first information about temporal sequences of coincidences at a sequence learner of each of the first computing modules responsive to receiving the coincidence information; and

generating second information by the at least one second computing module responsive to receiving the first information.

9. The computer-implemented method of claim 8, further comprising performing spatial pooling of the coincidences at the coincidence detector responsive to receiving the part of the first input.

10. The computer-implemented method of claim 8, wherein the coincidence information is generated at the coincidence detector based on spatial patterns learned by the first computing modules in the learning stage.

11. The computer-implemented method of claim 10, wherein the second information is generated by the at least one second computing module based on spatial patterns and temporal sequences learned in the learning stage.

12. The computer-implemented method of claim 10, wherein the sequence learner generates the first information based on a conditional probability table.

13. A computer-readable medium having instructions therein that are executable by a processor, the instructions comprising instructions to:

receive a first input data at a hierarchy of computing modules, the hierarchy comprising a plurality of first computing modules at a first level of the hierarchy and at least one second computing module at a second level of the hierarchy in a learning stage;

learn spatial patterns and temporal sequences in the first input data by the hierarchy of computing modules in the learning stage;

receive second input data at the hierarchy of computing modules in an inference stage subsequent to the learning stage;

generate coincidence information at a coincidence detector of each of the first computing modules responsive to receiving a part of the first input;

generate first information about temporal sequences of coincidences at a sequence learner of each of the first computing modules responsive to receiving the coincidence information; and

generate second information by the at least one second computing module responsive to receiving the first information.

14. The computer-readable medium of claim 13, further comprising instructions to:

perform spatial pooling of the coincidences at the coincidence detector responsive to receiving the part of the first input.

15. The computer-readable medium of claim 13, wherein the coincidence information is generated at the coincidence detector based on spatial patterns learned by the first computing modules in the learning stage.

16. The computer-readable medium of claim 13, wherein the second information is generated by the at least one second computing module based on spatial patterns and temporal sequences learned in the learning stage.