Methods, Architecture, and Apparatus for Implementing Machine Intelligence and Hierarchical Memory Systems

• US 20080201286A1
• Filed: 02/29/2008
• Published: 08/21/2008
• Est. Priority Date: 12/10/2004
• Status: Active Grant

• Current Assignee: Numenta, Inc.
• Original Assignee: Numenta, Inc.
• Inventors: Hawkins, Jeffrey, George, Dileep
• Granted Patent: US 8,175,981 B2
• Time in Patent Office: Days
• US Class Current: 706/46
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Sophisticated memory systems and intelligent machines may be constructed by creating an active memory system with a hierarchical architecture. Specifically, a system may comprise a plurality of individual cortical processing units arranged into a hierarchical structure. Each individual cortical processing unit receives a sequence of patterns as input. Each cortical processing unit processes the received input sequence of patterns using a memory containing previously encountered sequences with structure and outputs another pattern. As several input sequences are processed by a cortical processing unit, it will therefore generate a sequence of patterns on its output. The sequence of patterns on its output may be passed as an input to one or more cortical processing units in next higher layer of the hierarchy. A lowest layer of cortical processing units may receive sensory input from the outside world. The sensory input also comprises a sequence of patterns.

Claims

1. A computer-based memory system comprising:

a first lower level processing unit receiving a first portion of an input data and generating a first output representing first probabilities that patterns and sequences in the first portion of the input data correspond to first stored patterns and sequences;

a second lower level processing unit receiving a second portion of the input data and generating a second output representing second probabilities that patterns and sequences in the second portion of the input data correspond to second stored patterns and sequences; and

an upper level processing unit associated with the first and second lower level processing units, the upper level processing unit generating a third output based on the first and the second outputs, the third output representing third probabilities of causes to the input data.

2. The computer-based memory system of claim 1, said memory system further comprising:

sensory units coupled to the first lower level processing unit and the second lower level processing unit for providing the input data to the first lower level processing unit and the second lower level processing unit.

3. The computer-based memory system of claim 1, wherein the first lower level processing unit, the second lower level processing unit, and the upper level processing unit each comprises a probability table for generating the first output, the second output or the third output.

4. The computer-based memory system of claim 1, wherein the upper level processing unit generates a prediction output based on the first output and the second output at a first time, the prediction output representing predicted input data at a second time subsequent to the first time, the first lower level processing unit receiving the prediction output for generating the first output at the second time to generate the first output at a third time subsequent to the second time.

5. The computer-based memory system of claim 1, wherein the first lower level processing unit, the second lower level processing unit, and the upper level processing unit form a Bayesian belief net.

6. The computer-based memory system of claim 1, wherein the first lower level processing unit generates a prediction output based on the first portion of the input data at a first time, the prediction output representing predicted input data at a second time subsequent to the first time, the prediction output fed back to the first lower level processing unit at the second time to generate the first output at a third time subsequent to the second time.

7. The computer-based memory system of claim 6, wherein the prediction output is used for resolving ambiguities in the first portion of the input data received at the first lower level processing unit.

8. The computer-based memory system of claim 6, wherein the prediction output resolves ambiguities in the first portion of the input data received at the first lower level processing unit.

9. The computer-based memory system of claim 1, wherein the upper level processing module associates the causes of the input data to object names.

10. The computer-based memory system of claim 1, further comprising an intermediate level processing module coupled between the first lower level processing module and the upper level processing module, the intermediate level processing module generating a fourth output, the upper level processing module receiving the fourth output and generating the third output based on the fourth output.

11. The computer-based memory system of claim 1, wherein the upper level processing unit determines whether a combination of the first output and the second output corresponds to third stored patterns and sequences, the first and second stored patterns and sequences representing a first level of structures in the input data, and the third stored patterns and sequences representing a second level of structures in the input data that are formed by the first level of primitive structures.

12. The computer-based memory system of claim 1, the first level of structures comprise vertical lines, horizontal lines, corners, boxes, and parallel lines;

and the second level of structures comprise letters.

13. A computer-based memory system, comprising:

a lower level processing unit receiving an input data and generating a first output representing probabilities that patterns and sequences in the input data correspond to first stored patterns and sequences, the first stored patterns and sequences representing a first level of structures in the input data; and

an upper level processing unit associated with the lower level processing units, the upper level processing unit generating a second output representing probabilities that patterns and sequences in the first output correspond to second stored patterns and sequences, the second stored patterns and sequences representing a second level of structures in the input data that are formed by the first level of structures.

14. The memory system of claim 13, further comprising:

sensory units coupled to the lower level processing unit for providing the input data to the lower level processing unit.

15. The computer-based memory system of claim 13, wherein the lower level processing unit and the upper level processing unit each comprises a probability table for generating the first output and the second output.

16. The computer-based memory system of claim 13, wherein the upper level processing unit generates a prediction output based on the first output at a first time, the prediction output representing predicted input data at a second time subsequent to the first time, the lower level processing unit receiving the prediction output for generating the first output at the second time to generate the first output at a third time subsequent to the second time.

17. The computer-based memory system of claim 16, wherein the prediction output is used for resolving ambiguities in the input data at the lower level processing unit.

18. The computer-based memory system of claim 13, wherein the lower level processing unit and the upper level processing unit form a Bayesian belief net.

19. The computer-based memory system of claim 13, wherein the lower level processing unit generates a prediction output based on the input data at a first time, the prediction output representing predicted input data at a second time subsequent to the first time, the prediction output fed back to the lower level processing unit at the second time to generate the first output at a third time subsequent to the second time.

20. The computer-based memory system of claim 19, wherein the prediction output resolves ambiguities in the input data.

21. The computer-based memory system of claim 13 wherein the second output indicates object names.

22. The computer-based memory system of claim 13, further comprising an intermediate level processing module coupled between the lower level processing module and the upper level processing module, the intermediate level processing module receiving the first output and generating a third output based on the first output, the upper level processing module receiving the third output and generating the second output based on the third output.