Methods, architecture, and apparatus for implementing machine intelligence and hierarchical memory systems

• US 8,175,981 B2
• Filed: 02/29/2008
• Issued: 05/08/2012
• Est. Priority Date: 12/10/2004
• Status: Active Grant

• Current Assignee: Numenta, Inc.
• Original Assignee: Numenta, Inc.
• Inventors: Hawkins, Jeffrey C, George, Dileep
• Primary Examiner(s): Chen, Alan
• Assistant Examiner(s): Wong, Lut
• Application Number:
  US12/040,849
• Publication Number:
  US 20080201286A1
• Time in Patent Office: 1,530 Days
• Field of Search: 706/12, 706/45
• US Class Current: 706/12
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Sophisticated memory systems and intelligent machines may be constructed by creating an active memory system with a hierarchical architecture. Specifically, a system may comprise a plurality of individual cortical processing units arranged into a hierarchical structure. Each individual cortical processing unit receives a sequence of patterns as input. Each cortical processing unit processes the received input sequence of patterns using a memory containing previously encountered sequences with structure and outputs another pattern. As several input sequences are processed by a cortical processing unit, it will therefore generate a sequence of patterns on its output. The sequence of patterns on its output may be passed as an input to one or more cortical processing units in next higher layer of the hierarchy. A lowest layer of cortical processing units may receive sensory input from the outside world. The sensory input also comprises a sequence of patterns.

Claims

1. A computer-based memory system comprising:

a first lower level processing unit receiving a first portion of an input data and generating a first output representing first probabilities that sequences of patterns in the first portion of the input data correspond to first learned sequences of patterns, the first learned sequences of patterns including a first level of structures;

a second lower level processing unit receiving a second portion of the input data and generating a second output representing second probabilities that sequences of patterns in the second portion of the input data correspond to second learned sequences of patterns, the second learned sequences of patterns including the first level of structures; and

an upper level processing unit associated with the first and second lower level processing units, the upper level processing unit generates a prediction by determining whether sequences of combinations of the first output and the second output correspond to third learned sequences of patterns, wherein the third learned sequences of patterns include a second level of structures, and wherein the prediction is indicative of subsequent patterns to be received at the first and second lower level processing units or a subsequent combination of the first output and the second output to be received at the upper level processing unit.

2. The computer-based memory system of claim 1, said memory system further comprising:

sensory units coupled to the first lower level processing unit and the second lower level processing unit for providing the input data to the first lower level processing unit and the second lower level processing unit.

3. The computer-based memory system of claim 1, wherein the first lower level processing unit, the second lower level processing unit, and the upper level processing unit each comprises a probability table for generating the first output, the second output or the third output.

4. The computer-based memory system of claim 1, wherein the first lower level processing unit, the second lower level processing unit, and the upper level processing unit form a Bayesian belief net.

5. The computer-based memory system of claim 1, wherein the first lower level processing unit generates a prediction output based on the first portion of the input data at a first time, the prediction output representing predicted input data at a second time subsequent to the first time, the prediction output fed back to the first lower level processing unit at the second time to generate the first output at a third time subsequent to the second time.

6. The computer-based memory system of claim 5, wherein the prediction output resolves ambiguities in the first portion of the input data received at the first lower level processing unit.

7. The computer-based memory system of claim 1, wherein the upper level processing module associates further generates an output indicating causes of the input data.

8. The computer-based memory system of claim 1, the first level of structures comprise vertical lines, horizontal lines, corners, boxes, and parallel lines;

and the second level of structures comprise letters.

9. A computer-implemented method, comprising:

at a first lower level processing unit, receiving a first portion of an input data and generating a first output representing first probabilities that sequences of patterns in the first portion of the input data correspond to first learned sequences of patterns, the first learned sequences of patterns including a first level of structures;

at a second lower level processing unit, receiving a second portion of the input data and generating a second output representing second probabilities that sequences of patterns in the second portion of the input data correspond to second learned sequences of patterns, the second learned sequences of patterns including the first level of structures; and

at an upper level processing unit associated with the first and second lower level processing units, generating a prediction by determining whether sequences of combinations of the first output and the second output correspond to third learned sequences of patterns, wherein the third learned sequences of patterns include a second level of structures, and wherein the prediction is indicative of subsequent patterns to be received at the first and second lower level processing units or a subsequent combination of the first output and the second output to be received at the upper level processing unit.

10. The method of claim 9, further comprising:

at sensory units, generating the input data to the first lower level processing unit and the second lower level processing unit.

11. The method of claim 9, wherein the first lower level processing unit, the second lower level processing unit, and the upper level processing unit each comprises a probability table for generating the first output, the second output or the third output.

12. The method of claim 9, wherein the first lower level processing unit, the second lower level processing unit, and the upper level processing unit form a Bayesian belief net.

13. The method of claim 9, further comprising generating a prediction output at the first lower level processing unit based on the first portion of the input data at a first time, the prediction output representing predicted input data at a second time subsequent to the first time, the prediction output fed back to the first lower level processing unit at the second time to generate the first output at a third time subsequent to the second time.

14. The method of claim 13 wherein the prediction output resolves ambiguities in the first portion of the input data received at the first lower level processing unit.

15. The method of claim 9, further comprising generating an output indicating causes of the input data at the upper level processing module.

16. The method of claim 9, wherein the first level of structures comprise vertical lines, horizontal lines, corners, boxes, and parallel lines;

and the second level of structures comprise letters.