Trainable hierarchical memory system and method

• US 20070005531A1
• Filed: 06/06/2005
• Published: 01/04/2007
• Est. Priority Date: 06/06/2005
• Status: Active Grant

• Current Assignee: Numenta, Inc.
• Original Assignee: Numenta, Inc.
• Inventors: Hawkins, Jeffrey, George, Dileep
• Granted Patent: US 7,739,208 B2
• Time in Patent Office: Days
• US Class Current: 706/16
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Memory networks and methods are provided. Machine intelligence is achieved by a plurality of linked processor units in which child modules receive input data. The input data are processed to identify patterns and/or sequences. Data regarding the observed patterns and/or sequences are passed to a parent module which may receive as inputs data from one or more child modules. the parent module examines its input data for patterns and/or sequences and then provides feedback to the child module or modules regarding the parent-level patterns that correlate with the child-level patterns. These systems and methods are extensible to large networks of interconnected processor modules.

Claims

1. A memory network of interconnected processing modules, comprising:

a child processing module for receiving input data, the child processing module generating lower level characterization data that represent patterns and/or sequences found in the input data, the child module outputting the lower level characterization data;

a parent processing module for receiving the lower level characterization data, the parent processing module generating higher level characterization data that represent patterns and/or sequences found in the lower level characterization data;

the parent module providing feedback to the child module for linking the higher level characterization data to the lower level characterization data.

2. The memory network of claim 1, wherein:

the child processing module produces the lower level characterization data by identifying, labeling, and quantifying the relative and/or absolute frequency of patterns and/or sequences in the input data.

3. The memory network of claim 1, wherein:

the parent processing module produces the higher level characterization by identifying, labeling, and quantifying the relative and/or absolute frequency of patterns and/or sequences in the lower level characterization data.

4. The memory network of claim 1, wherein:

the parent module provides feedback information about a currently active higher level data characterization.

5. The memory network of claim 4, wherein:

the child module comprises a memory that stores a correlation between the lower level data characterization and the corresponding currently active higher level data characterization based on the feedback information from the parent module.

6. The memory network of claim 5, wherein the correlation comprises a conditional probability table.

7. The memory network of claim 4, wherein:

the child module comprises a memory that stores a transition of a currently active lower level data characterization along with the feedback information.

8. The memory network of claim 7, wherein the correlation comprises a conditional transition probability matrix.

9. The memory network of claim 1, wherein the input data comprises data from one or more sensors.

10. The memory network of claim 1, wherein:

the input data comprise grandchild processing module output data that characterize patterns and/or sequences observed in a data set received by a grandchild module.

11. A method for training a memory network comprising the step of:

receiving training data comprising a world of patterns at a child processing module;

generating lower level characterization data that represent the patterns in the input data,

outputting the lower level characterization data to a parent module;

generating higher level characterization data that represent patterns in the lower level characterization data;

providing feedback to the child module for linking the higher level characterization data to the lower level characterization data.

12. The method of claim 11, further comprising the step of querying the memory network about the world of patterns.

13. The method of claim 11, further comprising the step of

storing in a child memory a correlation between the lower level data characterization and a corresponding currently active higher level data characterization based on the feedback from the parent module.

14. The method of claim 13, wherein the correlation comprises a conditional probability matrix.

15. The method of claim 14, wherein the conditional probability matrix is used to fill-in missing information in the input data.

16. The method of claim 11 wherein the training patterns comprises video sequences.

17. The method of claim 11 wherein the training data comprise visual data and/or patterns of the visual world.

18. The method of claim 14, further comprising the steps of:

receiving input data comprising a specific pattern at the child module; and

using the conditional probability matrix to predict the higher level characterization data that will be produced by the parent module in response to the specific pattern.

19. The method of claim 18, wherein the input data comprises visual patterns.

20. A memory network of claim 17 where the relative movement of visual patterns and sensors was used to improve the accuracy of answering recognition related queries of claim 18.

21. An interpretive framework method for training a memory system, comprising the steps of:

processing a discrete stream of input data at each of two or more lower tier processing modules, wherein the processing at each lower tier processing module comprises identifying and labeling lower tier patterns and/or sequences in the input data stream;

passing the lower tier sequence identity label from each the two or more lower tier processing modules to an upper tier processing module;

processing the lower tier sequence identity label data in the upper tier processing module, the processing of the lower tier sequence identity label data comprising identifying and labeling upper tier sequences and quantifying the relative and/or absolute frequency of the upper tier sequences; and

providing feedback to the lower tier modules regarding the relative and/or absolute frequency of one of the upper tier sequences in relation to the occurrence of a specific lower tier sequence.

22. The method of claim 21, wherein the step of processing discrete streams of input data further comprises:

quantizing a plurality of first input data at a first of the two or more lower tier processing modules, the quantizing comprising categorizing the plurality of first input data such that similar input data are categorized together;

identifying and quantifying the relative and/or absolute frequency of sequences in the plurality of first input data; and

assigning a lower tier identity label to each sequence.

23. The method of claim 21, wherein:

the step of processing the lower tier sequence identity label further comprises;

quantizing the lower tier sequence identity labels at the upper tier processing module, the quantizing comprising categorizing the lower tier sequence identity labels such that similar lower tier sequence identity labels are categorized together;

identifying and quantifying the relative and/or absolute frequency of sequences in the lower tier sequence identity labels; and

assigning an upper tier identity label to each sequence; and

24. The method of claim 21, wherein:

the step of processing discrete streams of input data further comprises quantizing a plurality of first input data at a first of the two or more lower tier processing modules, the quantizing comprising categorizing the plurality of first input data such that similar input data are categorized together;

identifying and quantifying the relative and/or absolute frequency of sequences in the plurality of first input data; and

assigning a lower tier identity label to each sequence; and

the step of processing the lower tier sequence identity label further comprises quantizing the lower tier sequence identity labels at the upper tier processing module, the quantizing comprising categorizing the lower tier sequence identity labels such that similar lower tier sequence identity labels are categorized together;

identifying and quantifying the relative and/or absolute frequency of sequences in the lower tier sequence identity labels; and

assigning an upper tier identity label to each sequence.

25. A data processing structure, comprising:

an upper tier processing module;

a first and a second lower tier processing module, each of which receives a discrete stream of input data, the first and the second lower tier processor modules each comprising a memory and a processor that categorizes the received input data into discrete categories and identifies, labels, and quantifies the relative and/or absolute frequency of sequences observed in the categorized input data;

two-way links that separately connect the first lower tier processing module to the upper tier processing module and the second lower tier processing module to the upper tier processing module.

26. A memory network, comprising:

a first lower tier processing module having a first receptive field size and a second lower tier processing module having a second receptive field size;

an upper tier processing module having a third receptive field size that at least as large as the sum of the first and the second receptive field sizes;

external inputs feeding a discrete input data stream to each of the first and the second lower tier processing modules;

data links between the upper tier processing module and the lower tier processing modules, wherein each data link passes processed lower tier sequence information from one of the first tier processing modules to the second tier processing module and feedback information from the second tier module to the one first tier processing module, the lower tier sequence information comprising an identity label and a relative and/or absolute frequency for each data sequence experienced by the one first tier processing module and the feedback information comprising a relative and/or absolute frequency correlation between each data sequence experienced by the one first tier processing module and a second tier sequence experienced by the second tier module.