Trainable hierarchical memory system and method

• US 20100191684A1
• Filed: 03/31/2010
• Published: 07/29/2010
• Est. Priority Date: 06/06/2005
• Status: Active Grant

• Current Assignee: Numenta, Inc.
• Original Assignee: Numenta, Inc.
• Inventors: Hawkins, Jeffrey C., George, Dileep
• Granted Patent: US 8,103,603 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Memory networks and methods are provided. Machine intelligence is achieved by a plurality of linked processor units in which child modules receive input data. The input data are processed to identify patterns and/or sequences. Data regarding the observed patterns and/or sequences are passed to a parent module which may receive as inputs data from one or more child modules. the parent module examines its input data for patterns and/or sequences and then provides feedback to the child module or modules regarding the parent-level patterns that correlate with the child-level patterns. These systems and methods are extensible to large networks of interconnected processor modules.

Claims

1-18. -18. (canceled)

19. A computer-implemented memory network including at least one processor for producing an identifier associated with spatial patterns and temporal sequences in input data, comprising:

two or more children processing modules for receiving spatially different parts of the input data, each child processing module generating lower level characterization data that represents spatial patterns and temporal sequences in the different parts of the input data; and

a parent processing module associated with the two or more children processing modules for receiving the lower level characterization data, the parent processing module generating and outputting higher level characterization data that represents spatial patterns and temporal sequences in the lower level characterization data, the parent processing module providing feedback information to the two or more children processing modules for associating the higher level characterization data with the lower level characterization data in a learning process, the parent processing module outputting the second level characterization data as the identifier in a querying process subsequent to the learning process.

20. The memory network of claim 19, wherein each child processing module generates the lower level characterization data by identifying the spatial patterns and temporal sequences in a spatially different part of the input data, and quantifying frequency of the spatial patterns and temporal sequences in the spatially different part of the input data.

21. The memory network of claim 19, wherein the parent processing module generates the higher level characterization data by identifying the spatial patterns and temporal sequences in the lower level characterization data and quantifying frequency of the spatial patterns and temporal sequences in the lower level characterization data.

22. The memory network of claim 19, wherein the feedback information comprises the higher level characterization data generated by the parent processing module.

23. The memory network of claim 22, wherein each child processing module receives the feedback information from the parent processing module to generate and store a correlation between the lower level characterization data and the higher level characterization data generated by the parent processing module based on the lower level characterization data.

24. The memory network of claim 23, wherein the correlation is represented by a conditional probability table.

25. The memory network of claim 22, wherein each child processing module stores a transition of the lower level characterization data along with the feedback information.

26. The memory network of claim 25, wherein the correlation is represented by a conditional transition probability matrix.

27. The memory network of claim 19, further comprising two or more grandchildren processing modules, each grandchild processing module associated with at least one child processing module for generating and outputting at least a part of the input data to the at least one child processing module.

28. The memory network of claim 19, wherein the two or more children processing modules and the parent processing module form a Bayesian belief network.

29. A computer-implemented method for training a memory network and producing an identifier associated with spatial patterns and temporal sequences in input data, the method comprising the steps of:

training the memory network comprising;

receiving spatially different parts of training data at two or more children processing modules, the training data comprising spatial patterns and temporal sequences;

generating lower level characterization data that represents the spatial patterns and temporal sequences in the training data at the two or more children processing modules;

outputting the lower level characterization data at the two or more children processing modules to a parent processing module;

generating higher level characterization data that represents spatial patterns and temporal sequences in the lower level characterization data at the parent processing module;

providing feedback based on the higher level characterization data to the two or more children processing modules for generating a correlation between the higher level characterization data and the lower level characterization data; and

storing the correlation between the lower level characterization data and the second level characterization data; and

generating and outputting from the parent processing module the identifier based on the stored correlation between the lower level characterization data and the higher level characterization data responsive to the two or more children processing modules receiving spatially different parts of the input data and providing the lower level characterization data to the parent processing module.

30. The method of claim 29, wherein the correlation is represented by a conditional probability matrix.

31. The method of claim 29, further comprising filling-in missing information in the input data based on the conditional probability matrix.

32. The method of claim 29, wherein the training data comprises video sequences.

33. The method of claim 29, wherein the training data comprises visual data.

34. The method of claim 29, wherein the input data comprises visual patterns.

35. The method of claim 34, where the visual data comprises patterns with relative movements.

36. The method of claim 29, wherein the two or more children processing modules and the parent processing module form a Bayesian belief network.

37. A computer-implemented memory network comprising at least one processor, the memory network producing an identifier associated with spatial patterns and temporal sequences in input data, the memory network comprising:

a first lower tier processing module having a first receptive field for receiving a first spatial part of the input data, the first lower tier processing module generating first lower tier information representing spatial patterns and temporal sequences in the first spatial part of the input data;

a second lower tier processing module having a second receptive field for receiving a second spatial part of the input data, the second lower tier processing module generating second lower tier information representing spatial patterns and temporal sequences in the second spatial part of the input data; and

an upper tier processing module having a third spatial receptive field covering a range of the input data larger than the first spatial receptive field or the second spatial receptive field, the upper tier processing module coupled to the first lower tier processing module and the second lower tier processing module, the upper tier processing module identifying and outputting the identifier based on spatial patterns and temporal sequences in the first and second lower tier information, the upper tier processing module generating feedback information provided to the first and second lower tier processing modules for associating the first and second lower tier information with the identifier.

38. The memory network of claim 37, wherein the first lower tier processing module, the second lower tier processing module, and the upper tier processing module form a Bayesian belief network.