Apparatus and method for detecting or recognizing pattern by employing a plurality of feature detecting elements

• US 20020038294A1
• Filed: 06/12/2001
• Published: 03/28/2002
• Est. Priority Date: 06/16/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Matsugu, Masakazu
• Granted Patent: US 7,054,850 B2
• Time in Patent Office: Days
• US Class Current: 706/20
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding

Abstract

A pattern detecting apparatus has a plurality of hierarchized neuron elements to detect a predetermined pattern included in input patterns. Pulse signals output from the plurality of neuron elements are given specific delays by synapse circuits associated with the individual elements. This makes it possible to transmit the pulse signals to the neuron elements of the succeeding layer through a common bus line so that they can be identified on a time base. The neuron elements of the succeeding layer output the pulse signals at output levels based on a arrival time pattern of the plurality of pulse signals received from the plurality of neuron elements of the preceding layer within a predetermined time window. Thus, the reliability of pattern detection can be improved, and the number of wires interconnecting the elements can be reduced by the use of the common bus line, leading to a small scale of circuit and reduced power consumption.

Claims

1. A pattern detecting apparatus comprising:

input means for inputting a pattern; and

pattern detecting means that comprises a plurality of signal processing elements and performs detection related to a plurality of predetermined features on a pattern input by the input means so as to detect a predetermined pattern included in the pattern,

wherein each of the plurality of signal processing elements outputs a pulse signal to another signal processing element or outside in response to an input from the input means or another signal processing element, and

predetermined ones among the plurality of signal processing elements output pulse signals with outputs based on arrival time patterns of a plurality of pulse signals input within a predetermined time range.

2. A pattern detecting apparatus according to claim 1, wherein

the plurality of signal processing elements comprises:

a feature detection element that belongs to a feature detection layer for extracting a predetermined feature, and a feature integration element that belongs to a feature integration layer that integrates outputs from the feature detection layer according to a predetermined method and outputs a result of the integration, and

the predetermined ones among the plurality of signal processing elements are the feature detection elements that receive inputs from a plurality of the feature integration elements.

3. A pattern detecting apparatus according to claim 2, wherein the feature detection element outputs a pulse signal at an output level based on the degree of salience or disposition of the features.

4. A pattern detecting apparatus according to claim 2, wherein the feature integration element outputs, in a predetermined time range, a pulse signal at an output level based on a total sum of inputs from the plurality of feature detection elements that are associated with an identical feature.

5. A pattern detecting apparatus according to claim 2, wherein a plurality of the feature detection layers and the feature integration layers are alternately connected in a cascade.

6. A pattern detecting apparatus according to claim 5, wherein each of the feature integration elements of the feature integration layers has a local receptive field structure for receiving the signals from a plurality of feature detection elements which exist in a local range in a feature detection layer of a preceding stage and which individually detect an identical feature.

7. A pattern detecting apparatus according to claim 5, wherein feature detection elements of the feature detection layer receive signals from feature integration elements associated with different features in a feature integration layer in the preceding stage to detect a higher-order feature.

8. A pattern detecting apparatus according to claim 2, wherein at least some of the feature detection layers comprise a plurality of filters for performing a local spatial frequency analysis related to a component in a predetermined direction.

9. A pattern detecting apparatus according to claim 2, wherein the plurality of feature detection elements existing in the same receptive field of the feature integration elements output pulses in phase-synchronization with each other for a predetermined pattern.

10. A pattern detecting apparatus according to claim 2, wherein the feature detection elements detect the presence of a predetermined associated feature according as whether a pulse is present in a predetermined time window range intrinsic to the pulse in an arrival time pattern of the plurality of pulse signals.

11. A pattern detecting apparatus according to claim 2, wherein the feature detection layer comprises timing elements appendant to the feature detection elements in the layer, and the timing elements output pulses at predetermined pulse intervals to issue signals of the phase synchronization for feature detection calculating elements on the basis of the output from the feature integration elements on the same receptive field of a layer in the preceding stage.

12. A pattern detecting apparatus according to claim 11, wherein the timing elements output predetermined timing pulses to some feature detection elements in the feature detection layer comprising the timing elements, and to feature integration elements in the feature integration layer in the preceding stage that exists in the same receptive field of the feature detection element.

13. A pattern detecting apparatus according to claim 2, wherein each feature detection element in the feature detection layer outputs a pulse signal with a phase based on a feature to be detected thereby.

14. A pattern detecting apparatus according to claim 2, comprising a plurality of the feature detection layers and the feature integration layers, wherein a signal processing element of the last layer issues a signal representing a result of the detection of the predetermined pattern.

15. A pattern detecting apparatus according to claim 1, wherein the plurality of signal processing elements are connected through the intermediary of connecting means, and the connecting means carries out predetermined modulation on an output pulse signal of one of the signal processing elements and transmits the modulated output pulse signal to the other of the signal processing elements.

16. A pattern detecting apparatus according to claim 15, wherein the modulation is implemented to delay a pulse phase.

17. A pattern detecting apparatus according to claim 16, wherein the connecting means decreases the delay amount of the pulse phase as the number of events with input pulses representing patterns of the same category increases.

18. A pattern detecting apparatus according to claim 16, wherein the delay amount of the pulse phase remains substantially constant regardless of the type of a feature.

19. A pattern detecting apparatus according to claim 15, wherein the modulation is pulse width modulation or frequency modulation.

20. A pattern detecting apparatus according to claim 1, wherein the predetermined ones among the plurality of signal processing elements output pulse signals at output levels based on the weighted sum obtained by multiplying a plurality of pulse signals received within the predetermined time range by predetermined weighting coefficient values, which temporally change, and adding the results.

21. A pattern detecting apparatus according to claim 1, wherein the output level is based on the phase, frequency, amplitude, or pulse width of the pulse signal.

22. An image processing apparatus adapted to control an operation for processing an image to be processed, on the basis of a result of the detection of a predetermined pattern from among patterns of the image to be processed, the result of the detection being obtained by the pattern detecting apparatus according to claim 1.

23. A pattern detecting apparatus according to claim 2, further comprising a feature position detection layer that receives an output from the feature integration layer and outputs information regarding the position where a predetermined feature or pattern exists.

24. A pattern detecting apparatus according to claim 23, wherein the number of the feature position detection layers is smaller than that of the feature integration layers, and the feature position detection layers are respectively connected to predetermined portions of the feature integration layers.

25. A pattern detecting apparatus according to claim 1, wherein

the pattern detecting means has a plurality of processing layers composed of a plurality of neuron elements arranged in parallel as the signal processing elements, the neuron elements receiving a plurality of signals and outputting pulse signals,

the pulse signals output from a plurality of neuron elements of another layer to at least one of the neuron elements of a predetermined one of the processing layers are input through the intermediary of a bus line common to synaptic connection means provided for each of the plurality of neuron elements, and to the plurality of neuron elements, and

the synaptic connection means imparts a specific pulse phase shift amount to the pulse signals output from the plurality of neuron elements.

26. A pattern detecting apparatus according to claim 1, wherein

the pattern detecting means comprises:

a plurality of processing means for implementing processing for different resolutions or scale levels on patterns received from the input means; and

multiplex processing means for coupling the outputs of the plurality of processing means,

wherein each of the plurality of processing means comprises a plurality of feature detection elements, as the signal processing elements, that detect and output a plurality of features associated with individual points obtained by sampling the input data according to a predetermined method, and

the multiplex processing means couples the outputs of the plurality of feature detection elements.

27. A pattern detecting apparatus according to claim 26, wherein

the plurality of processing means make up a hierarchical network structure, and

the multiplex processing means selects a resolution or scale level, or makes setting for coupling processed outputs of a plurality of resolutions or scale levels on the basis of a predetermined plurality of outputs among the outputs of the feature detection elements that correspond to the processing results in respective intermediate hierarchies of the processing means for a plurality of resolutions or scale levels.

28. A pattern detecting apparatus according to claim 27, wherein the multiplex processing means selects a resolution or scale level that indicates a maximum response output among the processing results obtained at a predetermined intermediate hierarchy of the plurality of processing means for different resolutions or scale levels.

29. A pattern detecting apparatus according to claim 27, wherein the multiplex processing means selects a resolution or a scale level by coupling a predetermined plurality of results among the processing results obtained at the intermediate hierarchies of the plurality of processing means for different resolutions or scale levels.

30. A pattern detecting apparatus according to claim 26, wherein each of the plurality of processing means comprises a plurality of hierarchical processing layers, and the multiplex processing means refers to intermediate processing results at processing layers of different hierarchical levels thereby to select a resolution or scale level.

31. A pattern detecting apparatus according to claim 29, wherein each of the plurality of processing means comprises a plurality of hierarchical processing layers, and the coupling processing refers to different resolutions or scale levels for the values obtained by coupling the processing results of different computing elements at the same hierarchical level of the processing means, and provides an output related to a predetermined resolution or scale level.

32. A pattern detecting apparatus according to claim 26, wherein the plurality of processing means have feature detection layers for detecting features at different resolutions or scale levels on each of a predetermined plurality of feature categories.

33. A pattern detecting apparatus according to claim 32, wherein the multiplex processing means is provided for each feature detection element of the feature detection layer.

34. A pattern detecting apparatus according to claim 32, wherein the feature detection layer locally performs spatial filtering for different spatial frequencies.

35. A pattern detecting apparatus according to claim 32, wherein each feature detection element of the feature detection layer detects a plurality of features at different resolutions or scale levels in a local, identical region of input data.

36. A pattern detecting apparatus according to claim 32, wherein the individual feature detection elements of the feature detection layer have selectivity on a plurality of different resolutions or scale levels on the same geometric feature category, the selectivity thereof overlapping each other.

37. A pattern detecting apparatus according to claim 32, wherein the feature detection layer includes a plurality of computing elements having sensitivity features approximated by a basis function that has locally different directional selectivities at different resolutions or scale levels for a feature category to be detected.

38. A pattern detecting apparatus according to claim 2, wherein the feature detection layer detects a plurality of features for a plurality of resolutions or scale levels for a pattern received from the input means.

39. A pattern detecting apparatus according to claim 38, wherein the feature integration element comprises an element for sub-sampling feature data in a local receptive field region, and a population coding element for integrating the outputs of the sub-sampling elements that extend over a plurality of resolutions or scale levels.

40. A pattern detecting apparatus according to claim 38, wherein

the feature detection layer comprises:

a feature detection element that detects a plurality of features of different resolutions or scale levels only on low-order features, and coverts the outputs of the detection pulse signals associated with the features on the basis of the resolutions or scale levels so that the outputs are provided with phases different in the direction of a time base; and

a scale conversion feature extracting element that converts the phases of the output signals from the element thereby to obtain feature detection signals that remain invariant against resolutions or scale levels.

41. A pattern detecting apparatus according to claim 26, wherein

the processing means comprises:

a plurality of feature detection elements for detecting a plurality of features at individual points obtained by sampling the input data according to a predetermined method, and

control means that integrates the plurality of the outputs of the feature detection elements of the plurality of processing means at different resolutions or scale levels so as to carry out control related to resolutions or scale levels.

42. A pattern detecting apparatus according to claim 41, wherein the control means controls the setting of a predetermined resolution or scale level.

43. A pattern detecting apparatus according to claim 41, wherein the control means controls the activation degree of the feature detection element on the basis of a resolution or a scale level.

44. A pattern detecting apparatus according to claim 41, wherein the control means converts or copies a signal of a predetermined resolution or scale into a signal of another resolution or scale level according to a predetermined method, and distributes the resulting signal in a learning mode.

45. A pattern detecting apparatus according to claim 26, wherein

the plurality of processing means comprises a plurality of channels that individually implement processing at different scale levels or resolutions by employing groups of neurons having hierarchical structures, and

the multiplexing means comprises a group of population coding neurons that integrate outputs of a plurality of channels.

46. A pattern detecting apparatus according to claim 45, wherein an optimal scale level or resolution is set on the basis of the processing results of the group of population coding neurons.

47. A pattern detecting apparatus according to claim 45, wherein the groups of neurons having the hierarchical structures comprise groups of sub-sampling neurons that perform sub-sampling for each category.

48. A pattern detecting apparatus according to claim 1, wherein the pattern detecting means comprises:

a plurality of feature detection elements in a plurality of hierarchies for detecting a plurality of features in correspondence with points obtained by sampling the patterns received from the input means according to a predetermined method, and

fixation region setting control means for controlling the setting of a fixation region associated with an output of a lower layer of the plurality of hierarchies on the basis of the distributions of feedback signals from an upper layer of the plurality of hierarchies.

49. A pattern detecting apparatus according to claim 48, wherein the fixation region setting control means updates a set position or size of a fixation region.

50. A pattern detecting apparatus according to claim 48, wherein

the pattern detecting means comprises salience level detecting elements for detecting the salience of features, and a coupling means for coupling the elements and transmitting signals, and forms a plurality of element layers for low-order to high-order features,

the connecting means comprises feedback connecting means for transmitting signals from an element layer for a high-order feature to an element layer for a feature of a lower order, and

the fixation region setting control means controls the setting of a fixation region for low-order feature data or input data on the basis of the feature salience level and a signal transmission amount obtained by the feedback connecting means.

51. A pattern detecting apparatus according to claim 50, wherein the fixation region setting control means comprises:

priority level calculating means for determining the priority level of a fixation position at sampling points of each piece of input data on the basis of a signal transmission amount received from the feedback connecting means and the salience level of a low-order feature; and

fixation position setting means for setting fixation positions in a descending order of priority levels on the basis of a distribution of priority levels.

52. A pattern detecting apparatus according to claim 51, wherein the fixation region setting control means comprises:

counting means for counting the number of searches for fixation positions; and

control means for controlling a permissible range of priority levels wherein fixation positions can be set by the fixation position setting means on the basis of the number of searches for fixation positions.

53. A pattern detecting apparatus according to claim 51, wherein

the detecting means comprises a plurality of processing channels associated with a plurality of scale levels or resolutions, and

the fixation region setting control means controls the size of a fixation region on the basis of the processing channel to which a feature selected based on the priority level belongs.

54. A pattern detecting apparatus according to claim 50, wherein the fixation region setting control means controls the setting of a fixation region as an active receptive field of a feature detection element that belongs to a low-order feature detection layer.

55. A pattern detecting apparatus according to claim 50, wherein the fixation region setting control means receives a feedback connection from an upper layer that outputs information regarding the position of an object that belongs to the category to be recognized and regarding a probability of existence, and a feedback connection from an intermediate layer that outputs information regarding the position of a medium-order feature of an object of a category to be recognized and regarding a probability of existence, and gives priority to a feedback input from the upper layer when searching for the object, or to a feedback input from the intermediate layer when recognizing the object.

56. A pattern detecting apparatus according to claim 50, wherein the fixation region setting control means reduces a temporal change in the central position of a fixation region when a predetermined degree of gaze is high.

57. A pattern detecting apparatus according to claim 55, wherein the degree of attention takes a monotone increase function value of a feedback signal level from the upper layer.

58. A pattern detecting apparatus according to claim 2, further comprising:

feedback connecting means for transmitting signals from an upper layer to a lower layer in a hierarchical network structure combining the feature detection layers and the feature integration layers; and

fixation region setting control means that issues control signals related to a fixation region on the basis of a signal from the feedback connecting means and the salience level of a feature,

wherein the feature detection layer extracts a plurality of features for each of a plurality of resolutions or scale levels, and

the feature detection layer or the feature integration layer comprises a salience detecting element for detecting the salience of a feature.

59. A pattern detecting apparatus according to claim 58, wherein the fixation region setting control means sets the size of a fixation region on the basis of a detected scale level associated with the pattern that belongs to a category to be recognized.

60. An imaging apparatus comprising:

fixation region setting means for setting a fixation region;

determining means for determining photographing conditions on the basis of a fixation region set by the fixation region setting means; and

memory means for storing model data regarding an object to be photographed,

wherein the fixation region setting means sequentially updates a fixation region to search for a fixation region that meets a predetermined requirement regarding the model data thereby to set the fixation region under the control by the fixation region setting control means of the pattern detecting apparatus according to claim 49.

61. An imaging apparatus according to claim 60, further comprising initializing means for initializing a fixation region in a photographing standby state.

62. An imaging apparatus according to claim 60, further comprising displaying means for displaying an object to be photographed that is associated with a fixation region set by the fixation region setting means.

63. An imaging apparatus comprising according to claim 60, further comprising detecting means for detecting a shutter drive signal, wherein the fixation region setting means resumes a search for a fixation region if no shutter drive signal is detected by the detecting means within a predetermined time after a fixation region is set.

64. An imaging apparatus comprising:

fixation region setting means for setting a fixation region;

determining means for determining photographing conditions on the basis of a fixation region set by the fixation region setting means; and

fixation position detecting means for detecting a fixation position on the basis of a user'"'"'s visual axis,

wherein the fixation region setting means searches for and sets a fixation region on the basis of the fixation position under the control by the fixation region setting control means of the pattern detecting apparatus according to claim 49.

65. An imaging apparatus according to claim 64, wherein the fixation region setting means explores a neighborhood of the fixation position.

66. A pattern detecting method comprising the steps of:

receiving a pattern from an input section, and

subjecting the received pattern to detection on a predetermined plurality of features thereby to detect a predetermined pattern included in the pattern by employing a plurality of signal processing elements,

wherein the step of subjecting the received pattern to detection includes the steps of

outputting a pulse signal to another signal processing element or outside from each of the plurality of signal processing elements in response to an input from the input section or another signal processing element, and

outputting pulse signals, from predetermined ones among the plurality of signal processing elements, at output levels based on arrival time patterns of a plurality of pulse signals input within a predetermined time range.