Image recognition apparatus using neural network processing

• US 9,361,534 B2
• Filed: 07/25/2012
• Issued: 06/07/2016
• Est. Priority Date: 08/05/2011
• Status: Active Grant

• Current Assignee: Megachips Corporation
• Original Assignee: Megachips Corporation
• Inventors: Mizuno, Yusuke, Togo, Daisuke
• Primary Examiner(s): Chen, Wenpeng
• Application Number:
  US14/236,739
• Publication Number:
  US 20140169631A1
• Time in Patent Office: 1,413 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An image recognition apparatus determines whether an image of a pedestrian is captured in a frame of video data captured by a vehicle mounted camera. A pre-processing unit determines a detection block from within a frame, and cuts out block image data corresponding to the detection block from the frame. Block data with a predetermined size that is smaller than the size of the detection block is created from the block image data. A neuro calculation unit executes neuro calculation on the block data, and calculates an output synapse. A post-processing unit determines whether a pedestrian exists within the detection block on the basis of the output synapse. When a pedestrian is detected, the post-processing unit creates result data, which is obtained by superimposing the detection block within which the pedestrian was detected onto the frame.

Claims

1. An image recognition apparatus configured to determine whether or not an object which is to be detected is included in a frame, comprising:

circuitry configured to

generate first calculation image data from a first area image in the frame, a size of the first calculation image data being smaller than that of the first area image;

calculate a neuro calculation value which indicates whether or not the object which is to be detected is included in the first area image by performing neuro calculation on a pixel value of the first calculation image data;

generate result data which indicates whether or not the object which is to be detected is included in the frame by using the neuro calculation value;

carry out at least one intermediate synapse calculation process of performing neuro calculation on synapses in an input layer to calculate synapses in an intermediate layer;

perform neuro calculation on the synapses in the intermediate layer to calculate an output synapse;

output the output synapse as the neuro calculation value,

wherein each of the synapses in the input layer in the first intermediate synapse calculation process is pixel values of the first calculation image data,

wherein the circuitry is further configured to

designate N intermediate synapses from the synapses in the intermediate layer as an object for calculation (N being a natural number greater than one);

carry out a selecting process of selecting one input synapse from the synapses in the input layer;

determine a weighting coefficient associated with the selected input synapse and the K-th intermediate synapse (K being a natural number equal to or smaller than N);

perform an intermediate operation process of adding a result of multiplication of the selected input synapse by the determined weighting coefficient, to the K-th intermediate synapse; and

perform respective intermediate operation processes on the first to N-th intermediate synapses, in parallel,

wherein the selecting process and the intermediate operation process are repeated until all of input synapses are selected.

2. The image recognition apparatus according to claim 1, further comprising:

a first block buffer configured to store the first calculation image data; and

a second block buffer configured to store second calculation image data generated from a second area image in the frame by the circuitry, the second area image being different from the first area image, wherein

the circuitry generates the second calculation image data and stores the second calculation image data in the second block buffer while performing neuro calculation by using the first calculation image data stored in the first block buffer, and

the circuitry generates the first calculation image data and stores the first calculation image data in the first block buffer while performing neuro calculation by using the second calculation image data stored in the second block buffer.

3. The image recognition apparatus according to claim 1, wherein the circuitry is further configured to:

statistically process pixel values in the first area image to calculate a block statistical value; and

normalize pixel values in the first area image based on the block statistical value and a standard distribution data of pixel values in the first area image.

4. The image recognition apparatus according to claim 3, wherein the circuitry is further configured to:

statistically process pixel values in the frame to calculate a frame statistical value,

normalize pixel values in the frame based on the frame statistical value and a standard distribution data of pixel values in the frame, and

normalize pixel values in the first area image by using the frame which is normalized.

5. The image recognition apparatus according to claim 1, further comprising:

an input interface configured to receive a plurality of frames;

a first frame buffer; and

a second frame buffer, wherein

the input interface stores a frame which is input next to a first frame stored in the first frame buffer into the second frame buffer while the first calculation image data is generated from the first frame.

6. The image recognition apparatus according to claim 1, wherein the circuitry is further configured to:

determine a weighting coefficient used for a next intermediate operation process in a period of the intermediate operation process.

7. The image recognition apparatus according to claim 1, wherein

the synapses in the input layer are divided into a plurality of input groups, and

the circuitry is further configured to select one from the plurality of input groups,

an input synapse on which the intermediate operation process is to be performed from the selected input group, and

a next input group after all of input synapses included in the selected input group are selected.

8. The image recognition apparatus according to claim 7, further comprising:

an input synapse buffer configured to store the synapses in the input layer; and

a first coefficient table configured to store weighting coefficients used by the circuitry, wherein

the circuitry is configured to read out all of input synapses included in the selected input group from the input synapse buffer by a one-time reading process, and reads out weighting coefficients determined from the first coefficient table by a one-time reading process.

9. The image recognition apparatus according to claim 7, wherein the circuitry is further configured to:

carry out an intermediate operation process which is carried out on an input synapse lastly selected from the selected input group, and a process of selecting the next input group, in parallel.

10. The image recognition apparatus according to claim 1, wherein the circuitry is further configured to:

use intermediate synapses which are calculated immediately before, as input synapses when each of the second and subsequent intermediate operation processes is performed.

11. The image recognition apparatus according to claim 1, wherein the circuitry is further configured to:

perform sigmoid calculation on intermediate synapses after intermediate calculation processes on all of input synapses are finished, wherein

an approximate function obtained by linearly approximating a sigmoid function is used in the sigmoid calculation.

12. An image recognition apparatus configured to determine whether or not an object which is to be detected is included in a frame, comprising:

circuitry configured to

generate first calculation image data from a first area image in the frame, a size of the first calculation image data being smaller than that of the first area image;

calculate a neuro calculation value which indicates whether or not the object which is to be detected is included in the first area image by performing neuro calculation on a pixel value of the first calculation image data;

generate result data which indicates whether or not the object which is to be detected is included in the frame by using the neuro calculation value;

carry out at least one intermediate synapse calculation process of performing neuro calculation on synapses in an input layer to calculate synapses in an intermediate layer;

perform neuro calculation on the synapses in the intermediate layer to calculate an output synapse;

output the output synapse as the neuro calculation value, wherein

each of the synapses in the input layer in the first intermediate synapse calculation process is pixel values of the first calculation image data,

the circuitry is further configured to

set Q partial addition values which are used for calculation of the output synapse;

select Q intermediate synapses from the synapses in the intermediate layer, and perform a partial addition process of adding a result of multiplication of the R-th (R is a natural number equal to or smaller than Q) intermediate synapse included in the Q intermediate synapses by a weighting coefficient associated with the R-th intermediate synapse, to the R-th partial addition value; and

calculate a total value of the Q partial addition values after partial addition processes on all of intermediate synapses included in the synapses in the intermediate layer are finished, to calculate the output synapse, wherein

the partial addition processes using the Q intermediate synapses are carried out in parallel.

13. The image recognition apparatus according to claim 12, further comprising:

an intermediate synapse buffer configured to store the synapses in the intermediate layer; and

a second coefficient table configured to store weighting coefficients respectively associated with the synapses in the intermediate layer, wherein

the circuitry is configured to read out the Q intermediate synapses from the intermediate synapse buffer by a one-time reading process, and reads out the weighting coefficients respectively associated with the Q intermediate synapses from the second coefficient table by a one-time reading process.

14. The image recognition apparatus according to claim 12, wherein the circuitry is further configured to:

designate another Q intermediate synapses and weighting coefficients which are used for the next partial addition process in a period in which the partial addition process is carried out.

15. The image recognition apparatus according to claim 12, wherein the circuitry is further configured to:

perform sigmoid calculation on the total value of the Q partial addition values to calculate the output synapse after partial addition processes on all of the intermediate synapses are finished, wherein

an approximate function obtained by linearly approximating a sigmoid function is used in the sigmoid calculation.

16. The image recognition apparatus according to claim 12, further comprising:

a first block buffer configured to store the first calculation image data; and

a second block buffer configured to store second calculation image data generated from a second area image in the frame by the circuitry, the second area image being different from the first area image, wherein

the circuitry is further configured to

generate the second calculation image data and store the second calculation image data in the second block buffer while performing neuro calculation by using the first calculation image data stored in the first block buffer, and

generate the first calculation image data and store the first calculation image data in the first block buffer while performing neuro calculation by using the second calculation image data stored in the second block buffer.

17. The image recognition apparatus according to claim 12, wherein the circuitry is further configured to:

statistically process pixel values in the first area image to calculate a block statistical value; and

normalize pixel values in the first area image based on the block statistical value and a standard distribution data of pixel values in the first area image.

18. The image recognition apparatus according to claim 17, wherein the circuitry is further configured to:

statistically process pixel values in the frame to calculate a frame statistical value;

normalize pixel values in the frame based on the frame statistical value and a standard distribution data of pixel values in the frame; and

normalize pixel values in the first area image by using the frame which is normalized.

19. The image recognition apparatus according to claim 12, further comprising:

an input interface configured to receive a plurality of frames;

a first frame buffer; and

a second frame buffer,

wherein the input interface stores a frame which is input next to a first frame stored in the first frame buffer into the second frame buffer while the first calculation image data is generated from the first frame.

20. An image recognition apparatus configured to determine whether or not an object which is to be detected is included in a frame, comprising:

pre-processing circuitry configured to generate first calculation image data from a first area image in the frame, a size of the first calculation image data being smaller than that of the first area image;

neuro calculation circuitry configured to calculate a neuro calculation value which indicates whether or not the object which is to be detected is included in the first area image by performing neuro calculation on a pixel value of the first calculation image data; and

a post-processing circuitry configured to generate result data which indicates whether or not the object which is to be detected is included in the frame by using the neuro calculation value,

wherein the neuro calculation circuitry includes

intermediate synapse calculation circuitry configured to carry out at least one intermediate synapse calculation process of performing neuro calculation on synapses in an input layer to calculate synapses in an intermediate layer, and

output synapse calculation circuitry configured to perform neuro calculation on the synapses in the intermediate layer to calculate an output synapse, and output the output synapse as the neuro calculation value, wherein

each of the synapses in the input layer in the first intermediate synapse calculation process includes pixel values of the first calculation image data,

the intermediate synapse calculation circuitry includes

object-for-calculation designation circuitry configured to designate N intermediate synapses from the synapses in the intermediate layer, (N being a natural number greater than one) as an object for calculation,

input synapse selection circuitry configured to carry out a selecting process of selecting one input synapse from the synapses in the input layer, and

intermediate operation circuitry configured to determine a weighting coefficient associated with the selected input synapse and the K-th (K is a natural number equal to or smaller than N) intermediate synapse, and perform an intermediate operation process of adding a result of multiplication of the selected input synapse by the determined weighting coefficient, to the K-th intermediate synapse,

wherein the intermediate operation circuitry performs respective intermediate operation processes on the first to N-th intermediate synapses, in parallel, and

wherein the selecting process and the intermediate operation process are repeated until all of input synapses are selected.

21. The image recognition apparatus according to claim 20, wherein the neuro calculation circuitry is implemented by a configurable processor.

22. An image recognition apparatus configured to determine whether or not an object which is to be detected is included in a frame, comprising:

pre-processing circuitry configured to generate first calculation image data from a first area image in the frame, a size of the first calculation image data being smaller than that of the first area image;

neuro calculation circuitry configured to calculate a neuro calculation value which indicates whether or not the object which is to be detected is included in the first area image by performing neuro calculation on a pixel value of the first calculation image data; and

post-processing circuitry configured to generate result data which indicates whether or not the object which is to be detected is included in the frame by using the neuro calculation value,

the neuro calculation circuitry includes

intermediate synapse calculation circuitry configured to carry out at least one intermediate synapse calculation process of performing neuro calculation on synapses in an input layer to calculate synapses in an intermediate layer, and

output synapse calculation circuitry configured to perform neuro calculation on the synapses in the intermediate layer to calculate an output synapse, and output the output synapse as the neuro calculation value,

wherein each of the synapses in the input layer in the first intermediate synapse calculation process is pixel values of the first calculation image data, the output synapse calculation circuitry includes

partial addition value setting circuitry configured to set Q partial addition values which are used for calculation of the output synapse,

partial addition value calculation circuitry configured to select Q intermediate synapses from the synapses in the intermediate layer, and perform a partial addition process of adding a result of multiplication of the R-th (R is a natural number equal to or smaller than Q) intermediate synapse included in the Q intermediate synapses by a weighting coefficient associated with the R-th intermediate synapse, to the R-th partial addition value, and

summing circuitry configured to calculate a total value of the Q partial addition values after partial addition processes on all of intermediate synapses included in the synapses in the intermediate layer are finished, to calculate the output synapse,

wherein the partial addition processes using the Q intermediate synapses are carried out in parallel.

23. The image recognition apparatus according to claim 22, wherein the neuro calculation circuitry is implemented by a configurable processor.