Image recognition system and semiconductor integrated circuit

• US 10,074,187 B2
• Filed: 08/25/2016
• Issued: 09/11/2018
• Est. Priority Date: 02/26/2014
• Status: Active Grant

• Current Assignee: Socionext Inc.
• Original Assignee: Socionext Inc.
• Inventors: Sasagawa, Yukihiro, Tetsukawa, Tatsuya, Mi, Michael Bi, Ping, Chua Tien, Nakanishi, Ryuta, Nojiri, Naoki
• Primary Examiner(s): Woldemariam, Aklilu
• Application Number:
  US15/247,337
• Publication Number:
  US 20160364882A1
• Time in Patent Office: 747 Days
• Field of Search: 382103, 382169, 382700, 348135, 348169, 348700, 708523, 708316, 708524, 708490, 708603
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

An image recognition system for detecting and tracking at least an image portion associated with a predefined object from a moving picture is configured to be able to perform: an object detection processing step of detecting the object; a tracking point specification processing step of specifying a predetermined point as a tracking point; a tracking target recognition processing step of recognizing an actual tracking target based on the tracking point; a tracking processing step of tracking the tracking target; and a determination processing step of determining the type of the tracking target'"'"'s behavior. The tracking point specification processing step and the determination processing step are implemented by software, while the object detection processing step, the tracking target recognition processing step, and the tracking processing step are implemented by hardware.

Claims

1. An image recognition system for detecting and tracking at least an image portion associated with a predefined object from a moving picture, wherein

the image recognition system is configured to be able to perform:

an object detection processing step of detecting the object from the moving picture;

a tracking point specification processing step of specifying at least a predetermined point associated with the object as a tracking point;

a tracking target recognition processing step of recognizing an actual tracking target based on the tracking point;

a tracking processing step of tracking the tracking target; and

a determination processing step of determining the type of the tracking target'"'"'s behavior based on a result of the tracking processing step,

wherein the tracking point specification processing step and the determination processing step are implemented by software, while the object detection processing step, the tracking target recognition processing step, and the tracking processing step are implemented by hardware,

the object detector performing the object detection processing step includes at least a feature quantity calculating unit, a plurality of posterior event memories storing posterior events, a posterior event updater, and a score calculating unit,

the feature quantity calculating unit calculates a feature quantity based on the moving picture,

the posterior event memories store at least both of a posterior event in an object image and a posterior event in a non-object image,

the posterior event updater updates the posterior event in the object image and the posterior event in the non-object image in accordance with a pattern of the feature quantity, and

the score calculating unit retrieves, from the posterior event memories, a posterior event corresponding to an arbitrary pattern of the feature quantity obtained from the feature quantity calculating unit, and calculates a score corresponding to the arbitrary pattern of the feature quantity.

2. The image recognition system of claim 1, wherein the object is a human.

3. The image recognition system of claim 1, further comprising:

hardware further including;

a tracking target recognizer capable of performing the tracking target recognition processing step,

a tracker capable of performing the tracking processing step, and

a CPU capable of performing the tracking point specification processing step and the determination processing step.

4. The image recognition system of claim 1, wherein

the tracker performing the tracking processing step performs repetitive calculation of a luminance gradient of image fragments,

the repetitive calculation of the luminance gradient being performed by sequentially referring to a memory storing the image fragments,

the image recognition system includes a plurality of calculating units performing the repetitive calculation of the luminance gradient, and associates the calculating units with the independent image fragments, respectively, so as to perform the repetitive calculation of the luminance gradient of the image fragments in parallel.

5. A semiconductor integrated circuit performing a processing step of detecting and tracking at least an image portion associated with a predefined object from a moving picture, the semiconductor integrated circuit comprising:

an object detector configured to perform a processing step of detecting the object from the moving picture,

a tracking target recognizer configured to perform a processing step of recognizing at least a predetermined point associated with the object as an actual tracking target,

a tracker configured to perform a processing step of tracking the tracking target,

a CPU configured to performing a processing step of specifying the predetermined point, and a processing step of determining the type of the tracking target'"'"'s behavior based on a result of tracking obtained by the tracker,

a direct memory access (DMA) controller configured to transfer data to the object detector, the tracking target recognizer, and the tracker, the data being required by the object detector, the tracking target recognizer and the tracker to perform their respective processing steps, and

a data bus configured to couple together the object detector, the tracking target recognizer, the tracker, the CPU, and the DMA controller,

wherein the object detector includes at least a feature quantity calculating unit, a plurality of posterior event memories storing posterior events, a posterior event updater, and a score calculating unit,

the feature quantity calculating unit calculates a feature quantity based on the moving picture,

the posterior event memories store at least both of a posterior event in an object image and a posterior event in a non-object image,

the posterior event updater updates the posterior event in the object image and the posterior event in the non-object image in accordance with a pattern of the feature quantity, and

the score calculating unit retrieves, from the posterior event memories, a posterior event corresponding to an arbitrary pattern of the feature quantity obtained from the feature quantity calculating unit, and calculates a score corresponding to the arbitrary pattern of the feature quantity.

6. The semiconductor integrated circuit of claim 5, wherein

the CPU controls the operation of each of the object detector, the tracking target recognizer, the tracker, and the DMA controller.

7. The semiconductor integrated circuit of claim 6, further comprising:

a control bus coupled to the object detector, the tracking target recognizer, the tracker, and the DMA controller, and

a sequencer configured to relay the operation control by the CPU to the control bus.

8. The semiconductor integrated circuit of claim 5, wherein

the CPU performs control such that the processing step performed by the object detector finishes within a predetermined unit time, and performs repeatedly, based on a result of the processing step performed by the object detector, a processing step of specifying the predetermined point, control of the operation of the tracking target recognizer, control of the operation of the tracker, and a processing step of determining the type of the tracking target'"'"'s behavior in each of a plurality of unit times continuous with the predetermined unit time.

9. The semiconductor integrated circuit of claim 8, wherein

the CPU performs at least two of the processing step of specifying the predetermined point, the control of the operation of the tracking target recognizer, the control of the operation of the tracker, and the processing step of determining the type of the tracking target'"'"'s behavior in parallel in any one of the plurality of unit times.

10. The semiconductor integrated circuit of claim 9, wherein

the CPU performs control such that, when the object detector performs the processing step again after the predetermined unit time has passed, this processing step is performed in a unit time during which at least two of the processing step of specifying the predetermined point, the control of the operation of the tracking target recognizer, the control of the operation of the tracker, and the processing step of determining the type of the tracking target'"'"'s behavior are performed.