Pedestrian detection device and pedestrian detection method

• US 10,354,160 B2
• Filed: 06/29/2015
• Issued: 07/16/2019
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Shigemura, Shusaku
• Primary Examiner(s): Beg, Samah A
• Application Number:
  US15/327,607
• Publication Number:
  US 20180144207A1
• Time in Patent Office: 1,478 Days
• Field of Search: None
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 15/66 Sonar tracking systems
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/87 Combinations of sonar systems
• CPC Class Codes: G01S 15/93 for anti-collision purposes
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 2013/9324 Alternative operation using ultrasonic waves
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G01S 2013/93275 in the bumper area
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 30/1914 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries
• CPC Class Codes: G06V 30/242 Division of the character sequences into groups prior to recognition; Selection of dictionaries
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A pedestrian detection device on a vehicle having a vehicular camera and a sonar or a radar includes: an identification dictionary storage unit that stores multiple identification dictionaries of shape features of a pedestrian; a detection result accumulation unit that searches the captured image for one feature to detect the pedestrian, and accumulates a detection result for each identification dictionary; a pedestrian position accumulation unit that detects the pedestrian using the sonar or the radar, and accumulates a pedestrian position; an identification dictionary selection unit that selects a detection identification dictionary having the detection result consistent with the pedestrian position; and a pedestrian detection unit that searches the captured image for one feature described in the detection identification dictionary, and detects the pedestrian.

Claims

1. A pedestrian detection device mounted on a vehicle having a vehicular camera and at least one of a sonar and a radar, and analyzing a captured image captured by the vehicular camera to detect a pedestrian in the captured image, the pedestrian detection device comprising:

an identification dictionary storage unit that, using a processor, stores a plurality of kinds of identification dictionaries having different numerical values obtained by converting a plurality of features of a shape of a pedestrian in a previously captured image into a numerical form;

a detection result accumulation unit that, using the processor, searches the plurality of kinds of identification dictionaries stored in the identification dictionary storage unit for a part of the captured image having one of the features described in the identification dictionaries to detect the pedestrian in the captured image, and accumulates a detection position, where the pedestrian is detected, as a detection result for each of the identification dictionaries;

a pedestrian position accumulation unit that, using the processor, detects the pedestrian disposed around the vehicle using at least one of the sonar and the radar, and accumulates a pedestrian position where the pedestrian is detected;

an identification dictionary selection unit that, using the processor, compares the pedestrian position accumulated in the pedestrian position accumulation unit with the detection position of the pedestrian accumulated as the detection result for each of the identification dictionaries in the detection result accumulation unit, and selects one of the identification dictionaries having the detection result consistent with the pedestrian position as a detection identification dictionary;

a pedestrian detection unit that, using the processor, searches for a part of the captured image having one of the features described in the detection identification dictionary, and detects the pedestrian in the captured image;

a detection accuracy evaluation unit that, using the processor, accumulates the detection position of the pedestrian obtained by the detection identification dictionary and the pedestrian position detected using at least one of the sonar and the radar, and evaluates a detection accuracy of the detection identification dictionary; and

a reselection necessity determination unit that, using the processor, determines a necessity of reselection of the detection identification dictionary based on an evaluation result obtained by the detection accuracy evaluation unit, wherein;

the detection result accumulation unit accumulates the detection result for each of the identification dictionaries when the reselection necessity determination unit determines that reselection of the detection identification dictionary is necessary.

2. The pedestrian detection device according to claim 1, wherein:

the detection result accumulation unit acquires information about a distance between the vehicle and the pedestrian based on the pedestrian position where the pedestrian is detected in the captured image, and accumulates the detection position of the pedestrian as the detection result of the pedestrian when the distance to the pedestrian falls within a threshold distance.

3. The pedestrian detection device according to claim 1, wherein:

the pedestrian detection unit includes;

a feature value extraction unit that converts a target image, shown in a part of the captured image to be searched, into a numerical form with respect to the plurality of features of the shape of the pedestrian to extract a plurality of feature values from the target image; and

an inner product value calculation unit that calculates an inner product value between the plurality of numerical values described in each of the identification dictionaries and the plurality of feature values extracted from the target image; and

the pedestrian detection unit detects the pedestrian by determining that the target image includes the pedestrian when the inner product value obtained by the inner product value calculation unit exceeds a predetermined threshold.

4. The pedestrian detection device according to claim 3 further comprising:

a detection result confirmation unit that confirms, using the processor and at least one of the sonar and the radar, whether the detection result of the pedestrian obtained by the pedestrian detection unit is correct or incorrect;

a correct detection feature value accumulation unit that, using the processor, accumulates, as correct detection feature values, the feature values obtained from the target image in which the pedestrian is detected when the pedestrian is detected by the pedestrian detection unit and the detection result confirmation unit determines that the detection result is correct;

an incorrect detection feature value accumulation unit that, using the processor, accumulates, as incorrect detection feature values, the feature values obtained from the target image in which the pedestrian is detected when the pedestrian is detected by the pedestrian detection unit and the detection result confirmation unit determines that the detection result is incorrect;

a learned identification dictionary generation unit that, using the processor, generates a learned identification dictionary by learning the plurality of numerical values of the detection identification dictionary so as to increase the inner product value obtained by the inner product value calculation unit with respect to the correct detection feature values and to reduce the inner product value obtained by the inner product value calculation unit with respect to the incorrect detection feature values; and

a learned identification dictionary addition unit that, using the processor, adds the learned identification dictionary to the identification dictionary storage unit as the identification dictionary.

5. The pedestrian detection device according to claim 4 further comprising:

an identification dictionary deleting unit that, using the processor, determines whether the number of kinds of the identification dictionaries stored in the identification dictionary storage unit reaches a predetermined upper limit number, and deletes a learned identification dictionary added at an oldest timing by the learned identification dictionary addition unit from the learned identification dictionaries stored in the identification dictionary storage unit when the number of kinds of the identification dictionaries reaches the upper limit number.

6. A pedestrian detection method applied to a vehicle having a vehicular camera and at least one of a sonar and a radar, and analyzing a captured image captured by the vehicular camera to detect a pedestrian included in the captured image, the pedestrian detection method comprising:

searching a plurality of kinds of identification dictionaries, described by converting a plurality of features of a shape of a pedestrian in a previously captured image into a numerical form, for a part of the captured image having one of the features described in the identification dictionaries to detect the pedestrian in the captured image; and

accumulating a detection position, where the pedestrian is detected, as a detection result for each of the identification dictionaries;

detecting the pedestrian disposed around the vehicle using at least one of the sonar and the radar; and

accumulating a pedestrian position where the pedestrian is detected;

comparing the pedestrian position accumulated in the accumulating of the pedestrian position with the detection position of the pedestrian accumulated as the detection result for each of the identification dictionaries in the accumulating of the detection result; and

selecting one of the identification dictionaries having the detection result consistent with the pedestrian position as a detection identification dictionary;

searching for a part of the captured image having one of the features described in the detection identification dictionary to detect the pedestrian in the captured image;

accumulating the detection position of the pedestrian obtained by the detection identification dictionary and the pedestrian position detected using at least one of the sonar and the radar, and evaluating a detection accuracy of the detection identification dictionary;

determining a necessity of reselection of the detection identification dictionary based on an evaluation result obtained in the evaluating of the detection accuracy; and

accumulating the detection position for each of the identification dictionaries when determining that reselection of the detection identification dictionary is necessary.

7. A pedestrian detection device mounted on a vehicle having a vehicular camera and at least one of a sonar and a radar, and analyzing a captured image captured by the vehicular camera to detect a pedestrian in the captured image, the pedestrian detection device comprising:

an identification dictionary storage unit that, using a processor, stores a plurality of kinds of identification dictionaries having different numerical values obtained by converting a plurality of features of a shape of a pedestrian in a previously captured image into a numerical form;

a detection result accumulation unit that, using the processor, searches the plurality of kinds of identification dictionaries stored in the identification dictionary storage unit for a part of the captured image having one of the features described in the identification dictionaries to detect the pedestrian in the captured image, and accumulates a detection position, where the pedestrian is detected, as a detection result for each of the identification dictionaries;

a pedestrian position accumulation unit that, using the processor, detects the pedestrian disposed around the vehicle using at least one of the sonar and the radar, and accumulates a pedestrian position where the pedestrian is detected;

an identification dictionary selection unit that, using the processor, compares the pedestrian position accumulated in the pedestrian position accumulation unit with the detection position of the pedestrian as the detection result for each of the identification dictionaries in the detection result accumulation unit, and selects one of the identification dictionaries having the detection result consistent with the pedestrian position as a detection identification dictionary;

a pedestrian detection unit that, using the processor, searches for a part of the captured image having one of the features described in the detection identification dictionary, and detects the pedestrian in the captured image; and

a detection accuracy evaluation unit that, using the processor, accumulates the detection position of the pedestrian obtained by the detection identification dictionary and the pedestrian position detected using at least one of the sonar and the radar, and evaluates a detection accuracy of the detection identification dictionary, wherein;

the identification dictionary selection unit selects the detection identification dictionary based on the detection accuracy.

8. A pedestrian detection method applied to a vehicle having a vehicular camera and at least one of a sonar and a radar, and analyzing a captured image captured by the vehicular camera to detect a pedestrian included in the captured image, the pedestrian detection method comprising:

searching a plurality of kinds of identification dictionaries, described by converting a plurality of features of a shape of the pedestrian in the captured image into a numerical form, for a part of the captured image having one of the features described in the identification dictionaries to detect the pedestrian in the captured image; and

accumulating a detection position, where the pedestrian is detected, as a detection result for each of the identification dictionaries;

detecting the pedestrian disposed around the vehicle using at least one of the sonar and the radar; and

accumulating a pedestrian position where the pedestrian is detected;

comparing the pedestrian position accumulated in the accumulating of the pedestrian position with the detection position of the pedestrian as the detection result for each of the identification dictionaries in the accumulating of the detection result; and

selecting one of the identification dictionaries having the detection result consistent with the pedestrian position as a detection identification dictionary;

searching for a part of the captured image having one of the features described in the detection identification dictionary to detect the pedestrian in the captured image; and

accumulating the detection position of the pedestrian obtained by the detection identification dictionary and the pedestrian position detected using at least one of the sonar and the radar, and evaluating a detection accuracy of the detection identification dictionary, wherein;

the selecting of the detection identification dictionary is performed based on the detection accuracy.