Image processing device, object detection device, image processing method

• US 10,373,024 B2
• Filed: 04/02/2015
• Issued: 08/06/2019
• Est. Priority Date: 04/02/2015
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Nakamura, Katsuyuki, Akiyama, Yasuhiro, Kagehiro, Tatsuhiko
• Primary Examiner(s): Woldemariam, Aklilu K
• Application Number:
  US15/562,535
• Publication Number:
  US 20180107901A1
• Time in Patent Office: 1,587 Days
• Field of Search: 382103, 382117, 382118, 382167, 382190, 382209, 382276, 348241, 348239
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

An image processing device for evaluating a classifier which recognizes a target object within an input image and classifies the target object into one among a plurality of classes, includes: a classification unit which obtains a classification performance of the classifier by using the classifier and classifying the target object included in each of a plurality of verification images in which a class of the target object is known, and outputting one among the plurality of classes for each of the verification images; and an evaluation unit which outputs an evaluation result of the classifier based on the classification performance of the classifier obtained with the classification unit.

Claims

1. An image processing device for evaluating a classifier which performs recognition with respect to a plurality of classes, comprising:

a classification unit which obtains a classification performance of the classifier by using the classifier to classify a plurality of different target objects respectively included in a plurality of verification images in which the respective classes of the target objects are known, and output one of the classes among the plurality of classes for each of the verification images;

an evaluation unit which outputs a plurality of respective evaluation results of the classifier for the plurality of classes based on the classification performance of the classifier obtained with the classification unit, and identifies at least one of the classes with insufficient training data based on the respective evaluation results of each of the at least one of the classes not satisfying a predetermined threshold value,

a training data generation unit which generates training data including at least one training image for use in learning of the classifier based on the evaluation results; and

a learning unit which performs learning of the classifier based on the at least one training image,

wherein the training data generation unit generates each of the at least one training image by synthesizing, via physically based rendering, object data including geometric information and material information of a target object corresponding to the identified at least one of the classes with insufficient training data, and background data including a global image or three-dimensional shape information.

2. The image processing device according to claim 1, wherein:

the plurality of classes are configured by including a plurality of superordinate categories, and a plurality of sub categories obtained by additionally subdividing each of the plurality of superordinate categories;

the classification unit obtains, as the classification performance of the classifier, a plurality of respective evaluations value based on a predetermined evaluation standard for each of the superordinate categories and each of the sub categories; and

the evaluation unit outputs, based on the evaluation values, the evaluation results which identifies at least one among the plurality of superordinate categories and at least one among the plurality of sub categories as the at least one of the classes with insufficient training data.

3. The image processing device according to claim 2, wherein:

the evaluation unit determines the at least one among the plurality of sub categories based on the evaluation values and a correlation between each of the sub categories.

4. The image processing device according to claim 1, wherein:

the training data generation unit outputs the at least one training image, and at least one piece of information among coordinate information of a respective target object in each of the at least one training image, coordinate information of a specific part of the respective target object, class information of the respective target object, and attribute information of the respective target object.

5. The image processing device according to claim 1, wherein:

the training data generation unit stores history information of previously used parameters which were previously used when generating a previous at least one training image, and generates the at least one training image by using parameters which are different from the previously used parameters based on the history information.

6. The image processing device according to claim 1, further comprising:

an output unit which determines whether a learning state of the classifier by the learning unit satisfies a predetermined termination condition, wherein;

the learning unit terminates the learning of the classifier when the output unit determines that the learning state of the classifier has satisfied the termination condition.

7. The image processing device according to claim 1, further comprising:

a notification unit which notifies a learning state of the classifier by the learning unit to a user.

8. The image processing device according to claim 7, further comprising:

a reception unit which receives an instruction from the user and performs an operation for improving a classification accuracy of the target object by the classifier.

9. An object detection device, comprising:

an object detection unit which detects an object in an image input from a camera by using the classifier which was subject to learning by using the image processing device according to claim 1; and

a danger determination unit which determines a danger of an object detected by the object detection unit.

10. An image processing method executed by a computer for evaluating a classifier which performs recognition with respect to a plurality of classes, comprising:

obtaining a classification performance of the classifier by using the classifier to classify a plurality of different target objects respectively included in a plurality of verification images in which the respective classes of the target objects are known, and output one of the classes among the plurality of classes for each of the verification images;

outputting a plurality of respective evaluation results of the classifier for the plurality of classes based on the classification performance of the classifier, and

identifying at least one of the classes with insufficient training data based on the respective evaluation results of each of the at least one of the classes not satisfying a predetermined threshold value;

generating at least one training image for use in learning of the classifier based on the evaluation results; and

performing learning of the classifier based on the at least one training image, wherein each of the at least one training image is generated by synthesizing,

via physically based rendering, object data including geometric information and material information of a target object corresponding to the identified at least one of the classes with insufficient training data, and background data including a global image or three-dimensional shape information.