Learning method and learning device for integrating object detection information acquired through V2V communication from other autonomous vehicle with object detection information generated by present autonomous vehicle, and testing method and testing device using the same
First Claim
1. A learning method for generating integrated object detection information on an integrated target space including a first target space and a second target space, by integrating first object detection information on the first target space generated by a first vehicle and second object detection information on the second target space generated by a second vehicle, comprising steps of:
- (a) a learning device, if the first object detection information on the first target space and the second object detection information on the second target space are acquired by processing a first original image on the first target space and a second original image on the second target space, instructing a concatenating network included in a Deep Neural Network (DNN) to generate one or more pair feature vectors including information on one or more pairs of first original Regions-of-Interest (ROIs) included in the first target space and second original ROIs in the second target space;
(b) the learning device instructing a determining network included in the DNN to apply one or more fully-connected operations to the pair feature vectors, to thereby generate (i) one or more determination vectors including information on probabilities of the first original ROIs and the second original ROIs included in each of the pairs being appropriate to be integrated and (ii) one or more box regression vectors including information on each of relative 3-dimensional locations of integrated ROIs, corresponding to at least part of the pairs, comparing to each of original 3-dimensional locations of each component of said at least part of the pairs, on the integrated target space;
(c) the learning device instructing a loss unit to generate an integrated loss by referring to the determination vectors, the box regression vectors and their corresponding Ground Truths (GTs), and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN, whereinat the step of (a), a specific pair feature vector, which is one of the pair feature vectors, includes (i) first class information of a first specific object included in the first target space, (ii) feature values of a first specific original ROI including the first specific object, (iii) 3-dimensional coordinate values of a first specific original bounding box corresponding to the first specific original ROI, (iv) 3-dimensional coordinate values of the first specific original ROI, (v) second class information of a second specific object included in the second target space, (vi) feature values of a second specific original ROI including the second specific object, and (vii) 3-dimensional coordinate values of a second specific original bounding box corresponding to the second specific original ROI, and (viii) 3-dimensional coordinate values of the second specific original ROI, andat the step of (b), a specific determination vector, which is one of the determination vectors and corresponds to the specific pair feature vector, includes information on a probability of the first specific original ROI and the second specific original ROI being integrated on the integrated target space, and a specific box regression vector, which is one of the box regression vectors and corresponds to the specific pair feature vector, includes information on 3-dimensional coordinates of a specific integrated bounding box generated by merging the first specific original ROI and the second specific original ROI on the integrated target space.
1 Assignment
0 Petitions
Accused Products
Abstract
A learning method for generating integrated object detection information by integrating first object detection information and second object detection information is provided. And the method includes steps of: (a) a learning device instructing a concatenating network to generate one or more pair feature vectors; (b) the learning device instructing a determining network to apply FC operations to the pair feature vectors, to thereby generate (i) determination vectors and (ii) box regression vectors; (c) the learning device instructing a loss unit to generate an integrated loss by referring to the determination vectors, the box regression vectors and their corresponding GTs, and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN.
60 Citations
18 Claims
-
1. A learning method for generating integrated object detection information on an integrated target space including a first target space and a second target space, by integrating first object detection information on the first target space generated by a first vehicle and second object detection information on the second target space generated by a second vehicle, comprising steps of:
-
(a) a learning device, if the first object detection information on the first target space and the second object detection information on the second target space are acquired by processing a first original image on the first target space and a second original image on the second target space, instructing a concatenating network included in a Deep Neural Network (DNN) to generate one or more pair feature vectors including information on one or more pairs of first original Regions-of-Interest (ROIs) included in the first target space and second original ROIs in the second target space; (b) the learning device instructing a determining network included in the DNN to apply one or more fully-connected operations to the pair feature vectors, to thereby generate (i) one or more determination vectors including information on probabilities of the first original ROIs and the second original ROIs included in each of the pairs being appropriate to be integrated and (ii) one or more box regression vectors including information on each of relative 3-dimensional locations of integrated ROIs, corresponding to at least part of the pairs, comparing to each of original 3-dimensional locations of each component of said at least part of the pairs, on the integrated target space; (c) the learning device instructing a loss unit to generate an integrated loss by referring to the determination vectors, the box regression vectors and their corresponding Ground Truths (GTs), and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN, wherein at the step of (a), a specific pair feature vector, which is one of the pair feature vectors, includes (i) first class information of a first specific object included in the first target space, (ii) feature values of a first specific original ROI including the first specific object, (iii) 3-dimensional coordinate values of a first specific original bounding box corresponding to the first specific original ROI, (iv) 3-dimensional coordinate values of the first specific original ROI, (v) second class information of a second specific object included in the second target space, (vi) feature values of a second specific original ROI including the second specific object, and (vii) 3-dimensional coordinate values of a second specific original bounding box corresponding to the second specific original ROI, and (viii) 3-dimensional coordinate values of the second specific original ROI, and at the step of (b), a specific determination vector, which is one of the determination vectors and corresponds to the specific pair feature vector, includes information on a probability of the first specific original ROI and the second specific original ROI being integrated on the integrated target space, and a specific box regression vector, which is one of the box regression vectors and corresponds to the specific pair feature vector, includes information on 3-dimensional coordinates of a specific integrated bounding box generated by merging the first specific original ROI and the second specific original ROI on the integrated target space. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A testing method for generating integrated object detection information for testing on an integrated target space for testing including a first target space for testing and a second target space for testing, by integrating first object detection information for testing on the first target space for testing generated by a first vehicle for testing and second object detection information for testing on the second target space for testing generated by a second vehicle for testing, comprising steps of:
-
(a) on condition that (1) a learning device, if first object detection information for training on a first target space for training and second object detection information for training on a second target space for training have been acquired by processing a first original image for training on the first target space for training and a second original image for training on the second target space for training, has instructed a concatenating network included in a Deep Neural Network (DNN) to generate one or more pair feature vectors for training including information on one or more pairs for training of first original Regions-of-Interest (ROIs) for training included in the first target space for training and second original ROIs for training in the second target space for training;
(2) the learning device has instructed a determining network included in the DNN to apply one or more fully-connected operations to the pair feature vectors for training, to thereby generate (i) one or more determination vectors for training including information on probabilities for training of the first original ROIs for training and the second original ROIs for training included in each of the pairs for training being appropriate to be integrated and (ii) one or more box regression vectors for training including information on each of relative 3-Dimensional locations for training of integrated ROIs for training, corresponding to at least part of the pairs for training, comparing to each of original 3-Dimensional locations for training of each component of said at least part of the pairs for training, on an integrated target space for training;
(3) the learning device has instructed a loss unit to generate an integrated loss by referring to the determination vectors for training, the box regression vectors for training and their corresponding Ground Truths (GTs), and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN, a testing device installed on the first vehicle, if the first object detection information for testing on the first target space for testing and the second object detection information for testing on the second target space for testing are acquired by processing a first original image for testing on the first target space for testing and a second original image for testing on the second target space for testing, instructing the concatenating network included in the DNN to generate one or more pair feature vectors for testing including information on one or more pairs for testing of first original ROIs for testing included in the first target space for testing and second original ROIs for testing in the second target space for testing;(b) the testing device instructing the determining network included in the DNN to apply the FC operations to the pair feature vectors for testing, to thereby generate (i) one or more determination vectors for testing including information on probabilities for testing of the first original ROIs for testing and the second original ROIs for testing included in each of the pairs for testing being appropriate to be integrated and (ii) one or more box regression vectors for testing including information on each of relative 3-dimensional locations for testing of integrated ROIs for testing, corresponding to at least part of the pairs for testing, comparing to each of original 3-Dimensional locations for testing of each component of said at least part of the pairs for testing, on the integrated target space for testing; (c) the testing device instructing a merging unit to generate the integrated object detection information for testing by merging at least part of the pairs for testing of first original bounding boxes for testing and second original bounding boxes for testing by referring to the determination vectors for testing and the box regression vectors for testing. - View Dependent Claims (7, 8, 9)
-
-
10. A learning device for generating integrated object detection information on an integrated target space including a first target space and a second target space, by integrating first object detection information on the first target space generated by a first vehicle and second object detection information on the second target space generated by a second vehicle, comprising:
-
at least one memory that stores instructions; and at least one processor configured to execute the instructions to perform processes of (I) if the first object detection information on the first target space and the second object detection information on the second target space are acquired by processing a first original image on the first target space and a second original image on the second target space, instructing a concatenating network included in a Deep Neural Network (DNN) to generate one or more pair feature vectors including information on one or more pairs of first original Regions-of-Interest (ROIs) included in the first target space and second original ROIs in the second target space;
(II) instructing a determining network included in the DNN to apply one or more fully-connected operations to the pair feature vectors, to thereby generate (i) one or more determination vectors including information on probabilities of the first original ROIs and the second original ROIs included in each of the pairs being appropriate to be integrated and (ii) one or more box regression vectors including information on each of relative 3-dimensional locations of integrated ROIs, corresponding to at least part of the pairs, comparing to each of original 3-dimensional locations of each component of said at least part of the pairs, on the integrated target space;
(III) instructing a loss unit to generate an integrated loss by referring to the determination vectors, the box regression vectors and their corresponding Ground Truths (GTs), and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN, whereinat the process of (I), a specific pair feature vector, which is one of the pair feature vectors, includes (i) first class information of a first specific object included in the first target space, (ii) feature values of a first specific original ROI including the first specific object, (iii) 3-dimensional coordinate values of a first specific original bounding box corresponding to the first specific original ROI, (iv) 3-dimensional coordinate values of the first specific original ROI, (v) second class information of a second specific object included in the second target space, (vi) feature values of a second specific original ROI including the second specific object, and (vii) 3-dimensional coordinate values of a second specific original bounding box corresponding to the second specific original ROI, and (viii) 3-dimensional coordinate values of the second specific original ROI, and at the process of (II), a specific determination vector, which is one of the determination vectors and corresponds to the specific pair feature vector, includes information on a probability of the first specific original ROI and the second specific original ROI being integrated on the integrated target space, and a specific box regression vector, which is one of the box regression vectors and corresponds to the specific pair feature vector, includes information on 3-dimensional coordinates of a specific integrated bounding box generated by merging the first specific original ROI and the second specific original ROI on the integrated target space. - View Dependent Claims (11, 12, 13, 14)
-
-
15. A testing device for generating integrated object detection information for testing on an integrated target space for testing including a first target space for testing and a second target space for testing, by integrating first object detection information for testing on the first target space for testing generated by a first vehicle for testing and second object detection information for testing on the second tamet space for testing generated by a second vehicle for testing, comprising:
-
at least one memory that stores instructions; and at least one processor configured to execute the instructions to perform processes of;
(I) on condition that (1) a learning device, if first object detection information for training on a first target space for training and second object detection information for training on a second target space for training have been acquired by processing a first original image for training on the first target space for training and a second original image for training on the second target space for training, has instructed a concatenating network included in a Deep Neural Network (DNN) to generate one or more pair feature vectors for training including information on one or more pairs for training of first original Regions-of-Interest (ROIs) for training included in the first target space for training and second original ROIs for training in the second target space for training;
(2) the learning device has instructed a determining network included in the DNN to apply one or more fully-connected operations to the pair feature vectors for training, to thereby generate (i) one or more determination vectors for training including information on probabilities for training of the first original ROIs for training and the second original ROIs for training included in each of the pairs for training being appropriate to be integrated and (ii) one or more box regression vectors for training including information on each of relative 3-Dimensional locations for training of integrated ROIs for training, corresponding to at least part of the pairs for training, comparing to each of original 3-Dimensional locations for training of each component of said at least part of the pairs for training, on an integrated target space for training;
(3) the learning device has instructed a loss unit to generate an integrated loss by referring to the determination vectors for training, the box regression vectors for training and their corresponding Ground Truths (GTs), and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN, if the first object detection information for testing on the first target space for testing and the second object detection information for testing on the second target space for testing are acquired by processing a first original image for testing on the first target space for testing and a second original image for testing on the second target space for testing, instructing the concatenating network included in the DNN to generate one or more pair feature vectors for testing including information on one or more pairs for testing of first original ROIs for testing included in the first target space for testing and second original ROIs for testing in the second target space for testing;
(II) instructing the determining network included in the DNN to apply the FC operations to the pair feature vectors for testing, to thereby generate (i) one or more determination vectors for testing including information on probabilities for testing of the first original ROIs for testing and the second original ROIs for testing included in each of the pairs for testing being appropriate to be integrated and (ii) one or more box regression vectors for testing including information on each of relative 3-dimensional locations for testing of integrated ROIs for testing, corresponding to at least part of the pairs for testing, comparing to each of original 3-Dimensional locations for testing of each component of said at least part of the pairs for testing, on the integrated target space for testing;
(III) instructing a merging unit to generate the integrated object detection information for testing by merging at least part of the pairs for testing of first original bounding boxes for testing and second original bounding boxes for testing by referring to the determination vectors for testing and the box regression vectors for testing. - View Dependent Claims (16, 17, 18)
-
Specification