Method and device of neural network operations using a grid generator for converting modes according to classes of areas to satisfy level 4 of autonomous vehicles

• US 10,460,210 B1
• Filed: 01/22/2019
• Issued: 10/29/2019
• Est. Priority Date: 01/22/2019
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Insu, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, SukHoon, Sung, Myungchul, Yeo, Donghun, Ryu, Wooju, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Hsieh, Ping Y
• Application Number:
  US16/253,996
• Time in Patent Office: 280 Days
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method of neural network operations by using a grid generator is provided for converting modes according to classes of areas to satisfy level 4 of autonomous vehicles. The method includes steps of: (a) a computing device, if a test image is acquired, instructing a non-object detector to acquire non-object location information for testing and class information of the non-objects for testing by detecting the non-objects for testing on the test image; (b) the computing device instructing the grid generator to generate section information by referring to the non-object location information for testing; (c) the computing device instructing a neural network to determine parameters for testing; (d) the computing device instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, to thereby generate one or more neural network outputs.

Claims

1. A method for neural network operations by using a grid generator, comprising steps of:

(a) a computing device, if a test image is acquired, instructing a non-object detector to acquire non-object location information for testing, including information on where non objects for testing are located on the test image, and class information of the non-objects for testing, including information on classes of the non-objects for testing, by detecting the non-objects for testing on the test image;

(b) the computing device instructing the grid generator to generate section information, which includes information on a plurality of subsections in the test image, by referring to the non-object location information for testing;

(c) the computing device instructing a neural network to determine parameters for testing, to be used for applying the neural network operations to either (i) at least part of the subsections including each of the objects for testing and each of non-objects for testing corresponding to said each of the objects for testing, or (ii) each of sub-regions, in each of said at least part of the subsections, where said each of the non-objects for testing is located, by referring to parameters for training which have been learned by using information on non-objects for training whose corresponding class information is same as or similar to that of the non-objects for testing; and

(d) the computing device instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, corresponding to each of said at least part of the subsections, to thereby generate one or more neural network outputs.

2. The method of claim 1, wherein, at the step of (b), the grid generator divides the test image by using a dynamic template, and wherein the dynamic template is provided by adjusting one or more boundaries included therein, such that at least one of the non-objects for testing is included in each of said at least part of the subsections of the dynamic template.

3. The method of claim 2, wherein the grid generator manages information on the dynamic template, and wherein the dynamic template includes at least part of (i) at least one first boundary in a first direction and (ii) at least one second boundary in a second direction, some of which are adjustable.

4. The method of claim 2, wherein the dynamic template includes row groups and one or more column groups, wherein the row groups have an upper row group, a middle row group including at least part of the column groups, and a lower row group, and wherein at least one of the non-objects for testing is included in each of said at least part of the subsections formed by the row groups and the column groups.

5. The method of claim 1, before the step (a), comprising a step of:

(a0) the computing device instructing the neural network to learn the parameters for training by using information on each of the non-objects for training included in one or more training images.

6. The method of claim 5, wherein, at the step of (a0), the neural network learns the parameters for training by using information on each of the non-objects for training included in each of at least part of partial areas, from a first partial area to a k-th partial area, included in the training images, and store class information of the non-objects for training coupled with their corresponding parameters for training.

7. The method of claim 6, wherein, on condition that there are first location information to k-th location information corresponding to the first partial area to the k-th partial area in each of the training images, the training images, in which respective relative variations of at least part of the first location information to the k-th location information are equal to or smaller than a predetermined threshold value, are selected.

8. The method of claim 1, wherein, at the step of (c), (i) on condition that a ratio of an area of a specific object for testing to a specific subsection including the specific object for testing is smaller than a predetermined threshold value, the computing device instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one specific parameter for training among the parameters for training and (ii) on condition that the ratio of that is the same as or larger than the predetermined threshold value, the computing device instructs the neural network (ii-1) to determine a first part of the specific parameter for testing, to be used for applying the neural network operations to a sub-region, in the specific subsection, where a specific non-object for testing is located, by referring to the specific parameter for training among the parameters for training and (ii-2) to determine a second part of the specific parameter for testing, to be used for applying the neural network operations to a sub-region, in the specific subsection, where the specific object for testing is located, by referring to at least one additional parameter for training having been separately learned by using additional objects for training whose corresponding class information is the same as or similar to that of the specific object for testing.

9. The method of claim 1, wherein, at the step of (d), the computing device instructs the neural network to generate the neural network outputs by applying one or more convolutional operations and one or more deconvolutional operations to the test image with the parameters for testing.

10. The method of claim 9, wherein the test image represents road driving circumstances, the objects for testing represent objects existing on a road, and the neural network outputs are used for autonomous driving.

11. A computing device for neural network operations by using a grid generator, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to;

perform processes of (I) instructing a non-object detector to acquire non-object location information for testing, including information on where non-objects for testing are located on a test image, and class information of the non-objects for testing, including information on classes of the non-objects for testing, by detecting the non-objects for testing on the test image, (II) instructing the grid generator to generate section information, which includes information on a plurality of subsections in the test image, by referring to the non-object location information for testing, (III) instructing a neural network to determine parameters for testing, to be used for applying the neural network operations to either (i) at least part of the subsections including each of the objects for testing and each of non-objects for testing corresponding to said each of the objects for testing, or (ii) each of sub-regions, in each of said at least part of the subsections where said each of the non-objects for testing is located, by referring to parameters for training which have been learned by using information on non-objects for training whose corresponding class information is same as or similar to that of the non-objects for testing, and (IV) instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, corresponding to each of said at least part of the subsections, to thereby generate one or more neural network outputs.

12. The computing device of claim 11, wherein, at the process of (II), the grid generator divides the test image by using a dynamic template, and wherein the dynamic template is provided by adjusting one or more boundaries included therein, such that at least one of the non-objects for testing is included in each of said at least part of the subsections of the dynamic template.

13. The computing device of claim 12, wherein the grid generator manages information on the dynamic template, and wherein the dynamic template includes at least part of (i) at least one first boundary in a first direction and (ii) at least one second boundary in a second direction, some of which are adjustable.

14. The computing device of claim 12, wherein the dynamic template includes row groups and one or more column groups, wherein the row groups have an upper row group, a middle row group including at least part of the column groups, and a lower row group, and wherein at least one of the non-objects for testing is included in each of said at least part of the subsections formed by the row groups and the column groups.

15. The computing device of claim 11, before the process of (I), wherein the processor further performs a process of:

(0) instructing the neural network to learn the parameters for training by using information on each of the non-objects for training included in one or more training images.

16. The computing device of claim 15, wherein, at the process of (0), the neural network learns the parameters for training by using information on each of the non-objects for training included in each of at least part of partial areas, from a first partial area to a k-th partial area, included in the training images, and store class information of the non-objects for training coupled with their corresponding parameters for training.

17. The computing device of claim 16, wherein, on condition that there are first location information to k-th location information corresponding to the first partial area to the k-th partial area in each of the training images, the training images, in which respective relative variations of at least part of the first location information to the k-th location information are equal to or smaller than a predetermined threshold value, are selected.

18. The computing device of claim 11, wherein, at the process of (III), (i) on condition that a ratio of an area of a specific object for testing to a specific subsection including the specific object for testing is smaller than a predetermined threshold value, the processor instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one specific parameter for training among the parameters for training and (ii) on condition that the ratio of that is the same as or larger than the predetermined threshold value, the processor instructs the neural network (ii-1) to determine a first part of the specific parameter for testing, to be used for applying the neural network operations to a sub-region, in the specific subsection, where a specific non-object for testing is located, by referring to the specific parameter for training among the parameters for training and (ii-2) to determine a second part of the specific parameter for testing, to be used for applying the neural network operations to a sub-region, in the specific subsection, where the specific object for testing is located, by referring to at least one additional parameter for training having been separately learned by using additional objects for training whose corresponding class information is the same as or similar to that of the specific object for testing.

19. The computing device of claim 11, wherein, at the process of (IV), the processor instructs the neural network to generate the neural network outputs by applying one or more convolutional operations and one or more deconvolutional operations to the test image with the parameters for testing.

20. The computing device of claim 19, wherein the test image represents road driving circumstances, the objects for testing represent objects existing on a road, and the neural network outputs are used for autonomous driving.