METHOD AND APPARATUS FOR RECONSTRUCTING 3D MICROSTRUCTURE USING NEURAL NETWORK

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
0Forward
Citations 
0
Petitions 
1
Assignment
First Claim
1. A method of generating a 3D microstructure using a neural network, the method comprising:
 configuring an initial 3D microstructure;
obtaining a plurality of crosssectional images by disassembling the initial 3D microstructure in at least one direction of the initial 3D microstructure;
generating first output feature maps with respect to at least one layer of the neural network based on each of the plurality of crosssectional images to the neural network;
generating second output feature maps with respect to the at least one layer in the neural network based on a 2D original image;
generating a 3D gradient by applying a loss value based on the first output feature maps and the second output feature maps to a backpropagation algorithm in the neural network; and
generating a final 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient.
1 Assignment
0 Petitions
Accused Products
Abstract
A method of generating a 3D microstructure using a neural network includes configuring an initial 3D microstructure; obtaining a plurality of crosssectional images by disassembling the initial 3D microstructure in at least one direction of the initial 3D microstructure; obtaining first output feature maps with respect to at least one layer that constitutes the neural network by inputting each of the crosssectional images to the neural network; obtaining second output feature maps with respect to at least one layer by inputting a 2D original image to the neural network; generating a 3D gradient by applying a loss value to a backpropagation algorithm after calculating the loss value by comparing the first output feature maps with the second output feature maps; and generating a final 3D microstructure based on the 2D original image by reflecting the 3D gradient to the initial 3D microstructure.
0 Citations
No References
No References
17 Claims
 1. A method of generating a 3D microstructure using a neural network, the method comprising:
configuring an initial 3D microstructure; obtaining a plurality of crosssectional images by disassembling the initial 3D microstructure in at least one direction of the initial 3D microstructure; generating first output feature maps with respect to at least one layer of the neural network based on each of the plurality of crosssectional images to the neural network; generating second output feature maps with respect to the at least one layer in the neural network based on a 2D original image; generating a 3D gradient by applying a loss value based on the first output feature maps and the second output feature maps to a backpropagation algorithm in the neural network; and generating a final 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient.  View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
 10. An apparatus configured to generate a 3D microstructure using a neural network, the apparatus comprising:
a memory configured to store at least one program; and at least one processor configured to process data in the neural network by executing the at least one program, wherein the at least one processor is configured to; configure an initial 3D microstructure; obtain a plurality of crosssectional images by disassembling the initial 3D microstructure in at least one direction; generate first output feature maps with respect to at least one layer of the neural network based on each of the plurality of crosssectional images to the neural network; generate second output feature maps with respect to the at least one layer in the neural network based on a 2D original image; generate a 3D gradient by applying a loss value based on the first output feature maps with the second output feature maps to a backpropagation algorithm in the neural network; and generate a final 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient.  View Dependent Claims (11, 12, 13, 14, 15, 16, 17)
1 Specification
This application is based on and claims priority from Korean Patent Application No. 1020180130242, filed on Oct. 29, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.
The disclosure relates to methods and apparatuses for reconstructing a threedimensional (3D) microstructure using a neural network, and more particularly, to methods and apparatuses for reconstructing a 3D microstructure from a twodimensional (2D) original image using a convolution neural network.
Recently, many studies have been conducted related to design and discovery of new material. In order to detect an interrelation of structurecharacteristic of a new material or physical properties of the new material that may not be observable on a 2D image of a material, there is need to obtain a 3D structure of the material.
However, when many crosssections of the new material are needed to obtain a 3D structure, excessive energy consumption and loss of image quality may occur.
Also, in the related art, a method of reconstructing a 3D microstructure of the material is based on a probability determination, but does not use convolution neural network. Therefore, there have been problems with accuracy due to the difference between a shape of particles that constitute generated 3D microstructure and that of a 2D original image.
Provided are methods and apparatuses for generating a 3D microstructure using a neural network.
Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.
According to an aspect of the disclosure, there is provided a method of generating a 3D microstructure using a neural network, the method comprising: configuring an initial 3D microstructure; obtaining a plurality of crosssectional images by disassembling the initial 3D microstructure in at least one direction of the initial 3D microstructure; generating first output feature maps with respect to at least one layer of the neural network based on each of the crosssectional images to the neural network; generating second output feature maps with respect to at least the one layer in the neural network based on a 2D original image; generating a 3D gradient by applying a loss value based on the first output feature maps and the second output feature maps to a backpropagation algorithm in the neural network; and generating a final 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient.
The generating of the 3D gradient may comprise: generating 2D gradients respectively corresponding to each of the plurality of crosssectional images by applying the loss value to the backpropagation algorithm, the loss value being calculated by comparing the first output feature maps with the second output feature maps.
The at least one direction may comprise one of a first direction parallel to an xy plane, a second direction parallel to a yz plane, or a third direction parallel to an xz plane of the initial 3D microstructure, and the plurality of crosssectional images may comprise a plurality of first crosssectional images obtained by disassembling the initial 3D microstructure in the first direction parallel to the xy plane, a plurality of second crosssectional images obtained by disassembling the initial 3D microstructure in the second direction parallel to the yz plane, and a plurality of third crosssectional images obtained by disassembling the initial 3D microstructure in the third direction parallel to the xz plane.
The generating of the 3D gradient may comprise: generating 2D gradients respectively corresponding to each of the plurality of crosssectional images by applying the loss value to the backpropagation algorithm; generating a first 3D gradient from the 2D gradients respectively corresponding to the plurality of first crosssectional images; generating a second 3D gradient from the 2D gradients respectively corresponding to the plurality of second crosssectional images; generating a third 3D gradient from the 2D gradients respectively corresponding to the plurality of third crosssectional images; and generating a final 3D gradient by adding the first 3D gradient, the second 3D gradient, and the third 3D gradient.
The initial 3D microstructure may have a pixel size of a×b×c (a, b, and c respectively are natural numbers), the total number of the first crosssectional images, the second crosssectional images, and the third crosssectional images is a+b+c.
The generating of the final 3D microstructure may comprise: generating an intermediate 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient; generating a new 3D gradient based on the intermediate 3D microstructure when the loss value is greater than a predetermined value and modifying the intermediate 3D microstructure based on the new 3D gradient; and determining the intermediate 3D microstructure as the final 3D microstructure when the loss value is less than the predetermined value.
The generating of the 3D gradient may comprise: obtaining a first gram matrix using the first output feature maps respectively output from at least one layer; and obtaining a second gram matrix using the second output feature maps respectively output from at least one layer, wherein the loss value is calculated by comparing the first gram matrix with the second gram matrix.
The neural network may comprise a convolution neural network (CNN).
According to another aspect of the disclosure, there is provided a nontransitory computerreadable recording medium comprising a computerreadable program for executing a method of generating a 3D microstructure using a neural network, the method comprising: configuring an initial 3D microstructure; obtaining a plurality of crosssectional images by disassembling the initial 3D microstructure in at least one direction of the initial 3D microstructure; generating first output feature maps with respect to at least one layer of the neural network based on each of the crosssectional images to the neural network; generating second output feature maps with respect to at least the one layer in the neural network based on a 2D original image; generating a 3D gradient by applying a loss value based on the first output feature maps and the second output feature maps to a backpropagation algorithm in the neural network; and generating a final 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient.
According to another aspect of the disclosure, there is provided an apparatus configured to generate a 3D microstructure using a neural network, the apparatus comprising: a memory configured to store at least one program; and at least one processor configured to process data in the neural network by executing the at least one program, wherein the at least one processor is configured to: configure an initial 3D microstructure; obtain a plurality of crosssectional images by disassembling the initial 3D microstructure in at least a direction; generate first output feature maps with respect to at least one layer of the neural network based on each of the crosssectional images to the neural network; generate second output feature maps with respect to the at least one layer in the neural network based on a 2D original image; generate a 3D gradient by applying a loss value based on the first output feature maps with the second output feature maps to a backpropagation algorithm in the neural network; and generate a final 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient.
The at least one processor may generate 2D gradients respectively corresponding to each of the plurality of crosssectional images by applying the loss value to the backpropagation algorithm, the loss value being calculated by comparing the first output feature maps with the second output feature maps; and generate the 3D gradient by stacking the 2D gradients.
The at least one direction may comprise one of a first direction parallel to an xy plane, a second direction parallel to a yz plane, or a third direction parallel to an xz plane of the initial 3D microstructure, and the plurality of crosssectional images comprises a plurality of first crosssectional images obtained by disassembling the initial 3D microstructure in the first direction parallel to the xy plane, a plurality of second crosssectional images obtained by disassembling the initial 3D microstructure in the second direction parallel to the yz plane, and a plurality of third crosssectional images obtained by disassembling the initial 3D microstructure in the third direction parallel to the xz plane.
The at least one processor may be further configured to: generate 2D gradients respectively corresponding to each of the plurality of crosssectional images by applying the loss value to the backpropagation algorithm; generate a first 3D gradient from 2D gradients respectively corresponding to the first crosssectional images; generate a second 3D gradient from the 2D gradients respectively corresponding to the second crosssectional images; generates a third 3D gradient from the 2D gradients respectively corresponding to the third crosssectional images; and generate a final 3D gradient by adding the first 3D gradient, the second 3D gradient, and the third 3D gradient.
The initial 3D microstructure may have a pixel size of a×b×c (a, b, and c respectively are natural numbers), the total number of first crosssectional images, second crosssectional images, and third crosssectional images is a+b+c.
The at least one processor may be further configured to: generate an intermediate 3D microstructure by modifying the initial 3D microstructure based on the 3D gradient; generate a new 3D gradient based on the intermediate 3D microstructure when the loss value is greater than a predetermined value and modifies the intermediate 3D microstructure based on the new 3D gradient, and determine the intermediate 3D microstructure as the final 3D microstructure when the loss value is less than the predetermined value.
The at least one processor may be further configured to: obtain a first gram matrix using the first output feature maps respectively output from at least one layer; and obtain a second gram matrix using the second output feature maps respectively output from at least one layer, wherein the loss value is calculated by comparing the first gram matrix with the second gram matrix.
The neural network may comprise a convolution neural network (CNN).
According to another aspect of the disclosure, there is provided an apparatus configured to generate a 3D microstructure of material using a neural network, the apparatus comprising: a memory configured to store at least one program; and at least one processor configured to process data in the neural network by executing the at least one program, wherein the at least one processor is configured to: receive a first 3D microstructure image; obtain a plurality of crosssectional images from the first 3D microstructure; generates a plurality of first output feature map information with respect to at least one layer of the neural network based on the plurality of crosssectional images, each of the plurality of first output features map information uniquely corresponding to one of the plurality of crosssectional images; receives a 2D original surface image of the material; generates a second output feature map information with respect to at least one layer of the neural network based on the 2D original surface image; generates a 3D gradient by comparing each of the plurality of first output feature map information with the second output feature map information; and generates a second 3D microstructure based on the generated 3D gradient.
In generating the 3D gradient, the at least one processor may be further configured to: calculate a plurality of loss values based on results of the comparing each of the plurality of first output feature map information with the second output feature map information; generate a plurality of 2D gradients corresponding to each of the plurality of crosssectional images by applying the plurality of loss values to a backpropagation algorithm in the neural network; and stack the plurality of 2D gradients to generate the 3D gradient.
The 2D original surface image may be an image captured by an image capturing device.
According to another aspect of the disclosure, there is provided a method of generating a 3D microstructure of a material using a neural network, the method comprising: receiving a first 3D microstructure image; obtaining a plurality of crosssectional images from the first 3D microstructure; generating a plurality of first output feature map information with respect to at least one layer of the neural network based on the plurality of crosssectional images, each of the plurality of first output features map information uniquely corresponding to one of the plurality of crosssectional images; receiving a 2D original surface image of the material; generating a second output feature map information with respect to at least one layer of the neural network based on the 2D original surface image; generating a 3D gradient by comparing each of the plurality of first output feature map information with the second output feature map information; and generating a second 3D microstructure based on the generated 3D gradient.
The generating the 3D gradient may comprise: calculating a plurality of loss values based on results of the comparing each of the plurality of first output feature map information with the second output feature map information; generating a plurality of 2D gradients corresponding to each of the plurality of crosssectional images by applying the plurality of loss values to a backpropagation algorithm in the neural network; and stacking the plurality of 2D gradients to generate the 3D gradient.
The 2D original surface image may be an image captured by an image capturing device.
These and/or other aspects will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings in which:
Terminologies used herein are selected as commonly used by those of ordinary skill in the art in consideration of functions of the current embodiment, but may vary according to the technical intention, precedents, or a disclosure of a new technology. Also, in particular cases, some terms are arbitrarily selected by the applicant, and in this case, the meanings of the terms will be described in detail at corresponding parts of the specification. Accordingly, the terms used in the specification should be defined not by simply the names of the terms but based on the meaning and contents of the whole specification.
In the descriptions of the embodiments, it will be understood that, when an element is referred to as being connected to another element, it may include electrically connected when the element is directly connected to the other element and when the element is indirectly connected to the other element by intervening a constituent element. Also, it should be understood that, when a part “comprises” or “includes” a constituent element in the specification, unless otherwise defined, it is not excluding other elements but may further include other elements.
It will be further understood that the term “comprises” or “includes” should not be construed as necessarily including various constituent elements and various operations described in the specification, and also should not be construed that portions of the constituent elements or operations of the various constituent elements and various operations may not be included or additional constituent elements and operations may further be included.
The descriptions of the embodiments should not be interpreted as limiting the scope of right, and embodiments that are readily inferred from the detailed descriptions and embodiments by those of ordinary skill in the art will be construed as being included in the inventive concept. Hereinafter, references will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings.
A neural network refers to a class of models that may be used to solve problems by adjusting the binding strength of synapses through learning.
Referring to
The neural network 1 may be realized as an architecture having a plurality of layers including an input image, feature maps, and an output. In the neural network 1, the input image performs a convolution computation with a filter referred to as a kernel, and feature maps are output as a result. The output feature maps generated at this process are input feature maps, and the input feature maps reperform a convolution computation with a kernel, and, as a result, new feature maps are output. In this way, as a result of repeatedly performing the convolution computations, a recognition result with respect to characteristics of input image may be finally output through the neural network 1.
For example, when an image having a size of 24×24 is input to the neural network 1, the input image may be output as four channel feature maps having a size of 20×20 through convolution computations with a kernel. Also, four channel feature maps having a size of 10×10 may be output by using some of the pixel values of the four channel feature maps having a size of 20×20 through a subsampling. The subsampling method may include a maxpooling method, an averagepooling method, etc.
Thereafter, a size of the feature maps having a size of 10×10 is reduced through repeated convolution computation and subsampling computation with a kernel, and finally, global characteristics of the input image is output. The neural network 1 may filter and output touch characteristics that may represent the whole image from an input image by repeatedly performing the convolution computation and the subsampling computation with the kernel and may draw a recognition result with respect to a final input image by inputting the output global characteristics to a fully connected layer.
Referring to
A convolution computation with respect to the first feature map FM1 and a kernel may be performed, and as a result, the second feature map FM2 may be generated. The kernel may be a weight defined on each element and may filter characteristics of the first feature map FM1 by performing a convolution computation with the first feature map FM1. The kernel performs a convolution computation with windows (or tiles) of the first feature map FM1 while shifting the first feature map FM1 with a sliding window method. For each shift, weights included in the kernel respectively may be multiplied by or added to pixel values of windows overlapped in the first feature map FM1. As the first feature map FM1 and the kernel are convoluted, a single channel of the second feature map FM2 may be generated. In
The second feature map FM2 may correspond to an input feature map of the next layer. For example, the second feature map FM2 may be an input feature map of a pooling (or a subsampling) layer.
In
Referring to
The 2D original image 310 may be acquired from a material 300 for generating a 3D microstructure. A surface image of the material 300 for generating a 3D microstructure may be readily acquired, but, when a crosssection of the material 300 is cut many times to understand an internal 3D microstructure, excessive energy may be consumed and loss of an image may occur. Accordingly, the disclosure provides a method of generating an internal 3D microstructure of the material 300 by using a neural network based on a surface image, that is, the 2D original image 310 of the material 300.
In detail, the 2D original image 310 may be a surface image at an xy plane, a yz plane, and an xz plane of the material 300 for generating a 3D microstructure according to an embodiment, but the embodiment is not limited thereto.
The 2D original image 310 may be obtained by using an imaging device.
According to an embodiment, the imaging device may be a scanning electron microscope (SEM) or an optical microscope. The SEM may obtain a 2D scanning image that shows a surface shape of the material 300 by injecting an electron beam onto the material 300.
The 2D original image 310 may be input to the convolution neural network 320 that is learned in advance. Since the 2D original image 310 is input to the convolution neural network 320, feature maps may be output from each layer that constitutes the convolution neural network 320. In order to generate a 3D microstructure, the neural network apparatus may use feature maps of the 2D original image 310 output from at least one arbitrary layer that constitutes the convolution neural network 320. Referring to
Referring to
In order to configure the initial 3D microstructure 330, a neural network apparatus may define a size of a final 3D microstructure to be generated in advance. The initial 3D microstructure 330 may be an arbitrary 3D microstructure configured to have the same or equal size of the defined final 3D microstructure.
After the size of the 3D microstructure is determined, a plurality of crosssectional images may be obtained by cutting the initial 3D microstructure 330 in at least a direction. Referring to
Each of the crosssectional images 340 may be input to the learned convolution neural network 320. Since each of the crosssectional images 340 is input to the learned convolution neural network 320, feature maps may be output from layers that constitute the learned convolution neural network 320. The neural network apparatus may use the crosssectional images 340 output from the at least one arbitrary layer that constitutes the learned convolution neural network 320 to generate a 3D microstructure.
The neural network apparatus may obtain second output feature maps 360 from an arbitrary 1^{th }layer by inputting a first crosssectional image 3401 of the crosssectional images 340 to a neural network. The neural network apparatus may repeat this operation with respect to each of the crosssectional images 340, and finally, may obtain second output feature maps 370 from the 1^{th }layer by inputting an N^{th }crosssectional image 340N of the crosssectional images 340 to the neural network.
The neural network apparatus may calculate a loss value of the 2D original image 310 in at least one layer that constitutes the convolution neural network 320 with respect to each of the crosssectional images 340. In detail, the neural network apparatus may calculate a loss value by comparing the first output feature maps 350 of the 2D original image 310 output from an arbitrary 1^{th }layer with the second output feature maps 370 of each of the crosssectional images 340 of the initial 3D microstructure 330.
For example, the neural network apparatus may calculate a loss value by comparing the first output feature maps 350 with the second output feature maps 360 of the first crosssectional image 3401 of the crosssectional images 340. Likewise, the neural network apparatus may calculate a loss value by comparing the first output feature maps 350 with the N^{th }crosssectional image 340N of the crosssectional images 340.
The neural network apparatus may generate a 2D gradient by backpropagating the calculated loss value. Accordingly, the neural network apparatus may generate 2D gradients respectively corresponding to the crosssectional images 340. Finally, the neural network apparatus may generate a 3D gradient by stacking the 2D gradients. The generated 3D gradient is reflected to the initial 3D microstructure 330, and thus, may update the initial 3D microstructure 330. The loss value calculation and the generation of gradient will be described in detail below with reference to
Referring to
At this point, feature maps 440 of the 2D original image 410 output from each layer of the learned convolution neural network 420 may be vectorized in a matrix type. A matrix vectorized of the feature maps 440 of the 2D original image 410 output from an arbitrary l^{th }layer of the learned convolution neural network 420 is referred to as an S_{1}.
Likewise, feature maps of the crosssectional images 430 output from each layer of the learned convolution neural network 420 may be vectorized in a matrix type. A matrix vectorized of feature maps 450 of the crosssectional images 430 output from an arbitrary l^{th }layer of the learned convolution neural network 420 is referred to as a F_{1}.
Next, a loss value L that indicates a difference between the 2D original image 410 and the crosssectional image 430 at the arbitrary layer may be calculated by using the matrix S_{l }of the 2D original image 410 and the matrix F_{l }of the crosssectional image 430 of an initial 3D microstructure.
The loss value L at the layer may be calculated such that gram matrixes are obtained with respect to the matrix S_{l }and the matrix F_{l}, and a square of Frobenius norm between a gram matrix G(S_{l}) of the matrix S_{l }and a gram matrix G(F_{l}) of the gram matrix F_{l }is obtained. That is, a loss value L^{l }at a l^{th }layer may be expressed as Equation 1 below.
L^{l}=Σ(G(S_{l})−G(G_{l}))^{2} [Equation 1]
The gram matrix may include information indicating a relationship of the feature maps generated through a convolution layer of the learned convolution neural network 420. The gram matrix may be expressed as Equation 2 below.
When a l^{th }layer of the learned convolution neural network 420 includes c_{l }number of filters, the c_{l }number of feature maps generated by passing through the l^{th }layer respectively may have a size of M_{l }obtained by multiplying a width W_{l }and a height H_{l}. Accordingly, the c_{l }number of feature maps generated by passing through the l^{th }layer may be vectorized to a matrix A^{l }having a size of [c_{l}×M_{l}] (c_{l }is the number of channels). When a gram matrix is obtained by using the matrix A^{l}, the size of the gram matrix may be [c_{l}×c_{l}].
Referring to Equation 2, a gram matrix G_{ij}^{l }may be obtained by an inner production of A_{ik}^{l }indicating a vectorized feature map by passing through an i^{th }filter in and A_{jk}^{l }indicating a vectorized feature map by passing through j^{th }filter in a l^{th }convolution layer.
According to Equation 2, a gram matrix of a matrix S_{l }that vectorizes the feature maps 450 of the 2D original image 410 output from the l^{th }layer may be expressed as Equation 3 below.
Likewise, a gram matrix of matrix F_{l }that vectorizes the feature maps 450 of the crosssectional images 430 of the initial 3D microstructure output from the l^{th }layer may be expressed as Equation 4 below.
A loss value L^{l }indicating a difference between the 2D original image 410 and the crosssectional image 430 in the l^{th }layer according to Equation 1 may be calculated by using the gram matrix of the matrix S_{l }calculated according to Equation 3 and the gram matrix of the matrix F_{l }calculated according to Equation 4.
The neural network apparatus may generate a 2D gradient 460 by backpropagating the loss value L^{l }at the l^{th }layer to the learned convolution neural network 420, as described in detail with reference to
Meanwhile, in
Referring to
The learned convolution neural network 520 may be, for example, a VGG19 network, but is not limited thereto. The VGG19 network may be configured of total 16 convolution layers, 5 pooling layers, and 3 fully connected layers. The learned convolution neural network 520 may be configured of convolution layers and pooling layers without including fully connected layers.
When the 2D original image 510 is input to the learned convolution neural network 520, feature maps output from each of the pooling layers of the learned convolution neural network 520 may be obtained.
Likewise, when the crosssectional image 530 of an initial 3D microstructure is input to the learned convolution neural network 520, feature maps output from each of the pooling layers of the learned convolution neural network 520 may be obtained.
At this point, a loss value L^{l }may be calculated by using the feature maps of the 2D original image 510 and the feature maps of the crosssectional image 530 output from a l^{th }pooling layer of the learned convolution neural network 520. The process of calculating the loss value L^{l }may be the same as the process described with reference to
A final loss value L indicating a difference between the 2D original image 510 and the crosssectional image 530 of an initial 3D microstructure may be obtained by adding all the loss values L^{l }calculated in each of the pooling layers of the learned convolution neural network 520. The loss L may be expressed as Equation 5 below.
Next, the neural network apparatus may calculate a gradient by applying a loss value to a backpropagation algorithm. The backpropagation algorithm is an algorithm that renews a neural network so that the loss value is minimum by sequentially performing partial differential in a direction opposite to the output direction.
In detail, a 2D gradient
540 may be generated from the loss value L by using a gradient descent algorithm that calculates a gradient of error in the backpropagation algorithm. {right arrow over (x)} may be a vector including information of pixels constituting the crosssectional image 530 of an initial 3D microstructure.
Referring to
In this case, the plurality of crosssectional images may include first crosssectional images 630 obtained by disassembling the initial 3D microstructure 620 in a direction parallel to the xy plane, second crosssectional images 640 obtained by disassembling the initial 3D microstructure 620 in a direction parallel to the yz plane, and third crosssectional images 650 obtained by disassembling the initial 3D microstructure 620 in a direction parallel to the xz plane.
The neural network apparatus may generate 2D gradients respectively corresponding to a plurality of crosssectional images by inputting a 2D original image 610 and each of the plurality of crosssectional images to a convolution neural network. The 2D gradients may be generated from a loss value using a gradient descent algorithm, and the magnitude of the 2D gradients may be equal to the size of the crosssectional image.
In detail, the neural network apparatus may calculate a loss value between the 2D original image 610 and each of the first crosssectional images 630, and may generate 2D gradients 660 corresponding to the first crosssectional images 630 from the loss value.
Likewise, the neural network apparatus may calculate a loss value between the 2D original image 610 and each of the second crosssectional images 640, and may generate 2D gradients 670 corresponding to the second crosssectional images 640 from the loss value. Finally, the neural network apparatus may calculate a loss value between the 2D original image 610 and each of the third crosssectional images 650, and may generate 2D gradients 680 corresponding to the third crosssectional images 650 from the loss value.
Surface images at an xy plane, a yz plane, and an xz plane of a material to generate a 3D microstructure may be different from each other. Accordingly, the 2D original image may be a plurality of 2D original images, for example, a first 2D original image which is a surface image at the xy plane of the material, a second 2D original image which is a surface image at the yz plane of the material, and a third 2D original image which is a surface image at the xz plane of the material.
In this case, the neural network apparatus may generate 2D gradients by calculating loss values between the first 2D original image and each of the first crosssectional images 630, loss values between the second 2D original image and each of the second crosssectional images 640, and loss values between the third 2D original image and each of the third crosssectional images 650.
Next, the neural network apparatus may generate a 3D gradient by stacking the 2D gradients respectively corresponding to the crosssectional images.
In detail, the neural network apparatus may generate a first 3D gradient 661 by stacking 2D gradients 660 respectively corresponding to the first crosssectional images 630. Likewise, the neural network apparatus may generate a second 3D gradient 671 by stacking 2D gradients 670 respectively corresponding to the second crosssectional images 640 and may generate a third 3D gradient 681 by stacking 2D gradients 680 respectively corresponding to the third crosssectional images 650.
Finally, the neural network apparatus may generate a final 3D gradient 690 by adding the first 3D gradient 661, the second 3D gradient 671, and the third 3D gradient 681. The neural network apparatus may update the initial 3D microstructure by reflecting the final 3D gradient 690 to the initial 3D microstructure.
Referring to
Assuming that a pixel size in a zaxis direction of the initial 3D microstructure is N, the number of first crosssectional images 710 obtained by disassembling the initial 3D microstructure in a direction parallel to the xy plane may be N. Likewise, assuming that a pixel size in an xaxis direction of the initial 3D microstructure is N, the number of second crosssectional images 720 obtained by disassembling the initial 3D microstructure in a direction parallel to the yz plane may be N, and also, assuming that a pixel size in a yaxis direction of the initial 3D microstructure is N, the number of third crosssectional images 730 obtained by disassembling the initial 3D microstructure in a direction parallel to the xz plane may be N.
The neural network apparatus may generate 2D gradients respectively corresponding to the first crosssectional images 710. For example, a 2D gradient 7401 corresponding to a first crosssectional image 7101 of the first crosssectional images 710 may be _{xy,1}, and a 2D gradient 740N corresponding to an N^{th }crosssectional image 710N of the first crosssectional images 710 may be _{xy,N}.
Also, the neural network apparatus may generate 2D gradients 750 corresponding to the second crosssectional images 720. For example, a 2D gradient 7501 corresponding to a first crosssectional image 7201 of the second crosssectional images 720 may be _{yz,1}, and a 2D gradient 750N corresponding to an N^{th }crosssectional image 720N of the first crosssectional images 720 may be _{yz,N}.
Likewise, the neural network apparatus may generate 2D gradients 760 corresponding to the third crosssectional images 730. For example, a 2D gradient 7601 corresponding to a first crosssectional image 7301 of the second crosssectional images 730 may be _{xz,1}, and a 2D gradient 760N corresponding to an N^{th }crosssectional image 730N of the first crosssectional images 730 may be _{xz,N}.
Next, the neural network apparatus may generate a 3D gradient 741 _{xy }by stacking the _{xy,1 }through _{xy,N }which are 2D gradients generated with respect to each of the N number of the first crosssectional images 710.
Likewise, the neural network apparatus may generate a 3D gradient 751 _{yz }by stacking the _{yz,1 }through _{yz,N }which are 2D gradients generated with respect to each of the second crosssectional images 720, and also, may generate a 3D gradient 761 _{xy }by stacking the _{xz,1 }through _{xz,N }which are 2D gradients generated with respect to each of the third crosssectional images 730.
Finally, the neural network apparatus may generate a final 3D gradient 770 _{total }by adding all of the 3D gradients _{xy}, _{yz}, and _{xz}. The neural network apparatus may update the initial 3D microstructure by reflecting the final 3D gradient 770 to the initial 3D microstructure.
The neural network apparatus may again obtain a plurality of crosssectional images by disassembling the updated 3D microstructure in directions respectively parallel to the xy plane, the yz plane, and the xz plane and may calculate loss values between a 2D original image and each of the plurality of crosssectional images.
At this point, a loss value may be determined as to whether the loss value is greater than a predetermined threshold value or not. The predetermined threshold value may be a reference value for determining the degree of similarity between a 2D original image and a crosssectional image of an initial 3D microstructure and may be defined in advance.
When the loss value is greater than the predetermined threshold value, a new final 3D gradient may be generated by using the loss value. The new final 3D gradient may be reflected to the updated initial 3D microstructure, and thus, the updated initial 3D microstructure may be reupdated.
When the loss value is less than the predetermined threshold value, the updated initial 3D microstructure may be determined as the final 3D microstructure.
Through the algorithm described above, a final 3D microstructure may be generated based on a 2D original image by repeatedly updating the initial 3D microstructure.
In detail, the neural network apparatus 800 may be a device that generates a 3D microstructure by using a neural network.
The neural network apparatus 800 may be realized as various kinds of devices, such as personal computers (PCs), mobile devices, embedded devices, etc. As practical examples, smart phones, tablet devices, augmented reality (AR) devices, internet of things (IoT) devices, autonomous vehicles, robotics, or medical devices that perform a voice recognition, an image recognition, an image classification using a neural network, but the devices are not limited thereto. Furthermore, the neural network apparatus 800 may correspond to an exclusive hardware (HW) accelerator mounted on the devices described above. The neural network apparatus 800 may be a HW accelerator, such as a neural processing unit (NPU), a tensor processing unit (TPU), an neural engine, etc. which is an exclusive module for driving a neural network, but is not limited thereto.
Referring to
The processor 810 may control overall functions for executing the neural network apparatus 800. For example, the processor 810 controls the neural network apparatus 800 by executing programs stored in the memory 820 of the neural network apparatus 800. The processor 810 may be realized as a central processing unit (CPU), a graphics processing unit (GPU), or an application processor (AP) included in the neural network apparatus 800, but is not limited thereto.
The memory 820 is hardware that stores various data processing in the neural network apparatus 800. For example, the memory 820 may store data processed and data to be processed in the neural network apparatus 800. Also, the memory 820 may store applications to be driven by the neural network apparatus 800 or drivers, etc. The memory 820 may include random access memory (RAM), such as dynamic RAM or static RAM, readonly memory (ROM), electrically erasable programmable readonly memory (EEPROM), a CDROM, a blue ray, or other optical disc storage, a hard disc drive (HDD), a solid state drive (SSD) or a flash memory.
The processor 810 may read/write data, for example, image data, feature map
data, etc. to be processed in the neural network apparatus 800 from the memory 820 and may execute a neural network by using the read/write data.
For example, the processor 810 may obtain a 2D original image obtained by using an SEM or an optical microscope from the memory 820. The processor 810 may define a size of a final 3D microstructure to be generated and may configure an arbitrary 3D microstructure having a size equal to that of the final 3D microstructure as an initial 3D microstructure.
The processor 810 may obtain a plurality of crosssectional images by disassembling the initial 3D microstructure in directions parallel to the xy plane, the yz plane, and the xz plane of the initial 3D microstructure. The processor 810 may obtain output feature map data in at least a layer than constitutes a neural network by inputting a 2D original image and a plurality of crosssectional images of an initial 3D microstructure into the neural network. Unlike the neural network 1 depicted in
The processor 810 may calculate a loss value that indicates a difference between a 2D original image and a crosssectional image of an initial 3D microstructure by using the output feature map data. The processor 810 may calculate a final loss value L by adding all loss values between the 2D original image and a crosssectional image of the initial 3D microstructure of an arbitrary l^{th }layer of the neural network.
The processor 810 may obtain a 2D gradient from a gradient descent algorithm by backpropagating the loss value L to the neural network. Afterwards, the processor 810 may obtain a 3D gradient by stacking 2D gradients that are calculated with respect to crosssectional images by disassembling the initial 3D microstructure in directions parallel to the xy plane, the yz plane, and the xz plane of the initial 3D microstructure.
The processor 810 may update the initial 3D microstructure by reflecting the final 3D gradient to the initial 3D microstructure.
According to another embodiment, the processor 810 may receive a first 3D microstructure image of the material as a input 3D microstructure image. According to an embodiment, the first 3D microstructure image may be a template representing the 3D structure of the material. Further, the processor 810 may obtain a plurality of crosssectional images from the first 3D microstructure. According to an embodiment, the processor 810 may dissect the first microstructure image at different locations to obtain the plurality of crosssectional images. According to an embodiment, the different locations may be at an uniform interval.
Further, the processor 810 may generate a plurality of first output feature map information with respect to at least one layer of the neural network based on the plurality of crosssectional images, each of the plurality of first output features map information uniquely corresponding to one of the plurality of crosssectional images.
Moreover, the processor 810 may receive a 2D image of the material. According to an embodiment, the 2D image 310 may be an actual image of a surface of the material captured using an imaging device. According to an embodiment, the imaging device may be a scanning electron microscope (SEM) or an optical microscope. The SEM may obtain a 2D scanning image that shows a surface shape of the material 300 by injecting an electron beam onto the material 300.
The processor 810 may generate a second output feature map information with respect to at least one layer of the neural network based on the 2D image to the neural network.
According to an embodiment, the plurality of first output feature map information and the second output features map information may be a data set in which various characteristics of input data are expressed. The plurality of first output feature map information and the second output features map information may have elements of 2D matrix or elements of 3D matrix, and a pixel value may be defined in each of the elements. The plurality of first output feature map information and the second output features map information may have a width W (or a column), a height H (a row), and a depth D. At this point, the depth D may correspond to the number of channels.
According to an embodiment, the processor 810 may generate a 3D gradient by comparing each of the plurality of first output feature map information with the second output feature map information and generate a second 3D microstructure based on the generated 3D gradient. According to an embodiment, the second 3D microstructure may be a output as s final 3D microstructure of the material.
According to an embodiment, the processor 810 may calculate a plurality of loss values based on results from comparing each of the plurality of first output feature map information with the second output feature map information, generate a plurality of 2D gradients corresponding to each of the plurality of crosssectional images by applying the plurality of loss values to a backpropagation algorithm in the neural network and stack the plurality of 2D gradients to generate the 3D gradient.
According to another embodiment, the processor 810 may repeat the operations of described above using the second 3D microstructure as the new input microstructure image until the loss value is below a threshold.
In operation 910, the neural network apparatus 800 may configure an initial 3D microstructure. In order to configure the initial 3D microstructure, a size of a final 3D microstructure to be generated may be defined in advance. The initial 3D microstructure may be an arbitrary 3D microstructure configured to have a size equal to that of the defined final 3D microstructure.
In operation 920, the neural network apparatus 800 may obtain a plurality of crosssectional images of the initial 3D microstructure by disassembling the initial 3D microstructure in at least one direction. In detail, the plurality of crosssectional images may include first crosssectional images obtained by disassembling the initial 3D microstructure in a direction parallel to an xy plane of the initial 3D microstructure, second crosssectional images obtained by disassembling the initial 3D microstructure in a direction parallel to a yz plane of the initial 3D microstructure, and third crosssectional images obtained by disassembling the initial 3D microstructure in a direction parallel to an xz plane of the initial 3D microstructure.
In operation 930, the neural network apparatus 800 may obtain first output feature maps with respect to at least one layer that constitutes a neural network by inputting each of the crosssectional images to the neural network. The neural network may correspond to a convolution neural network that is learned in advance.
In operation 940, the neural network apparatus 800 may obtain second output feature maps with respect to at least one layer by inputting a 2D original image to the neural network. The 2D original image may be obtained by using an SEM or an optical microscope and may be stored in a memory. The 2D original image may be surface images at an xy plane, a yz plane, and an xz plane of a material to generate a 3D microstructure.
In operation 950, the neural network apparatus 800 may calculate a loss value by comparing the first output feature maps with the second output feature maps and may generate a 3D gradient by applying the loss value to a backpropagation algorithm. The neural network apparatus 800 may generate the 3D gradient by stacking 2D gradients generated from the loss value that is obtained with respect to each of the crosssectional images obtained by disassembling the initial 3D microstructure in a direction. The neural network apparatus 800 may generate a final 3D gradient by adding all 3D gradients respectively corresponding to crosssectional images obtained by disassembling the initial 3D microstructure in a direction parallel to the xy plane, crosssectional images obtained by disassembling the initial 3D microstructure in a direction parallel to the yz plane, and crosssectional images obtained by disassembling the initial 3D microstructure in a direction parallel to the xz plane of the initial 3D microstructure.
In operation 960, the neural network apparatus 800 may generate a final 3D microstructure based on a 2D original image by reflecting the 3D gradient to the initial 3D microstructure. The neural network apparatus 800 may update the initial 3D microstructure by reflecting the final 3D gradient to the initial 3D microstructure.
The disclosure may provide a method and apparatus for generating a 3D microstructure using a neural network. In detail, the 3D microstructure may be generated based on a 2D original image using the neural network. Accordingly, it is unnecessary to cut a crosssection of a material many times to generate a 3D microstructure, and thus, it is efficient in view of energy consumption. Also, it is possible to generate a 3D microstructure so that a crosssection of the generated 3D microstructure is similar to a particle shape in a 2D original image.
Embodiments of the disclosure can also be implemented in the form of a computerreadable recording medium comprising commands executable by a computer, such as program modules, being executed by a computer. The computer readable medium may be any usable medium that may be accessed by a computer, and may include any usable medium, such as, a volatile and nonvolatile medium and a discrete type and nondiscrete type medium. Also, the computer readable medium may include all computer storing media and communication media. The computer storing medium may include any medium, such as, a volatile and nonvolatile medium and a discrete type and nondiscrete type medium that is realized by a method or technique for storing information, such as, a computer readable command, a data structure, a program module, or other data. The communication medium may include other data of modulated signal, such as, a computer readable command, a data structure, a program module, or a carrier signal, or other transmission mechanism, and an arbitrary information medium.
Also, in the specification, a “unit” may be a hardware component, such as a processor or a circuit and/or a software component that is executed by the hardware component like the processor.
The embodiments are examples, and thus, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention. Therefore, the embodiments should be considered in descriptive sense only and not for purposes of limitation. For example, each constituent element that is described as a singular form may be embodied in distribution forms. Also, constituent elements that are described in distribution forms may be embodied in a combined form.
Therefore, the scopes of the exemplary embodiments are defined not by the detailed description but by the appended claims, and all differences within the scope will be construed as being included in the exemplary embodiments.