LEARNING METHOD AND RECORDING MEDIUM

• US 20190332939A1
• Filed: 07/11/2019
• Published: 10/31/2019
• Est. Priority Date: 03/16/2017
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: ALLETTO, STEFANO, RIGAZIO, LUCA, TSUKIZAWA, SOTARO
• Granted Patent: US 11,687,773 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 19/18 the unit being a set of transform coefficients
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/513 Processing of motion vectors

Abstract

A learning method includes an input process to input, to a neural network, a first image and a second image that constitute a moving image and that are temporally adjacent to each other, where the second image is an image subsequent to the first image with a predetermined time interval therebetween, a learning process to cause the neural network to use the first image and the second image and learn to output a transformation matrix applied to all pixels of the first image and used to convert the first image into the second image, and an output process to output, as a result of estimation of motion between the first image and the second image, a motion amount image generated from the transformation matrix and representing an amount of motion of each of the pixels of the first image that continues until the predetermined time interval elapses.

Claims

1. A learning method comprising:

inputting, to a neural network, a first image and a second image that constitute a moving image and that are temporally adjacent to each other, the second image being an image subsequent to the first image with a predetermined time interval therebetween;

causing the neural network to use the first image and the second image and learn to output a transformation matrix applied to all pixels of the first image and used to convert the first image into the second image; and

outputting, as a result of estimation of motion between the first image and the second image, a motion amount image generated from the transformation matrix and representing an amount of motion of each of the pixels of the first image that continues until the predetermined time interval elapses.

5. The learning method according to claim 1, wherein a first neural network that constitutes the neural network includes at least one convolutional layer and at least one fully connected layer, and

wherein the fully connected layer outputs at least eight of nine parameters functioning as nine coefficients that constitute the transformation matrix.

6. The learning method according to claim 1, wherein a first neural network that constitutes the neural network includes at least two convolutional layers, and

wherein the final layer of the at least two convolutional layers outputs at least eight of nine parameters functioning as nine coefficients that constitute the transformation matrix.

7. The learning method according to claim 5, wherein the second neural network that constitutes the neural network includes a compression layer consisting of at least one convolutional layer, an expansion layer located subsequent to the compression layer and consisting of at least one deconvolutional layer, and a single convolutional layer located subsequent to the expansion layer, and

wherein the single convolutional layer outputs two parameters indicative of the amount of motion of each of the pixels constituting the second motion amount image.

2. A learning method comprising:

inputting, to a neural network, a first image and a second image that constitute a moving image and that are temporally adjacent to each other, the second image being an image subsequent to the first image with a predetermined time interval therebetween;

i) causing a first neural network that constitutes the neural network to use the first image and the second image and learn to output a first motion amount image representing a first amount of motion of each of pixels of the first image that continues until the predetermined time interval elapses and ii) causing a second neural network that constitutes the neural network and that differs from the first neural network to use the first image, the second image, and the first motion amount image and learn to output a second motion amount image representing a second amount of motion of each of the pixels of the first image that continues until the predetermined time interval elapses; and

outputting the second motion amount image as a result of estimation of motion between the first image and the second image.

3. The learning method according to claim 2, wherein causing the first and second neural networks to learn to output the motion amount images involves causing the first neural network to use the first image and the second image and learn to output a transformation matrix applied to all the pixels of the first image and used to convert the first image into the second image and causing the first neural network to learn to output the first motion amount image generated from the transformation matrix.

4. The learning method according to claim 3, wherein causing the first and second neural networks to learn to output the motion amount images involves calculating a combined error by combining a first error representing an error between a first estimated image that is generated from the transformation matrix and the first image and that is an image subsequent to the first image with the predetermined time interval therebetween and the second image with a second error representing an error between a second estimated image that is generated from the second motion amount image and the first image and that is an image subsequent to the first image with the predetermined time interval therebetween and the second image and determining, as weights of the first neural network and second neural network that have carried out learning, weights of the first neural network and the second neural network with a minimum combined error when the combined error is minimized.

8. A non-transitory computer-readable recording medium storing a program which causes a computer to:

input, to a neural network, a first image and a second image that constitute a moving image and that are temporally adjacent to each other, where the second image is an image subsequent to the first image with a predetermined time interval therebetween;

cause the neural network to use the first image and the second image and learn to output a transformation matrix applied to all pixels of the first image and used to convert the first image into the second image; and

output, as a result of estimation of motion between the first image and the second image, a motion amount image generated from the transformation matrix and representing an amount of motion of each of the pixels of the first image that continues until the predetermined time interval elapses.

9. A non-transitory computer-readable recording medium storing a program which causes a computer to:

input, to a neural network, a first image and a second image that constitute a moving image and that are temporally adjacent to each other, the second image being an image subsequent to the first image with a predetermined time interval therebetween;

i) cause a first neural network that constitutes the neural network to use the first image and the second image and learn to output a first motion amount image representing a first amount of motion of each of pixels of the first image that continues until the predetermined time interval elapses and ii) cause a second neural network that constitutes the neural network and that differs from the first neural network to use the first image, the second image, and the first motion amount image and learn to output a second motion amount image representing a second amount of motion of each of the pixels of the first image that continues until the predetermined time interval elapses; and

output the second motion amount image as a result of estimation of motion between the first image and the second image.