Method and apparatus for generating a displacement map of an input dataset pair
First Claim
1. A method for generating a displacement map of a first input dataset and a second input dataset of an input dataset pair, each input dataset having at least one spatial dimension and/or time dimension, the method comprising the steps ofprocessing the first input dataset and the second input dataset by means of a neural network based feature extractor (25) so as to generate a feature map hierarchy (50) comprising a base pair of feature maps (20e, 30e) and a refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d), each pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d, 20e, 30e) constituting one level of the feature map hierarchy (50), the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) having less coarse resolution in all of their at least one spatial dimension and/or time dimension than the base pair of feature maps (20e, 30e);
- generating an initial displacement map in a displacement generating operation for the base pair of feature maps (20e, 30e) of the feature map hierarchy (50) based on matching a first feature map of the base pair of feature maps (20e, 30e) with a second feature map of the base pair of feature maps (20e, 30e);
in a displacement refinement operation,upscaling the initial displacement map in all of its at least one spatial dimension and/or time dimension to the scale of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) of the feature map hierarchy (50) with a respective upscaling factor and multiplying the values of the initial displacement map with the respective upscaling factor to generate an upscaled initial displacement map,generating a warped version of the first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) by carrying out a warping operation using the upscaled initial displacement map on a first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) of the feature map hierarchy (50),matching the warped version of the first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) with the second feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) so as to obtain a correction displacement map for the warped version of the first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) and the second feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d), which correction displacement map is added to the upscaled initial displacement map so as to obtain an updated displacement map.
1 Assignment
0 Petitions
Accused Products
Abstract
The invention is a method and an apparatus for generating a displacement map of a first input dataset and a second input dataset of an input dataset pair. The apparatus comprises
- a neural network based feature extractor (25) for processing the first input dataset and the second input dataset so as to generate a feature map hierarchy (50) comprising a base pair of feature maps (20e, 30e) and a refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d),
- a displacement unit comprising a first comparator unit for obtaining an initial displacement map using the base pair of feature maps (20e, 30e); and
- a displacement refinement unit for obtain an updated displacement map for the refinement pair of feature maps (20d, 30d).
44 Citations
12 Claims
-
1. A method for generating a displacement map of a first input dataset and a second input dataset of an input dataset pair, each input dataset having at least one spatial dimension and/or time dimension, the method comprising the steps of
processing the first input dataset and the second input dataset by means of a neural network based feature extractor (25) so as to generate a feature map hierarchy (50) comprising a base pair of feature maps (20e, 30e) and a refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d), each pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d, 20e, 30e) constituting one level of the feature map hierarchy (50), the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) having less coarse resolution in all of their at least one spatial dimension and/or time dimension than the base pair of feature maps (20e, 30e); -
generating an initial displacement map in a displacement generating operation for the base pair of feature maps (20e, 30e) of the feature map hierarchy (50) based on matching a first feature map of the base pair of feature maps (20e, 30e) with a second feature map of the base pair of feature maps (20e, 30e); in a displacement refinement operation, upscaling the initial displacement map in all of its at least one spatial dimension and/or time dimension to the scale of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) of the feature map hierarchy (50) with a respective upscaling factor and multiplying the values of the initial displacement map with the respective upscaling factor to generate an upscaled initial displacement map, generating a warped version of the first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) by carrying out a warping operation using the upscaled initial displacement map on a first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) of the feature map hierarchy (50), matching the warped version of the first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) with the second feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) so as to obtain a correction displacement map for the warped version of the first feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d) and the second feature map of the refinement pair of feature maps (20b, 30b, 20c, 30c, 20d, 30d), which correction displacement map is added to the upscaled initial displacement map so as to obtain an updated displacement map. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
Specification