Dense feature scale detection for image matching

• US 10,552,968 B1
• Filed: 09/22/2017
• Issued: 02/04/2020
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Wang, Shenlong, Luo, Linjie, Zhang, Ning, Li, Jia
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US15/712,990
• Time in Patent Office: 865 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Dense feature scale detection can be implemented using multiple convolutional neural networks trained on scale data to more accurately and efficiently match pixels between images. An input image can be used to generate multiple scaled images. The multiple scaled images are input into a feature net, which outputs feature data for the multiple scaled images. An attention net is used to generate an attention map from the input image. The attention map assigns emphasis as a soft distribution to different scales based on texture analysis. The feature data and the attention data can be combined through a multiplication process and then summed to generate dense features for comparison.

Claims

1. A method comprising:

identifying, using one or more processors of a machine, an image;

generating a plurality of scaled images from the image;

generating image feature datasets for the plurality of scaled images; and

generating a dense feature dataset by combining the image feature datasets with attention values of an attention map, the attention values being one or more numerical values that modify values of the dense feature dataset based at least in part on the scale of the plurality of scaled images;

tracking an object depicted in the images of an image sequence using the dense feature dataset;

generating a modified image sequence by applying an image effect to the object depicted in the image sequence, the image effect applied to the depiction of the object tracked in the images of the image sequence; and

publishing, on a network site, the modified image sequence as an electronic message.

2. The method of claim 1, wherein the attention values are a range of numerical values in a distribution, and wherein the image feature datasets are combined using a multiplication operation and an addition operation.

3. The method of claim 1, wherein the electronic message is an ephemeral message that is inaccessible through the network site after expiry of a timer associated with the ephemeral message.

4. The method of claim 1, wherein the plurality of scaled images comprises a first scaled image and a second scaled image;

and

wherein the first scaled image is used to generate a first set of attention values and a first image feature dataset; and

wherein the second scaled image is used to generate a second set of attention values and a second image feature dataset.

5. The method of claim 4, wherein the first set of attention values and the first image feature dataset are multiplied together to produce a first multiplication output;

and wherein the second set of attention values and the second image feature dataset are multiplied together to produce a second multiplication output.

6. The method of claim 5, further comprising:

summing the first multiplication output and the second multiplication output to generate a dense feature dataset.

7. The method of claim 6, wherein the dense feature dataset comprises a plurality of vectors for a plurality of pixels of the image.

8. The method of claim 1, wherein the image feature dataset and the attention map are generated using one or more convolutional neural networks.

9. The method of claim 8, further comprising:

identifying a source image having a source dense feature;

identifying a target image having a target dense feature; and

training the one or more convolutional neural networks by at least maximizing a product of the source dense feature and the target dense feature.

10. The method of claim 9, wherein maximizing the product comprises adjusting parameters in the one or more convolutional networks to maximize the product.

11. The method of claim 10, wherein the product is an inner product.

12. A system comprising:

one or more processors of a machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

identifying an image;

generating a plurality of scaled images from the image;

generating image feature datasets for the plurality of scaled images; and

generating a dense feature dataset by combining the image feature datasets with attention values of an attention map, the attention values being one or more numerical values that modify values of the dense feature dataset based at least in part on the scale of the plurality of scaled images;

tracking an object depicted in the images of an image sequence using the dense feature dataset;

generating a modified image sequence by applying an image effect to the object depicted in the image sequence, the image effect applied to the depiction of the object tracked in the images of the image sequence; and

publishing, on a network site, the modified image sequence as an electronic message.

13. The system of claim 12, wherein the attention values are a range of numerical values in a distribution.

14. The system of claim 12, wherein the image feature datasets are combined using a multiplication operation and an addition operation.

15. The system of claim 14, wherein the plurality of scaled images comprises a first scaled image and a second scaled image;

and wherein the first scaled image is used to generate a first set of attention values and a first image feature dataset; and

wherein the second scaled image is used to generate a second set of attention values and a second image feature dataset.

16. The system of claim 15, wherein the first set of attention values and the first image feature dataset are multiplied together to produce a first multiplication output;

and wherein the second set of attention values and the second image feature dataset are multiplied together to produce a second multiplication output.

17. The system of claim 16, the operations further comprising:

summing the first multiplication output and the second multiplication output to generate a dense feature dataset.

18. The system of claim 17, wherein the dense feature dataset comprises a plurality of vectors for a plurality of pixels of the image.

19. A non-transitory machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

identifying an image;

generating a plurality of scaled images from the image;

generating image feature datasets for the plurality of scaled images; and

generating a dense feature dataset by combining the image feature datasets with attention values of an attention map, the attention values being one or more numerical values that modify values of the feature dataset based at least in part on the scale of the plurality of scaled images; and

tracking an object depicted in the images of an image sequence using the dense feature dataset;

generating a modified image sequence by applying an image effect to the object depicted in the image sequence, the image effect applied to the depiction of the object tracked in the images of the image sequence; and

publishing, on a network site, the modified image sequence as an electronic message.

20. The non-transitory machine-readable storage device of claim 19, wherein the attention values are a range of numerical values in a distribution;

and wherein the image feature datasets are combined using a multiplication operation and an addition operation.