REAL-TIME TRACKING-COMPENSATED IMAGE EFFECTS

• US 20190087660A1
• Filed: 09/15/2017
• Published: 03/21/2019
• Est. Priority Date: 09/15/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Hare, Samuel Edward, Poliakov, Fedir, Wang, Guohui, Xiong, Xuehan, Yang, Jianchao, Yang, Linjie, Anilkumar, Shah Tanmay
• Granted Patent: US 10,474,900 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

Claims

1. A method comprising:

generating, using an image sensor, an image sequence;

generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;

generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images;

displaying, on a display device of the machine, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.

2. The method of claim 1, wherein the modified keyframe images are generated in a first pipeline and the modified non-keyframe images are generated in a second pipeline, wherein the second pipeline is asynchronous from the first pipeline.

3. The method of claim 2, wherein the machine learning scheme is a convolutional neural network.

4. The method of claim 1, further comprising:

publishing the modified image sequence as an ephemeral message on a social media network.

5. The method of claim 1, wherein each flow map is generated by a non-keyframe image and a preceding keyframe image, the non-keyframe image and the preceding keyframe image separated by one or more additional non-keyframe images.

6. The method of claim 1, wherein the preceding keyframe images used to generate the flow maps have existing modified keyframe images, and the flow maps are applied to the existing modified keyframe images.

7. The method of claim 1, wherein the modified image sequence is displayed on the machine in one or more of:

real time or near real time.

8. The method of claim 1, wherein the modified keyframe images and the modified non-keyframe images are ordered in the modified image sequence according to an ordering of images in the image sequence.

9. The method of claim 1, further comprising:

generating the modified image sequence by using the modified keyframe images and the modified non-keyframe images.

10. The method of claim 9, wherein the modified keyframe images and the modified non-keyframe images are image masks used to apply one or more visual effects to the image sequence.

11. The method of claim 1, further comprising:

generating the flow maps using a motion tracking scheme that describes transformations of image features across a plurality of images.

12. A system comprising:

one or more processors of a machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

generating an image sequence;

generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;

generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images; and

displaying, on a display device of a machine, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.

13. The system of claim 12, wherein the modified keyframe images are generated in a first pipeline and the modified non-keyframe images are generated in a second pipeline, wherein the second pipeline is asynchronous from the first pipeline

14. The system of claim 13, wherein the machine learning scheme is a convolutional neural network.

15. The system of claim 13, wherein the first pipeline and the second pipeline operate in parallel.

16. The system of claim 12, wherein each flow map is generated by a non-keyframe image and a preceding keyframe image, the non-keyframe image and the preceding keyframe image separated by one or more additional non-keyframe images.

17. The system of claim 12, wherein the preceding keyframe images used to generate the flow maps have existing modified keyframe images, and the flow maps are applied to the existing modified keyframe images.

18. The system of claim 12, wherein the modified image sequence is displayed on the machine in one or more of:

real time or near real time.

19. A machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

generating, using one or more processors of a machine, an image sequence;

generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;

generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images; and

displaying, on a display device of the machine, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.

20. The machine-readable storage device of claim 19, wherein the modified keyframe images are generated in a first pipeline and the modified non-keyframe images are generated in a second pipeline, wherein the second pipeline is asynchronous to the first pipeline.