SYSTEMS AND METHODS FOR PROCESSING AND FORMATTING VIDEO FOR INTERACTIVE PRESENTATION

• US 20180007286A1
• Filed: 07/01/2016
• Published: 01/04/2018
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snapchat Inc (Snap, Inc.)
• Inventors: Li, Jia, Litke, Nathan, Paredes, Jose Jesus (Joseph), Sheth, Rahul, Szeto, Daniel, Xu, Ning, Yang, Jianchao
• Granted Patent: US 10,623,662 B2
• Time in Patent Office: Days
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Systems and methods are described for receiving, at a computing device, a video comprising a plurality of frames and determining, by the computing device, that vertical cropping should be performed for the video. For each frame of the plurality of frames, the computing device processes the video by analyzing the frame to determine a region of interest in the frame, wherein the frame is a first frame, cropping the first frame based on the region of interest in the frame to produce a vertically cropped frame for the video, determining a second frame immediately preceding the first frame, and smoothing a trajectory from the second frame to the vertically cropped frame. The vertically cropped frame is displayed to a user instead of the first frame.

Claims

1. A method comprising:

causing display to a user, by a computing device, of a video comprising a sequence of frames;

determining, by the computing device, that vertical cropping should be performed for the video, the determining being based on a user interaction with the computing device that includes turning the computing device to a different orientation or angle;

for each frame of the sequence of frames, processing the video by;

analyzing the frame, by the computing device, to determine a region of interest in the frame, wherein the frame is a first frame;

cropping, by the computing device, the first frame based on the region of interest in the frame, to produce a vertically cropped frame for the video;

determining, by the computing device, a second frame immediately adjacent the first frame in the sequence of frames; and

smoothing, by the computing device, a trajectory from the second frame to the vertically cropped frame;

wherein the vertically cropped frame is caused to be displayed to a user instead of the first frame.

4. The method of claim 1, wherein analyzing the frame, by the computing device, to determine a region of interest in the frame comprises:

analyzing the frame to determine there is no scene change from the second frame to the first frame; and

determining the region of interest in the first frame based on a region of interest in the second frame.

5. The method of claim 4, wherein determining the region of interest in the first frame based on a region of interest in the second frame is determined using a visual tracking method.

6. The method of claim 1, wherein analyzing the frame to determine a region of interest in the frame comprises:

analyzing the first frame and determining there is a scene change from the second frame to the first frame;

performing a saliency analysis of the first frame;

generating a saliency map of the first frame;

projecting the saliency map on a horizontal axis;

analyzing the saliency map to determine a most salient window of a predetermined size;

setting the region of interest to the most salient window of the predetermined size.

7. The method of claim 6, wherein determining there is a scene change is based on matching a color histogram.

8. The method of claim 6, wherein determining there is a scene change is based on matching a directional histogram.

9. The method of claim 1, wherein analyzing the frame to determine a region of interest in the frame comprises:

detecting a user interaction with the frame; and

setting the region of interest to the location of interaction by the user.

11. The method of claim 1, further comprising:

providing the vertically cropped frame to a user as part of the video.

12. The method of claim 1, wherein the video is a landscape video.

21. The method of claim 1, wherein analyzing the frame to determine a region of interest in the frame comprises:

detecting a user interaction with a display screen of the computing device buy touching an area of interest on the display screen; and

setting the region of interest to the location of the user touching of the area of interest on the display screen.

22. The method of claim 1, wherein cropping the first frame based on a region of interest in the frame further comprises enlarging the region of interest to produce the vertically cropped frame for the video.

23. The method of claim 22, wherein the region of interest is enlarged based on a user interaction with a touch screen on a display of the computing device.

24. The method of claim 1, wherein analyzing the frame to determine the region of interest in the frame comprises determining the region of interest based on a user interaction with the frame or based on a region of interest of the second frame.

2. (canceled)

3. (canceled)

10. (canceled)

13. A computing device comprising:

a display;

at least one processor; and

a non-transitory computer readable medium coupled with the processor, the computer readable medium comprising instructions stored thereon that are executable by the at least one processor to cause a computing device to perform operations comprising;

causing display to a user, of a video comprising a sequence of frames;

determining that vertical cropping should be performed for the video, the determining being based on a user interaction with the computing device that includes turning the computing device to a different orientation or angle;

for each frame of the sequence of frames, processing the video by;

analyzing the frame to determine a region of interest in the frame, wherein the frame is a first frame;

cropping the first frame based on the region of interest in the frame, to produce a vertically cropped frame for the video;

determining a second frame immediately preceding the first frame; and

smoothing a trajectory from the second frame to the vertically cropped frame;

wherein the vertically cropped frame is displayed to a user on the display instead of the first frame.

15. The computing device of claim 13, wherein analyzing the frame to determine a region of interest in the frame comprises:

analyzing the frame to determine there is no scene change from the second frame to the first frame; and

determining the region of interest in the first frame based on a region of interest in the second frame using a visual tracking method.

16. The computing device of claim 13, wherein analyzing the frame to determine a region of interest in the frame comprises:

analyzing the first frame and determining there is a scene change from the second frame to the first frame;

performing a saliency analysis of the first frame;

generating a saliency map of the first frame;

projecting the saliency map on a horizontal axis;

analyzing the saliency map to determine a most salient window of a predetermined size;

setting the region of interest to the most salient window of the predetermined size.

17. The computing device of claim 13, wherein analyzing the frame to determine a region of interest in the frame comprises:

detecting a user interaction with the frame; and

setting the region of interest to the location of interaction by the user.

18. The computing device of claim 13, the operations further comprising:

providing the vertically cropped frame to a user as part of the video.

19. The computing device of claim 13, wherein the video is a landscape video.

14. (canceled)

20. A non-transitory computer readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

causing display to a user, of a video comprising a sequence of frames;

determining that vertical cropping should be performed for the video, the determining being based on a user interaction with the computing device that includes turning the computing device to a different orientation or angle;

for each frame of the sequence of frames, processing the video by;

analyzing the frame to determine a region of interest in the frame, wherein the frame is a first frame;

cropping the first frame based on the region of interest in the frame, to produce a vertically cropped frame for the video;

determining a second frame immediately preceding the first frame; and

smoothing a trajectory from the second frame to the vertically cropped frame;

wherein the vertically cropped frame is displayed to a user instead of the first frame.