Processing and formatting video for interactive presentation

• US 10,623,662 B2
• Filed: 07/01/2016
• Issued: 04/14/2020
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Li, Jia, Litke, Nathan, Paredes, Jose Jesus (Joseph), Sheth, Rahul Bhupendra, Szeto, Daniel, Xu, Ning, Yang, Jianchao
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Hasan, Syed Y
• Application Number:
  US15/201,049
• Publication Number:
  US 20180007286A1
• Time in Patent Office: 1,383 Days
• Field of Search: 386278, 386230, 348143, 348 42, 3482181, 37524008, 725109, 725 12, 345660, 345173, 382195, 382190
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Systems and methods are described for receiving, at a computing device, a video comprising a plurality of frames and determining, by the computing device, that vertical cropping should be performed for the video. For each frame of the plurality of frames, the computing device processes the video by analyzing the frame to determine a region of interest in the frame, wherein the frame is a first frame, cropping the first frame based on the region of interest in the frame to produce a vertically cropped frame for the video, determining a second frame immediately preceding the first frame, and smoothing a trajectory from the second frame to the vertically cropped frame. The vertically cropped frame is displayed to a user instead of the first frame.

Claims

1. A method comprising:

causing, by a computing device, display of a video comprising a sequence of frames on a display of the computing device;

detecting, by the computing device, user interaction with the display of the computing device during display of the video comprising the sequence of frames, the user interaction detected during display of a current frame of the sequence of frames;

determining, by the computing device, that the user interaction indicates a region of interest in the current frame of the sequence of frames of the displayed video;

detecting, by the computing device, that the computing device has been moved from a first angle or orientation to a second angle or orientation during the display of the video comprising the sequence of frames;

determining, by the computing device, that the displayed video should be cropped according to the user interaction indicating the region of interest to display a portion of each frame of the sequence of frames;

cropping, by the computing device, the current frame of the sequence of frames being displayed to generate a cropped current frame displaying the region of interest;

enlarging the cropped current frame based on a display size corresponding to the second orientation or angle of the computing device;

causing the enlarged cropped current frame to be displayed as part of the sequence of frames of the displayed video;

for each frame of the sequence of frames after the current frame;

cropping, by the computing device, the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame and enlarging the cropped frame based on the display size corresponding to the second angle or orientation of the computing device during display of the video; and

causing the enlarged cropped frame to replace the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video on the display of the computing device instead of the frame.

2. The method of claim 1, wherein before cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame the method comprises:

analyzing the frame to determine whether there is a scene change from a previous frame immediately preceding the frame;

based on determining there is no scene change from the previous frame. using a region of interest of the previous frame as a region of interest for the frame;

based on determining there is a scene change from the previous frame, determining a region of interest for the frame based a saliency analysis of the frame;

wherein using the region of interest of the previous frame as the region of interest for the frame comprises using a visual tracking method to determine the region of interest from the previous frame.

3. The method of claim 2, wherein determining the region of interest based on a saliency analysis further comprises:

generating a saliency map of the frame;

projecting the saliency map on a horizontal axis;

analyzing the saliency map to determine a most salient window of a predetermined size; and

setting the region of interest to the most salient window of the predetermined size.

4. The method of claim 3, wherein determining the most salient window of a predetermined size is based on an aspect ratio of the display of the computing device.

5. The method of claim 2, wherein analyzing the frame to determine whether there is a scene change is based on matching a color histogram or matching a directional histogram.

6. The method of claim 2, wherein using the regions of interest of the previous frame as the region of interest for the frame comprises using a tracking method comprising at least one of an image based tracking method, motion estimations, optical flows, particle filters, and deep learning methods.

7. The method of claim 2, further comprising:

smoothing a trajectory from the previous frame to the enlarged cropped frame before causing the enlarged cropped frame to be displayed.

8. The method of claim 1, wherein the video is a landscape video.

9. The method of claim 1, wherein the enlarged cropped frame is displayed on the display of the computing device as a first part of a split screen and the frame is displayed as a second part of the split screen, wherein first part of the split screen is larger than the second part of the split screen.

10. The method of claim 1, further comprising:

storing the enlarged cropped frame.

11. The method of claim 1, further comprising:

splitting the display of the computing device into a first part and a second part based on the second angle or orientation;

causing the enlarged cropped frame to be displayed in the first part of the display of the computing device; and

causing an original frame to be displayed in the second part of the display of the computing device, the original frame the frame from which the enlarged cropped frame was generated.

12. The method of claim 1, further comprising:

splitting the display of the computing device into a first part and a second part based on the second angle or orientation;

causing the enlarged cropped frame to be displayed in the first part of the display of the computing device; and

causing the video comprising the sequence of frames to be displayed in the second part of the display of the computing device.

13. The method of claim 12, further comprising:

detecting a second user interaction with the second part of the display of the computing device during display of the sequence of frames, the user interaction detected during display of a second current frame;

determining that the second user interaction indicates a changed region of interest in the second current frame of the sequence of frames being displayed in the second part of the display of the computing device;

cropping the second current frame of the sequence of frames being displayed in the second part of the display of the computing device to generate a cropped second current frame displaying the region of interest;

enlarging the cropped second current frame based on a display size corresponding to the first part of the display of the computing device; and

causing the enlarged cropped second current frame to be displayed in the first part of the display of the computing device.

14. The method of claim 13, further comprising:

for each frame of the sequence of frames after the second current frame;

cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame and enlarging the cropped frame based on the display size corresponding to first part of the display of the computing device during display of the video; and

causing the enlarged cropped frame to replace the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video in the first part of the display of the computing device instead of the frame.

15. A computing device comprising:

a display;

at least one processor; and

a non-transitory computer readable medium coupled with the processor, the computer readable medium comprising instructions stored thereon that are executable by the at least one processor to cause a computing device to perform operations comprising;

causing display of a video comprising a sequence of frames on a display of the computing device;

detecting user interaction with the display of the computing device during display of the video comprising the sequence of frames, the user interaction detected during display of a current frame of the sequence of frames,

determining that the user interaction indicates a region of interest in the current frame of the sequence of frames of the displayed video;

detecting that the computing device has been moved from a first angle or orientation to a second angle or orientation during the display of the video comprising the sequence of frames;

determining that the displayed video should be cropped according to the user interaction indicating the region of interest to display a portion of each frame of the sequence of frames;

cropping the current frame of the sequence of frames being displayed to generate a cropped current frame displaying the region of interest;

enlarging the cropped current frame based on a display size corresponding to the second orientation or angle of the computing device;

causing the enlarged cropped current frame to be displayed as part of the sequence of frames of the displayed video;

for each frame of the sequence of frames after the current frame;

cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame and enlarging the cropped frame based on the display size corresponding to the second angle or orientation of the computing device during display of the video; and

causing the enlarged cropped frame to replace the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video on the display of the computing device instead of the frame.

16. The computing device of claim 15, wherein before cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame, the operations comprise:

analyzing the frame to determine whether there is a scene change from a previous frame immediately preceding the frame;

based on determining there is no scene change from the previous frame, using a region of interest of the previous frame as a region of interest for the frame;

based on determining there is a scene change from the previous frame, determining a region of interest for the frame based a saliency analysis of the frame;

wherein using the region of interest of the previous frame as the region of interest for the frame comprises using a visual tracking method to determine the region of interest from the previous frame.

17. The computing device of claim 16, wherein determining the region of interest based on a saliency analysis further comprises:

generating a saliency map of the frame;

projecting the saliency map on a horizontal axis;

analyzing the saliency map to determine a most salient window of a predetermined size; and

setting the region of interest to the most salient window of the predetermined size.

18. The computing device of claim 16, wherein determining the region of interest in the frame is further based on detecting a user interaction with the display of the computing device indicating a region of interest, and the method further comprises:

setting the region of interest to the location of interaction with the display.

19. The computing device of claim 15, wherein the video is a landscape video.

20. A non-transitory computer readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

causing display of a video comprising a sequence of frames on a display of the computing device;

detecting user interaction with the display of the computing device during display of the video comprising the sequence of frames, the user interaction detected during display of a current frame of the sequence of frames;

determining that the user interaction indicates a region of interest in the current frame of the sequence of frames of the displayed video;

detecting that the computing device has been moved from a first angle or orientation to a second angle or orientation during the display of the video comprising the sequence of frames;

determining that the displayed video should be cropped according to the user interaction indicating the region of interest to display a portion of each frame of the sequence of frames;

cropping the current frame of the sequence of frames being displayed to generate a cropped current frame displaying the region of interest;

enlarging the cropped current frame based on a display size corresponding to the second orientation or angle of the computing device;

causing the enlarged cropped current frame to be displayed as part of the sequence of frames of the displayed video;

for each frame of the sequence of frames after the current frame;

cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame and enlarging the cropped frame based on the display size corresponding to the second angle or orientation of the computing device during display of the video; and

causing the enlarged cropped frame to replace the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video on the display of the computing device instead of the frame.