Distributed image stabilization

• US 10,341,561 B2
• Filed: 09/11/2015
• Issued: 07/02/2019
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Puntambekar, Amit, Coward, Michael Hamilton
• Primary Examiner(s): Segura, Cynthia
• Application Number:
  US14/852,449
• Publication Number:
  US 20170078574A1
• Time in Patent Office: 1,390 Days
• Field of Search: 34820899
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

In a distributed video encoding system, a video is encoded by splitting into video segments and encoding the segments using multiple encoders. Prior to segmenting the video for distributed video encoding, image stabilization is performed on the video. For each frame in the video, a corresponding transform operation is first computed based on an estimated camera movement. Next, the video is segmented into multiple video segments and the corresponding per-frame transform information for the multiple video segments. The video segments are then distributed to multiple processing nodes that perform the image stabilization of the corresponding video segment by applying the corresponding transform. The results from all the stabilized video segments are then stitched back together for further video encoding operation.

Claims

1. A computer-implemented method for stabilizing captured video frames, implemented in a distributed computer network, comprising:

analyzing, using a single one of one or more master processors, frames of a video to generate a list of per-frame transform for mitigating camera movement for each frame;

partitioning, after analyzing, using the one or more master processors, the video into multiple video segments, wherein each video segment is assigned a bit rate based in part on a complexity of each video segment, wherein the complexity of each video segment is associated with an amount of bits in order to achieve a uniform quality in the video, and wherein each video segment comprises a plurality of consecutive frames and one or more overlapping frames of the video;

dividing, using the one or more master processors, the list of per-frame transform into multiple portions corresponding to the multiple video segments;

providing, from the one or more master processors to a plurality of worker processors, a different one of the multiple video segments generated from the partitioning, metadata indicating the one or more overlapping frames, and a corresponding portion of the list of per-frame transform;

performing, by the plurality of worker processors, image stabilization on the received video segment and the corresponding portion of the list of per-frame transform to produce stabilized video segments, wherein a first worker processor of the plurality of worker processors produces a stabilized video segment including the one or more overlapping frames and a second worker processor of the plurality of worker processors skips the one or more overlapping frames based on the metadata;

conveying the stabilized video segments from the plurality of worker processors to the one or more master processors; and

generating, by the one or more master processors, a stabilized video from the stabilized video segments.

2. The computer-implemented method of claim 1, wherein the analyzing frames includes:

analyzing motion vectors associated with the frames to identify a global component to the motion vectors.

3. The computer-implemented method of claim 1, wherein the analyzing the frames is performed on compressed representation of the frames.

4. The computer-implemented method of claim 3, wherein the analyzing the frames includes decompressing at least some of the frames at least partially.

5. The computer-implemented method of claim 3, wherein the analyzing the frames includes applying a rotational transform to at least some of the frames.

6. The computer-implemented method of claim 1, wherein the transform for mitigating camera movement includes compensating for at least one of a left-right movement, an up-down movement, and a rotational movement.

7. The computer-implemented method of claim 1, wherein the generating the stabilized video from the stabilized video segments includes smoothing camera movement across contiguous stabilized video segments.

8. The computer-implemented method of claim 1, wherein the partitioning into multiple video segments includes creating each video segment by including an intra encoded frame.

9. A non-transitory computer-readable medium storing instructions that, upon execution, cause at least one computer processor to perform a method comprising:

analyzing, using a single one of one or more master processors, frames of a video to generate a list of per-frame transform for mitigating camera movement for each frame;

partitioning, after analyzing, using the one or more master processors, the video into multiple video segments, wherein each video segment is assigned a bit rate based in part on a complexity of each video segment, wherein the complexity of each video segment is associated with an amount of bits in order to achieve a uniform quality in the video, and wherein each video segment comprises a plurality of consecutive frames and one or more overlapping frames of the video;

dividing, using the one or more master processors, the list of per-frame transform into multiple portions corresponding to the multiple video segments;

providing, from the one or more master processors, to a plurality of worker processors, a different one of the multiple video segment generated from the partitioning, metadata indicating the one or more overlapping frames, and a corresponding portion of the list of per-frame transform;

performing, by the plurality of worker processors, image stabilization on the received video segment and the corresponding portion of the list of per-frame transform to produce stabilized video segments, wherein a first worker processor of the plurality of worker processors produces a stabilized video segment including the one or more overlapping frames and a second worker processor of the plurality of worker processors skips the one or more overlapping frames based on the metadata;

conveying the stabilized video segments from the plurality of worker processors to the one or more master processors; and

generating, by the one or more master processors, a stabilized video from the stabilized video segments.

10. The non-transitory computer-readable medium of claim 9, wherein the analyzing frames includes:

analyzing motion vectors associated with the frames to identify a global component to the motion vectors.

11. The non-transitory computer-readable medium of claim 9, wherein the analyzing the frames includes analyzing a compressed representation of the frames.

12. The non-transitory computer-readable medium of claim 11, wherein the analyzing the frames includes decompressing at least some of the frames at least partially.

13. The non-transitory computer-readable medium of claim 11, wherein the analyzing the frames includes applying a rotational transform to at least some of the frames.

14. The non-transitory computer-readable medium of claim 9, wherein the transform for mitigating camera movement includes compensating for at least one of a left-right movement, an up-down movement, and a rotational movement.

15. A computer system, comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the system to perform;

analyzing frames of a video to generate a list of per-frame transform for mitigating camera movement for each frame;

partitioning the video, after analyzing, into multiple video segments, wherein each video segment is assigned a bit rate based in part on a complexity of each video segment, wherein the complexity of each video segment is associated with an amount of bits in order to achieve a uniform quality in the video, and wherein each video segment comprises a plurality of consecutive frames and one or more overlapping frames of the video;

dividing the list of per-frame transform into multiple portions corresponding to the multiple video segments;

providing a different one of the multiple video segment generated from the partitioning, metadata indicating the one or more overlapping frames, and a corresponding portion of the list of per-frame transform;

performing image stabilization on the received video segment and the corresponding portion of the list of per-frame transform to produce stabilized video segments, wherein a first worker processor of the plurality of worker processors produces a stabilized video segment including the one or more overlapping frames and a second worker processor of the plurality of worker processors skips the one or more overlapping frames based on the metadata;

conveying the stabilized video segments; and

generating a stabilized video from the stabilized video segments.

16. The computer system of claim 15, wherein the analyzing the frames includes:

analyzing motion vectors associated with the frames to identify a global component to the motion vectors.

17. The computer system of claim 15, wherein the analyzing the frames includes analyzing a compressed representation of the frames.

18. The computer system of claim 15, wherein the analyzing the frames includes decompressing at least some of the frames at least partially.

19. The computer system of claim 15, wherein the analyzing the frames includes applying a rotational transform to at least some of the frames.

20. The computer system of claim 15, wherein the-transform for mitigating camera movement includes compensating for at least one of a left-right movement, an up-down movement, and a rotational movement.