Parallel processing of a video frame

• US 10,003,811 B2
• Filed: 09/01/2015
• Issued: 06/19/2018
• Est. Priority Date: 09/01/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sadhwani, Shyam, Prabhu, Sudhakar, Greenbaum, Carol, Mandal, Saswata, Wu, Yongjun
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US14/842,823
• Publication Number:
  US 20170064320A1
• Time in Patent Office: 1,022 Days
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements

Abstract

A graphics pipeline with components that process frames by portions (e.g., pixels or rows) or slices to reduce end-to-end latency. Components of a pipeline process portions of a same frame at the same time. For example, as graphics data for a frame is being generated and fills a framebuffer, once a certain portion of video data less than the whole frame (slice or sub-frame) becomes available, before the corresponding frame is finished filling the framebuffer, the next pipeline component after the framebuffer, for instance a video processor for color conversion or an encoder, begins to process the portion of the frame. While one portion of a frame is accumulating in the frame buffer, another portion of the same frame is being encoded by an encoder, and another portion of the frame might be being packaged by a multiplexer, and a network socket might start streaming the multiplexed portion.

Claims

1. A computing device comprising:

processing hardware and storage hardware, the storage hardware storing an application that when executed by the processing hardware generates video frames;

a display;

a framebuffer configured to store the video frames generated by the processing hardware, the framebuffer operationally connected with an encoder, wherein each video frame comprises segments, the computing device configured to display the framebuffer on the display at intervals that correspond to a refresh rate; and

the encoder configured to compress the video frames, wherein the computing device is configured to generate a signal when a segment of a video frame is stored in the framebuffer, wherein, based on the signal, the encoder receives a segment of a video frame from the framebuffer before other segments of the video frame have been fully generated and stored in the framebuffer, wherein the encoder begins encoding the received segment of the video frame before the video frame completely fills the framebuffer, and wherein during a single interval the video frame has;

a first segment being stored into the framebuffer, and a second segment being compressed by the encoder from the framebuffer.

2. A computing device according to claim 1, further comprising a multiplexer configured to multiplex the compressed video frames with audio data, wherein the multiplexer receives a compressed segment of the video frame and begins multiplexing the compressed segment with the audio data before receiving from the encoder a next compressed segment of the video frame.

3. A computing device according to claim 2, wherein during a single interval the video frame has:

the first segment being stored into the framebuffer, the second segment being compressed by the encoder from the framebuffer, and a third segment being multiplexed with the audio data by the multiplexer.

4. A computing device according to claim 1, wherein the video frames are partitioned into segments, wherein a given video frame comprises a first segment and a second segment, and wherein the first segment is compressed and outputted by the encoder before the encoder receives the second segment.

5. A computing device according to claim 4, further comprising a multiplexer that prepares the video frames for transmission by the computing device, wherein the given video frame comprises a third segment, and wherein the multiplexer is multiplexing the third segment while the encoder is compressing the first segment.

6. A computing device according to claim 1, wherein the encoder begins compressing the segment of the video frame from the framebuffer before the video frame is finished being generated by an application being executed by the processing hardware.

7. A computing device according to claim 6, wherein the encoder implements a video compression algorithm that performs both inter-frame and intra-frame encoding of video frame segments.

8. A computing device according to claim 1, wherein the displaying from the frame buffer and the encoding are both synchronized with a refresh signal that cycles at the refresh rate.

9. A method, performed by a computing device, to perform concurrent encoding and multiplexing of video frames, the method comprising:

receiving, from processing hardware of the computing device, a second portion of a video frame at a framebuffer, and based on the complete second portion being stored in the framebuffer, generating a signal;

displaying each of the video frames from the framebuffer as they are respectively stored in the framebuffer;

after generating the signal, beginning to receive, from the processing hardware of the computing device, a first portion of a video frame at the framebuffer, wherein the first and second portions of the video frame are received from a source that generates the video frames and renders the video frames to the framebuffer; and

based on the signal, encoding, by a video encoding hardware module, the second portion of the video frame received from the framebuffer, wherein the encoding of the second portion begins before the first portion of the video frame is fully stored in the framebuffer.

10. A method according to claim 9, further comprising, while encoding the second portion and while the first portion is being written to the framebuffer, formatting a third portion of the video frame into a video streaming container.

11. A method according to claim 10, further comprising, while encoding the second portion and while the first portion is being written to the framebuffer:

transmitting a network protocol frame comprising a portion of the video streaming container that comprises a third portion of the video frame, the third portion having been encoded by the compression module and having been multiplexed with audio data.

12. A method according to claim 9, wherein the framebuffer operates as a frontbuffer and/or a backbuffer, and wherein the framebuffer is connected with a display driver of the computing device to display video frames from the framebuffer without the displayed video frames passing through the compression hardware module.

13. A method according to claim 9, wherein processing of the portions are synchronized by a timer or interrupt signal.

14. A method according to claim 9, wherein a vertical-sync signal drives the displaying and wherein the displaying is synchronized with the vertical-sync signal.

15. A computing device comprising:

a graphics pipeline comprising a framebuffer, an encoder, and a multiplexor, the graphics pipeline configured to receive video frames newly generated to the frame buffer by application software on the computing device and to output a video stream comprised of the video frames encoded multiplexed with audio data, wherein the framebuffer supplies video frame portions to a video encoder and the encoder supplies encoded video frame portions to the multiplexer; and

the computing device configured such that, when operating, the framebuffer will receive a first portion of a video frame while the encoder encodes a second portion of the video frame and while the multiplexer concurrently multiplexes audio data with a third portion of the video frame with audio data, the computing device further configured such that, when operating, the multiplexor multiplexes the encoded third portion of the video frame before receiving the first portion of the video frame.

16. A computing device according to claim 15, wherein the encoder implements either an H.264/AVC (advanced video coding) or an H.265/HEVC (high efficiency video coding) standard.

17. A computing device according to claim 15, wherein the multiplexor packages portions of the video frames encoded by the encoder into a video streaming format.

18. A computing device according to claim 15, further comprising processing hardware configured to generate the video frames.

19. A computing device according to claim 15, wherein the framebuffer, the encoder, and the multiplexor are synchronized in correspondence with a vertical-sync signal.