Video encoding for reduced streaming latency

• US 9,894,393 B2
• Filed: 08/31/2015
• Issued: 02/13/2018
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Petersen, Mark, Tran, Peter, Youel, Jeffrey S.
• Primary Examiner(s): TELAN, MICHAEL R
• Application Number:
  US14/841,431
• Publication Number:
  US 20170064344A1
• Time in Patent Office: 897 Days
• Field of Search: None
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4382 Demodulation or channel decoding, e.g. QPSK demodulation analog front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/4385 Multiplex stream processing, e.g. multiplex stream decrypting
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/6336 directed to decoder
• CPC Class Codes: H04N 21/845 Structuring of content, e.g. decomposing content into time segments
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A client device requests a streaming video from a storage device, which encodes the video'"'"'s frame in communication packets according to an encoding protocol for transmission to the client. The client device decodes the communication packets to obtain frame segments and a frame marking header indicating a boundary of the video frame relative to other video frames in the video. The communication packets may also include other segment headers indicating a relative ordering of corresponding frame segments. The client device determines, based on the frame marking header, whether a sufficient number of communication packets have been received to render a video frame. In response to determining that the sufficient number of communication packets has been received, the client device assembles the video frame from the frame segments corresponding to the video frame and presents at least a portion of the video by rendering the assembled video frame.

Claims

1. A non-transitory computer-readable medium comprising instructions for streaming a video, the instructions executable by a processor and comprising instructions for:

receiving, from a client device, a request to stream the video for playback by the client device;

accessing a video frame from video frames included in the video, the video frame being a last frame in a group of pictures included in the video, the group of pictures comprising an intra-coded video frame and one or more inter-coded video frames;

segmenting the video frame into a plurality of frame segments;

identifying a frame boundary segment immediately preceding a frame segment in another group of pictures included in the video;

generating segment headers indicating a sequence order of the frame segments, the segment headers comprising a frame marking header, the frame marking header indicating a boundary of the video frame relative to other video frames in the video and including a flag identifying the frame boundary segment;

generating communication packets each having a payload comprising one of the segment headers and a corresponding one of the frame segments, the communication packets comprising a communication packet with a payload comprising the frame boundary segment and the frame marking header; and

transmitting the communication packets to the client device for playback of the video frame, the client device rendering the video frame using the frame marking header.

2. The computer-readable medium of 1, wherein generating the segment headers comprises:

determining a total number of frame segments included in the frame segments; and

generating the frame marking header specifying the total number of frame segments.

3. The computer-readable medium of 1,

wherein generating the segment headers comprises:

identifying the frame boundary segment at an end of the sequence order of the frame segments.

4. The computer-readable medium of 1, wherein the instructions further comprise instructions for:

generating a subsequent set of communication packets comprising frame segments of a next video frame immediately following the video frame; and

transmitting the subsequent set of communication packets corresponding to the next video frame after transmitting all the communication packets corresponding to the video frame.

5. A camera comprising:

an image sensor;

a processor; and

a non-transitory computer-readable medium comprising instructions for transferring a video captured by the image sensor, the instructions executable by a processor and comprising instructions for;

receiving, from a client device, a request to stream the video for playback by the client device;

accessing a video frame from video frames included in the video, the video frame being a last frame in a group of pictures included in the video, the group of pictures comprising an intra-coded video frame and one or more inter-coded video frames;

segmenting the video frame into a plurality of frame segments;

identifying a frame boundary segment immediately preceding a frame segment in another group of pictures included in the video;

generating segment headers each indicating a sequence order of the frame segments, the segment headers comprising a frame marking header, the frame marking header indicating a boundary of the video frame relative to other video frames in the video and including a flag identifying the frame boundary segment;

generating communication packets each having a payload comprising one of the segment headers and a corresponding one of the frame segments, the communication packets comprising a communication packet with a payload comprising the frame boundary segment and the frame marking header; and

transmitting the communication packets to the client device for playback of the video frame, the client device rendering the video frame using the frame marking header.

6. The camera of claim 5, wherein generating the segment headers comprises:

determining a total number of frame segments included in the frame segments; and

generating the frame marking header specifying the total number of frame segments.

7. The camera of claim 5,

wherein generating the segment headers comprises:

identifying the frame boundary segment at an end of the sequence order of the frame segments.

8. The camera of claim 5, wherein the instructions further comprise instructions for:

generating a subsequent set of communication packets comprising frame segments of a next video frame immediately following the video frame; and

transmitting the subsequent set of communication packets corresponding to the next video frame after transmitting all the communication packets corresponding to the video frame.