Backfill of video stream

• US 10,135,897 B2
• Filed: 01/24/2017
• Issued: 11/20/2018
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kirkby, Ronald Loren, Nelson, Gregory Rourk, Duffy, Gregory R.
• Primary Examiner(s): Lai, Michael C
• Application Number:
  US15/414,553
• Publication Number:
  US 20170201353A1
• Time in Patent Office: 665 Days
• CPC Class Codes: H04L 1/1877 for semi-reliable protocols, e.g. for less sensitive applications like streaming video buffer level management for video bitstream control arrangements H04N21/44004
• CPC Class Codes: H04L 47/193 at the transport layer, e.g. TCP related
• CPC Class Codes: H04L 47/2433 Allocation of priorities to traffic types
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A data streaming system includes one or more processors, memory, and a circular buffer stored in the memory. The system includes a media loader process that retrieves frames from a media source and writes the retrieved frames in chronological order to the circular buffer. The system includes a forward dispatch process that transmits frames from the circular buffer to a destination server in chronological order in real-time. The system also includes a backfill dispatch process that executes in response to receiving a backfill request from the destination server, transmitting frames from the circular buffer to the destination server in reverse chronological order.

Claims

1. A camera system, comprising:

one or more processors;

memory having a limited capacity;

a camera element configured to continuously capture video data in real-time and store the captured video in the memory;

a microphone configured to continuously capture audio data in real-time and store the captured audio in the memory; and

a communication interface;

wherein the one or more processors are configured to;

continuously capture video and audio data in real-time via the camera element and the microphone, respectively;

while capturing the video and audio data in real-time, continuously store the captured video and audio data in the memory having the limited capacity;

continuously send the continuously captured video and audio data in real-time to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; and

while sending the continuously captured video and audio data in real time to the remote server system;

determine that a portion of the video and audio data sent to the remote server system was not received by the remote server system, thereby creating a gap in the video and audio data stored at the remote server system and available for access by the client access device; and

when at least some of the portion is still available in the memory having the limited capacity, resend the at least some of the portion from the memory to the remote server system via the communication interface to fill at least some of the gap.

2. The camera system of claim 1, wherein the client access device is configured to display video and audio data sent from the camera system to the remote server system and stored at the remote server system.

3. The camera system of claim 1, wherein the client access device is configured to display video and audio data sent from the camera system to the client access device in real time via the remote server system.

4. The camera system of claim 1, wherein resending the portion occurs concurrently while continuing to capture video and audio data at the camera element.

5. The camera system of claim 1, wherein resending the portion occurs concurrently while continuing to continuously send the video and audio data in real-time to the remote server system.

6. The camera system of claim 1, wherein segments of the captured audio and video are timestamped, and determining that a portion of the video and audio data sent to the remote server system has not been received by the remote server system includes determining which timestamped segments of the captured audio and video have not been acknowledged by the remote server system.

7. The camera system of claim 1, wherein determining that a portion of the video and audio data sent to the remote server system has not been received by the remote server system includes receiving a backfill request from the remote server system.

8. The camera system of claim 1, wherein data that was stored more recently is resent before data that was stored earlier.

9. The camera system of claim 1, wherein data that was stored earlier is resent before data that was stored more recently.

10. The camera system of claim 1, wherein the camera system comprises the remote server system.

11. A method, comprising:

at a camera system having one or more processors, memory having a limited capacity, a camera element configured to continuously capture video data in real-time and store the captured video in the memory, a microphone configured to continuously capture audio data in real-time and store the captured audio in the memory, and a communication interface;

continuously capturing video and audio data in real-time via the camera element and the microphone, respectively;

while capturing the video and audio data in real-time, continuously storing the captured video and audio data in the memory having the limited capacity;

continuously sending the continuously captured video and audio data in real-time to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; and

while sending the continuously captured video and audio data in real time to the remote server system;

determining that a portion of the video and audio data sent to the remote server system was not received by the remote server system, thereby creating a gap in the video and audio data stored at the remote server system and available for access by the client access device; and

when at least some of the portion is still available in the memory having the limited capacity, resending the at least some of the portion from the memory to the remote server system via the communication interface to fill at least some of the gap.

12. The method of claim 11, wherein determining that a portion of the video and audio data sent to the remote server system has not been received by the remote server system includes receiving a backfill request from the remote server system.

13. The method of claim 11, wherein data that was stored more recently is resent before data that was stored earlier.

14. The method of claim 11, further comprising at the remote server system:

receiving a stream of audio and video from the camera element;

storing the received stream of audio and video data in memory at the remote server system; and

providing access to the received audio and video data to the client access device.

15. The method of claim 11, wherein the camera system comprises the remote server system, further comprising:

receiving by the remote server system a stream of audio and video from the camera element;

storing the received stream of audio and video data in memory at the remote server system; and

providing by the remote server system access to the received audio and video data to the client access device.

16. The method of claim 15, further comprising, at the remote server system:

identifying the gap in the received audio and video; and

in response to identifying the gap, sending a backfill request to the camera element.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a camera system having one or more processors, memory having a limited capacity, a camera element configured to continuously capture video data in real-time and store the captured video in the memory, a microphone configured to continuously capture audio data in real-time and store the captured audio in the memory, and a communication interface, wherein the one or more programs comprise instructions for:

continuously capturing video and audio data in real-time via the camera element and the microphone, respectively;

while capturing the video and audio data in real-time, continuously storing the captured video and audio data in the memory having the limited capacity;

continuously sending the continuously captured video and audio data in real-time to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; and

while sending the continuously captured video and audio data in real time to the remote server system;

determining that a portion of the video and audio data sent to the remote server system was not received by the remote server system, thereby creating a gap in the video and audio data stored at the remote server system and available for access by the client access device; and

when at least some of the portion is still available in the memory having the limited capacity, resending the at least some of the portion from the memory to the remote server system via the communication interface to fill at least some of the gap.

18. The computer readable storage medium of claim 17, wherein resending the portion occurs concurrently while continuing to capture video and audio data at the camera element.

19. The computer readable storage medium of claim 17, wherein resending the portion occurs concurrently while continuing to send the video and audio data from the memory to the remote server system.

20. The computer readable storage medium of claim 17, wherein segments of the captured audio and video are timestamped, and determining that a portion of the video and audio data sent to the remote server system was not received by the remote server system includes determining which timestamped segments of the captured audio and video have not been acknowledged by the remote server system.