Backfill of video stream

• US 10,708,334 B2
• Filed: 11/19/2018
• Issued: 07/07/2020
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kirkby, Ronald Loren, Nelson, Gregory Rourk, Duffy, Gregory R.
• Primary Examiner(s): Lai, Michael C
• Application Number:
  US16/195,453
• Publication Number:
  US 20190141105A1
• Time in Patent Office: 596 Days
• CPC Class Codes: H04L 1/1877 for semi-reliable protocols, e.g. for less sensitive applications like streaming video buffer level management for video bitstream control arrangements H04N21/44004
• CPC Class Codes: H04L 47/193 at the transport layer, e.g. TCP related
• CPC Class Codes: H04L 47/2433 Allocation of priorities to traffic types
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

This application is directed to a camera system having one or more processors, memory having a limited capacity, a camera element and a microphone. The camera system captures video and audio data in real-time via the camera element and the microphone, respectively. While capturing the video and audio data in real-time, the captured video and audio data are stored in the memory having the limited capacity, and sent to a remote server system via the communication interface for storage by the remote server system. The camera system determines that a portion of the video and audio data was not available at the remote server system. In accordance with a determination that at least some of the portion is still available in the memory having the limited capacity, the camera system resends the at least some of the portion from the memory to the remote server system via the communication interface.

Claims

1. A system, comprising:

(1) a camera system, further comprising;

one or more processors;

a camera element configured to capture video data in real-time;

a microphone configured to capture audio data in real-time;

first memory having a limited capacity and configured to store the video and audio data captured by the camera element and the microphone, respectively; and

a communication interface;

wherein the camera system is configured to;

capture the video and audio data in real-time via the camera element and the microphone, respectively;

while capturing the video and audio data in real-time, store the captured video and audio data in the first memory having the limited capacity;

send the captured video and audio data to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; and

in response to a determination that a portion of video and audio data captured by the camera system is not stored at the remote server system, send at least the portion of video and audio data for view on the client access device, wherein the portion of the video and audio data is associated with a gap in the video and audio data stored at the remote server system; and

(2) a second memory of the client access device for storing an application executable by the client access device, the application being installed on the client access device and configured to, when executed by the client access device;

access the video and audio data stored at the remote server system; and

enable display of the video and audio data including the portion of the video and audio data that is resent by the camera system.

2. The system of claim 1, wherein the camera system is configured to:

determine that additional space is needed in the first memory for storing the captured video and audio data; and

in accordance with a determination that additional space is needed in the first memory, overwrite oldest video and audio data of the captured video and audio data on the first memory with the video and audio data being captured in real-time.

3. The system of claim 1, wherein the camera system is configured to:

provide at least some of the portion of the video and audio data stored in the first memory to the client access device includes by resending the at least some of the portion of the video and audio data stored in the first memory to the remote server system, thereby allowing the client access device to access the at least some of the portion of the video and audio data via the remote server system.

4. The system of claim 3, wherein resending the at least some of the portion occurs concurrently while continuing to capture video and audio data at the camera system.

5. The system of claim 3, wherein resending the at least some of the portion occurs concurrently while continuing to continuously send the video and audio data in real-time to the remote server system.

6. The system of claim 1, wherein segments of the captured audio and video are timestamped, and the camera system is configured to determine that the portion of the video and audio data is not available at the remote server system by determining which timestamped segments of the captured audio and video have not been acknowledged by the remote server system.

7. The system of claim 1, wherein the camera system is configured to:

determine that a portion of the video and audio data was is not available at the remote server system;

determine that at least some of the portion of the video and audio data is still available in the first memory having the limited capacity; and

provide the at least some of the portion of the video and audio data stored in the first memory to the client access device.

8. The system of claim 1, wherein the client access device is configured to display video and audio data sent from the camera system to the remote server system and stored at the remote server system.

9. The system of claim 1, wherein the client access device is configured to display video and audio data sent from the camera system to the client access device in real time via the remote server system.

10. A method, comprising:

capturing video and audio data in real-time via a camera element and a microphone of a camera system, respectively, the camera system having one or more processors, memory having a limited capacity, the camera element, the microphone, and a communication interface;

while capturing the video and audio data in real-time, storing by the camera system the captured video and audio data in the memory having the limited capacity;

sending the captured video and audio data by the camera system to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; and

in response to a determination that a portion of video and audio data captured by the camera system is not stored at the remote server system, sending by the camera system at least the portion of video and audio data for view on the client access device, wherein the portion of the video and audio data is associated with a gap in the video and audio data stored at the remote server system;

accessing the video and audio data stored at the remote server system via an application installed on and executable by the client access device; and

enabling, by the application of the client access device, display of the video and audio data including the portion of the video and audio data that is resent by the camera system.

11. The method of claim 10, further comprising:

determining that additional space is needed in the memory for storing the captured video and audio data; and

in accordance with a determination additional space is needed in the memory, overwriting oldest video and audio data of the captured video and audio data on the memory with the video and audio data being captured in real-time.

12. The method of claim 10, further comprising:

in response to a determination that a portion of video and audio data captured by the camera system is not stored at the remote server system, resending by the camera system at least some of the portion of video and audio data stored in the memory, wherein data that was stored in the memory more recently is resent before data that was stored earlier.

13. The method of claim 10, further comprising:

in response to a determination that a portion of video and audio data captured by the camera system is not stored at the remote server system, resending by the camera system at least some of the portion of video and audio data stored in the memory, wherein data that was stored earlier in the memory is resent before data that was stored more recently.

14. The method of claim 10, further comprising at the remote server system:

receiving a stream of video and audio data from the camera system;

storing the received stream of video and audio data in memory at the remote server system; and

providing access to the received video and audio data to the client access device.

15. The method of claim 14, further comprising, at the remote server system:

identifying the gap in the received audio and video; and

in response to identifying the gap, sending a backfill request to the camera system.

16. The method of claim 10, wherein the memory of the camera system includes a circular buffer, and the captured video and audio data is stored in the circular buffer and sent to the remote server system according to a chronological order.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a camera system having one or more processors, memory having a limited capacity, a camera element, a microphone, and a communication interface, the non-transitory computer readable storage medium including a second memory of a client access device for storing an application installed on and executable by the client access device, wherein the one or more programs comprise instructions for:

capturing video and audio data in real-time via the camera element and the microphone, respectively;

while capturing the video and audio data in real-time, storing the captured video and audio data in the memory having the limited capacity; and

sending the captured video and audio data to a remote server system via the communication interface for storage by the remote server system, wherein the video and audio data stored at the remote server system is subsequently accessible by a client access device; and

in response to a determination that a portion of video and audio data captured by the camera system is not stored at the remote server system, sending at least the portion of video and audio data for view on the client access device, wherein the portion of the video and audio data is associated with a gap in the video and audio data stored at the remote server system;

accessing the video and audio data stored at the remote server system via the application of the client access device; and

enabling, by the application of the client access device, display of the video and audio data including the portion of the video and audio data that is resent by the camera system.

18. The computer readable storage medium of claim 17, wherein the one or more programs further comprise instructions for:

determining that additional space is needed in the memory for storing the captured video and audio data; and

in accordance with a determination additional space is needed in the memory, overwriting oldest video and audio data of the captured video and audio data on the memory with the video and audio data being captured in real-time.

19. The computer readable storage medium of claim 18, further comprising:

resending by the camera system at least some of the portion of video and audio data stored in the memory, wherein sending the captured video and audio data to the remote server system is given a higher priority than resending the at least some of the portion to the remote server system.

20. The computer readable storage medium of claim 17, wherein the captured video and audio data is stored in the memory and sent to the remote server system according to a chronological order, and at least some of the portion is resent to the remote server system according to a reverse chronological order until reaching a position in the memory where content has been overwritten by video and audio data that are newly recorded.