Systems and methods for performing smooth visual search of media encoded for adaptive bitrate streaming via hypertext transfer protocol using trick play streams

• US 9,210,481 B2
• Filed: 02/07/2014
• Issued: 12/08/2015
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Original Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Inventors: Braness, Jason, Soroushian, Kourosh, van der Schaar, Auke Sjoerd
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US14/175,929
• Publication Number:
  US 20140250473A1
• Time in Patent Office: 669 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/322 used signal is digitally coded
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/26258 for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list retrieval of multimedia data based on playlists G06F16/40
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/42607 for processing the incoming bitstream
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8455 : involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 : involving a specific file format, e.g. MP4 format
• CPC Class Codes: H04N 21/8543 : using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80

Abstract

Systems and methods for performing smooth visual search of media encoded for adaptive bitrate streaming using trick play streams are disclosed. One embodiment of the invention includes requesting and buffering portions of video from at least one of the alternative streams using a playback device, decoding the buffered portions of video using a decoder on the playback device, receiving at least one user instruction directing the playback device to perform a visual search of the media, requesting and buffering potions of video from the trick play stream using the playback device, and decoding the buffered portions of the trick play stream using a decoder on the playback device.

Claims

1. A method of performing visual search of encoded media using a trick play stream using a playback device, where the media is encoded as a plurality of alternative streams used during normal playback and a trick play stream used during visual search, the method comprising:

requesting and buffering portions of video from at least one of the alternative streams using the playback device, wherein;

the alternative streams and a trick play stream are stored in separate Extensible Binary Markup Language (EBML) container files,

each of the EBML container files comprises a plurality of Cluster elements, where each Cluster element contains a portion of encoded video, and

the portion of encoded video in each of the Cluster elements for the alternative streams and the trick play stream commences with an intra frame;

decoding the buffered portions of video using a decoder on the playback device;

receiving at least one user instruction directing the playback device to perform a visual search of the media;

requesting and receiving an index that identifies the location of each frame in the portion of trick play stream video within the separate EBML container file containing the trick play stream, wherein the index is packed into the separate EBML container file containing the trick play stream;

requesting and buffering portions of video from the trick play stream using the playback device; and

decoding the buffered portions of the trick play stream using a decoder on the playback device.

2. The method of claim 1, further comprising measuring the current streaming conditions by measuring the time taken to receive requested portions of a stream from the time at which the portions were requested using the playback device.

3. The method of claim 2, wherein requesting and buffering portions of video from at least one of the alternative streams using a playback device further comprises requesting and buffering portions of video from at least one of the alternative streams based upon the bitrates of the alternative streams and the measured streaming conditions using the playback device.

4. The method of claim 1, wherein each frame of the trick play stream is an intra frame starting a closed group of pictures (GOP).

5. The method of claim 1, wherein:

the at least one user instruction specifies the rate and direction of the visual search; and

requesting and buffering potions of video from the trick play stream using the playback device further comprises requesting frames from the trick play stream based upon the rate and direction of the visual search specified by the at least one user instruction.

6. The method of claim 1, wherein:

requesting portions of video from a stream further comprising requesting portions of files from remote servers via Hypertext Transfer Protocol (HTTP) byte range requests using the playback device.

7. The method of claim 1, wherein the received at least one user instruction indicates a desired rate for the visual search, and wherein the method further comprises:

manipulating, using the playback device, the timecodes assigned to the buffered frames of the trick play stream prior to providing the buffered frames of the trick play stream to a decoder on the playback device, wherein the timecodes are manipulated in order to achieve a rate of playback based on the desired rate for the visual search; and

providing, using the playback device, the buffered frames of the trick play stream to the decoder on the playback device.

8. The method of claim 1, the intra frame in each of the Cluster elements for the trick play stream has a time code corresponding to at least one Cluster element of the alternative streams, wherein the time codes indicate start times of the encoded video within the Cluster elements.

9. The method of claim 1, wherein each Cluster element includes at least one closed group of pictures.

10. The method of claim 1, wherein the portions of encoded video in each of the Cluster elements of the alternative streams have the same duration.

11. The method of claim 10, wherein the portions of encoded video in each of the Cluster elements of the alternative streams have a 2 second duration and each of the Cluster element of the trick play stream include 64 frames.

12. The method of claim 1, wherein each Cluster element contains a time code and each encoded frame of the portion of encoded video contained within the Cluster element is contained within a separate element.

13. The method of claim 12, wherein the first element containing a frame in the Cluster element contains the intra frame.

14. The method of claim 13, wherein the first element containing a frame is a BlockGroup element that contains a Block element, which specifies the time code attribute of the intra frame relative to the time code of the Cluster element.

15. The method of claim 12, wherein all of the elements that contain frames in the Cluster elements of the EBML container file containing the trick play stream contain intra frames.

16. The method of claim 1, wherein the index is a EBML container file containing the trick play stream includes at least one modified Cues element containing an index that references Clusters elements containing portions of encoded video within the EBML container file and the elements containing frames within the referenced Clusters elements.

17. The method of claim 16, further comprising:

identifying frames to retrieve from the EBML container file based upon a visual search rate using the Cues element; and

wherein requesting and buffering potions of video from the trick play stream using the playback device further comprises requesting and buffering elements that contain frames from the EBML container file containing the trick play stream using the playback device.

18. The method of claim 17, further comprising updating the size attribute of Cluster elements retrieved by the playback device.

19. The method of claim 1, further comprising modifying the time stamps of the buffered portions of the trick play stream prior to providing the portions of the trick play stream to the decoder for decoding.

20. The method of claim 1, further comprising retrieving a top level index file using the playback device that identifies the alternative streams used during normal playback of the encoded video and identifies the trick play stream used during visual search of the encoded video.