METHOD AND APPARATUS FOR STREAMING MEDIA DATA PROCESSING, AND STREAMING MEDIA PLAYBACK EQUIPMENT

• US 20140049689A1
• Filed: 09/21/2012
• Published: 02/20/2014
• Est. Priority Date: 12/05/2011
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Guangzhou Ucweb Computer Technology Company, Ltd., UC Technology Company Limited
• Inventors: Liang, Jie
• Granted Patent: US 8,670,072 B1
• Time in Patent Office: Days
• US Class Current: 348/515
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

The present invention provides a streaming media data processing method. The method includes: based on a stream index in streaming media data, separating the streaming media data stream into audio stream data and video stream data and respectively buffering them in an audio stream data queue and a video stream data queue; respectively decoding audio data buffered in the audio stream data queue and video data buffered in the video stream data queue; based on a play callback timestamp of the decoded audio data and a system time of a streaming media playback equipment, determining an audio/video synchronization time; based on a comparison result between a video frame timestamp and a sum of the determined audio/video synchronization time and a video refresh time, processing and displaying each frame in the decoded video stream data according to a predetermined processing method in accordance with the comparison result. The method may avoid the asynchrony between video and audio steams resulted from the asynchrony between audio time and the system time and, at the same time, the synchronization between the video time and the system time when displaying the stream data, which may greatly reduce the repeated video pauses in the video display.

Claims

1. A streaming media data processing method, comprising:

based on a stream index in streaming media data, separating the streaming media data into audio stream data and video stream data and respectively buffering them in an audio stream data queue and a video stream data queue;

respectively decoding audio data buffered in the audio stream data queue and video data buffered in the video stream data queue;

based on a play callback timestamp of the decoded audio data and a system time of a streaming media playback equipment, determining an audio/video synchronization time;

based on a comparison result between a video frame timestamp and a sum of the determined audio/video synchronization time and a video refresh time, processing and displaying each frame in the decoded video stream data according to a predetermined processing method in accordance with the comparison result.

2. The streaming media data processing method according to claim 1, wherein decoding the video data buffered in the video stream data queue further includes:

when a video frame is determined to be a key data frame or the video frame timestamp is larger than or equal to the sum of the determined audio/video synchronization time and the decoding time of the video frame, decoding the video frame; and

when the video frame is determined not to be a key data frame or the video frame timestamp is less than the sum of the determined audio/video synchronization time and the decoding time of the video frame, discarding the video frame.

3. The streaming media data processing method according to claim 1, wherein, when the play callback timestamp of the decoded audio data is inconsistent with the system time of the streaming media playback equipment, the audio/video synchronization time is determined to be half of the sum of the callback timestamp and the system time of the streaming media playback equipment.

4. The streaming media data processing method according to claim 1, wherein, based on the comparison result between the video frame timestamp and the sum of the determined audio/video synchronization time and the video refresh time, processing and displaying each frame in the decoded video stream data according to the predetermined processing method in accordance with the comparison result further includes:

when the video frame timestamp is greater than the sum of the determined audio/video synchronization time and the video refresh time, waiting for displaying the video frame data;

when the video frame time stamp is equal to the sum of the determined audio/video synchronization time and the video refresh time, displaying the video frame; and

when the video frame time stamp is less than the sum of the determined audio/video synchronization time and the video refresh time, discarding the video frame.

5. The streaming media data processing method according to claim 1, further comprising:

before decoding the audio stream data, determining whether a data buffer queue in the audio playing module is fully filled with the audio stream data;

when the data buffer queue in the audio playing module is fully filled with the audio stream data or all audio stream data is completely decoded, playing the audio steaming data;

otherwise,

continuously decoding the audio stream data buffered in the audio stream data queue.

6. The streaming media data processing method according to claim 4, wherein, when displaying the video frame, a pointer redirecting technique is used to perform a reference operation directly on a memory storing the video frame.

14. The streaming media data processing method according to claim 2, wherein, when the play callback timestamp of the decoded audio data is inconsistent with the system time of the streaming media playback equipment, the audio/video synchronization time is determined to be half of the sum of the callback timestamp and the system time of the streaming media playback equipment.

15. The streaming media data processing method according to claim 2, wherein, based on the comparison result between the video frame timestamp and the sum of the determined audio/video synchronization time and the video refresh time, processing and displaying each frame in the decoded video stream data according to the predetermined processing method in accordance with the comparison result further includes:

when the video frame timestamp is greater than the sum of the determined audio/video synchronization time and the video refresh time, waiting for displaying the video frame data;

when the video frame time stamp is equal to the sum of the determined audio/video synchronization time and the video refresh time, displaying the video frame; and

when the video frame time stamp is less than the sum of the determined audio/video synchronization time and the video refresh time, discarding the video frame.

16. The streaming media data processing method according to claim 2, further comprising:

before decoding the audio stream data, determining whether a data buffer queue in the audio playing module is fully filled with the audio stream data;

when the data buffer queue in the audio playing module is fully filled with the audio stream data or all audio stream data is completely decoded, playing the audio steaming data;

otherwise,

continuously decoding the audio stream data buffered in the audio stream data queue.

7. A streaming media data processing apparatus, comprising:

a data separating unit configured to separate the streaming media data into audio stream data and video stream data, based on a stream index in the streaming media data;

a buffering unit configured to respectively buffer the separated audio stream data and video stream data in an audio stream data queue and a video stream data queue;

an audio data decoding unit configured to decode the audio data buffered in the audio stream data queue;

a video data decoding unit configured to decode the video stream data buffered in the video stream data queue;

an audio/video synchronization time determining unit configured to determine an audio/video synchronization time, based on a play callback timestamp of the decoded audio data and a system time of the streaming media playback apparatus; and

a video frame processing unit configured to, based on a comparison result between a video frame timestamp and the sum of the determined audio/video synchronization time and a video refresh time, process and display each frame of the decoded video stream data according to a predetermined processing method in accordance with the comparison result.

8. The streaming media data processing apparatus according to claim 7, wherein, when the play callback timestamp of the decoded audio data is inconsistent with the system time of the streaming media playback apparatus, the audio/video synchronization time determining unit determines that audio/video synchronization time is half of the sum of the play callback timestamp and the system time.

9. The streaming media data processing apparatus according to claim 7, further comprising:

a time determining unit configured to, before decoding the video frame, determine whether the video frame is a key data frame or whether the video frame timestamp is larger than or equal to the sum of the determined audio/video synchronization time and decoding time of the video frame,

wherein;

when the video frame is a key data frame or the video frame timestamp is larger than or equal to the sum of the determined audio/video synchronization time and the decoding time of the video frame, the video data decoding unit decodes the video frame; and

when the video frame is not a key data frame or the video frame timestamp is less than the sum of the determined audio/video synchronization time and the decoding time of the video frame, the video data decoding unit discards the video frame.

10. The streaming media data processing apparatus according to claim 7, wherein, according to the predetermined processing method in accordance with the comparison result, to process and display the video frame further includes:

when the video frame timestamp is greater than the sum of determined audio/video synchronization time and the video refresh time, the video frame processing unit waits for displaying the video frame;

when the video frame timestamp is equal to the sum of determined audio/video synchronization time and the video refresh time, the video frame processing unit displays the video frame;

or

when the video frame timestamp is less than the sum of the determined audio/video synchronization time and the video refresh time, the video frame processing unit discards the video frame.

11. The streaming media data processing apparatus according to claim 7, further including:

an audio stream data display determining unit configured to, before displaying the audio stream data, determine whether a data buffer queue in an audio playing module has been fully filled with the audio steam data;

wherein

when the data buffer queue in the audio playing module has been fully filled with the audio steam data, the audio playing module plays the audio stream data;

otherwise, the audio data decoding unit continuously decodes the audio stream data.

12. The streaming media data processing apparatus according to claim 10, further comprising:

a memory reference unit configured to, when displaying the video data, use a pointer redirecting technique to perform a reference operation directly on a memory storing the video frame.

13. A streaming media data playback equipment, comprising a streaming media data processing apparatus according to claim 7.

17. The streaming media data processing apparatus according to claim 9, wherein, according to the predetermined processing method in accordance with the comparison result, to process and display the video frame further includes:

when the video frame timestamp is greater than the sum of determined audio/video synchronization time and the video refresh time, the video frame processing unit waits for displaying the video frame;

when the video frame timestamp is equal to the sum of determined audio/video synchronization time and the video refresh time, the video frame processing unit displays the video frame;

or

when the video frame timestamp is less than the sum of the determined audio/video synchronization time and the video refresh time, the video frame processing unit discards the video frame.