Decomposed multi-stream (DMS) techniques for video display systems

• US 9,582,238 B2
• Filed: 12/13/2010
• Issued: 02/28/2017
• Est. Priority Date: 12/14/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Rajamani, Krishnan, Wang, Xiaodong, Shaukat, Fawad
• Primary Examiner(s): Li, Tracy Y.
• Application Number:
  US12/966,495
• Publication Number:
  US 20110145879A1
• Time in Patent Office: 2,269 Days
• Field of Search: 375/240.01, 375/240.26
• US Class Current: 1/1
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G09G 2310/04 Partial updating of the display screen
• CPC Class Codes: G09G 2320/0613 The adjustment depending on the type of the information to be displayed
• CPC Class Codes: G09G 2320/10 Special adaptations of display systems for operation with variable images
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2352/00 Parallel handling of streams of display data
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04L 65/70 Media network packetisation

Abstract

This disclosure describes techniques that can improve the generation of a decomposed multi-stream (DMS) by a host device of a video display system and the display of a DMS by a client device of the video display system. The techniques may apply different frame rates to different streams within a DMS, and the frame rates may depend on the content. For example, one stream within a DMS may comprise a sequence of full-motion video information, which may be rendered at a relatively high frame rate. However, another stream within the DMS may be associated with a background of the display, various graphic user interface control windows or elements, or a display window that includes non-video content (such as e-mail or a document). The second stream in the DMS may be rendered at a much slower frame rate than that used for the sequence of full-motion video information.

Claims

1. A method comprising:

generating a decomposed multi-stream (DMS) via a host device of a video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;

generating information in the DMS that identifies which of the first content or the second content is on top in the overlapping area;

communicating the DMS from the host device to a client device of the video display system; and

adjusting the DMS to only include one of the first content or the second content in response to user input.

2. The method of claim 1, further comprising dynamically adjusting the first video frame rate and the second video frame rate in the DMS based on the first content and the second content.

3. The method of claim 1, wherein generating the DMS includes intercepting the first content from an application and capturing the second content from a display buffer in the host device.

4. The method of claim 3, further comprising dynamically reducing a capture rate associated with capturing the second content from the display buffer based on the first content and the second content.

5. The method of claim 1, wherein generating the DMS includes encoding the second content.

6. The method of claim 1, wherein generating the DMS includes reducing or eliminating data for either the first content or the second content in the overlapping area, wherein the reduced or eliminated data is below other data in the overlapping area.

7. The method of claim 1, wherein generating the DMS includes intercepting the first content or the second content from an application.

8. The method of claim 7, further comprising:

determining that intercepting the first content or the second content from the application is not possible; and

generating a signal composed stream (SCS) that includes the first content and the second content directly from a display buffer at the host device in response to determining that intercepting the first content or the second content from the application is not possible.

9. The method of claim 1, wherein the first content comprises a video sequence and the second content comprises output of a non-video application.

10. The method of claim 9, wherein the first video frame rate is greater than the second video frame rate.

11. The method of claim 1, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.

12. The method of claim 1, further comprising:

determining a bandwidth available between the host device and the client device; and

adjusting one or both of the first video frame rate and the second video frame rate based on the bandwidth available.

13. The method of claim 1, further comprising:

determining a bandwidth available between the host device and the client device; and

dynamically encoding one or both of the first content and the second content based on the bandwidth available.

14. The method of claim 1, further comprising dynamically encoding one or both of the first content and the second content.

15. A method comprising:

receiving a decomposed multi-stream (DMS) at a client device of a video display system from a host device of the video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;

rendering the first display content and the second display content on the client device; and

rendering only one of the first content or the second content in response to user input.

16. The method of claim 15, wherein the first content comprises a video sequence and the second content comprises output of a non-video application.

17. The method of claim 16, wherein the first video frame rate is greater than the second video frame rate.

18. The method of claim 15, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.

19. The method of claim 15, further comprising:

buffering the first content differently than the second content in the client device.

20. The method of claim 19, further comprising applying different latency rules for display of the first content by the client device and display of the second content by the client device.

21. A host device of a video display system, the host device comprising:

a decomposed multi-stream (DMS) generator that generates a DMS, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate, and wherein the DMS generator generates information in the DMS that identifies which of the first content or the second content is on top in the overlapping area and adjusts the DMS to only include one of the first content or the second content in response to user input; and

a transport unit that communicates the DMS from the host device to a client device of the video display system.

22. The host device of claim 21, wherein the DMS generator dynamically adjusts the first video frame rate and the second video frame rate in the DMS based on the first content and the second content.

23. The host device of claim 21, wherein the DMS generator intercepts the first content from an application and captures the second content from a display buffer in the host device.

24. The host device of claim 23, wherein the DMS generator dynamically reduces a capture rate associated with capturing the second content from the display buffer based on the first content and the second content.

25. The host device of claim 23, wherein the DMS generator uses an encoder to encode the second content in the DMS.

26. The host device of claim 21, wherein the DMS generator reduces or eliminates data for either the first content or the second content in the overlapping area, wherein the reduced or eliminated data is below other data in the overlapping area.

27. The host device of claim 21, wherein the DMS generator intercepts the first content or the second content from an application.

28. The host device of claim 27, wherein the DMS generator:

determines that intercepting the first content or the second content from the application is not possible; and

generates a signal composed stream (SCS) that includes the first content and the second content directly from a display buffer at the host device in response to determining that intercepting the first content or the second content from the application is not possible.

29. The host device of claim 21, wherein the first content comprises a video sequence and the second content comprises output of a non-video application.

30. The host device of claim 29, wherein the first video frame rate is greater than the second video frame rate.

31. The host device of claim 21, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.

32. The host device of claim 21, wherein the DMS generator:

determines a bandwidth available between the host device and the client device; and

adjusts one or both of the first video frame rate and the second video frame rate based on the bandwidth available.

33. The host device of claim 21, wherein the DMS generator:

determines a bandwidth available between the host device and the client device; and

dynamically encodes one or both of the first content and the second content based on the bandwidth available.

34. The host device of claim 21, wherein the DMS generator dynamically encodes one or both of the first content and the second content.

35. A client device of a video display system, the client device comprising:

a transport unit that receives a decomposed multi-stream (DMS) from a host device, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;

a display unit; and

a DMS render unit that renders the first display content and the second display content on the display unit, wherein the DMS render renders only one of the first content or the second content in response to user input.

36. The client device of claim 35, wherein the first content comprises a video sequence and the second content comprises output of a non-video application.

37. The client device of claim 36, wherein the first video frame rate is greater than the second video frame rate.

38. The client device of claim 35, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.

39. The client device of claim 35, wherein the display unit includes buffers that are configured to buffer the first content differently than the second content.

40. The client device of claim 35, wherein the render unit is configured to apply different latency rules for display of the first content by the display unit, and display of the second content by the display unit.

41. A non-transitory computer-readable storage medium comprising instructions that upon execution in a processor of a host device of a video display system, cause the host device to:

generate a decomposed multi-stream (DMS), wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;

generate information in the DMS that identifies which of the first content or the second content is on top in the overlapping area;

communicate the DMS from a host device to a client device of the video display system; and

adjusts the DMS to only include one of the first content or the second content in response to user input.

42. A non-transitory computer-readable storage medium comprising instructions that upon execution in a processor of a client device of a video display system, cause the client device to:

upon receiving a decomposed multi-stream (DMS) from a host device of the video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate,

render the first display content and the second display content on the client device, and

render only one of the first content or the second content in response to user input.

43. A host device of a video display system, the host device comprising:

means for generating a decomposed multi-stream (DMS), wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;

means for generating information in the DMS that identifies which of the first content or the second content is on top in the overlapping area;

means for communicating the DMS from the host device to a client device of the video display system; and

means for adjusting the DMS to only include one of the first content or the second content in response to user input.

44. A client device of a video display system, the client device comprising:

means for receiving a decomposed multi-stream (DMS) from a host device of the video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;

means for rendering the first display content and the second display content on the client device; and

means for rendering only one of the first content or the second content in response to user input.