Methods and apparatus for rendering multiple images using a limited rendering resource

• US 6,384,846 B1
• Filed: 02/18/1999
• Issued: 05/07/2002
• Est. Priority Date: 12/11/1998
• Status: Expired due to Fees

• Current Assignee: Hitachi America Limited (Hitachi, Ltd.)
• Original Assignee: Hitachi America Limited (Hitachi, Ltd.)
• Inventors: Hiroi, Kazushige
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Joseph, Thomas J
• Application Number:
  US09/252,709
• Time in Patent Office: 1,174 Days
• Field of Search: 345/781, 345/782, 345/473, 345/653, 345/654, 345/638, 345/619, 345/614, 345/637, 345/794, 345/795, 345/629
• US Class Current: 715/794
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 5/45 Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen

Abstract

The present invention addresses decoding and simultaneously displaying multiple images in different windows of a PC or other display device such as a television set. When more than two images are displayed, if the total processing load associated with decoding all of the images to be displayed is over 100%, then decoding priority is decided for each image as a function of display window size. By decoding and displaying images according to this priority, the user can view a desired image, e.g., a main picture image smoothly and comfortably, while other images displayed in smaller sub-windows may only be partially decoded during each frame time. When windows are of the same size, decoding priority is determined between the windows of the same size, as a function of the amount of decoder resources required to decode each image. As the amount of decoder resources needed to decode and image to be displayed in a window of a given size increases, the decoder priority is decreased relative to images which are to be displayed in the same size window. In one embodiment, the image assigned the highest decoding priority is always fully decoded while only portions of the other images may be decoded. The percentage of an image which is decoded will normally be lower for images with relatively low decoding priority than for images with relatively high decoding priority.

Claims

1. A method of rendering a plurality of images, the plurality of images including first and second images to be displayed in first and second windows of a display device, respectively, the first and second windows being of the same size, the rendering involving the use of an image rendering resource, the method comprising the steps of:

determining which of the first and second images will require more use of the rendering resource to render;

providing the one of the first and second images to be displayed which will require the least amount of use of the rendering resource to render, a higher priority in terms of access to the rendering resource than the one of the first and second images to be displayed which will require a greater use of the rendering resource to render; and

rendering at least a portion of the one of the first and second images to be displayed which is assigned the higher priority.

2. The method of claim 1, wherein the rendering resource is a processor, the determining step including the step of assessing the amount of processor time required to render the first and second images.

3. The method of claim 2, wherein the first and second images to be rendered are represented prior to image rendering in the form of encoded image data and wherein the rendering step includes the step of generating decoded imaged data.

4. The method of claim 3, wherein the step of assessing the amount of processor time required to render the first and second images includes the step of accessing information indicating the size of the first and second images represented by encoded image data prior to image rendering.

5. A method of rendering a plurality of images, the plurality of images including first and second images to be displayed in first and second windows of a display device, respectively, the rendering involving the use of an image rendering resource, the method comprising the steps of:

determining which of the first and second windows is larger;

providing the one of the first and second images to be displayed in the larger window a higher priority in terms of access to the rendering resource than the one of the first and second images to be displayed in the smaller window; and

rendering at least portions of the first and second images using the rendering resource as a function of the assigned rendering priorities.

6. The method of claim 5, wherein

the step of rendering includes the steps of:

rendering a first percentage of the image assigned the higher priority; and

rendering an equal or lesser percentage of the image being assigned the lower priority.

7. The method of claim 5, wherein the rendering step includes the step of:

fully rendering the image assigned the highest priority amongst the plurality of images.

8. The method of claim 7,

wherein the plurality of images to be rendered are represented as encoded digital image data prior to rendering;

wherein the rendering resource is a decoder; and

wherein the rendering step includes the step of using the decoder to decode the encoded digital image data to generate decoded image data.

9. The method of claim 6, wherein the plurality of images includes a third image, the third image being for display in a window of a third size, the method further comprising the step of:

comparing the size of the third window to the size of the first and second window; and

assigning, as a function of the size of the third window, a rendering priority to the third image.

10. The method of claim 9, wherein the second and third windows are of the same size, the method further comprising the step of:

determining the amount of at least one rendering resource required to render the second and third images; and

wherein the step of assigning a rendering priority is performed as a function of the amount of the at least one rendering resource required to render the second and third images.

11. The method of claim 10, wherein the at least one rendering resource is processor time.

12. The method of claim 10, wherein the determining step includes the step of:

obtaining original image size information indicating the original size of the second and third images.

13. The method of claim 11, wherein the third image requires more processor time to render than the second image, the step of assigning a rendering priority to the third image including the step of:

assigning the third image a lower rendering priority than the second image.

14. A method of allocating image rendering resources for processing data representing a plurality of different images, each of the plurality of images to be simultaneously displayed on a display screen, each image being associated with a different one of a plurality of windows of the display screen, the method comprising the steps of:

assigning image rendering priorities to images as a function of the size of the window in which the image is to be displayed, the larger the window in which the image is to be displayed, the greater the image rendering priority assigned to the image;

performing image rendering operations as a function of the image rendering priority assigned to each image, the higher the image rendering priority, the greater the probability that the image will be fully rendered; and

outputting data produced by the image rendering operations to a display device.

15. The method of claim 14, wherein the step of assigning the image rendering priorities includes the step of:

comparing the sizes of the display windows.

16. The method of claim 14, wherein the step of performing an image rendering operation includes the step of:

fully rendering the image assigned the highest image rendering priority; and

rendering, at most, only a portion of the image assigned the lowest image rendering priority.

17. The method of claim 14, further comprising the step of:

wherein the step of assigning image rendering priorities includes the step of assigning to images which are to be simultaneously displayed in windows of the same size, rendering priorities which are a function of processor loads associated with rendering said images.

18. The method of claim 17,

wherein first and second images are to be displayed in windows of the same size, the first image requiring less processor time to render than the second image, the step of assigning image rendering priorities including the step of assigning a higher image rendering priority to the first image than is assigned to the second image.

19. The method of claim 17, further comprising the step of assessing processor loads associated with rendering images to be displayed in the same size windows.

20. The method of claim 19, wherein the step of assessing processor loads is performed using information on the original size of an image which is being rendered and the size of the window in which the image being rendered will be displayed.

21. The method of claim 14, wherein the image rendering step includes the step of:

performing a decoding operation on encoded image data representing an image to be displayed.

22. A system for rendering and display images, comprising:

a display device for simultaneously displaying a plurality of images on a display screen, each image being displayed in a different display screen window;

means for receiving encoded data representing the images to be simultaneously displayed;

a decoder coupled to the receiving means, for decoding at least some of the received encoded image data;

decoder control means for controlling use of the decoder to decode encoded image data when all of the image data to be simultaneously displayed on a display screen can not be decoded in a limited amount of time available to decode the encoded image data, the data control means giving priority in terms of decoder access to encoded data representing images as a function of the size of the image window in which the data is to be displayed, the larger the window in which the data is to be displayed, the higher the assigned priority.

23. The system of claim 22, wherein the display device is a computer monitor.

24. The system of claim 22, wherein the display device is a television set.

25. The system of claim 22, further wherein the decoder control means includes:

means for giving priority in terms of decoder access to encoded data representing images to be displayed in the same size windows to the encoded image data which requires the least amount of decoder time to decode.

26. The system of claim 25, wherein the means for giving decoder priority includes:

means for determining the amount of decoder time required to processes a set of encoded image data.

27. The system of claim 25, wherein the means for determining the amount of decoder time includes a database of encoded image size information.

28. A method of allocating decoder resources amongst a plurality of data streams of encoded video images, each data stream to be decoded and different window of a display having a plurality of windows, the method comprising the steps of:

assigning the highest decoding priority to the data stream to be displayed in the largest of the plurality of windows;

assigning, as a function of the size of the window in which the images of a data stream are to be displayed, decoding priorities to each of the remaining data streams, the larger the window in which the image is to be displayed, the higher the assigned decoder priority.

29. The method of claim 28, wherein data streams which are displayed in equal sized windows are assigned decoder priority both as a function of window size and the amount of processing resources required to decoded the encoded images in the data streams, data streams requiring a larger amount of resources to decode being assigned a lower decoding priority than data streams requiring fewer resources to decode.

30. The method of claim 28 wherein the method is further directed to using decoder resources, the method further comprising the step of decoding encoded images included in said data streams, the decoding step including the steps of:

i. decoding all the encoded images in the data stream assigned the highest decoding priority are decoded; and

ii. decoding only a portion of the encoded images in the lowest priority data stream are decoded.

31. The method of claim 30,

wherein intra-coded frames included in the lowest priority data stream are decoded but not predictively or bi-directionally coded frames.

32. The method of claim 30, where an intermediate decoding priority is assigned to one of the data streams, the intra-coded and at least some predictively coded frames included in the data stream assigned intermediate decoding priority being decoded but not bi-directionally coded frames.