System and method for combining multiple video streams

• US 6,147,695 A
• Filed: 08/26/1998
• Issued: 11/14/2000
• Est. Priority Date: 03/22/1996
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Silicon Graphics Incorporated (Graphics Properties Holdings, Inc.)
• Inventors: Eitzmann, Gregory M., Warren, David, Jones, Michael T., Dignam, David L., Bowen, Andrew, Maxon, Dawn
• Primary Examiner(s): Chauhan, Ulka J.
• Application Number:
  US09/140,396
• Time in Patent Office: 811 Days
• Field of Search: 345/418, 345/419, 345/428, 345/433, 345/435, 345/439, 345/473, 345/475, 345/501, 345/503
• US Class Current: 345/503
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

An operation for combining multiple video streams permits combining any number of overlay images and base images regardless of processes performed upon one or more of the images. Specifically, where the base images are dynamically sized and resized to provide a constant frame rate (despite varying frame complexity), the process similarly treats overlay images and even other base images. In the dynamic sizing process, a rendering time is compared to high and low water marks. During dynamic resizing, two double buffering operations and a synchronization operation are performed. After dynamic sizing and resizing, the resulting resized images are combined together, regardless of the frame rate of the individual images. Consequently, multiple video streams at varying frame rates are combined at a constant frame rate.

Claims

1. A method for determining a resolution and rendering an image, comprising the steps of:

(a) rendering a current image frame into a frame buffer at an image frame resolution;

(b) computing a rendering time determined by how long it takes to render a previous image frame into said frame buffer;

(c) comparing said rendering time to a high-water mark,

wherein said high-water mark represents a second rendering time at which an image frame resolution would start to be reduced, and

reducing said resolution for a next image frame, if said rendering time exceeds said high-water mark; and

(d) comparing said rendering time to a low-water mark,

wherein said low-water mark represents a third rendering time at which an image frame resolution would start to be increased, and

expanding said resolution for a next image frame, if said rendering time is below said low-water mark and said image frame resolution does not equal a target resolution.

2. A method according to claim 1, further comprising the steps of:

(e) double buffering said current image frame,

wherein a first frame buffer receives said current image frame rendered by step (a), and a second frame buffer stores a previously-rendered image frame rendered by step (a);

(f) double buffering said image frame resolution,

wherein a buffered resolution coefficients register stores said image frame resolution, and

wherein a resolution coefficients register stores an image frame resolution for said previously-rendered image frame; and

(g) synchronizing said double buffering of said current image frame with said double buffering of said image frame resolution,

wherein said first frame buffer and said second frame buffer are swapped concurrently with a transfer of the content of said buffered resolution coefficients register to said resolution coefficients register.

3. A method for combining image frames 1 through n, wherein n is an integer greater than or equal to two, comprising the steps of:

(a) performing a partial rendering of image frames 1 through n;

(b) dynamically sizing image frames 1 through n by calculating one or more image frame resolutions, and dynamically resizing the resulting frames to generate resized image frames 1 through n, wherein said dynamic sizing and resizing for image frames 1 through n comprises the steps of

(1) rendering the current image frame into a frame buffer at an image frame resolution,

(2) computing a rendering time based on how long it takes to render a previous image frame into said frame buffer,

(3) comparing said rendering time to a high-water mark, wherein said high-water mark represents a second rendering time at which said image frame resolution would start to be reduced, and reducing said resolution for a next image frame if said rendering time exceeds said high-water mark, and

(4) comparing said rendering time to a low-water mark, wherein said low-water mark represents a third rendering time at which said image resolution would start to be increased, and expanding said resolution for a next image frame, if said rendering time is below said low-water mark and said image frame resolution does not equal a target resolution, and

(c) combining said resized image frames 1 through n to form a combined image frame for display.

4. A method according to claim 3, further comprising:

(d) repeating steps (a) through (c) if one or more new image frames (n+1) through m are desired to be combined with said combined image frame, wherein m is an integer greater than or equal to three.

5. A method according to claim 3, wherein step (a) comprises the steps of:

generating one or more vertices describing shapes of desired graphical objects;

performing geometry calculations on said vertices to generate one or more primitives; and

transmitting said primitives to a system for performing rasterization thereon.

6. A method according to claim 3, wherein step (b) comprises one of the steps of:

generating and contracting fragments for each of image frames 1 through n wherein the complexity of the image is greater than a threshold level; and

generating and expanding fragments for each of image frames 1 through n wherein the complexity of the image is less than a threshold level.

7. A method according to claim 6, wherein step (b) further comprises:

transforming each of said set of fragments 1 through n into corresponding pixels.

8. A method according to claim 3, wherein step (b) further comprises:

(5) double buffering said current image frame,

wherein a first frame buffer receives said current image frame rendered by step (1), and a second frame buffer stores a previously-rendered image frame rendered by step (1);

(6) double buffering said image frame resolution,

wherein a buffered resolution coefficients register stores said image frame resolution, and

wherein a resolution coefficients register stores an image frame resolution for said previously-rendered image frame; and

(7) synchronizing said double buffering of said current image frame with said double buffering of said image frame resolution,

wherein said first frame buffer and said second frame buffer are swapped concurrently with a transfer of the content of said buffered resolution coefficients register to said resolution coefficients register.

9. A method according to claim 3, further comprising one of the steps of:

using a push model,

wherein a rasterization system renders said image frames 1 through n to a plurality of frame buffers and pushes said image frames 1 through n to a display system for display; and

using a pull model,

wherein a rasterization system renders said image frames 1 through n to a plurality of frame buffers, and a display system extracts said image frames 1 through n for display.

10. A method according to claim 3, wherein step (c) comprises one of the steps of:

(1) using a binary display method, wherein as between a first image frame of image frames 1 through n and a second image frame of image frames 1 through n;

a pixel of said first image frame is displayed in lieu of a corresponding pixel of said second image frame if said first image frame pixel is opaque, and

said corresponding pixel of said second image frame is displayed if said first image frame pixel is non-opaque; and

(2) using a blending display method, wherein pixels of a first image frame of image frames 1 through n and corresponding pixels of a second image frame of image frames 1 through n are blended together.

11. A system for combining image frames 1 through n, wherein n is an integer greater than or equal to two, comprising:

means for performing a partial rendering of image frames 1 through n;

means for dynamically sizing image frames 1 through n by calculating one or more image frame resolutions, and dynamically resizing the resulting frames to generate resized image frames 1 through n, wherein said dynamic sizing and resizing means comprises

means for rendering the current image frame into a frame buffer at an image frame resolution,

means for computing a rendering time based on how long it takes to render a previous image frame into said frame buffer,

means for comparing said rendering time to a high-water mark, wherein said high-water mark represents a second rendering time at which said image frame resolution would start to be reduced, and wherein said means for comparing said rendering time to a high-water mark reduces said resolution for a next image frame, if said rendering time exceeds said high-water mark, and

means for comparing said rendering time to a low-water mark, wherein said low-water mark represents a third rendering time at which said image resolution would start to be increased, and wherein said means for comparing said rendering time to a low-water mark expands said resolution for a next image frame, if said rendering time is below said low-water mark and said image frame resolution does not equal a target resolution; and

means for combining said resized image frames 1 through n to form a combined image frame, and displaying said combined image frame.

12. A system according to claim 11, wherein said performing means comprises:

means for generating one or more vertices describing shapes of desired graphical objects;

means for performing geometry calculations on said vertices to generate one or more primitives; and

means for transmitting said primitives to a system for performing rasterization thereon.

13. A system according to claim 12, wherein said dynamic sizing and dynamic resizing means comprises:

means for generating and contracting fragments for each of image frames 1 through n wherein the complexity of the image is greater than a threshold level; and

means for generating and expanding fragments for each of image frames 1 through n wherein the complexity of the image is less than a threshold level.

14. A system according to claim 13, wherein said dynamic sizing and dynamic resizing means comprises:

means for transforming each of said set of fragments 1 through n into corresponding pixels.

15. A system according to claim 11, wherein said sizing and resizing means further comprises:

means for double buffering said current image frame,

wherein a first frame buffer receives said current image frame, and a second frame buffer stores a previously-rendered image frame;

means for double buffering said image frame resolution,

wherein a buffered resolution coefficients register stores said image frame resolution, and

wherein a resolution coefficients register stores an image frame resolution for said previously-rendered image frame; and

means for synchronizing said current image frame double buffering means and said image frame resolution double buffering means,

wherein said first frame buffer and said second frame buffer are swapped concurrently with a transfer of the content of said buffered resolution coefficients register to said resolution coefficients register.

16. A system according to claim 11, further comprising one of:

push model means, comprising;

a rasterization system for rendering said image frames 1 through n to a plurality of frame buffers and for pushing said image frames 1 through n to a display system for display; and

pull model means, comprising;

a rasterization system for rendering said image frames 1 through n to a plurality of frame buffers, and

a display system for extracting said image frames 1 through n for display.

17. A system according to claim 11, wherein said means for combining comprises:

(i) binary display means,

wherein as between a first image frame of image frames 1 through n and a second image frame of image frames 1 through n,

said binary display means displays a pixel of said first image frame in lieu of a corresponding pixel of said second image frame if said first image frame pixel is opaque, and

said binary display means displays said corresponding pixel of said second image frame if said first image frame pixel is non-opaque; and

(ii) blending display means,

wherein said blending display means blends pixels of a first image frame of image frames 1 through n and corresponding pixels of a second image frame of image frames 1 through n together to form said combined image frame.