Graphic system comprising a pipelined graphic engine, pipelining method and computer program product

• US 7,990,389 B2
• Filed: 05/10/2007
• Issued: 08/02/2011
• Est. Priority Date: 07/07/2003
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Barone, Massimiliano
• Primary Examiner(s): Nguyen; Hau H
• Application Number:
  US11/746,849
• Publication Number:
  US 20070285430A1
• Time in Patent Office: 1,545 Days
• Field of Search: 345/419, 345/474, 345/475, 345606-610
• US Class Current: 345/475
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

A graphic system includes a pipelined graphic engine for generating image frames for display. The pipelined graphic engine includes a geometric processing stage for performing motion extraction, and a rendering stage for generating full image frames at a first frame rate for display at a second frame rate. The second frame rate is higher than the first frame rate. A motion encoder stage receives motion information from the geometric processing stage, and produces an interpolated frame signal representative of interpolated frames. A motion compensation stage receives the interpolated frame signal from the motion encoder stage, and the full image frames from the rendering stage for generating the interpolated frames. A preferred application is in graphic systems that operate in association with smart displays through a wireless connection, such as in mobile phones.

Claims

1. A graphic system comprising:

a pipelined graphic engine for generating image frames for display, said pipelined graphic engine comprising

at least one geometric processing stage for performing motion extraction, and

a rendering stage for generating full image frames at a first frame rate for display at a second frame rate, the second frame rate being higher than the first frame rate;

a motion encoder stage for receiving motion information from said at least one geometric processing stage, and for producing an interpolated frame signal representative of interpolated frames;

a motion compensation stage for receiving the interpolated frame signal from said motion encoder stage, and the full image frames from said rendering stage for generating the interpolated frames;

a video decoder stage for receiving the full image frames for display at the second frame rate;

a frame buffer for receiving the full image frames from said video decoder stage, and for providing the full image frames to said motion compensation stage; and

a multiplexer downstream from said frame buffer and from said motion compensation stage for selecting a display order for the full image frames and the interpolated frames.

2. A graphic system of claim 1, further comprising a motion decoder stage upstream from said motion compensation stage.

3. A graphic system of claim 1, wherein said at least one geometric processing stage comprises a geometry stage, and a triangle setup stage connected to an output of said geometry stage.

4. A graphic system of claim 1, further comprising a video encoder stage downstream from said rendering stage for generating a compressed bitstream.

5. A graphic system of claim 4, further comprising a first transmitter downstream from said video encoder stage;

and a second transmitter downstream from said motion encoder stage.

6. A graphic system of claim 1, wherein said motion compensation stage is part of a display unit for displaying the full image frames at the second frame rate;

and wherein said rendering stage and said motion encoder stage are integrated as a self-contained device with the display unit.

7. A graphic system of claim 1, further comprising a perspective correction stage for performing a perspective correction so that said motion encoder stage produces the interpolated frame signal representative of interpolated frames.

8. A graphic system of claim 1, further comprising a perspective correction stage for performing a double perspective correction so that said motion encoder stage produces the interpolated frame signal representative of interpolated frames.

9. A graphic system of claim 8, wherein said perspective correction stage is configured for performing the following:

obtaining motion vectors from primitives of a previous frame in a first space;

projecting the motion vectors from the first space to a second space, wherein a center point of the primitive is unaffected by translation and rotation in the second space; and

producing the interpolated frame signal representative of an interpolated next frame by projecting the motion vectors to the interpolated next frame.

10. A graphic system of claim 9, wherein said perspective correction stage is further configured for performing the following:

projecting to the second space the internal pixels of the primitives of the previous frame;

moving from the previous frame to the interpolated next frame in the internal pixel second space of the interpolated next frame; and

applying the projection of the interpolated next frame on the internal pixels of the primitives to return to the first space.

11. A graphic system of claim 9, wherein the primitives comprise triangles.

12. A graphic system of claim 9, wherein the first space is a screen space.

13. A graphic system of claim 9, wherein the second space is a view space.

14. A graphic system of claim 9, wherein the second space is a common space.

15. A graphic system of claim 14, wherein said perspective correction stage is further configured for performing the following:

linearly interpolating the motion vectors from the primitives of the previous frame in the first space; and

dividing the motion vectors by the respective division factors to return to the first space.

16. A graphic system of claim 15, wherein the first space is a screen space.

17. A method for pipelining graphical operations in a graphic system comprising:

receiving processing instructions from a graphic application for a pipeline graphic engine comprising at least one geometric phase for performing motion extraction;

operating the pipeline graphic engine for generating full image frames at a first frame rate for display at a second frame rate, the second frame rate being higher than the first frame rate;

deriving from the at least one geometric phase motion information to be used as prediction information, and producing an interpolated frame signal representative of interpolated frames;

generating the interpolated frames for display based upon the interpolated frame signal and the full image frames;

producing depth buffer information associated with at least one currently computed frame; and

allotting at least a portion of pixels in the at least one currently computed frame to a set background pixels based on the depth buffer information;

wherein generating the interpolated frames is also based upon the motion information, the generating taking into account the depth buffer information and at least one of the forward and backward motion buffers by

setting a pixel color to a background value if a pixel belongs to the background, otherwise controlling orientation of two motion vectors to carry out a quantization to a pixel position if needed, then

if both vectors are within the forward frame and backward frame, then the pixel color is computed using the information addressed by the motion vectors, and

if only one vector falls in a valid area of a fully rendered forward frame or backward frame, then the pixel color is computed by using the valid information of the fully rendered frame.

18. A method of claim 17, wherein the motion information being used as the prediction information is derived from motion vectors related to at least one of a backward frame and a forward frame included in the motion information.

19. A method of claim 18, wherein the deriving comprises associating motion vectors in at least one of a backward frame and a forward frame to potentially visible image primitives in at least one interpolated frame.

20. A method of claim 19, further comprising quantizing the motion vectors, and storing the motion vectors in at least one of a forward motion buffer and a backward motion buffer.

21. A method of claim 20, further comprising producing contents of at least one of the forward and backward motion buffers using a triangle setup phase.

22. A method of claim 17, further comprising allotting a remaining portion of the pixels in the at least one currently computed frame to a set of pixels in at least one image primitive.

23. A method of claim 17, wherein if both the motion vectors fall outside the frames of reference, then no operation is performed.

24. A method of claim 17, further comprising performing a scan-conversion process based on a linear interpolation of the motion vectors.

25. A method of claim 24, wherein the scan-conversion process comprises a linear interpolation of motion vectors with a perspective correction.

26. A method of claim 17, wherein the graphic system further comprises a perspective correction stage for performing a perspective correction for producing the interpolated frame signal representative of interpolated frames.

27. A method of claim 17, wherein the graphic system further comprises a perspective correction stage for performing a double perspective correction for producing the interpolated frame signal representative of interpolated frames by performing the following:

obtaining motion vectors from primitives of a previous frame in a first space;

projecting the motion vectors from the first space to a second space, wherein a center point of the primitive is unaffected by translation and rotation in the second space; and

producing the interpolated frame signal representative of an interpolated next frame by projecting the motion vectors to the interpolated next frame.

28. A method of claim 27, wherein the perspective correction stage is further configured for performing the following:

projecting to the second space the internal pixels of the primitives of the previous frame;

moving from the previous frame to the interpolated next frame in the internal pixel second space of the interpolated next frame; and

applying the projection of the interpolated next frame on the internal pixels of the primitives to return to the first space.

29. A computer-readable storage medium having computer-executable instructions for causing a graphical system to perform steps comprising:

receiving processing instructions from a graphic application for a pipeline graphic engine comprising at least one geometric phase for performing motion extraction;

operating the pipeline graphic engine for generating full image frames at a first frame rate for display at a second frame rate, the second frame rate being higher than the first frame rate;

deriving from the at least one geometric phase motion information to be used as prediction information, and producing an interpolated frame signal representative of interpolated frames, the motion information being used as the pred information being derived from motion vectors related to at least one of a backward frame and a forward frame included in the motion information,

generating the interpolated frames for display based upon the interpolated frame signal and the full image frames; and

quantizing the motion vectors, and storing the motion vectors in at least one of a forward motion buffer and a backward motion buffer;

wherein generating the interpolated frames is also based upon the motion information, the generating taking into account the depth buffer information and at least one of the forward and backward motion buffers by

setting a pixel color to a background value if a pixel belongs to the background, otherwise controlling orientation of two motion vectors to carry out a quantization to a pixel position if needed, then

if both vectors are within the forward frame and backward frame, then the pixel color is computed using the information addressed by the motion vectors, and

if only one vector falls in a valid area of a fully rendered forward frame or backward frame, then the pixel color is computed by using the valid information of the fully rendered frame.

30. A computer-readable storage medium of claim 29, wherein the deriving comprises associating motion vectors in at least one of a backward frame and a forward frame to potentially visible image primitives in at least one interpolated frame.

31. A computer-readable storage medium of claim 29, further comprising:

producing depth buffer information associated with at least one currently computed frame; and

allotting at least a portion of pixels in the at least one currently computed frame to a set of background pixels based on the de th buffer information.

32. A computer-readable storage medium of claim 29, further comprising allotting a remaining portion of the pixels in the at least one currently computed frame to a set of pixels in at least one image primitive.

33. A computer-readable storage medium of claim 29, wherein if both the motion vectors fall outside the frames of reference, then no operation is performed.

34. A computer-readable storage medium of claim 29, further comprising producing contents of at least one of the forward and backward motion buffers using a triangle setup phase.

35. A computer-readable storage medium of claim 29, further comprising performing a scan-conversion process based on a linear interpolation of the motion vectors.

36. A computer-readable storage medium of claim 35, wherein the scan-conversion process comprises a linear interpolation of motion vectors with a perspective correction.

37. A computer-readable storage medium of claim 29, wherein the graphic system further comprises a perspective correction stage for performing a perspective correction for producing the interpolated frame signal representative of interpolated frames.

38. A computer-readable storage medium of claim 29, wherein the graphic system further comprises a perspective correction stage for performing a double perspective correction for producing the interpolated frame signal representative of interpolated frames by performing the following:

obtaining motion vectors from primitives of a previous frame in a first space;

projecting the motion vectors from the first space to a second space, wherein a center point of the primitive is unaffected by translation and rotation in the second space; and

producing the interpolated frame signal representative of an interpolated next frame by projecting the motion vectors to the interpolated next frame.

39. A graphic system comprising:

a pipelined graphic engine for generating image frames for display, said pipelined graphic engine comprising

at least one geometric processing stage for performing motion extraction, and

a rendering stage for generating full image frames at a first frame rate for display at a second frame rate, the second frame rate being higher than the first frame rate,

a motion encoder stage for receiving motion information from said at least one geometric processing stage, and for producing an interpolated frame signal representative of interpolated frames;

a motion compensation stage for receiving the interpolated frame signal from said motion encoder stage, and the full image frames from said rendering stage for generating the interpolated frames; and

a perspective correction stage for performing a double perspective correction so that said motion encoder stage produces the interpolated frame signal representative of interpolated frames, said perspective correction stage being configured for performing

obtaining motion vectors from primitives of a previous frame in a first space,

projecting, the motion vectors from the first space to a second space, wherein a center point of the primitive is unaffected by translation and rotation in the second space, and

producing the interpolated frame signal representative of an interpolated next frame by projecting the motion vectors to the interpolated next frame.

40. A graphic system of claim 39, further comprising a motion decoder stage upstream from said motion compensation stage.

41. A graphic system of claim 39, further comprising:

a video decoder stage for receiving the full image frames for display at the second frame rate; and

a frame buffer for receiving the full image frames from said video decoder stage, and for providing the full image frames to said motion compensation stage.

42. A graphic system of claim 39, wherein said at least one geometric processing stage comprises a geometry stage, and a triangle setup stage connected to an output of said geometry stage.

43. A graphic system of claim 39, further comprising a video encoder stage downstream from said rendering stage for generating a compressed bitstream.

44. A graphic system of claim 41, further comprising a multiplexer downstream from said frame buffer and from said motion compensation stage for selecting a display order for the full image frames and the interpolated frames.

45. A graphic system of claim 40, wherein said motion compensation stage is part of a display unit for displaying the full image frames at the second frame rate;

and wherein said rendering stage and said motion encoder stage are integrated as a self-contained device with the display unit.

46. A graphic system of claim 40, further comprising a perspective correction stage for performing a perspective correction so that said motion encoder stage produces the interpolated frame signal representative of interpolated frames.

47. A method for pipelining graphical operations in a graphic system comprising:

receiving processing instructions from a graphic application for a pipeline graphic engine comprising at least one geometric phase for performing motion extraction;

operating the pipeline graphic engine for generating full image frames at a first frame rate for display at a second frame rate, the second frame rate being higher than the first frame rate;

deriving from the at least one geometric phase motion information to be used as prediction information, and producing an interpolated frame signal representative of interpolated frames; and

generating the interpolated frames for display based upon the interpolated frame signal and the full image frames;

wherein producing the interpolated frame signal representative of interpolated frames is based on performing a double perspective correction by performing the following

obtaining motion vectors from primitives of a previous frame in a first space,

projecting the motion vectors from the first space to a second space, with a center point of the primitive is unaffected by translation and rotation in the second space, and

producing the interpolated frame signal representative of an interpolated next frame by projecting the motion vectors to the interpolated next frame.

48. A method of claim 47, wherein the motion information being used as the prediction information is derived from motion vectors related to at least one of a backward frame and a forward frame included in the motion information.

49. A method of claim 47, wherein the deriving comprises associating motion vectors in at least one of a backward frame and a forward frame to potentially visible image primitives in at least one interpolated frame.

50. A method of claim 47, further comprising quantizing the motion vectors, and storing the motion vectors in at least one of a forward motion buffer and a backward motion buffer.

51. A method of claim 47, further comprising:

producing depth buffer information associated with at least one currently computed frame; and

allotting at least a portion of pixels in the at least one currently computed frame to a set of background pixels based on the depth buffer information.

52. A method of claim 51, further comprising allotting a remaining portion of the pixels in the at least one currently computed frame to a set of pixels in at least one image primitive.