Video processing system and method with dynamic tag architecture

• US 7,869,666 B2
• Filed: 07/15/2005
• Issued: 01/11/2011
• Est. Priority Date: 01/13/2005
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Ruggiero, Carl J.
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US11/183,227
• Publication Number:
  US 20060153475A1
• Time in Patent Office: 2,006 Days
• Field of Search: 382298-308
• US Class Current: 382/303
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/46 for receiving on more than one standard at will deflecting circuits of multi-standard receivers H04N3/27
• CPC Class Codes: H04N 7/012 Conversion between an interlaced and a progressive signal for graphics images G09G2310/0229
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59
• CPC Class Codes: H04N 7/0145 the interpolation being class adaptive, i.e. it uses the information of class which is determined for a pixel based upon certain characteristics of the neighbouring pixels
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

An image processing system and method, in which a plurality of image processing operations are dynamically controlled based on dynamically changing tag data associated with pixels being processed.

Claims

1. An image processing system, comprising:

a pixel processing subsystem configured to receive input pixels and process the input pixels so as to produce output pixels;

an image data classifier operatively coupled with the pixel processing subsystem, where the image data classifier is configured to interact with the pixel processing subsystem at multiple different times prior to output of the output pixels, and where the pixel processing subsystem is configured to dynamically tune its pixel processing operations based on such interactions with the image data classifier; and

a controller operatively coupled with the pixel processing subsystem and image data classifier and configured to dynamically and unequally allocate processing times for the pixel processing subsystem.

2. The system of claim 1, where the pixel processing subsystem includes a deinterlacer.

3. The system of claim 1, where the pixel processing subsystem includes an image interpolator.

4. The system of claim 1, where the pixel processing subsystem includes a color processor.

5. The system of claim 1, where the image data classifier includes a motion estimator.

6. The system of claim 5, where the motion estimator is operatively coupled with a plurality of different image processing operators included in the pixel processing subsystem, and where the motion estimator is configured to determine motion data for pixels being applied as processing inputs to the plurality of different image processing operators.

7. The system of claim 1, where the image data classifier is configured to repeatedly update tag data associated with pixels being processed by the pixel processing subsystem and where, at multiple different times prior to output of the output pixels, pixel processing operations are dynamically controlled based on updated tag data.

8. The system of claim 7, where the pixel processing subsystem includes multiple different image processing operators.

9. The system of claim 8, where the pixel processing subsystem and image data classifier are configured so that tag data associated with pixels being processed by the pixel processing subsystem is updated prior to application of the pixels to a first one of the different image processing operators, and where, after the pixels are processed by said first image processing operator, the tag data associated with the pixels is again updated and then used to dynamically control a processing operation performed on the pixels at a second one of the different image processing operators.

10. The system of claim 1, where the pixel processing subsystem includes multiple pixel processing operators configured to process the input pixels in a sequential order in a first mode of operation, the image processing system further comprising a controller configured to effect a second mode of operation in which the sequential order is varied.

11. The system of claim 10, where the sequential order is varied by bypassing one or more of the multiple pixel processing operators.

12. The system of claim 11, where the controller effects bypassing of one or more pixel processing operators based on tag data obtained by the image data classifier.

13. The system of claim 11, where the controller effects bypassing of one or more pixel processing operators based on processing information associated with the pixels being processed by the pixel processing subsystem.

14. The system of claim 1, where the pixel processing subsystem includes multiple pixel processing operators configured to sequentially process the input pixels such that the input pixels are sequentially and successively modified by each of the pixel processing operators and then output from the pixel processing subsystem as the output pixels.

15. The system of claim 14, where the multiple pixel processing operators define a sequential processing pipeline capable of processing video data in a FIFO manner, such that during operation, multiple frames of video data are being processed within the pixel processing subsystem.

16. The system of claim 15, where the controller is configured to allocate processing time from one of the multiple frames of video data to another of the multiple frames of video data.

17. The system of claim 15, where the controller is configured to allocate processing time from one portion of one of the multiple video frames to another portion of said video frame.

18. The system of claim 15, where the controller is configured to allocate processing time from one of the multiple pixel processing operators to another of the multiple pixel processing operators.

19. The system of claim 15, where processing time allocations are effected based on tag data associated with pixels being processed by the pixel processing subsystem.

20. The system of claim 1, further comprising a shared motion estimator, where the pixel processing subsystem includes multiple pixel processing operators operatively coupled to the motion estimator and configured to be dynamically controlled based on motion-related tag data obtained from the shared motion estimator.

21. An integrated circuit, comprising:

image processing logic configured to sequentially execute a plurality of image processing operations on pixels of video data; and

controller and classifier logic configured to control operation of the image processing logic, and repeatedly update tag data associated with the pixels of video data, said updating being performed prior to execution of each of the plurality of image processing operations, and configured to execute one of the plurality of image processing operations by;

dynamically selecting, for each pixel or group of pixels to be processed by said one of the plurality of image processing operations, a particular implementation of said image processing operation; and

applying the particular implementation to the pixel or group of pixels,

where the particular implementation is selected from a plurality of loaded implementations of said image processing operation, and where the plurality of loaded implementations is varied from time to time during operation of the integrated circuit.

22. The integrated circuit of claim 21, where the plurality of image processing operations are configured to sequentially process the pixels of video data such that the pixels of video data are sequentially and successively processed by each of the plurality of image processing operations.

23. The integrated circuit of claim 22, where the plurality of image processing operations define a sequential processing pipeline capable of processing video data in a FIFO manner, such that during operation, multiple frames of video data are being processed within the image processing logic.

24. The integrated circuit of claim 23, where the controller and classifier logic is configured to dynamically and unequally allocate processing time for the image processing logic.

25. The integrated circuit of claim 24, where the controller and classifier logic is configured to allocate processing time from one of the multiple frames of video data to another of the multiple frames of video data.

26. The integrated circuit of claim 24, where the controller and classifier logic is configured to allocate processing time from one portion of one of the multiple video frames to another portion of said video frame.

27. The integrated circuit of claim 24, where the controller and classifier logic is configured to allocate processing time from one of the plurality of image processing operations to another of the plurality of image processing operations.

28. The integrated circuit of claim 24, where processing time allocations are effected based on the tag data.

29. The integrated circuit of claim 21, where the image processing logic and controller and classifier logic are configured so that each of the plurality of processing operations are dynamically controlled based on updated tag data.

30. The integrated circuit of claim 29, where the tag data includes classification information pertaining to characteristics of one or more pixels contained in the pixels of video data.

31. The integrated circuit of claim 29, where the tag data includes processing information pertaining to processing operations applied to one or more pixels contained in the pixels of video data.

32. The integrated circuit of claim 29, where the tag data includes source identification information associated with one or more pixels contained in the pixels of video data.

33. An image processing system, comprising:

image processing logic configured to sequentially execute a plurality of image processing operations on pixels of video data;

storage media configured to store the pixels and tag data associated with the pixels, the image processing system being configured to dynamically update the tag data between execution of the plurality of image processing operations, and where the image processing logic is configured such that each image processing operation is dynamically controlled based on the tag data; and

a controller operatively coupled with the image processing logic and configured to dynamically and unequally allocate processing times for the image processing logic.

34. The system of claim 33, further comprising a controller operatively coupled with the image processing logic and configured to dynamically vary an order in which the image processing operations are performed on the pixels of video data.

35. The system of claim 34, where the order is varied by bypassing one or more of the image processing operations.

36. The system of claim 35, where the controller effects bypassing of one or more image processing operations based on tag data associated with the pixels of video data.

37. The system of claim 36, where the tag data includes classification information pertaining to characteristics of the pixels of video data.

38. The system of claim 36, where the tag data includes processing information pertaining to processing operations applied to the pixels of video data.

39. The system of claim 36, where the tag data includes source identification information associated with the pixels of video data.

40. The system of claim 33, where the plurality of image processing operations define a sequential processing pipeline capable of processing video data in a FIFO manner, such that during operation, multiple frames of video data are being processed by the image processing system.

41. The system of claim 40, where the controller is configured to allocate processing time from one of the multiple frames of video data to another of the multiple frames of video data.

42. The system of claim 40, where the controller is configured to allocate processing time from one portion of one of the multiple video frames to another portion of said video frame.

43. The system of claim 40, where the controller is configured to allocate processing time from one of the multiple image processing operations to another of the multiple image processing operations.

44. The system of claim 40, where processing time allocations are effected based on tag data associated with pixels being processed by the image processing logic.

45. The system of claim 40, where the tag data includes classification information pertaining to characteristics of the pixels.

46. An image processing system, comprising:

a pixel processing subsystem configured to receive input pixels and process the input pixels so as to produce output pixels,

where the pixel processing subsystem includes multiple pixel processing operators configured to process the input pixels in a sequential order in a first mode of operation, the image processing system further comprising a controller configured to effect a second mode of operation in which the sequential order is varied; and

a controller operatively coupled with the pixel processing subsystem and configured to dynamically and unequally allocate processing times for the multiple pixel processing operators.