Video processing system and method with dynamic tag architecture

• US 20060153475A1
• Filed: 07/15/2005
• Published: 07/13/2006
• Est. Priority Date: 01/13/2005
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Ruggiero, Carl J.
• Granted Patent: US 7,869,666 B2
• Time in Patent Office: Days
• US Class Current: 382/276
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/46 for receiving on more than one standard at will deflecting circuits of multi-standard receivers H04N3/27
• CPC Class Codes: H04N 7/012 Conversion between an interlaced and a progressive signal for graphics images G09G2310/0229
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59
• CPC Class Codes: H04N 7/0145 the interpolation being class adaptive, i.e. it uses the information of class which is determined for a pixel based upon certain characteristics of the neighbouring pixels
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

An image processing system and method, in which a plurality of image processing operations are dynamically controlled based on dynamically changing tag data associated with pixels being processed.

Claims

1. An image processing system, comprising:

a pixel processing subsystem configured to receive input pixels and process the input pixels so as to produce output pixels; and

an image data classifier operatively coupled with the pixel processing subsystem, where the image data classifier is configured to interact with the pixel processing subsystem at multiple different times prior to output of the output pixels, and where the pixel processing subsystem is configured to dynamically tune its pixel processing operations based on such interactions with the image data classifier.

2. The system of claim 1, where the pixel processing subsystem includes a deinterlacer.

3. The system of claim 1, where the pixel processing subsystem includes an image interpolator.

4. The system of claim 1, where the pixel processing subsystem includes a color processor.

5. The system of claim 1, where the image data classifier includes a motion estimator.

6. The system of claim 5, where the motion estimator is operatively coupled with a plurality of different image processing operators included in the pixel processing subsystem, and where the motion estimator is configured to determine motion data for pixels being applied as processing inputs to the plurality of different image processing operators.

7. The system of claim 1, where the image data classifier is configured to repeatedly update tag data associated with pixels being processed by the pixel processing subsystem and where, at multiple different times prior to output of the output pixels, pixel processing operations are dynamically controlled based on updated tag data.

8. The system of claim 7, where the pixel processing subsystem includes multiple different image processing operators.

9. The system of claim 8, where the pixel processing subsystem and image data classifier are configured so that tag data associated with pixels being processed by the pixel processing subsystem is updated prior to application of the pixels to a first one of the different image processing operators, and where, after the pixels are processed by said first image processing operator, the tag data associated with the pixels is again updated and then used to dynamically control a processing operation performed on the pixels at a second one of the different image processing operators.

10. The system of claim 1, where the image processing system further comprises a controller operatively coupled with the pixel processing subsystem and image data classifier and configured to dynamically and unequally allocate processing times for the pixel processing subsystem.

11. The system of claim 1, where the pixel processing subsystem includes multiple pixel processing operators configured to process the input pixels in a sequential order in a first mode of operation, the image processing system further comprising a controller configured to effect a second mode of operation in which the sequential order is varied.

12. The system of claim 11, where the sequential order is varied by bypassing one or more of the multiple pixel processing operators.

13. The system of claim 12, where the controller effects bypassing of one or more pixel processing operators based on tag data obtained by the image data classifier.

14. The system of claim 12, where the controller effects bypassing of one or more pixel processing operators based on processing information associated with the pixels being processed by the pixel processing subsystem.

15. The system of claim 1, where the pixel processing subsystem includes multiple pixel processing operators configured to sequentially process the input pixels such that the input pixels are sequentially and successively modified by each of the pixel processing operators and then output from the pixel processing subsystem as the output pixels.

16. The system of claim 15, where the multiple pixel processing operators define a sequential processing pipeline capable of processing video data in a FIFO manner, such that during operation, multiple frames of video data are being processed within the pixel processing subsystem.

17. The system of claim 16, further comprising a controller configured to dynamically and unequally allocate processing time for the pixel processing subsystem.

18. The system of claim 17, where the controller is configured to allocate processing time from one of the multiple frames of video data to another of the multiple frames of video data.

19. The system of claim 17, where the controller is configured to allocate processing time from one portion of one of the multiple video frames to another portion of said video frame.

20. The system of claim 17, where the controller is configured to allocate processing time from one of the multiple pixel processing operators to another of the multiple pixel processing operators.

21. The system of claim 17, where processing time allocations are effected based on tag data associated with pixels being processed by the pixel processing subsystem.

22. The system of claim 1, further comprising a shared motion estimator, where the pixel processing subsystem includes multiple pixel processing operators operatively coupled to the motion estimator and configured to be dynamically controlled based on motion-related tag data obtained from the shared motion estimator.

23. A method of processing digital video signals, comprising:

receiving input pixels to be processed;

performing multiple processing operations on the input pixels, where the multiple processing operations are performed during a time interval determined in part by a desired video output rate; and

performing a classification analysis at an intermediate time during the time interval, the classification analysis yielding tag data that is used to dynamically vary one or more of the multiple processing operations, and where the tag data is generated on a per-pixel basis to enable pixel by pixel variation of the multiple processing operations.

24. The method of claim 23, where the multiple processing operations include a deinterlacing operation.

25. The method of claim 23, where the multiple processing operations include an interpolation operation.

26. The method of claim 23, where the multiple processing operations include a color processing operation.

27. The method of claim 23, further comprising, prior to performance of the multiple processing operations, analyzing the input pixels to obtain source identification information associated with the input pixels, and further comprising dynamically tuning one or more of the multiple processing operations based on the source identification information.

28. The method of claim 23, further comprising, for one of the multiple processing operations:

providing a plurality of implementations of such processing operation;

using selected ones of the plurality of implementations to perform the processing operation; and

modifying the plurality of implementations and continuing to perform the processing operation after modification of the plurality of implementations.

29. The method of claim 23, further comprising dynamically allocating processing time for performing pixel processing operations, where such allocation is based upon tag data associated with pixels being processed by the method.

30. The method of claim 29, where dynamically allocating processing time includes performing an unequal allocation, in which processing time is allocated from one frame of video data to another frame of video data.

31. The method of claim 29, where dynamically allocating processing time includes performing an unequal allocation, in which processing time is allocated from one portion of a frame of video data to another portion of the frame of video data.

32. The method of claim 29, where dynamically allocating processing time includes performing an unequal allocation, in which processing time is allocated from one of the multiple processing operations to another of the multiple processing operations.

33. A method of processing video data, comprising:

receiving input pixels to be processed;

performing multiple processing operations on the input pixels, where the multiple processing operations are performed during a time interval determined in part by a desired video output rate; and

repeatedly performing a classification analysis during the time interval as the input pixels are changed by the multiple processing operations, where results of the repeated analyses are used to dynamically vary one or more of the multiple processing operations.

34. The method of claim 33, where repeatedly performing a classification analysis includes repeatedly updating tag data associated with the input pixels.

35. The method of claim 33, further comprising, for one of the multiple processing operations:

providing a plurality of implementations of such processing operation;

using selected ones of the plurality of implementations to perform the processing operation; and

modifying the plurality of implementations and continuing to perform the processing operation after modification of the plurality of implementations.

36. The method of claim 35, where modifying the plurality of implementations is performed based on observed usage of the plurality of implementations.

37. The method of claim 35, where modifying the plurality of implementations is performed based on observed characteristics of pixels being processed using the method.

38. The method of claim 33, further comprising dynamically allocating processing time for the multiple processing operations, where such allocation is based upon tag data associated with pixels being processed by the method.

39. The method of claim 38, where dynamically allocating processing time includes performing an unequal allocation, in which processing time is allocated from one frame of video data to another frame of video data.

40. The method of claim 38, where dynamically allocating processing time includes performing an unequal allocation, in which processing time is allocated from one portion of a frame of video data to another portion of the frame of video data.

45. The method of claim 38, where dynamically allocating processing time includes performing an unequal allocation, in which processing time is allocated from one of the multiple processing operations to another of the multiple processing operations.

46. An integrated circuit, comprising:

logic circuitry instantiated in silicon, the logic circuitry being configured to;

receive input pixels to be processed;

perform multiple processing operations on the input pixels, where the multiple processing operations are performed during a time interval determined in part by a desired video output rate; and

repeatedly perform a classification analysis during the time interval as pixels are changed by the multiple processing operations, where results of the repeated analyses are used to dynamically vary one or more of the multiple processing operations.

47. An integrated circuit, comprising:

image processing logic configured to sequentially execute a plurality of image processing operations on pixels of video data; and

controller and classifier logic configured to control operation of the image processing logic, and repeatedly update tag data associated with the pixels of video data, said updating being performed prior to execution of each of the plurality of image processing operations.

48. The integrated circuit of claim 47, where the plurality of image processing operations are configured to sequentially process the pixels of video data such that the pixels of video data are sequentially and successively processed by each of the plurality of image processing operations.

49. The integrated circuit of claim 48, where the plurality of image processing operations define a sequential processing pipeline capable of processing video data in a FIFO manner, such that during operation, multiple frames of video data are being processed within the image processing logic.

50. The integrated circuit of claim 49, where the controller and classifier logic is configured to dynamically and unequally allocate processing time for the image processing logic.

51. The integrated circuit of claim 50, where the controller and classifier logic is configured to allocate processing time from one of the multiple frames of video data to another of the multiple frames of video data.

52. The integrated circuit of claim 50, where the controller and classifier logic is configured to allocate processing time from one portion of one of the multiple video frames to another portion of said video frame.

53. The integrated circuit of claim 50, where the controller and classifier logic is configured to allocate processing time from one of the plurality of image processing operations to another of the plurality of image processing operations.

54. The integrated circuit of claim 50, where processing time allocations are effected based on the tag data.

55. 4.2. The integrated circuit of claim 47, where the image processing logic and controller and classifier logic are configured so that each of the plurality of processing operations are dynamically controlled based on updated tag data.

56. The integrated circuit of claim 55, where the tag data includes classification information pertaining to characteristics of one or more pixels contained in the pixels of video data.

57. The integrated circuit of claim 55, where the tag data includes processing information pertaining to processing operations applied to one or more pixels contained in the pixels of video data.

58. The integrated circuit of claim 55, where the tag data includes source identification information associated with one or more pixels contained in the pixels of video data.

59. The integrated circuit of claim 47, where the image processing logic and controller and classifier logic are configured to execute one of the plurality of image processing operations by:

dynamically selecting, for each pixel or group of pixels to be processed by said one of the plurality of image processing operations, a particular implementation of said image processing operation; and

applying the particular implementation to the pixel or group of pixels,

where the particular implementation is selected from a plurality of loaded implementations of said image processing operation, and where the plurality of loaded implementations is varied from time to time during operation of the integrated circuit.

60. An image processing system, comprising:

image processing logic configured to sequentially execute a plurality of image processing operations on pixels of video data; and

storage media configured to store the pixels and tag data associated with the pixels, the image processing system being configured to dynamically update the tag data between execution of the plurality of image processing operations, and where the image processing logic is configured such that each image processing operation is dynamically controlled based on the tag data.

61. The system of claim 60, further comprising a controller operatively coupled with the image processing logic and configured to dynamically and unequally allocate processing times for the image processing logic.

62. The system of claim 60, further comprising a controller operatively coupled with the image processing logic and configured to dynamically vary an order in which the image processing operations are performed on the pixels of video data.

63. The system of claim 62, where the order is varied by bypassing one or more of the image processing operations.

64. The system of claim 63, where the controller effects bypassing of one or more image processing operations based on tag data associated with the pixels of video data.

65. The system of claim 64, where the tag data includes classification information pertaining to characteristics of the pixels of video data.

66. The system of claim 64, where the tag data includes processing information pertaining to processing operations applied to the pixels of video data.

67. The system of claim 64, where the tag data includes source identification information associated with the pixels of video data.

68. The system of claim 60, where the plurality of image processing operations define a sequential processing pipeline capable of processing video data in a FIFO manner, such that during operation, multiple frames of video data are being processed by the image processing system.

69. The system of claim 68, further comprising a controller configured to dynamically and unequally allocate processing time for the image processing logic.

70. The system of claim 69, where the controller is configured to allocate processing time from one of the multiple frames of video data to another of the multiple frames of video data.

71. The system of claim 69, where the controller is configured to allocate processing time from one portion of one of the multiple video frames to another portion of said video frame.

72. The system of claim 69, where the controller is configured to allocate processing time from one of the multiple image processing operations to another of the multiple image processing operations.

73. The system of claim 69, where processing time allocations are effected based on tag data associated with pixels being processed by the image processing logic.

74. The system of claim 69, where the tag data includes classification information pertaining to characteristics of the pixels.

75. An image processing system, comprising:

a pixel processing subsystem configured to receive input pixels and process the input pixels so as to produce output pixels; and

controller and classifier logic configured to control operation of the pixel processing subsystem, where the controller and classifier logic is configured to dynamically control the pixel processing subsystem so that pixel processing operations are variably controllable on a per-pixel basis.

76. An image processing system, comprising:

a pixel processing subsystem configured to receive input pixels and process the input pixels so as to produce output pixels; and

controller and classifier logic configured to control operation of the pixel processing subsystem, where the controller and classifier logic is configured to dynamically control the pixel processing subsystem so that pixel processing operations are variably controllable on a per-region basis within a video frame, such that a given pixel processing operation is applied differently to different portions of the video frame.

77. An image processing system, comprising:

a pixel processing subsystem configured to receive input pixels and process the input pixels so as to produce output pixels,

where the pixel processing subsystem includes multiple pixel processing operators configured to process the input pixels in a sequential order in a first mode of operation, the image processing system further comprising a controller configured to effect a second mode of operation in which the sequential order is varied.