General image processor

• US 6,289,138 B1
• Filed: 02/18/1998
• Issued: 09/11/2001
• Est. Priority Date: 04/30/1997
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Pulver, Mark, Gibson, Ian, Yip, Dominic
• Primary Examiner(s): Couso, Jose L.
• Assistant Examiner(s): Do, Anh Hong
• Application Number:
  US09/025,725
• Time in Patent Office: 1,301 Days
• Field of Search: 382/307, 382/308, 382/233, 712/248, 712/36, 708/190
• US Class Current: 382/307
• CPC Class Codes: G06F 9/3879 for non-native instruction execution, e.g. executing a command; for Java instruction set
• CPC Class Codes: G06F 9/3885 using a plurality of independent parallel functional units
• CPC Class Codes: G06F 9/3897 with adaptable data path
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

The present invention relates to an image processor (242) comprising a control register block (1470), a decoding block (1468), a data object processor (1462), and flow control logic. The control register block (1470) stores all the relevant information about the image processing operation. The decoding block (1468) decodes the information into configuration signals, which configure an input data object interface (1460). The input data object interface (1460) accepts and stores data objects from outside, and distributes these data objects to the data object processor (1462). For some image processing operations, the input data object interface (1460) may also generate addresses for data objects, so that the source of these data objects can provide the correct data objects. The data object processor (1462) performs arithmetic operations on the data objects received. The flow control logic controls the flow of data objects within the data object processing logic (1462). More particularly, the data object processor (1462) can comprise a number of identical data object sub-processors, each of which processes part of an incoming data object.

Claims

1. An apparatus for performing image processing operations on data objects, said apparatus including:

data source means for providing a stream of said data objects;

a plurality of operand source means for providing streams of operand objects, or providing operand objects in response to an address presented;

instruction means for selecting an image processing operation, and enabling or disabling a plurality of options in said image processing operation;

a configuration register for storing said image processing operation and options;

a register file for storing information necessary for performing said image processing operation;

decoding means connected to said configuration register for decoding said image processing operation and options;

a control signal register connected to said decoding means for storing the output of said decoding means;

input interface means, connected to said control signal register, said register file, said data source means and said plurality of said operand source means, for;

(a) accepting, storing and rearranging said data objects from said data source means, and said operand objects from said operand source means, in accordance with the output of said control signal register,

(b) generating addresses for said operand objects, in accordance with the output of said control signals register and said register file, and

(c) updating said information in said register file to reflect a current status of the image processing operation;

processing means, connected to said input interface means, said register file, and said control signal register, for performing arithmetic operations on the output of said input interface means in accordance with the output of said control signals register and said register file to produce processed data objects; and

data destination means connected to said processing means for receiving said processed data objects.

2. The apparatus according to claim 1, wherein said processing means further includes:

a plurality of identical channel processing means for performing said arithmetic operations on part of the output of said input interface means in accordance with the output of said control signal register and said register file; and

flow control means connected to said plurality of said channel processing means for controlling a flow of said data objects in said channel processing means by outputting enable signals in accordance with the output of said control signal register.

3. The apparatus according to claim 2, wherein said channel processing means further includes:

a plurality of identical arithmetic units connected to said flow control means for performing said arithmetic operations on said data objects in accordance with the output of said control signal register;

combining means, connected to said flow control means and said arithmetic units, for adding the outputs of a plurality of said arithmetic units and said register file in accordance with the output of said control signal register and said register file;

first post-processing means, connected to said combining means and said flow control means, for rounding an output of said combining means, finding the absolute value of said rounded output, and clamping of said absolute value in accordance with the output of said control signal register;

second post-processing means, connected to a selected plurality of arithmetic units and said first post-processing means, for selecting between the outputs of said arithmetic units and said first post-processing means and clamping the selected output, if necessary, in accordance with the output of said control signal register; and

routing logic, connected to a plurality of said arithmetic units, said combining means and said first post-processing means, for routing selected outputs of a plurality of said arithmetic units and said first post-processing means to selected inputs of a plurality of said arithmetic units and said combining means.

4. The apparatus according to any one of claims 1 to 3, further comprising a read-only memory (ROM) containing dividends of 255/x, where x is an integer ranging from 0 to 255.

5. The apparatus according to claim 2, wherein the number of said channel processing means in the processing means is four.

6. An apparatus for performing compositing between two streams of pixels and a stream of attenuation values, said apparatus including:

data source means for providing a first one of said streams of pixels;

first operand source means for providing a second one of said streams of pixels;

second operand source means for providing said stream of attenuation values;

instruction means for enabling or disabling a plurality of options in compositing operations;

a configuration register for storing a compositing operation and said options of said instruction means;

register file for storing information necessary for performing compositing operations;

decoding means connected to said configuration register for decoding said options and said compositing operation;

a control signal register connected to said decoding means for storing the output of said decoding means;

input interface means, connected to said control signal register, said register file, said data source means, said first operand source means and said second operand source means, for;

(a) accepting, storing and rearranging pixels from said data source means and said first operand source means, and attenuation values from said second operand source means in accordance with an output of said control signal register,

(b) generating a stream of data objects to replace the second stream of pixels from said first operand source in accordance with the output of said control signal register and said register file, and

(c) updating said information in said register file to reflect a current status of the compositing operation;

processing means, connected to said input interface means, said register file, and said control signal register, for performing arithmetic operations on an output of said input interface means in accordance with the output of said control signal register and said register file to produce composited pixels; and

data destination means connected to said processing means for receiving said composited pixels.

7. The apparatus according to claim 6, wherein the pixels from said data source means and said first operand source means are made up of four channels, where three of the four channels represent a color of each pixel and the remaining channel represents an opacity of each pixel.

8. The apparatus according to claim 7, wherein said options in the compositing operations include:

multiplying each of said color channels in a pixel from said data source with an opacity of that pixel;

multiplying each of the color channels in the pixel from said first operand source with an opacity of the pixel;

specifying whether the stream of pixels from said first operand source is replaced by streams of data objects generated by said input interface means;

specifying which compositing operator is used;

specifying whether the output of said processing means is clamped or wrapped; and

dividing each of the color channels in the composited pixel in said processing means with opacity of the composited pixel if such an option is enabled.

9. The apparatus according to claim 8, wherein the information in said register file includes:

offsets for a plus operator on the four channels;

start values of a blend on the four channels;

end values of the blend on the four channels; and

a length of said stream of pixels from said data source means.

10. The apparatus according to claim 9, wherein said stream of data objects generated by said input interface means is a sequence of integers from 0 to 255 for the length of the stream of pixels from said data source means.

11. The apparatus according to claim 10, wherein the arithmetic operations performed by said processing means include:

multiplying each of the color channels in the pixel from said data source with an opacity of the pixel to produce a first pre-multiplied pixel, if such option is enabled;

interpolating between a start value of the blend and an end value of the blend with said stream of data objects generated by said input interface means, if such option is enabled;

multiplying each of the color channels in the pixel from said first operand source or the blend from said interpolating with an opacity of the pixel or the blend to produce a second pre-multiplied pixel, if such option is enabled;

multiplying said attenuation value with said second pre-multiplied pixel to produce an attenuated pixel;

performing one of the following compositing operators on said first pre-multiplied pixel and said attenuated pixel to produce the composited pixel;

12. An apparatus for performing general color space conversion on streams of interval values, fraction values, and colour table values, said apparatus including:

data source means for providing said interval and fraction values;

operand source means for providing said color table values in response to said interval values presented;

instruction means for enabling or disabling a plurality of options for general color space conversion;

a configuration register for storing instruction means;

decoding means connected to said configuration register for decoding said instruction means;

a control signal register connected to said decoding means for storing the output of said decoding means;

input interface means, connected to said control signal register, said data source means, and said operand source means, for;

(a) accepting, storing, rearranging and outputting said interval values and said fraction values from said data source means in accordance with the output of said control signal register, and

(b) fetching said color table values from said operand source means using said interval value as an address, and storing, rearranging and outputting said color table values in accordance with the output of said control signal register;

processing means, connected to said input interface means and said control signal register, for performing arithmetic operations on the output of said input interface means in accordance with the output of said control signal register to produce a result color; and

data destination means connected to said processing means for receiving the result color.

13. The apparatus according to claim 12, wherein the options for a general color space conversion operation include specifying whether to apply a translation and clamping operation to said result color.

14. The apparatus according to claim 13, wherein said translation and clamping operation is calculated using the formula:

$\mathrm{out}=\begin{array}{cc}0,& \mathrm{if}<br><br>\text{\hspace{1em}<br><br>}<br><br>x\le <br><br>63,\\ 2<br><br>\left(x-64\right),& \mathrm{if}<br><br>\text{\hspace{1em}<br><br>}<br><br>64\le <br><br>x\le <br><br>191,<br><br>\text{\hspace{1em}<br><br>}\\ 255,& \mathrm{if}<br><br>\text{\hspace{1em}<br><br>}<br><br>192\le <br><br>x.\end{array}$

15. The apparatus according to claim 14, wherein said arithmetic operation performed in said processing means is a tri-linear interpolation of said color table values with said fraction values.

16. An apparatus for applying an affine image transformation on a source image, said apparatus including:

data source means for providing a kernel descriptor and weights of a weighting function;

first operand source means for providing index table values from a index table of said source image in response to an index table address presented;

second operand source means for providing at least one pixel of said source image in response to a pixel address presented;

an instruction means for enabling or disabling a plurality of options of said affine image transformation;

a configuration register for storing instruction means;

decoding means connected to said configuration register for decoding said instruction means;

a control signal register connected to said decoding means for storing an output of said decoding means;

a register file for storing information necessary for performing said affine image transformation;

input interface means connected to said control signal register, said register file, said data source means, and said operand source means for;

(a) accepting and storing said kernel descriptor and said weights of said weighting function from said data source means in accordance with the output of said control signal register,

(b) generating coordinates of pixels to be fetched from said source image in accordance to said kernel descriptor and outputs of said register file and said control signal register,

(c) calculating index table addresses from coordinates and the output of said register file,

(d) fetching index table entries from said first operand source means,

(e) calculating a pixel address from said index table entry and said coordinates,

(f) fetching at least one of pixels from said second operand means, and storing and rearranging said pixels in accordance with the output of said control signals register;

processing means, connected to said input interface means, said register file and said control signal register, for performing a plurality of arithmetic operations on the output of said input interface means in accordance with the output of said control signal register to produce a result pixel; and

data destination means connected to said processing means for receiving the result pixel.

17. The apparatus according to claim 16, wherein the kernel descriptor has two forms including a long kernel descriptor and a short kernel descriptor.

18. The apparatus according to claim 17, wherein:

the long kernel descriptor includes;

source image start co-ordinates,

a source image horizontal delta,

a source image vertical delta, and

binary points to truncate; and

the short kernel descriptor includes;

an integer part of source image start x-coordinate, and

binary point to truncate;

with the fraction part of said source image start x-coordinate assumed to be zero, said source image horizontal delta assumed to be 1 in the direction of the x-axis, and said source image vertical delta assumed to be 1 in the direction of the y-axis.

19. The apparatus according to claim 18, wherein:

said source image start co-ordinates are unsigned fixed point numbers with 24.24 resolution; and

said source image horizontal delta and said source image vertical delta are 2'"'"'s complement fixed point numbers with 24.24 resolution.

20. The apparatus according to claim 16, wherein the pixels in said source image include four channels, where three of the four channels represent a color of the pixel and the remaining channel represents an opacity of the pixel.

21. The apparatus according to claim 20, wherein the options of the affine image transformation include:

bi-linearly interpolating four surrounding source image pixels to determine an actually sampled value, or said sampled value is snapped to a closest source image pixel value;

specifying whether an offset is applied on any one of the said four channels;

specifying whether to multiply each of the color channels in said pixel from said source image with an opacity of said pixel from said source image;

specifying whether to clamp output values; and

specifying whether to take an absolute value of output values before wrapping or clamping.

22. The apparatus according to claim 21, wherein said plurality of arithmetic operations include:

multiplying each of the color channels in said plurality of pixels from said source image with an opacity of the pixel to produce a first pre-multiplied pixel, if such option is enabled;

bi-linearly interpolating the four surrounding source image pixels to determine the actually sampled value, if such option is enabled, otherwise the pixel fetched from said source image is taken as the actually sampled value;

applying a weighting function on a plurality of said actually sampled values to determine the internal result pixel;

rounding off the fraction part of an internal result pixel in accordance to said binary points to truncate in said kernel descriptor; and

taking the absolute value of said internal result pixel and clamping it, if such options are enabled.

23. The apparatus according to claim 22, wherein said weighting function is applied by adding together a two-dimensional array of sub-sample pixels in said source image and said offset, with each pixel given a different weight.

24. The apparatus according to claim 23, wherein said weights in the weighting functions are signed numbers.

25. The apparatus according to claim 23, wherein said information necessary for performing said affine image transformation includes:

a number of rows and columns in said two-dimensional array of sub-samples;

a base address of said index table of said source image;

an offset to be applied in said weighting function; and

a number of result pixels to produce.

26. An apparatus for applying a convolution using a convolution matrix to a source image, said apparatus including:

data source means for providing a kernel descriptor and coefficients of said convolution matrix;

first operand source means for providing index table values from an index table of said source image in response to an index table address presented;

second operand source means for providing pixels of said source image in response to a pixel address presented;

an instruction means for enabling or disabling a plurality of options in convolution;

a configuration register for storing said instruction means;

decoding means connected to said configuration register for decoding said instruction means;

a control signal register connected to said decoding means for storing the output of said decoding means;

a register file for storing information necessary for performing an affine image transformation;

an input interface means, connected to said control signal register, said register file, said data source means, and said operand source means, for;

(a) accepting and storing said kernel descriptor and said weights of a weighting function from said data source means in accordance with the output of said control signal register,

(b) generating coordinates of pixels to be fetched from said image in accordance to said kernel descriptor and outputs of said register file and said control signal register,

(c) calculating index table addresses from said coordinates and the output of said register file,

(d) fetching index table entry from said first operand source means,

(e) calculating pixel address from said index table entry and said coordinates,

(f) fetching a plurality of pixels from said second operand means, and storing and rearranging said pixels in accordance with the output of said control signals register;

processing means, connected to said input interface means, said register file and said control signal register, for performing arithmetic operation on the output of said input interface means in accordance with the output of said control signal register to produce a result pixel; and

data destination means connected to said processing means for receiving the result pixel.

27. The apparatus according to claim 26, wherein the kernel descriptor has two forms including a long kernel descriptor and a short kernel descriptor.

28. The apparatus according to claim 27, wherein:

the long kernel descriptor includes;

source image start co-ordinates,

source image horizontal delta,

source image vertical delta, and

binary points to truncate;

and the short kernel descriptor includes;

integer part of source image start x-coordinate, and

binary point to truncate;

with the fraction part of the source image start x-coordinate assumed to be zero, the source image horizontal delta assumed to be 1 in the direction of the x-axis, and the source image vertical delta assumed to be 1 in the direction of the y-axis.

29. The apparatus according to claim 28, wherein:

said source image start co-ordinates are unsigned fixed point numbers with 24.24 resolution, and

said source image horizontal delta and said source image vertical delta are 2'"'"'s complement fixed point numbers with 24.24 resolution.

30. The apparatus according to claim 26, wherein the pixels in said source image include four channels, three of the four channels representing a color of the pixel and the remaining channel representing an opacity of the pixel.

31. The apparatus according to claim 30, wherein the options for said convolution include:

bi-linearly interpolating four surrounding source image pixels to determine an actually sampled value, or the sampled value is snapped to a closest source image pixel value;

specifying whether to apply the offset on any one of the said four channels;

specifying whether to multiply each of the color channels in said pixel from said image with an opacity of said pixel from said image;

specifying whether to clamp output values; and

specifying whether to take absolute value of output values before wrapping or clamping.

32. The apparatus according to claim 31, wherein said plurality of arithmetic operations include:

multiplying each of the color channels in said plurality of pixels from said source image with an opacity of the pixel to produce a first pre-multiplied pixel, if such option is enabled;

bi-linearly interpolating the four surrounding source image pixels to determine the actually sampled value, if such option is enabled, otherwise the pixel fetched from said source image is taken as the actually sampled value;

applying a weighting function on a plurality of said actually sampled values to determine an internal result pixel;

rounding off the fraction part of said internal result pixel in accordance to said binary points to truncate in said kernel descriptor; and

taking an absolute value of said internal result pixel and clamping it, if such options are enabled.

33. The apparatus according to claim 32, wherein said weighting function is applied by adding together a two-dimensional array of sub-sample pixels in said source image and said offset, with each pixel given a different weight.

34. The apparatus according to claim 31, wherein said weights in the weighting functions are signed numbers.

35. The apparatus according to claim 32, wherein said information necessary for performing said affine image transformation includes:

a number of rows and columns in said two-dimensional array of sub-samples;

a base address of said index table of said source image;

an offset to be applied in said weighting function; and

a number of result pixels to produce.

36. An apparatus for performing a linear colour space conversion on a stream of source data objects with a conversion matrix, said apparatus including:

data source means for providing said source data objects;

operand source means for providing a plurality of coefficients of said conversion matrix in response to a line number presented;

instruction means for enabling or disabling a plurality of options in linear color space conversion;

a configuration register for storing said instruction means;

decoding means connected to said configuration register for decoding said instruction means;

a control signal register connected to said decoding means for storing the output of said decoding means;

input interface means, connected to said control signal register, said data source means, and said operand source means, for;

(a) accepting, storing and outputting said source data objects from said data source means in accordance with the output of said control signals register,

(b) generating said line number to fetch a plurality of said coefficients from operand source means in accordance to said control signal register, and

(c) storing, rearranging and outputting said coefficients in accordance with the output of said control signals register;

processing means, connected to said input interface means and said control signal register, for performing a plurality of arithmetic operations on the output of said input interface means in accordance with the output of said control signal register to produce a result data object; and

data destination means connected to said processing means for receiving the result data object.

37. The apparatus according to claim 36, wherein:

said conversion matrix has 4 rows and 5 columns,

said data objects and said result data objects are pixels with 4 channels, three of the four channels representing the color of the pixel and the remaining channel representing the opacity of the pixel, and

said linear color space conversion is defined by;

$\left[\begin{array}{c}{r}_x\\ r_y\\ r_z\\ r_o\end{array}\right]=\left[\begin{array}{ccccc}{b}_{o,o}& b_{o,1}& b_{o,2}& b_{o,3}& b_{o,4}\\ b_{1,o}& b_{1,1}& b_{1,2}& b_{1,3}& b_{1,4}\\ b_{2,o}& b_{2,1}& b_{2,2}& b_{2,3}& b_{2,4}\\ b_{3,o}& b_{3,1}& b_{3,2}& b_{3,3}& b_{3,4}\end{array}\right]<br><br>\left[\begin{array}{c}{a}_x\\ a_y\\ a_z\\ a_o\\ 255\end{array}\right],$

where r_i is the result pixel and a_i is the A operand pixel.

38. The apparatus according to claim 37, wherein the coefficients of said conversion matrix are signed fixed point number with 12.12 resolution.

39. The apparatus according to claim 38 wherein said plurality of options in said linear color space conversion include:

whether to multiply each of the color channels in said pixel from said data source means with an opacity of said pixel;

whether to clamp said result data objects to a predetermined minimum value overflow when they underflow, and to a predetermined maximum value when they oveerflow; and

whether to take absolute value of said result data object before wrapping or clamping.

40. The apparatus according to claim 39, wherein said arithmetic operations performed in said processing means include:

multiplying each of the color channels in the pixel from said data source with an opacity of the pixel to produce first pre-multiplied pixel, if such option is enabled;

multiplying two of the channels with two of the columns of said conversion matrix to produce first and second internal products;

adding said first and second internal products and the last column of said conversion matrix to form first internal sum;

multiplying the remaining two channels with the remaining two columns of said conversion matrix to produce third and fourth internal products;

adding said third and fourth internal products to said first internal sum to produce internal result;

rounding off the fraction part of said internal result; and

taking absolute value and clamping the rounded internal result if such options are enabled.