Apparatus and method for digital camera and recorder having a high resolution color composite image output

• US 5,852,502 A
• Filed: 05/31/1996
• Issued: 12/22/1998
• Est. Priority Date: 05/31/1996
• Status: Expired due to Fees

• Current Assignee: American Digital Switching
• Original Assignee: American Digital Switching
• Inventors: Beckett, John Patrick
• Primary Examiner(s): VU, KIM Y
• Application Number:
  US08/657,607
• Time in Patent Office: 935 Days
• Field of Search: 358/511, 358/512, 358/514, 358/515-516, 358/518-520, 358/530, 358/537, 358/538, 348/540, 348/33, 348/222, 348/317, 348/333, 348/336-337, 348/342, 382/167, 382/284
• US Class Current: 358/512
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 1/46 Colour picture communication systems
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light

Abstract

The present invention provides an apparatus and method for producing series of high resolution color composite images. The digital camera has an optical assembly that directs visual images to a high resolution monochrome sensor and a lower resolution color sensor. These two sensors, which produce a succession of frames at the same rate, are encrypted with a frame number and time code and are stored in a frame buffer. The contents of the frame buffer can be transferred to a mass storage device, or a color rendering processor that produces a composite image from the monochrome frames and color frames. During the processing of the composite image, the monochrome grayscale value becomes the composite frame grayscale value, the color frame hue value becomes the composite frame hue value, and the color frame saturation value becomes the composite frame saturation value. Alternatively, the monochrome frame grayscale value can be used to affect the composite frame hue and saturation values, or the composite frame hue and saturation values can be mapped from the color frame hue and saturation pixel linearly or non-linearly.

Claims

1. An apparatus for producing a color image from a visual input comprising:

an optical assembly for receiving said visual input;

a first sensor operably connected to said optical assembly, said first sensor producing a monochromatic image from said visual input;

a second sensor operably connected to said optical assembly, said second sensor producing a color image from said visual input;

a processor for producing a composite image from said monochromatic image and said color image;

wherein said first sensor is high resolution respective to said second sensor;

wherein said first sensor produces a first plurality of pixels representative of said visual image;

wherein said second sensor produces a second plurality of pixels representative of said visual image;

wherein each of said second plurality of pixels corresponds to at least one of said first plurality of pixels;

wherein each of said first plurality of pixels has a grayscale value;

wherein each of said second plurality of pixels has a hue value and a saturation value;

wherein said composite image comprises a third plurality of pixels, said third plurality of pixels respectively corresponding to said first plurality of pixels;

wherein each of said third plurality of pixels has a respective composite grayscale value, said respective composite grayscale value being equal to said respective grayscale value of said first plurality of pixels;

wherein each of said third plurality of pixels has a respective composite hue value, said respective composite hue value being said hue value of said second plurality of pixels; and

wherein each of said third plurality of pixels has a respective composite saturation value, said respective composite saturation hue value being said saturation value of said second plurality of pixels.

2. A method for reproducing visual images comprising the steps of:

directing a plurality of visual images to a first image sensor and a second image sensor;

producing a plurality of monochrome image frames respectively from said plurality of visual images, each of said plurality of monochromatic image frames having a first plurality of pixels;

producing a plurality of color image frames respectively from said plurality of visual images, each of said plurality of color image frames having a second plurality of pixels;

respectively combining said plurality of monochrome image frames with said plurality of color image frames to produce a plurality of composite image frames, each of said plurality of composite image frames having a plurality of composite pixels;

wherein the step of producing a plurality of monochrome image frames is producing a plurality of high resolution monochrome image frames, said first plurality of pixels is a plurality of high resolution pixels, said second plurality of pixels is a plurality of low resolution pixels, each of said plurality of low resolution pixels corresponding to at least one of said plurality of high resolution pixels, said plurality of composite pixels respectively corresponding to said high resolution pixels;

producing a hue variable for each low resolution pixel;

producing a saturation variable for each low resolution pixel;

producing a high resolution grayscale value variable for each high resolution pixel; and

producing a plurality of arrays for each of said plurality of high resolution pixels, each of said arrays containing a plurality of high resolution grayscale value variables adjacent to each of said high resolution pixels.

3. A method as in claim 2 wherein said step of combining further comprises the step of:

adjusting said hue variable of any one of said plurality of composite image pixels respectively based on one of said plurality of arrays and respectively based on one corresponding said hue variable.

4. A method as in claim 2 wherein said step of combining further comprises the step of:

adjusting said saturation variable of one of said plurality of composite image pixels respectively based on one of said plurality of arrays and respectively based on one corresponding said saturation variable.

5. A method for producing a color motion picture comprising the steps of:

directing a plurality of visual images to a first image sensor and a second image sensor;

producing a plurality of monochrome image frames respectively from said plurality of visual images;

producing a plurality of color image frames respectively from said plurality of visual images;

combining said plurality of monochrome image frames with said plurality of color images to produce a plurality of composite image frames, each of said plurality of composite image frames having a plurality of composite pixels;

recording said plurality of composite image frames;

viewing said plurality of color image frames;

selectively editing said plurality of color image frames whereby a portion of said plurality of color image frames are discarded or re-arranged to result in a resultant plurality of color image frames;

matching said resultant plurality of color image frames with respective said plurality of monochrome image frames to produce a resultant plurality of monochrome image frames; and

discarding any one of said plurality of monochrome image frames not corresponding to any one of said resultant plurality of monochrome image frames.

6. A method as in claim 5 wherein said step of producing said resultant plurality of monochrome image frames is producing a resultant plurality of high resolution monochrome image frames relative to said resultant plurality of color image frames, each of said plurality of high resolution monochromatic image frames having a first plurality of pixels, each of said resultant plurality of color image frames having a second plurality of pixels, each of said second plurality of pixels corresponding to at least one of said first plurality of pixels.

7. A method as in claim 6 wherein said step of combining said resultant plurality of monochrome images with said resultant plurality of color images further includes the steps:

generating a first grayscale value for each of said first plurality of pixels in each of said resultant plurality of high resolution monochrome image frames;

generating a second grayscale value for each of said second plurality of pixels in each of said plurality of color frames, said second grayscale value having less resolution than said first grayscale value;

combining respectively said first grayscale value and said second grayscale value to produce a composite grayscale value for each of said plurality of composite pixels in each of said plurality of composite image frames;

generating a hue value for each of said second plurality of pixels in each of said resultant plurality of color frames;

generating a saturation value for each of said second plurality of pixels in each of said resultant plurality of color frames;

producing a composite hue value for each of said plurality of composite pixels in each of said plurality of composite image frames from respective said hue value and said composite grayscale value; and

producing a composite saturation value for each of said plurality of composite pixels in each of said plurality of composite image frames from respective said saturation value and said composite grayscale value.

8. A method as in claim 7 wherein combining said first grayscale value and said second grayscale value is linear.

9. A method as in claim 7 wherein combining said first grayscale value and said second grayscale value is non-linear.

10. A method as in claim 6 wherein said step of said combining said resultant plurality of monochrome image frames with said resultant plurality of color images further includes the steps:

generating a grayscale value for each of said first plurality of pixels in each of said resultant plurality of high resolution monochrome image frames;

generating a hue value for each of said second plurality of pixels in each of said resultant plurality of color frames;

generating a saturation value for each of said second plurality of pixels in each of said resultant plurality of color frames;

generating an array for each of said first plurality of pixels in each of said resultant plurality of high resolution monochrome image frames, said array containing grayscale values of said first plurality of pixels adjacent to one of said first plurality of pixels;

generating a composite hue value for each of said plurality of composite pixels in each of said plurality of composite image frames from said hue value and said array;

generating a composite saturation value for each of said plurality of composite pixels in each of said plurality of composite image frames from said saturation value and said array; and

generating a composite grayscale value for each of said plurality of composite pixels in each of said plurality of composite image frames from said array.

11. An apparatus for producing a series of color images from a visual input comprising:

an optical assembly for receiving said visual input;

a first sensor operably connected to said optical assembly, said first sensor producing a plurality of monochrome image frames from said visual input;

a second sensor operably connected to said optical assembly, said second sensor producing a plurality of color image frames from said visual input, each of said plurality of color image frames respectively corresponding to one of said plurality of monochromatic image frames;

a processor for producing a plurality of composite image frames from said plurality of monochromatic image frames and said plurality of color image frames;

wherein said first sensor is high resolution respective to said second sensor;

wherein each frame of said plurality of monochromatic image frames has a first plurality of pixels;

wherein each frame of said plurality of color image frames has a second plurality of pixels;

wherein each of said second plurality of pixels corresponds to at least one of said first plurality of pixels;

a frame storage assembly operably connected to said first sensor by a monochrome data line, said monochrome data line transferring said plurality of monochrome image frames;

wherein said frame storage assembly being operationally connected to said second sensor by a color data line, said color data line transferring said plurality of color image frames;

a color image display for displaying said plurality of color image frames stored in said frame storage assembly;

means to selectively view said plurality of color image frames;

means to selectively discard at least one of said plurality of color image frames; and

means to automatically discard at least one of said plurality of monochrome image frames corresponding to said at least one of said plurality of color image frames.

12. An apparatus as in claim 11:

wherein said monochrome data line is a plurality of monochrome data lines;

wherein said first sensor further comprises a plurality of subdivisions, each of said plurality of subdivisions respectively corresponding to said plurality of monochrome data lines; and

wherein each of said plurality of monochrome frame images being comprised of said plurality of sub-frame images respectively corresponding to said plurality of subdivisions.

13. An apparatus as in claim 12:

wherein said plurality of monochrome data lines is a first plurality of monochrome data lines;

wherein said color data line is a first color data line;

said apparatus further comprising;

a color rendering processor operably connected to said frame storage assembly by a second plurality of monochromatic data lines, said color rendering processor operably connected to said frame storage assembly by a second color data line, said color rendering processor sequentially processing said plurality of sub-frame images with corresponding said plurality of color image frames to produce a plurality of composite image frames; and

a recorder to record said plurality of composite image frames for later real time playback.

14. An apparatus as in claim 12:

wherein said plurality of monochrome data lines is a first plurality of monochromatic data lines;

wherein said color data line is a first color data line;

said apparatus further comprising;

a plurality of color rendering processors, said plurality of color rendering processors respectively operably connected to said frame storage assembly by a second plurality of monochromatic data lines;

wherein each of said plurality of color rendering processors being operably connected to said frame storage assembly by a second color data line;

wherein said plurality of color rendering processors concurrently processes respective said plurality of subframes of each said plurality of monochrome image frames with said plurality of color image frames to produce a plurality of composite image frames in real time.

15. An apparatus as in claim 14 further comprising a recorder to record said plurality of composite image frames.

16. An apparatus as in claim 15 wherein said recorder is a digital mass storage device.

17. An apparatus as in claim 15 further comprising a display device for viewing said plurality of composite image frames.

18. An apparatus as in claim 17 wherein said display device transmits said plurality of composite image frames to a motion picture film recorder.

19. An apparatus as in claim 11 wherein said first sensor is a first charge coupled device and said second sensor is a second charge coupled device.

20. An apparatus as in claim 11 further comprising:

a lens assembly for receiving said visual input; and

a beam splitter for simultaneously directing said visual input to said first sensor and to said second sensor.

21. A method for reproducing visual images comprising the steps of:

directing a plurality of visual images to a first image sensor and a second image sensor;

producing a plurality of monochrome image frames respectively from said plurality of visual images, each of said plurality of monochromatic image frames having a first plurality of pixels;

producing a plurality of color image frames respectively from said plurality of visual images, each of said plurality of color image frames having a second plurality of pixels;

respectively combining said plurality of monochrome image frames with said plurality of color image frames to produce a plurality of composite image frames, each of said plurality of composite image frames having a plurality of composite pixels;

wherein the step of producing a plurality of monochrome image frames is producing a plurality of high resolution monochrome image frames, said first plurality of pixels is a plurality of high resolution pixels, said second plurality of pixels is a plurality of low resolution pixels, each of said plurality of low resolution pixels corresponding to at least one of said plurality of high resolution pixels, said plurality of composite pixels respectively corresponding to said high resolution pixels;

producing a hue variable for each low resolution pixel;

producing a saturation variable for each low resolution pixel;

producing a high resolution grayscale value variable for each high resolution pixel;

applying said hue variable to a portion of said plurality of composite image pixels;

applying said saturation variable to said portion of said plurality of composite image pixels; and

applying said high resolution grayscale value variables respectively to one of said plurality of composite image pixels.

22. A method as in claim 21 further comprising the step:

repeating said step of combining for each of said plurality of high resolution pixels in each said plurality of high resolution monochrome image frames.

23. A method for reproducing visual images comprising the steps of:

directing a plurality of visual images to a first image sensor and a second image sensor;

producing a plurality of monochrome image frames respectively from said plurality of visual images, each of said plurality of monochromatic image frames having a first plurality of pixels;

producing a plurality of color image frames respectively from said plurality of visual images, each of said plurality of color image frames having a second plurality of pixels;

respectively combining said plurality of monochrome image frames with said plurality of color image frames to produce a plurality of composite image frames, each of said plurality of composite image frames having a plurality of composite pixels;

wherein the step of producing a plurality of monochrome image frames is producing a plurality of high resolution monochrome image frames, said first plurality of pixels is a plurality of high resolution pixels, said second plurality of pixels is a plurality of low resolution pixels, each of said plurality of low resolution pixels corresponding to at least one of said plurality of high resolution pixels, said plurality of composite pixels respectively corresponding to said high resolution pixels;

producing a plurality of color determination variables for each low resolution pixel;

utilizing said plurality of color determination variables to control said step of combining said plurality of high resolution monochrome image frames with said plurality of color image frames;

producing a low resolution grayscale value variable for each pixel in each of said plurality of color image frames;

applying said hue variable to said portion of said plurality of composite image pixels;

applying said saturation variable to said portion of said plurality of composite image pixels; and

producing a composite grayscale value variable from said low resolution grayscale value variable and from at least one of said high resolution value variable.

24. A method for reproducing visual images comprising the steps of:

directing a plurality of visual images to a first image sensor and a second image sensor;

producing a plurality of monochrome image frames respectively from said plurality of visual images, each of said plurality of monochromatic image frames having a first plurality of pixels;

producing a plurality of color image frames respectively from said plurality of visual images, each of said plurality of color image frames having a second plurality of pixels;

respectively combining said plurality of monochrome image frames with said plurality of color image frames to produce a plurality of composite image frames, each of said plurality of composite image frames having a plurality of composite pixels;

viewing said plurality of color image frames;

selectively editing said plurality of color image frames whereby a portion of said plurality of color image frames are discarded to result in a resultant plurality of color image frames;

matching said resultant plurality of color image frames with respective said plurality of monochrome image frames to produce a resultant plurality of monochrome image frames; and

discarding any one of said plurality of monochrome image frames not corresponding to any one of said resultant plurality of monochrome image frames.

25. An apparatus for producing a color image from a visual input comprising:

an optical assembly for receiving said visual input;

a first sensor operably connected to said optical assembly, said first sensor producing a monochromatic image from said visual input;

a second sensor operably connected to said optical assembly, said second sensor producing a color image from said visual input;

a processor for producing a composite image from said monochromatic image and said color image;

wherein said first sensor is high resolution respective to said second sensor;

wherein said first sensor produces a first plurality of pixels representative of said visual image;

wherein said second sensor produces a second plurality of pixels representative of said visual image;

wherein each of said second plurality of pixels corresponds to at least one of said first plurality of pixels;

wherein each of said first plurality of pixels has a grayscale value;

wherein each of said second plurality of pixels has a hue value and a saturation value;

wherein said composite image comprises a third plurality of pixels, said third plurality of pixels respectively corresponding to said first plurality of pixels;

wherein each of said third plurality of pixels has a composite grayscale value;

wherein each of said third plurality of pixels has a composite hue value, said composite hue value being calculated from said hue value of one of said second plurality of pixels and from grayscale values of a group of pixels of said first plurality of pixels, said group of pixels being adjacent to each respective corresponding said first plurality of pixels; and

wherein each of said third plurality of pixels has a composite saturation value, said composite saturation value being calculated from said saturation value of one of said second plurality of pixels and from grayscale values of a group of pixels of said first plurality of pixels, said group of pixels being adjacent to each respective corresponding said first plurality of pixels.