Method and system for fusing images

• US 9,936,109 B2
• Filed: 10/24/2011
• Issued: 04/03/2018
• Est. Priority Date: 10/22/2010
• Status: Active Grant

• Current Assignee: University of New Brunswick
• Original Assignee: University of New Brunswick
• Inventors: Zhang, Yun
• Primary Examiner(s): Vu, Ngoc-Yen
• Application Number:
  US13/880,703
• Publication Number:
  US 20130335599A1
• Time in Patent Office: 2,353 Days
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A video imaging system including: a low resolution color digital video camera and a high resolution monochromatic digital video camera operably connected to a digital processing system.

Claims

1. A video imaging system comprising:

a low resolution colour digital video camera for providing a stream of low resolution multi-spectral video image frames comprising at least three spectral bands;

a high resolution monochromatic digital video camera for providing a stream of high resolution monochromatic video image frames; and

,

a digital processing system comprising an image fusing module for isolating corresponding pairs of video image frames from the two streams of video image frames where the video streams use the same frame rate, and fusing the pairs of video image frames to obtain a stream of colour video image frames having a higher resolution than the stream of video image frames from the low resolution colour digital video camera;

wherein the cameras are operably connected to the digital processing system, and

wherein the image fusing method used by the image fusing module is selected from the group consisting of an arithmetic based fusion method, a statistics based fusion method, and a ratio based fusion method, and further comprises, for a given pair of the isolated corresponding video image frames, fusing multi-spectral pixels in the multi-spectral video image frame of the given pair with the corresponding monochromatic pixels in the monochromatic video image frame of the given pair, where the fusing is performed in real time without user input or post processing.

2. The video imaging system according to claim 1 further comprising:

an object motion module for detecting a movement of an object within a field of view of at least one of the cameras, and

an object position module for determining a position of an object in an overlapping field of view of the cameras.

3. The system according to claim 1 further comprising:

an object motion module for detecting a movement of an object within a field of view of at least one of the cameras.

4. The system according to claim 1 further comprising:

an object position module for determining a position of an object in an overlapping field of view of the cameras.

5. The video imaging system according to claim 1 wherein the video cameras are selected from the group consisting of security, surveillance and monitoring video cameras.

6. The video imaging system according to claim 1 further comprising:

providing a three dimensional coordinate system for determining a position of an object in an overlap area of the fields of view of the cameras, by determining two dimensional coordinates of the object from a position of the object on at least one of the two images, and,

determining a distance from the cameras to the object in an axis perpendicular to the plane of the image frames from a parallax between two image frames of the stream of video image frames.

7. The video imaging system of claim 1, wherein the multi-spectral pixels are red, green and blue (RGB) colour pixels or red, green, blue and near infrared colour pixels.

8. A method comprising:

providing a stream of video image frames from a low resolution multi-spectral digital video camera;

providing a corresponding stream of video image frames from a high resolution monochromatic digital video camera where the video streams use the same frame rate;

isolating corresponding pairs of video image frames from the two streams of video image frames;

fusing the pairs of video image frames to obtain a stream of multi-spectral video image frames having a higher resolution than the stream of video image frames from the low resolution multi-spectral digital video camera;

wherein the image fusing method for fusing the two streams of video images is selected from the group consisting of an arithmetic based fusion method, a statistics based fusion method, and a ratio based fusion method, and further comprises, for a given pair of corresponding video image frames, fusing multi-spectral pixels in the multi-spectral video image frame of the given pair with the corresponding monochromatic pixels in the monochromatic video image frame of the given pair, where the fusing is performed in real time without user input or post processing.

9. The method according to claim 8 wherein the video cameras are selected from the group consisting of security, surveillance and monitoring video cameras.

10. The method according to claim 8 further comprising:

isolating two image frames from at least one of the cameras;

comparing the two image frames; and

,

identifying at least one difference in the two image frames.

11. The method according to claim 10 further comprising:

relating the at least one difference to motion of an object in the overlap area.

12. The method according to claim 8 further comprising:

providing a three dimensional coordinate system for determining a position of an object in an overlap area of the fields of view of the cameras, by determining two dimensional coordinates of the object from a position of the object on at least one of the two images; and

,

determining a distance from the cameras to the object in an axis perpendicular to the plane of the image frames from a parallax between two image frames of the stream of video image frames.

13. The video imaging system of claim 8, wherein the multi-spectral pixels are red, green and blue (RGB) colour pixels or red, green, blue and near infrared colour pixels.

14. An imaging system comprising:

a digital video camera comprising;

a low resolution colour digital sensor;

a high resolution monochromatic digital sensor;

a lens;

a beam splitter for directing incident light passing through the lens to the colour and monochromatic sensors; and

,

a digital processing system comprising an image fusing module for fusing a stream of video image frames from the colour digital sensor with a stream of video image frames from the monochromatic digital sensor, where the video streams use the same frame rate, and wherein the sensors are operably connected to the digital processing system; and

wherein the image fusing method used by the image fusing module is selected from the group consisting of an arithmetic based fusion method, a statistics based fusion method, and a ratio based fusion method, and further comprises isolating corresponding pairs of video image frames from the two streams of video image frames and for a given pair of corresponding video image frames, fusing multi-spectral pixels in the colour video image frame of the given pair with the corresponding monochromatic pixels in the monochromatic video image frame of the given pair, where the fusing is performed in real time without user input or post processing.

15. The imaging system according to claim 14 wherein the video camera is selected from the group consisting of security, surveillance and monitoring video camera.

16. A video imaging system comprising:

a colour digital video camera for providing a stream of low resolution multi-spectral video image frames;

a monochromatic digital video camera for providing a stream of high resolution monochromatic video image frames; and

,

a digital processing system comprising an image fusing module for isolating corresponding pairs of video image frames from the two streams of video image frames where the video streams use the same frame rate, and fusing the pairs of video image frames to obtain a stream of colour video image frames having a higher resolution than the stream of the low resolution multi-spectral video image frames from the multi-spectral digital video camera;

wherein the cameras are operably connected to the digital processing system, and

wherein the image fusing method used by the image fusing module is selected from the group consisting of an arithmetic based fusion method, a statistics based fusion method, and a ratio based fusion method, and further comprises for a given pair of corresponding video image frames, fusing multi-spectral pixels in the multi-spectral video image frame of the given pair with the corresponding monochromatic pixels in the monochromatic video image frame of the given pair, where the fusing is performed in real time without user input or post processing.

17. The video imaging system according to claim 16 further comprising:

an object motion module for detecting a movement of an object within a fields of view of at least one of the cameras, and

an object position module for determining a position of an object in the an overlapping field of view of the cameras.

18. The system according to claim 16 further comprising:

an object motion module for detecting a movement of an object within a field of view of at least one of the cameras.

19. The system according to claim 16 further comprising:

an object position module for determining a position of an object in an overlapping field of view of the cameras.

20. The video imaging system according to claim 16 wherein the video cameras are selected from the group consisting of security, surveillance and monitoring video cameras.

21. The video imaging system according to claim 16 further comprising:

providing a three dimensional coordinate system for determining a position of an object in an overlap area of the fields of view of the cameras, by determining two dimensional coordinates of the object from a position of the object on at least one of the two images, and,

determining a distance from the cameras to the object in an axis perpendicular to the plane of the image frames from a parallax between two image frames of the stream of video image frames.

22. The video imaging system of claim 16, wherein the multi-spectral pixels are red, green and blue (RGB) colour pixels or red, green, blue and near infrared colour pixels.

23. A method comprising:

providing a stream of low resolution multi-spectral digital video image frames;

providing a corresponding stream of high resolution monochromatic digital video image frames where the video streams use the same frame rate; and

,

isolating corresponding pairs of video image frames from the two streams of video image frames and fusing the two streams of video image frames to obtain a stream of colour video image frames having a higher resolution than the stream of low resolution multi-spectral digital video image frames;

wherein the image fusing method for fusing the two streams of digital video image frames is selected from the group consisting of an arithmetic based fusion method, a statistics based fusion method, and a ratio based fusion method, and further comprises, for a given pair of corresponding video image frames, fusing multi-spectral pixels in the multi-spectral video image frame of the given pair with the corresponding monochromatic pixels in the monochromatic video image frame of the given pair, where the fusing is performed in real time without user input or post processing.

24. The method according to claim 23 wherein the stream of low resolution colour digital video image frames is provided by a digital sensor and the stream of high resolution monochromatic digital video image frames is provided by the digital sensor.

25. The method according to claim 23 wherein the stream of low resolution colour digital video image frames and the stream of high resolution monochromatic digital video image frames are provided by a digital video camera.

26. The method according to claim 25 wherein the camera is selected from the group consisting of a security, surveillance and monitoring video cameras.

27. The method according to claim 23 further comprising:

isolating two image frames from at least one of the streams;

comparing the two image frames; and

,

identifying at least one difference in the two image frames.

28. The video imaging system of claim 23, wherein the multi-spectral pixels are red, green and blue (RGB) colour pixels or red, green, blue and near infrared colour pixels.

29. An imaging system comprising:

a digital video camera comprising;

a colour digital sensor;

a monochromatic digital sensor;

a lens; and

a beam splitter for directing incident light passing through the lens to the colour and monochromatic sensors; and

,

a digital processing system comprising an image fusing module for fusing a stream of low resolution video image frames from the colour digital sensor with a stream of high resolution video image frames from the monochromatic digital sensor, where the video streams use the same frame rate, and wherein the sensors are operably connected to the digital processing system; and

wherein the image fusing method used by the image fusing module is selected from the group consisting of an arithmetic based fusion method, a statistics based fusion method, and a ratio based fusion method, and further comprises isolating corresponding pairs of video image frames from the two streams of video image frames and for a given pair of corresponding video image frames, fusing multi-spectral pixels in the colour video image frame of the given pair with the corresponding monochromatic pixels in the monochromatic video image frame of the given pair, where the fusing is performed in real time without user input or post processing.

30. The imaging system according to claim 29 wherein the video camera is selected from the group consisting of security, surveillance and monitoring video camera.