Method and system for video stitching

• US 9,961,305 B2
• Filed: 11/21/2014
• Issued: 05/01/2018
• Est. Priority Date: 03/20/2014
• Status: Active Grant

• Current Assignee: Hangzhou Hikvision Digital Technology Company Limited
• Original Assignee: Hangzhou Hikvision Digital Technology Company Limited
• Inventors: Guo, Bin, Shi, Xing
• Primary Examiner(s): Hausmann, Michelle
• Assistant Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US14/890,263
• Publication Number:
  US 20160112676A1
• Time in Patent Office: 1,257 Days
• Field of Search: None
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2622 Signal amplitude transition in the zone between image portions, e.g. soft edges
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Disclosed are a method and a system for video stitching. After it is determined that initial mounting of cameras in a M×N video stitching scene is completed, a reference image is selected from M×N images corresponding to M×N cameras; where both M and N are positive integers, at least one of M and N is larger than 1, and a rotating mirror is set in front of a camera lens of each camera; an image whose position does not meet a requirement among respective images except for the selected reference image is adjusted by controlling a rotating mirror corresponding to the image to rotate according to the selected reference image; and video stitching is performed according to an adjustment result.

Claims

1. A method for video stitching, the method comprising:

selecting a reference image from M×

N images corresponding to M×

N cameras in a M×

N video stitching scene;

where both M and N are positive integers, at least one of M and N is larger than 1, and a rotating mirror is set in front of a camera lens of each camera;

adjusting a position of an image whose position does not meet a requirement of seamless stitching among respective images except for the selected reference image by controlling a rotating mirror corresponding to the image to rotate according to the selected reference image; and

performing video stitching;

wherein

the adjusting an image whose position does not meet a requirement among respective images except for the selected reference image by controlling a rotating mirror corresponding to the image to rotate according to the selected reference image comprises;

for each image x which is to be adjusted, performing;

determining whether the image x is adjacent to the selected reference image;

when the image x is adjacent to the selected reference image, then using the selected reference image as a reference image of the image x, and when the image x is not adjacent to the selected reference image, then using an image which has been adjusted and is adjacent to the image x as a reference image of the image x; and

adjusting the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x.

2. The method of claim 1, wherein

after selecting the reference image, the method further comprises:

obtaining a control parameter of a rotating mirror corresponding to the selected reference image, and controlling and adjusting respective mirrors except for the rotating mirror corresponding to the selected reference image according to the obtained control parameter.

3. The method of claim 1, wherein

the adjusting the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x comprises:

when the image x is located on the left of an image y which is the reference image of the image x,

determining a ratio A of a height of an overlapped part between the image x and the image y to a height of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the top left of the image y, then moving the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

when the ratio A is smaller than the first threshold and the overlapped part is on the bottom left of the image y, then moving the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

determining a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then moving the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then moving the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

4. The method of claim 1, wherein

the adjusting the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x comprises:

when the image x is located on the right of an image y which is the reference image of the image x,

determining a ratio A of a height of an overlapped part between the image x and the image y to a height of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the top right of the image y, then moving the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

when the ratio A is smaller than the first threshold and the overlapped part is on the bottom right of the image y, then moving the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

determining a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then moving the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then moving the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

5. The method of claim 1, wherein

the adjusting the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x comprises:

when the image x is located above an image y which is the reference image of the image x,

determining a ratio A of a width of an overlapped part between the image x and the image y to a width of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the top left of the image y, then moving the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

when the ratio A is smaller than the first threshold and the overlapped part is on the top right of the image y, then moving the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

determining a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then moving the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then moving the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

6. The method of claim 1, wherein

the adjusting the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x comprises:

when the image x is located below an image y which is the reference image of the image x,

determining a ratio A of a width of an overlapped part between the image x and the image y to a width of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the bottom left of the image y, then moving the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

when the ratio A is smaller than the first threshold and the overlapped part is on the bottom right of the image y, then moving the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

determining a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then moving the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then moving the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

7. The method of claim 3, wherein

controlling the rotating mirror corresponding to the image x to rotate comprises:

rotating the rotating mirror corresponding to the image x at a predetermined angle each time, determining whether the image x has met the requirement after each rotation, and if a determination is yes, then stopping rotating, or otherwise, continuing to rotate.

8. A system for video stitching, the system comprising:

M×

N cameras and a remote control apparatus;

where both M and N are positive integers, at least one of M and N is larger than 1, and a rotating mirror is set in front of a camera lens of each camera; and

the remote control apparatus is configured to select a reference image from M×

N images corresponding to the M×

N cameras in a M×

N video stitching scene;

adjust a position of an image whose position does not meet a requirement of seamless stitching among respective images except for the selected reference image by controlling a rotating mirror corresponding to the image to rotate according to the selected reference image; and

perform video stitching;

wherein

the remote control apparatus comprises;

a first processing module configured to select the reference image from the M×

N images corresponding to the M×

N cameras in the M×

N video stitching scene, and notify a second processing module; and

the second processing module configured to perform the following process for each image x which is to be adjusted;

determining whether the image x is adjacent to the selected reference image;

when the image x is adjacent to the selected reference image, then using the selected reference image as a reference image of the image x, and when the image x is not adjacent to the selected reference image, then using an image which has been adjusted and is adjacent to the image x as a reference image of the image x;

adjusting a position of the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x; and

performing the video stitching according to an adjustment result.

9. The system of claim 8, wherein

the first processing module is further configured to, after select the reference image, obtain a control parameter of a rotating mirror corresponding to the selected reference image, and control and adjust respective rotating mirrors except for a rotating mirror corresponding to the selected reference image according to the obtained control parameter.

10. The system of claim 8, wherein

when the image x is located on the left of an image y which is the reference image of the image x,

the second processing module is configured to determine a ratio A of a height of an overlapped part between the image x and the image y to a height of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the top left of the image y, then move the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

when the ratio A is smaller than the first threshold and the overlapped part is on the bottom left of the image y, then move the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold; and

the second processing module is configured to determine a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then move the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then move the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

11. The system of claim 8, wherein

when the image x is located on the right of an image y which is the reference image of the image x,

the second processing module is configured to determine a ratio A of a height of an overlapped part between the image x and the image y to a height of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the top right of the image y, then move the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold;

when the ratio A is smaller than the first threshold and the overlapped part is on the bottom right of the image y, then move the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold; and

the second processing module is configured to determine a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then move the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then move the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

12. The system of claim 8, wherein

when the image x is located above an image y which is the reference image of the image x,

the second processing module is configured to determine a ratio A of a width of an overlapped part between the image x and the image y to a width of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the top left of the image y, then move the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold; and

when the ratio A is smaller than the first threshold and the overlapped part is on the top right of the image y, then move the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold; and

the second processing module is configured to determine a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then move the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then move the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

13. The system of claim 8, wherein

when the image x is located below an image y which is the reference image of the image x,

the second processing module is configured to determine a ratio A of a width of an overlapped part between the image x and the image y to a width of the image y;

when the ratio A is smaller than a first threshold and the overlapped part is on the bottom left of the image y, then move the image x to the right by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold; and

when the ratio A is smaller than the first threshold and the overlapped part is on the bottom right of the image y, then move the image x to the left by controlling the rotating mirror corresponding to the image x to rotate, until the ratio A is larger than the first threshold; and

the second processing is configured to determine a ratio B of a size of the overlapped part between the image x and the image y to a size of the image y;

when the ratio B is smaller than a second threshold, then move the image x upwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than a third threshold, where the third threshold is larger than the second threshold; and

when the ratio B is larger than the third threshold, then move the image x downwards by controlling the rotating mirror corresponding to the image x to rotate, until the ratio B is larger than the second threshold and smaller than the third threshold.

14. A non-transitory computer readable storage medium storing one or more programs executable by a computer system, the one or more programs comprising instructions to:

select a reference image from M×

N images corresponding to M×

N cameras in a M×

N video stitching scene;

where both M and N are positive integers, at least one of M and N is larger than 1, and a rotating mirror is set in front of a camera lens of each camera;

adjust a position of an image whose position does not meet a requirement of seamless stitching among respective images except for the selected reference image by controlling a rotating mirror corresponding to the image to rotate according to the selected reference image; and

perform video stitching;

wherein

the adjusting an image whose position does not meet a requirement among respective images except for the selected reference image by controlling a rotating mirror corresponding to the image to rotate according to the selected reference image comprises;

for each image x which is to be adjusted, performing;

determining whether the image x is adjacent to the selected reference image;

when the image x is adjacent to the selected reference image, then using the selected reference image as a reference image of the image x, and when the image x is not adjacent to the selected reference image, then using an image which has been adjusted and is adjacent to the image x as a reference image of the image x; and

adjusting the image x to meet the requirement by controlling a rotating mirror corresponding to the image x to rotate according to the reference image of the image x.