Video signal processing device and method employing transformation matrix to generate composite image

• US 6,441,864 B1
• Filed: 11/10/1997
• Issued: 08/27/2002
• Est. Priority Date: 11/12/1996
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kurashige, Masafumi, Minami, Nobuyuki
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US08/966,915
• Time in Patent Office: 1,751 Days
• Field of Search: H04/N.974, H04/N.976, H04/N.975, 348/584, 348/585, 348/586, 348/587, 348/589, 348/591, 348/592, 348/598, 348/599, 345/419, 345/427
• US Class Current: 348/584
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

A video signal processing device includes an address generator to calculate a transformation matrix for three-dimensionally transforming an image to a “blue board” area. The transformation is based on corner positions of the blue board area of a video signal and those of an image area of a source video signal to be inserted. Address information is calculated for reading the source video signal from a memory based on an inverse matrix of the transformation matrix. In this way, the image area to be inserted can be three-dimensionally transformed and automatically inserted into the blue board area, without requiring an operator to input parameters indicative of a post-transformation state.

Claims

1. A video signal processing device for inserting a source video signal to a prescribed area of a video signal and forming a composite video signal, comprising:

corner detection means for detecting a blue board area from the video signal obtained by photographing an object with the blue board in the background of said object and for detecting each corner position of said area;

source corner specifying means for specifying each corner position of an image area of said source video signal which is inserted to said blue board area of said video signal;

address generation means for calculating a transformation matrix for three-dimensionally transforming said image area to said blue board area on the basis of the position information showing each corner position of said blue board area detected by said corner detection means and the position information showing each corner position of said image area specified by said source corner specifying means, calculating an inverse matrix of said transformation matrix, and calculating address information for reading said source video signal from a memory in which said source video signal is written;

image conversion means for forming a conversion source video signal which is three-dimensionally image transformed and located at the position in which said image area corresponds to said blue board area by writing the source video signal in the memory and reading out the source video signal written in the memory based on the address information generated by said address generation means; and

signal mixing means for forming a composite video signal in which an image area of said source video signal is inserted to said blue board area in said video signal by mixing said conversion source video signal and said video signal.

2. A video signal processing device according to claim 1, characterized by:

said transformation matrix is a matrix for mapping said image area in a three-dimensional space and further perspectively transforming it on a monitor screen.

3. A video signal processing device according to claim 1, characterized by:

said address generation means calculates the position vector of each corner of said image area on said memory on the basis of said position information from said source corner specifying means, and calculates said transformation matrix by setting a linear equation on each parameter of the transformation matrix based on the position vector of each corner of said image area and the position vector of each corner of said transformation matrix obtained by said position information from said corner detection means on a monitor screen and solving said linear equation.

4. A video signal processing device according to claim 3, characterized by:

said address generation means calculates an inverse matrix of said transformation matrix using the respective parameters of the calculated transformation matrix, and generates the address information for use in reading from said memory based on the respective parameters of said inverse matrix and addresses on the monitor screen generated in a raster scan order.

5. A video signal processing device according to claim 1, comprising:

input means for selecting said image area to be inserted to said blue board area in said video signal; and

characterized by;

said source corner specifying means specifies each corner position of an image area selected by said input means, and forms said composite video signal which has said selected image area inserted to said blue board area of said video signal.

6. A video signal processing method for inserting a source video signal to a prescribed area of a video signal and forming a composite video signal, comprising:

a corner detection step for detecting a blue board area from the video signal which is obtained by photographing an object with a blue board in the background of said object and detecting each corner position of said blue board area;

a source corner specifying step for specifying each corner position of an image area of said source video signal to be inserted to said blue board area of said video signal;

an address generation step for calculating a transformation matrix for three-dimensionally transforming said image area to said blue board area based on the position information showing each corner position of said blue board area detected by said corner detection step and the position information showing each corner position of said image area specified by said source corner specifying step, successively calculating an inverse matrix of said transformation matrix, and calculating address information for reading said source video signal from a memory in which said source video signal has been written;

an image conversion step for forming a conversion source video signal which has a picture area three-dimensionally transformed and located at a position corresponding to said blue board area by writing said source video signal in said memory and reading said source video signal written in said memory based on said address information generated by said address generation step; and

a signal mixing step for forming a composite video signal which has said image area of said source video signal inserted to said blue board area of said video signal by mixing said conversion source video signal and said video signal.

7. A video signal processing method according to claim 6, characterized by:

said transformation matrix is a matrix for mapping said image area to a three-dimensional space, and further perspectively transforming the mapped image area on a plane of a monitor screen.

8. A video signal processing method according to claim 7, characterized by:

said address generation step comprises calculating position vector of each corner of said image area on said memory based on said position information from said source corner specifying means; and

setting a linear equation on each parameter of said transformation matrix based on the position vector of each corner of said image area and the position vector of each corner of said blue board area on the monitor screen, obtained by said position information from said corner detection step, and calculating said transformation matrix by solving said linear equation.

9. A video signal processing method according to claim 8, characterized by:

said address generation step calculates an inverse matrix of said transformation matrix using each parameter of the calculated transformation matrix, and generates the address information for use in reading from said memory based on each parameter of said inverse matrix and the address on the monitor screen generated in a raster scan order.

10. A video signal processing method according to claim 6, comprising:

an area selection step for selecting said image area to be inserted to said blue board area in said video signal; and

characterized by;

said source corner specifying step specifies each corner position of the image area selected by said area selection step and forms said composite video signal which has said selected image area inserted to said blue board area of said video signal.

11. A video image processing device for conducting image processing on a source video image, comprising:

coordinate detection means for detecting coordinates of four corners of a first area specified in a first video image and coordinates of four corners of a second area specified in a second video image;

calculation means for calculating a three-dimensional transformation matrix for a three-dimensional image conversion from said coordinates of four corners of said first area and said coordinates of four corners of said second area in order to insert the video image of said second area into said first area by conducting three-dimensional image conversion on said second video image; and

image conversion means for conducting three-dimensional conversion processing on said second video image by using the three-dimensional transformation matrix calculated by said calculation means in order that the video image of the second area 3-D converted is inserted into said first area.

12. A video image processing device according to claim 11, characterized by:

said three-dimensional transformation matrix can be expressed by a 4-row 4-column matrix composed of 8 transformation parameters as follows;

$\begin{array}{cc}{T}_{33}^{\mathrm{\prime <br><br>}}=\left[\begin{array}{ccc}{a}_{11}/a_{33}& a_{12}/a_{33}& a_{13}/a_{33}\\ a_{21}/a_{33}& a_{22}/a_{33}& a_{23}/a_{33}\\ a_{31}/a_{33}& a_{32}/a_{33}& 1\end{array}\right]=\left[\begin{array}{ccc}{a}_{11}^{\mathrm{\prime <br><br>}}& a_{12}^{\mathrm{\prime <br><br>}}& a_{13}^{\mathrm{\prime <br><br>}}\\ a_{21}^{\mathrm{\prime <br><br>}}& a_{22}^{\mathrm{\prime <br><br>}}& a_{23}^{\mathrm{\prime <br><br>}}\\ a_{31}^{\mathrm{\prime <br><br>}}& a_{32}^{\mathrm{\prime <br><br>}}& 1\end{array}\right].& \left(68\right)\end{array}$

13. A video image processing device according to claim 12, characterized by:

said calculation means obtains said 8 parameter values by use of the following equation;

$\left[\begin{array}{ccc}{X}_i·<br><br>K_i& Y_i·<br><br>K_i& K_i\end{array}\right]=\left[\begin{array}{ccc}{X}_i^{\mathrm{\prime <br><br>}}& Y_i^{\mathrm{\prime <br><br>}}& 1\end{array}\right]·<br><br>T_{33}^{\mathrm{\prime <br><br>}}=\left[\begin{array}{ccc}{X}_i^{\mathrm{\prime <br><br>}}& Y_i^{\mathrm{\prime <br><br>}}& 1\end{array}\right]·<br><br>\left[\begin{array}{ccc}{a}_{11}^{\mathrm{\prime <br><br>}}& a_{12}^{\mathrm{\prime <br><br>}}& a_{13}^{\mathrm{\prime <br><br>}}\\ a_{21}^{\mathrm{\prime <br><br>}}& a_{22}^{\mathrm{\prime <br><br>}}& a_{23}^{\mathrm{\prime <br><br>}}\\ a_{31}^{\mathrm{\prime <br><br>}}& a_{32}^{\mathrm{\prime <br><br>}}& 1\end{array}\right]$

 

where (X₁, Y₁) (X₂, Y₂) (X₃, Y₃) (X₄, Y₄) are the coordinates of four corners of the first area, (X′

₁, Y′

₁) (X′

₂, Y′

₂) (X′

₃, Y′

₃) (X′

₄, Y′

₄) are the coordinates of four corners of the second area, and K₁, K₂, K₃, K₄ are parameters of a homogeneous coordinate system.

14. A video image processing device according to claim 11, comprising:

chroma key detection means for detecting an area having the prescribed color of said first video image as said first area and forming a chroma key signal corresponding to said first area; and

characterized by;

said coordinate detection means detects the coordinates of 4 corners of said first area on the basis of said chroma key signal.

15. A video image processing device according to claim 14, characterized by:

said image conversion means further conducts said three-dimensional conversion on said key signal corresponding to the second video image.

16. A video image processing device according to claim 15, comprising:

mixing means for mixing said first video image and said second video image 3-D converted on the basis of said key signal corresponding to the second video image.

17. A video image processing device according to claim 11, characterized by:

said image conversion means provides memory means to memorize said second video image and read address generation means to supply a read address which corresponds to said transformation matrix to said memory means.

18. A video image processing device according to claim 17, characterized by:

said read address generation means calculates said read address on the basis of the inverse matrix of said transformation matrix.

19. A video image processing method for conducting image processing on a source video image, comprising the steps of:

detecting coordinates of four corners of a first area specified in a first video image and coordinates of four corners of a second area specified in a second video image;

calculating a three-dimensional transformation matrix for conducting a three-dimensional image conversion from the coordinates of four corners of said first area and the coordinates of four corners of said second area in order that the video image of said second area is inserted into said first area by conducting the three-dimensional image conversion on said second video image; and

conducting three-dimensional conversion processing on said second video image using said three-dimensional transformation matrix in order that the video image of the second area 3-D converted is inserted into said first area.

20. A video image processing method according to claim 19, further comprising:

detecting an area having a prescribed color of said first video image as said first area and forming a chroma key signal corresponding to said first area; and

detecting the coordinates of four corners of said first area on the basis of said chroma key signal.