Imaging processing method for mapping video source information onto a displayed object

• US 6,021,229 A
• Filed: 10/24/1997
• Issued: 02/01/2000
• Est. Priority Date: 11/14/1995
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Maruyama, Kazuhiro, Takashima, Akihiro, Minami, Nobuyuki
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US08/849,970
• Time in Patent Office: 830 Days
• Field of Search: 345/421.22, 345/425-427, 345/430-438, 345/441-443, 345/502, 345/507, 345/348-349, 345/419, 382/153-154, 382/285, 382/277, 382/276, 382/293, 395/86, 348/580-588
• US Class Current: 382/285
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

An image processing apparatus and the method for mapping a plurality of images which are on the two-dimensional plane on the corresponding faces of a solid in the three-dimensional virtual space. The mapping image on the side face being close to the object image displaying face is formed in accordance with the shape of object image displaying face of the solid on which the object image is mapped, and the image mapped on each face is displaced in accordance with the movement of the solid in the three-dimensional space. Therefore, the image that as if a desired image is stuck to the faces of the solid moving in a virtual space can be displayed on a screen surface by easily operating by an operator.

Claims

1. A video processing apparatus, comprising:

first video processing means for performing a spatial transformation on a first source video by reading said first source video from a first memory in accordance with a first read address data in order to generate a first two-dimensional transform video;

second video processing means for performing a spatial transformation on a second source video by reading said second source video from a second memory in accordance with a second read address data in order to generate a second two-dimensional transform video;

third video processing means for performing a spatial transformation on a third source video by reading said third source video from a third memory in accordance with a third read address data in order to generate a third two-dimensional transform video;

composite means for combining said first, second and third two-dimensional transform video to generate a two-dimensional composite video; and

control means for generating said first read address data, said second read address data and said third read address data, and for modifying said second read address data and said third read address data in accordance with said spatial transformation to be performed on said first source video so that said two-dimensional composite video image appears as a desired three-dimensional object.

2. The video processing apparatus according to claim 1,

wherein said spatial transformation is performed as a mapping transformation for mapping source video onto a predetermined face of said three-dimensional object, a three-dimensional transformation for transforming said source video mapped on said predetermined face of said three-dimensional object into three-dimensional space, and a perspective transformation for perspective-transforming said source video transformed in three dimensional space onto a screen surface.

3. The video processing apparatus according to claim 2, wherein

said mapping transformation includes a calculation algorithm including, a first mapping transformation matrix for mapping said first source video on a first face of said three dimensional object, a second mapping transformation matrix for mapping said second source video on a second face of said three dimensional object and third mapping transformation matrix for mapping said third source video on a third face of said three dimensional object, and wherein

said three dimensional transforation is defined by a three dimensional transformation matrix, and

said perspective transformation is defined by a perspective transformation matrix.

4. The video processing apparatus according to claim 2, wherein:

said first transformation matrix designated as M_A, is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.A =L.sub.A ·

T.sub.rs

where, L_A ;

a matrix for performing a linear displacement on said first source video,

T_rs ;

a matrix for performing a rate/skew transformation on said first source video;

said second transformation matrix, designated as M_B, is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.B =L.sub.BO ·

S.sub.B ·

R.sub.BX ·

R.sub.BZ ·

L.sub.B

where, L_BO ;

a matrix for performing a linear displacement on said second source video based upon a crop operation,

S_B ;

a matrix for performing a magnification/reduction of said second source video

R_BY ;

a matrix for performing a rotation of the second source video around the y-axis

R_BZ ;

a matrix for performing a rotation of the second source video around the Z-axis

L_B ;

a matrix for performing a linear displacement on said second source video;

said third transformation matrix M_C is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.C =L.sub.CO ·

S.sub.C R.sub.CX ·

R.sub.CZ ·

L.sub.C

where, L_CO ;

a matrix for performing a linear displacement on said third source video based upon a crop operation,

S_C ;

a matrix for performing magnification/reduction of said third source video,

R_CX ;

a matrix for performing a rotation of the third source video around the X-axis,

R_CZ ;

a matrix for performing a rotation of the third source video around the Z-axis,

L_C ;

a matrix for performing a linear displacement on said third source video.

5. The video processing apparatus according to claim 4 wherein:

said three-dimensional transformation matrix T_O is expressed by the following equation;

##EQU40## where, r₁₁ to r₃₃ ;

parameters for transforming the source video for x-, y-, and z-axis of the three-dimensional coordinates of xyz

l_x ;

a parameter for linearly displacing the source video in the x-axis direction;

l_y ;

parameter for linearly displacing the source video in the y-axis direction;

l_z ;

parameter for linearly displacing the source video in the z-axis direction; and

s;

a parameter for magnifying or reducing the scale of source video;

and wherein said perspective transformation matrix P_O is expressed by the following equation;

##EQU41## where, P_Z ;

a perspective value.

6. The video processing apparatus according to claim 5,

wherein said control means does not use the z-axis element of said three-dimensional transformation matrix in the calculation of said read address data.

7. The video processing apparatus according to claim 5, wherein

a calculation algorithm of said mapping transformation further includes a fourth mapping transformation matrix for mapping said first source video on an opposite face of said first face of said three-dimensional object, a fifth mapping transformation matrix for mapping said second source video on an opposite face of said second face of said three-dimensional object and sixth mapping transformation matrix for mapping said third source video on an opposite face of said third face of said three-dimensional object.

8. The video processing apparatus according to claim 5, wherein

said first source video to be transformed to said first two-dimensional transform video by the transformation matrix T_A is expressed by the equation:

space="preserve" listing-type="equation">T.sub.A =M.sub.A T.sub.O P.sub.O

said second source video to be transformed to said second two-dimensional transform video.

9. The video processing apparatus according to claim 8, wherein:

a fourth mapping transformation matrix designated as M_A '"'"' is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.A '"'"'=L.sub.A '"'"'·

T.sub.rs

where, L_A '"'"';

a matrix for performing a linear displacement on said first source video,

T_rs ;

a matrix for performing a rate/skew transformation on said first source video;

a fifth mapping transformation matrix M_B '"'"' designated as is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.B '"'"'=L.sub.BO ·

S.sub.B ·

R.sub.BX ·

R.sub.BZ ·

L.sub.B '"'"'

where, L_BO ;

a matrix for performing a linear displacement on said second source video based upon a crop operation,

S_B ;

a matrix for performing a magnification/reduction of said second source video

R_BY ;

a matrix for performing a rotation of the second source video around the y-axis,

R_BZ ;

a matrix for performing a rotation of the second source video around the Z-axis,

L_B '"'"';

a matrix for performing a linear displacement of said second source video;

a sixth mapping transformation matrix M_C '"'"' is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.C '"'"'=L.sub.CO ·

S.sub.C ·

R.sub.CX ·

R.sub.CZ ·

L.sub.C '"'"'

where, L_CO ;

a matrix for performing a linear displacement on said third source video based upon a crop operation,

S_C ;

a matrix for performing a magnification/reduction of the third source video,

R_CX ;

a matrix for performing a rotation of the third source video around the x-axis;

R_CZ ;

a matrix for performing a rotation of the third source video around the z-axis,

L_C '"'"';

a matrix for performing a linear displacement on said third source video.

10. The video processing apparatus according to claim 1, further comprising:

means for implementing a reduction rate priority mode for modifying the content of, said first source video, said second source video, and/or said third source video when said first source video, said second source video, and/or said third source video are transformed to said first, second and third transform video, respectively; and

means for implementing a crop priority mode for reducing or magnifying the modified content of said first source video, said second source video, and/or said third source video into a size corresponding to a size of a respective face of said two-dimensional composite video image when said first source video, said second source video, and/or said third source video are transformed to said first, second or third transform videos, respectively.

11. The video processing apparatus according to claim 5, wherein

said first two-dimensional transform video written in a first memory, said second two-dimensional transform video written in a second memory, and said third two-dimensional transform video written in a third memory are each read out from each of said memories in accordance with an inverse matrix of said transformation matrix T_O.

12. The video processing apparatus according to claim 7, wherein said control means further comprises:

face detect means for detecting which of said first face or said opposite face of the first face is to be viewed as visible before performing said spatial transformation on said first source video; and

selecting means for selecting said first mapping transformation matrix or said fourth mapping transformation matrix based on the resultant of detection of said face detect means.

13. A video processing method, comprising the steps of:

performing a spatial transformation on a first source video by reading said first source video from a first memory in accordance with a first read address data in order to generate a first two-dimensional transform video;

performing a spatial transformation on a second source video by reading said second source video from a second memory in accordance with a second read address data in order to generate a second two-dimensional transform video;

performing a spatial transformation on a third source video by reading said third source video from a third memory in accordance with a third read address data in order to generate a third two-dimensional transform video;

combining said first, second and third two-dimensional transform video to generate a two-dimensional composite video;

generating said first read address data, said second read address data and said third read address data; and

modifying said second read address data and said third read address data in accordance with said spatial transformation to be performed on said first source video so that said two-dimensional composite video image appears as a desired three-dimensional object.

14. The video processing method according to claim 13, wherein said spatial transformation is performed as a mapping transformation for mapping source video onto a predetermined face of said three-dimensional object, a three-dimensional transformation for transforming said source video mapped on said predetermined face of said three-dimensional object into three-dimensional space, and a perspective transformation for perspective-transforming said source video transformed in three dimensional space onto a screen surface.

15. The video processing method according to claim 14, wherein

said mapping transformation includes a calculation algorithm including, a first mapping transformation matrix for mapping said first source video on a first face of said three dimensional object, a second mapping transformation matrix for mapping said second source video on a second face of said three dimensional object and third mapping transformation matrix for mapping said third source video on a third face of said three dimensional object, and wherein

said three dimensional transforation is defined by a three dimensional transformation matrix, and

said perspective transformation is defined by a perspective transformation matrix.

16. The video processing method according to claim 14, wherein:

said first transformation matrix designated as M_A, is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.A =L.sub.A ·

T.sub.rs

where, L_A ;

a matrix for performing a linear displacement on said first source video,

T_rs ;

a matrix for performing a rate/skew transformation on said first source video;

said second transformation matrix, designated as M_B, is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.B =L.sub.BO ·

S.sub.B ·

R.sub.BX ·

R.sub.BZ ·

L.sub.B

where, L_BO ;

a matrix for performing a linear displacement on said second source video based upon a crop operation,

S_B ;

a matrix for performing a magnification/reduction of said second source video

R_BY ;

a matrix for performing a rotation of the second source video around the y-axis

R_BZ ;

a matrix for performing a rotation of the second source video around the Z-axis

L_B ;

a matrix for performing a linear displacement on said second source video;

said third transformation matrix M_C is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.C =L.sub.CO ·

S.sub.C ·

R.sub.CX ·

R.sub.CZ ·

L.sub.C

where, L_CO ;

a matrix for performing a linear displacement on said third source video based upon a crop operation,

S_C ;

a matrix for performing magnification/reduction of said third source video,

R_CX ;

a matrix for performing a rotation of the third source video around the x-axis,

R_CZ ;

a matrix for performing a rotation of the third source video around the Z-axis,

L_C ;

a matrix for performing a linear displacement on said third source video.

17. The video processing method according to claim 16, wherein:

said three-dimensional transformation matrix is expressed by the following equation;

##EQU42## where, r₁₁ to r₃₃ ;

parameters for transforming the source video for x-, y-, and z-axis of the three-dimensional coordinates of xyz

l_x ;

a parameter for linearly displacing the source video in the x-axis direction;

l_y ;

a parameter for linearly displacing the source video in the y-axis direction;

l_z ;

a parameter for linearly displacing the source video in the z-axis direction; and

s;

a parameter for magnifying or reducing the scale of source video;

and wherein said perspective transformation matrix P_O is expressed by the following equation;

##EQU43## where, P_Z ;

a perspective value.

18. The video processing method according to claim 17,

wherein said control means does not use the z-axis element of said three-dimensional transformation matrix in the calculation of said read address data.

19. The video processing method according to claim 17, wherein

a calculation algorithm of said mapping transformation further includes a fourth mapping transformation matrix for mapping said first source video on an opposite face of said first face of said three-dimensional object, a fifth mapping transformation matrix for mapping said second source video on an opposite face of said second face of said three-dimensional object and sixth mapping transformation matrix for mapping said third source video on an opposite face of said third face of said three-dimensional object.

20. The video processing method according to claim 17, wherein

said first source video to be transformed to said first two-dimensional transform video by the transformation matrix T_A is expressed by the equation:

space="preserve" listing-type="equation">T.sub.A =M.sub.A ·

T.sub.O ·

P.sub.O

said second source video to be transformed to said second two-dimensional transform video by the transformation matrix T_B is expressed by the equation;

space="preserve" listing-type="equation">T.sub.B =M.sub.B T.sub.O ·

P.sub.O ;

and

said third source video to be transformed to said third two-dimensional transform video by the transformation matrix T_C is expressed by the equation;

space="preserve" listing-type="equation">T.sub.C =M.sub.C ·

T.sub.O P.sub.O.

21.

21. The video processing method according to claim 20, wherein:

a fourth mapping transformation matrix designated as M_A '"'"' is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.A '"'"'=L.sub.A '"'"'·

T.sub.rs

where, L_A '"'"';

a matrix for performing a linear displacement on said first source video,

T_rs ;

a matrix for performing a rate/skew transformation on said first source video;

a fifth mapping transformation matrix M_B '"'"' designated as is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.B '"'"'=L.sub.BO ·

S.sub.B ·

R.sub.BX ·

R.sub.BZ ·

L.sub.B '"'"'

where, L_BO ;

a matrix for performing a linear displacement on said second source video based upon a crop operation,

S_B ;

a matrix for performing a magnification/reduction of said second source video

R_BY ;

a matrix for performing a rotation of the second source video around the y-axis,

R_BZ ;

a matrix for performing a rotation of the second source video around the Z-axis,

L_B '"'"';

a matrix for performing a linear displacement of said second source video;

a sixth mapping transformation matrix M_C '"'"' is expressed by the following equation;

space="preserve" listing-type="equation">M.sub.C'"'"' =L.sub.CO ·

S.sub.C ·

R.sub.CX ·

R.sub.CZ ·

L.sub.C '"'"'

where, L_CO ;

a matrix for performing a linear displacement on said third source video based upon a crop operation,

S_C ;

a matrix for performing a magnification/reduction of the third source video,

R_CX ;

a matrix for performing a rotation of the third source video around the x-axis;

R_CZ ;

a matrix for performing a rotation of the third source video around the z-axis,

L_C '"'"';

a matrix for performing a linear displacement on said third source video.

22. The video processing method according to claim 13, further comprising the steps of:

implementing a reduction rate priority mode for modifying the content of, said first source video, said second source video, and/or said third source video when said first source video, said second source video, and/or said third source video are transformed to said first, second and third transform video, respectively; and

implementing a crop priority mode for reducing or magnifying the modified content of said first source video, said second source video, and/or said third source video into a size corresponding to a size of a respective face of said two-dimensional composite video image when said first source video, said second source video, and/or said third source video are first source video transformed to said first, second or third transform videos, respectively.

23. The video processing method according to claim 17, wherein said first two-dimensional transform video written in a first memory, said second two-dimensional transform video written in a second memory, and said third two-dimensional transform written in a third memory are each read out from each of said memories in accordance with an inverse matrix of said transformation matrix T_O.

24. The video processing method according to claim 19, further comprising the steps of:

detecting which of said first face or said opposite face of said first face is to be viewed as visible before performing said spatial transformation on said first source video; and

selecting said first mapping transformation matrix or said fourth mapping transformation matrix based on the resultant of detection.

25. A video processing apparatus, comprising:

first video processing means for performing a first image transformation on a first source image data in accordance with first transform parameters to generate a first two-dimensional transform video;

second video processing means for performing a second image transformation on a second source image data in accordance with second transform parameters to generate a second two-dimensional transform video;

third video processing means for performing a third image transformation on a third source image data in accordance with third transform parameters to generate a third two-dimensional transform video;

composite means for combining said first, second and third two-dimensional transform video to generate a two-dimensional composite video; and

control means for generating said first transform parameters, said second transform parameters and said third transform parameters, and for modifying said second transform parameters and said third transform parameters based on a modification in a first transform operation to be performed on said first source image data.

26. A video processing method, comprising the steps of:

performing a first image transformation on a first source image data in accordance with first transform parameters to generate a first two-dimensional transform video;

performing a second image transformation on a second source image data in accordance with second transform parameters to generate a second two-dimensional transform video;

performing a third image transformation on a third source image data in accordance with third transform parameters to generate a third two-dimensional transform video;

combining said first, second and third two-dimensional transform video to generate a two-dimensional composite video;

generating said first transform parameters, said second transform parameters and said third transform parameters; and

modifying said second transform parameters and said third transform parameters based on a modification in a first transform operation to be performed on said first source image data.