Picture generator and picture generation method

• US 6,147,716 A
• Filed: 01/20/1999
• Issued: 11/14/2000
• Est. Priority Date: 05/23/1997
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nagano, Hidetoshi, Totsuka, Takashi, Ohki, Mitsuharu
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US09/233,762
• Time in Patent Office: 664 Days
• Field of Search: 348/572, 348/573, 348/575, 348/576, 348/578, 348/580, 348/581, 348/582, 348/584, 348/585, 348/586, 348/587, 348/588, 348/589, 348/590, 348/591, 348/592, 348/25, 348/26 , 348/722, 345/435, 358/22, 364/518, 364/559, 382/284, 395/121, 318/628
• US Class Current: 348/584
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

The present invention relates to a picture generator and a picture generation method. In detail the picture generator provides a mobile photograph device acquires first video data by photographing a performer in front of a predetermined background, a camera parameter calculator acquires the viewpoint of the photograph device, a first video data generator generates second video data corresponding to the projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of the photographing device acquired in the camera parameter calculator is applied, a data delay device acquires third video data by delaying the first video data corresponding to the processing time of the camera parameter calculator and the first video data generator, a first video data synthesizer acquires first synthetic video data by synthesizing the third video data and the second video data, a second video data synthesizer acquires second synthetic video data by synthesizing the first video data and the second video data, and a picture display that displays a picture for assisting the performance of the performer based upon the second synthetic video data. Hereby, a picture for assisting the movement of the performer is hardly delayed and the performer can readily perform.

Claims

1. A picture generator, comprising:

mobile photographing means for photographing a performer in front of a predetermined background having a predetermined color to obtain first video data, said background including a plurality of inconspicuous markers each having a color similar to the predetermined color of the predetermined background;

camera parameter calculating means for acquiring viewpoint information of said photographing means including three-dimensional position and orientation of the photographing means on the basis of predetermined world coordinates;

first video data generating means for generating second video data corresponding to a projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of said photographing means acquired in said camera parameter calculating means is applied;

data delay means for acquiring third video data delayed corresponding to time for processing said first video data in said camera parameter calculating means and said first video data generating means;

first key signal generating means for generating a first key signal for separating a first picture element having the predetermined color of the predetermined background or a color similar thereto and a second picture element which is not the predetermined color of the predetermined background, based on said first video data;

second key generating means for generating a second key signal for separating the first picture element having the predetermined color of the predetermined background or a color similar thereto and the second picture element which is not the predetermined color of the predetermined background, based on said first video data and for acquiring a picture switching signal for fetching either of said first video data or said second video data based upon said first video data;

first video data synthesizing means for acquiring first synthetic video data by synthesizing said third video data and said second video data on the basis of said first key signal;

second video data synthesizing means for acquiring second synthetic video data by switching between said first video data and said second video data on the basis of said second key signal and synthesizing them; and

picture display means for displaying a picture for assisting the performance of said performer based upon said second synthetic video data.

2. A picture generator according to claim 1, wherein said mobile photographing means further comprises a turning base, on which the mobile photographing means is put, which turns said mobile photographing means using a gear, and wherein an orientation of said mobile photographing means is based upon a quantity in which said gear is rotated.

3. A picture generator comprising:

mobile photographing means for photographing a performer in front of a predetermined background having a predetermined color to obtain first video data, said background including a plurality of inconspicuous markers each having a color similar to the predetermined color of the predetermined background;

camera parameter calculating means for acquiring viewpoint information of said photographing means including three-dimensional position and orientation of the photographing means on the basis of predetermined world coordinates using three-dimensional position and orientation of said markers obtained from said first video data on the basis of coordinates relative to said photographing means, and predetermined three-dimensional position information of said markers relative to said world coordinates;

first video data generating means for generating second video data corresponding to a projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of said photographing means acquired in said camera parameter calculating means is applied;

camera parameter forecasting means for forecasting the viewpoint of said photographing means at current time or at time posterior to the current time based upon the viewpoint information of said photographing means acquired in said camera parameter calculating means;

second video data generating means for generating fourth video data corresponding to the projected picture of said predetermined object to which perspective transformation in a case viewed from the viewpoint forecast in said camera parameter forecasting means of said photographing means is applied;

data delay means for acquiring third video data delayed corresponding to time for processing said first video data in said camera parameter calculating means and said first video data generating means;

first key signal generating means for generating a first key signal for separating a first picture element having the predetermined color of the predetermined background or a color similar thereto and a second picture element which is not the predetermined color of the predetermined background, based on said first video data;

second key generating means for generating a second key signal for separating the first picture element having the predetermined color of the predetermined background or a color similar thereto and the second picture element which is not the predetermined color of the predetermined background, based on said first video data and for acquiring a picture switching signal for fetching either of said first video data or said fourth video data based upon said first video data;

first video data synthesizing means for acquiring first synthetic video data by synthesizing said third video data and said second video data on the basis of said first key signal;

second video data synthesizing means for acquiring second synthetic video data by switching between said first video data and said fourth video data on the basis of said second key signal and synthesizing them; and

picture display means for displaying a picture for assisting the performance of said performer based upon said second synthetic video data.

4. The picture generator according to claim 3, wherein said camera parameter forecasting means linearly forecasts a position of the photographing means at current time based upon positional information (T1, T2) of the photographing means at the past two times according to a formula T2+(T2-T1), and

wherein said camera parameter forecasting means linearly forecasts an orientation of the photographing means at current time based upon orientation information (R1, R2) of the photographing means at the past two times according to a formula R2+(R2-R1).

5. A picture generation method, comprising the steps of:

acquiring first video data by photographing a performer in front of a predetermined background having a predetermined color with mobile photographing means, said background including a plurality of inconspicuous markers each having color similar to the predetermined color of the background;

acquiring viewpoint information of said photographing means including three-dimensional position and orientation of the photographing means on the basis of predetermined world coordinates using three-dimensional position information of said markers obtained from said first video data on the basis of coordinates relative to said photographing means, and predetermined three-dimensional position information of said markers relative to said world coordinates;

generating second video data corresponding to a projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of said photographing means is applied;

acquiring third video data by delaying said first video data corresponding to processing time in the step for acquiring the viewpoint information of said photographing means and the step for generating said second video data;

generating a first key signal for separating a first picture element having the predetermined color of the predetermined background or a color similar thereto and a second picture element which is not the predetermined color of the predetermined background, based on said first video data;

generating a second key signal for separating the first picture element having the predetermined color of the predetermined background or a color similar thereto and the second picture element which is not the predetermined color of the predetermined background, based on said first video data and for acquiring a picture switching signal for fetching either of said first video data or said second video data based upon said first video data;

acquiring first synthetic video data by synthesizing said third video data and said second video data on the basis of said first key signal;

acquiring second synthetic video data by switching between said first video data and said second video data on the basis of said second key signal and synthesizing them; and

displaying a picture for assisting the performance of said performer based upon said second synthetic video data.

6. The picture generating method of claim 5, further comprising the steps of:

turning said mobile photographing means using a gear, wherein the orientation of the photographing means is based upon a quantity in which the gear is rotated.

7. A picture generation method comprising the steps of:

acquiring first video data by photographing a performer in front of a predetermined background having a predetermined color with mobile photographing means said background including a plurality of inconspicuous markers each having color similar to the predetermined color of the background;

acquiring viewpoint information of said photographing means including three-dimensional position and orientation of the photographing means on the basis of predetermined world coordinates using three-dimensional position information of said markers obtained from said first video data on the basis of coordinates relative to said photographing means, and predetermined three-dimensional position information of said markers relative to said world coordinates;

generating second video data corresponding to a projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of said photographing means is applied;

acquiring third video data by delaying said first video data corresponding to processing time in the step for acquiring the viewpoint information of said photographing means and the step for generating said second video data;

forecasting a viewpoint of said photographing means at current time or at time posterior to the current time based upon the viewpoint information of said photographing means acquired in the step for acquiring the viewpoint of said photographing means;

generating fourth video data corresponding to the projected picture of said predetermined object to which perspective transformation in a case viewed from said forecast viewpoint of said photographing means is applied;

generating a first key signal for separating a first picture element having the predetermined color of the predetermined background or a color similar thereto and a second picture element which is not the predetermined color of the predetermined background, based on said first video data;

generating a second key signal for separating the first picture element having the predetermined color of the predetermined background or a color similar thereto and the second picture element which is not the predetermined color of the predetermined background, based on said first video data and for acquiring a picture switching signal for fetching either of said first video data or said fourth video data based upon said first video data;

acquiring first synthetic video data by synthesizing said third video data and said second video data on the basis of said first key signal;

acquiring second synthetic video data by switching between said first video data and said fourth video data on the basis of said second key signal and synthesizing them; and

displaying a picture for assisting the performance of said performer based upon said second synthetic video data.

8. The picture generating of claim 7, wherein said step of forecasting the viewpoint of said photographing means further comprising the steps of:

linearly forecasting a position of the photographing means at current time based upon positional information (T1, T2) of the photographing means at the past two times according to a formula T2+(T2-T1), and

linearly forecasting an orientation of the photographing means at current time based upon orientation information (R1, R2) of the photographing means at the past two times according to a formula R2+(R2-R1).

9. A picture generator, comprising:

a mobile camera for photographing a performer in front of a predetermined background having a predetermined color to obtain first video data, said background including a plurality of inconspicuous markers each having a color similar to the predetermined color of the predetermined background;

a camera parameter calculator for acquiring viewpoint information of said mobile camera including three-dimensional position and orientation of the mobile camera on the basis of predetermined world coordinates;

a first video data generator for generating second video data corresponding to a projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of said mobile camera acquired in said camera parameter calculator is applied;

a data delay device for acquiring third video data delayed corresponding to time for processing said first video data in said camera parameter calculator and said first video data generator;

a first key signal generator for generating a first key signal for separating a first picture element having the predetermined color of the predetermined background or a color similar thereto and a second picture element which is not the predetermined color of the predetermined background, based on said first video data;

a second key generator for generating a second key signal for separating the first picture element having the predetermined color of the predetermined background or a color similar thereto and the second picture element which is not the predetermined color of the predetermined background, based on said first video data and for acquiring a picture switching signal for fetching either of said first video data or said second video data based upon said first video data;

a first video data synthesizer for acquiring first synthetic video data by synthesizing said third video data and said second video data on the basis of said first key signal;

a second video data synthesizer for acquiring second synthetic video data by switching between said first video data and said second video data on the basis of said second key signal and synthesizing them; and

a picture display for displaying a picture for assisting the performance of said performer based upon said second synthetic video data.

10. A picture generator according to claim 9, wherein said mobile camera further comprises a turning base, on which the mobile camera is put, which turns said mobile camera using a gear, and wherein an orientation of said mobile camera is based upon a quantity in which said gear is rotated.

11. A picture generator comprising:

a mobile camera for photographing a performer in front of a predetermined background having a predetermined color to obtain first video data, said background including a plurality of inconspicuous markers each having a color similar to the predetermined color of the predetermined background;

a camera parameter calculator for acquiring viewpoint information of said mobile camera including three-dimensional position and orientation of the mobile camera on the basis of predetermined world coordinates using three-dimensional position and orientation of said markers obtained from said first video data on the basis of coordinates relative to said mobile camera, and predetermined three-dimensional position information of said markers relative to said world coordinates;

a first video data generator for generating second video data corresponding to a projected picture of a predetermined object to which perspective transformation in a case viewed from the viewpoint of said mobile camera acquired in said camera parameter calculator is applied;

a camera parameter forecaster for forecasting the viewpoint of said mobile camera at current time or at time posterior to the current time based upon the viewpoint information of said mobile camera acquired in said camera parameter calculator;

a second video data generator for generating fourth video data corresponding to the projected picture of said predetermined object to which perspective transformation in a case viewed from the viewpoint forecast in said camera parameter forecaster of said mobile camera is applied;

a data delay device for acquiring third video data delayed corresponding to time for processing said first video data in said camera parameter calculator and said first video data generator;

a first key signal generator for generating a first key signal for separating a first picture element having the predetermined color of the predetermined background or a color similar thereto and a second picture element which is not the predetermined color of the predetermined background, based on said first video data;

a second key generating means for generating a second key signal for separating the first picture element having the predetermined color of the predetermined background or a color similar thereto and the second picture element which is not the predetermined color of the predetermined background, based on said first video data and for acquiring a picture switching signal for fetching either of said first video data or said fourth video data based upon said first video data;

a first video data synthesizer for acquiring first synthetic video data by synthesizing said third video data and said fourth video data on the basis of said first key signal;

a second video data synthesizer for acquiring second synthetic video data by switching between said first video data and said fourth video data on the basis of said second key signal and synthesizing them; and

a picture display for displaying a picture for assisting the performance of said performer based upon said second synthetic video data.

12. The picture generator according to claim 11, wherein said camera parameter forecaster linearly forecasts a position of the mobile camera at current time based upon positional information (T1, T2) of the mobile camera at the past two times according to a formula T2+(T2-T1), and

wherein said camera parameter forecaster linearly forecasts an orientation of the mobile camera at current time based upon orientation information (R1, R2) of the mobile camera at the past two times according to a formula R2+(R2-R1).