Image processing apparatus, image processing method, studio apparatus, storage medium, and program

• US 20040041822A1
• Filed: 09/04/2003
• Published: 03/04/2004
• Est. Priority Date: 03/13/2001
• Status: Abandoned Application

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Sato, Hiroaki, Iizuka, Yoshio, Noro, Hideo, Kawai, Tomoaki, Ono, Eita
• Application Number:
  US10/654,014
• Publication Number:
  US 20040041822A1
• Time in Patent Office: Days
• US Class Current: 345/634
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

This invention provides an image processing method, image processing apparatus, storage medium, and program, which can remove the boundary between real and virtual worlds. To this end, an apparatus has a image input device (101), the image input parameters of which are controllable, a position/posture sensor (102) for acquiring the image input parameters, a CG data management unit (108) for managing CG (computer graphics) data, a CG geometric information calculation unit (110) for calculating CG geometric information upon virtually laying out the CG data in the real world, a CG image generator (109) for generating a CG image from the viewpoint of the image input device (101), a image composition device (113) for compositing a real image and CG image, and a data processing device (107) for changing the image input parameters using the image input parameters and the CG geometric information.

Claims

1. An image processing method comprising:

a image input step of taking an image using image input means, a image input parameter of which is controllable;

a image input parameter acquisition step of acquiring the image input parameter;

a CG data management step of managing CG (computer graphics) data;

a CG geometric information calculation step of calculating CG geometric information upon virtually laying out the CG data in a real world;

a CG image generation step of generating a CG image from a viewpoint of the image input means;

a image composition step of compositing a real image and the CG image; and

a image input parameter control step of changing the image input parameter using the image input parameter and the CG geometric information.

2. The method according to claim 1, further comprising the instruction input step of inputting an instruction from an operator, and wherein the image input parameter control step includes the step of changing the image input parameter additionally using the instruction from the instruction input step.

3. The method according to claim 1, further comprising the instruction input step of inputting an instruction from an operator, and wherein the CG geometric information calculation step includes the step of calculating the CG geometric information additionally using the instruction from the instruction input step.

4. The method according to claim 1, wherein a plurality of image input means equivalent to the image input means are equipped, and said method further comprises the viewpoint selection step of selecting one of the plurality of image input means.

5. The method according to claim 1, further comprising:

a display geometric information measurement step of measuring a display parameter of display means;

a second CG image generation step of generating a second CG image from a viewpoint of the display means; and

a display step of displaying the second CG image on the display means.

6. The method according to claim 1, wherein the image input parameter is a image input position of the image input means.

7. The method according to claim 1, wherein the image input parameter is a image input direction of the image input means.

8. The method according to claim 5, wherein the display parameter is a display position of the display means.

9. The method according to claim 5, wherein the display parameter is a display direction of the display means.

10. The method according to claim 1, wherein the display means is an HMD (head-mounted display).

11. The method according to claim 10, wherein the HMD is a see-through HMD of a type that allows external light to pass through a region where no image is displayed.

12. An image processing apparatus comprising:

a image input means, a image input parameter of which is controllable;

a image input parameter acquisition means that acquires the image input parameter;

a CG data management means that manages CG (computer graphics) data;

a CG geometric information calculation means that calculates CG geometric information upon virtually laying out the CG data in a real world;

a CG image generation means that generates a CG image from a viewpoint of said image input means;

a image composition means that composites a real image and the CG image; and

a image input parameter control means that changes the image input parameter using the image input parameter and the CG geometric information.

13. An image processing method comprising:

a image input step of image inputting an image using image input means;

a studio set step of forming a background;

a display step of displaying an image using display means that a staff member associated with an image process wears;

a first measurement step of measuring a image input parameter of the image input means;

a second measurement step of measuring a display parameter of the display means;

a CG data management step of managing CG (computer graphics) data;

a first CG image generation step of generating a CG image from a viewpoint of the image input means;

a image composition step of compositing an image taken by the image input means, and the CG image generated in the first CG image generation step;

a second CG image generation step of generating a CG image from a viewpoint of the display means;

a image superimpose step of superimposing the CG image on a real space that can be seen from the display means;

a image broadcast step of broadcasting an image composited in the image composition step;

a viewer information management step of managing viewer information;

a scenario management step of setting the viewer information in a portion of a scene; and

a prohibited region processing step of controlling a range in which a CG object is present.

14. The method according to claim 13, wherein the display means is an HMD (head-mounted display).

15. The method according to claim 13, wherein the image input parameter is a image input position of the image input means.

16. The method according to claim 13, wherein the image input parameter is a image input direction of the image input means.

17. The method according to claim 13, wherein the display parameter is a display position of the display means.

18. The method according to claim 13, wherein the display parameter is a display direction of the display means.

19. The method according to claim 13, wherein viewers can be displayed as virtual viewer in the image composition step and the scenario management step.

20. The method according to claim 19, further comprising the limitation step of limiting the number of CG characters displayed as viewer.

21. The apparatus according to claim 20, further comprising the control step of controlling to display a count result of all pieces of viewer information to the virtual viewer.

22. The method according to claim 20, further comprising the selection step of selecting viewers to be displayed as the virtual viewer by drawing.

23. The method according to claim 21, wherein the control step includes the step of controlling to display the count result of the viewer information on only a display screen of the display means.

24. The method according to claim 19, wherein the viewers can help a player via the virtual viewer.

25. An image processing apparatus comprising:

a image input means that image input an image;

a studio set means that forms a background;

a display means, worn by a staff member associated with an image process, for displaying an image;

a first measurement means that measures a image input parameter of said image input means;

a second measurement means that measures a display parameter of said display means;

a CG data management means that manages CG (computer graphics) data;

a first CG image generation means that generates a CG image from a viewpoint of said image input means;

a image composition means that composites an image taken by said image input means, and the CG image generated by said first CG image generation means;

a second CG image generation means that generates a CG image from a viewpoint of said display means;

a image superimpose means that superimposes the CG image on a real space that can be seen from said display means;

a image broadcast means that broadcasts an image composited by said image composition means;

a viewer information management means that manages viewer information;

a scenario management means that sets the viewer information in a portion of a scene; and

a prohibited region processing means that controls a range in which a CG object is present.

26. A studio apparatus comprising an image processing apparatus cited in claim 25.

27. A storage medium that stores a computer-readable control program for controlling an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a image input step of taking an image using image input means, a image input parameter of which is controllable;

a image input parameter acquisition step of acquiring the image input parameter;

a CG data management step of managing CG (computer graphics) data;

a CG geometric information calculation step of calculating CG geometric information upon virtually laying out the CG data in a real world;

a CG image generation step of generating a CG image from a viewpoint of the image input means;

a image composition step of compositing a real image and the CG image; and

a image input parameter control step of changing the image input parameter using the image input parameter and the CG geometric information.

28. A storage medium that stores a computer-readable control program for controlling an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a image input step of image inputting an image using image input means;

a studio set step of forming a background;

a display step of displaying an image using display means that a staff member associated with an image process wears;

a first measurement step of measuring a image input parameter of the image input means;

a second measurement step of measuring a display parameter of the display means;

a CG data management step of managing CG (computer graphics) data;

a first CG image generation step of generating a CG image from a viewpoint of the image input means;

a image composition step of compositing an image taken by the image input means, and the CG image generated in the first CG image generation step;

a second CG image generation step of generating a CG image from a viewpoint of the display means;

a image superimpose step of superimposing the CG image on a real space that can be seen from the display means;

a image broadcast step of broadcasting an image composited in the image composition step;

a viewer information management step of managing viewer information;

a scenario management step of setting the viewer information in a portion of a scene; and

a prohibited region processing step of controlling a range in which a CG object is present.

29. A computer-readable control program for controlling an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a image input step of taking an image using image input means, a image input parameter of which is controllable;

a image input parameter acquisition step of acquiring the image input parameter;

a CG data management step of managing CG (computer graphics) data;

a CG geometric information calculation step of calculating CG geometric information upon virtually laying out the CG data in a real world;

a CG image generation step of generating a CG image from a viewpoint of the image input means;

a image composition step of compositing a real image and the CG image; and

a image input parameter control step of changing the image input parameter using the image input parameter and the CG geometric information.

30. A computer-readable control program for controlling an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a image input step of image inputting an image using image input means;

a studio set step of forming a background;

a display step of displaying an image using display means that a staff member associated with an image process wears;

a first measurement step of measuring a image input parameter of the image input means;

a second measurement step of measuring a display parameter of the display means;

a CG data management step of managing CG (computer graphics) data;

a first CG image generation step of generating a CG image from a viewpoint of the image input means;

a image composition step of compositing an image taken by the image input means, and the CG image generated in the first CG image generation step;

a second CG image generation step of generating a CG image from a viewpoint of the display means;

a image superimpose step of superimposing the CG image on a real space that can be seen from the display means;

a image broadcast step of broadcasting an image composited in the image composition step;

a viewer information management step of managing viewer information;

a scenario management step of setting the viewer information in a portion of a scene; and

a prohibited region processing step of controlling a range in which a CG object is present.

31. An image processing method comprising:

a tracking step of measuring a position/posture of an object such as a performer or the like; and

an affecting CG data step of reflecting the position/posture obtained in the tracking step in CG (computer graphics) data to be superimposed on an image of the object.

32. An image processing apparatus comprising:

a tracking means that measures a position/posture of an object such as a performer or the like; and

an affecting CG data means that reflects the position/posture obtained by said tracking means in CG (computer graphics) data to be superimposed on an image of the object.

33. An image processing method for measuring a position/posture of an object such as a performer or the like, and reflecting the measured position/posture in CG (computer graphics) data to be superimposed on an image of the object to display the CG data on display means, comprising:

a image input step of image inputting the object using image input means;

a CG image generation step of generating a CG image from a viewpoint of the image input means on the basis of a image input parameter of the image input means and a display parameter of the display means;

a image composition step of compositing a real image of the object taken by the image input means with the CG image generated in the CG image generation step, and displaying a composite image on the display means; and

a prohibited region processing step of limiting in the image composition step a range in which the CG image is present.

34. The method according to claim 33, wherein the CG image generation step comprises the scenario management step of managing scenario information used to generate the CG.

35. The method according to claim 33, wherein the image input parameter is a image input position of the image input means.

36. The method according to claim 33, wherein the image input parameter is a image input direction of the image input means.

37. The method according to claim 33, wherein the display parameter is a display position of the display means.

38. The method according to claim 33, wherein the display parameter is a display direction of the display means.

39. The method according to claim 33, wherein the display means is an HMD (head-mounted display).

40. An image processing apparatus for measuring a position/posture of an object such as a performer or the like, and reflecting the measured position/posture in CG (computer graphics) data to be superimposed on an image of the object to display the CG data on display means, comprising:

image input means that image input the object;

CG image generation means that generates a CG image from a viewpoint of said image input means on the basis of a image input parameter of said image input means and a display parameter of the display means;

image composition means that composites a real image of the object taken by said image input means with the CG image generated by said CG image generation means, and displaying a composite image on the display means; and

prohibited region processing means that limits in an image composition process of said image composition means a range in which the CG image is present.

41. A storage medium that stores a computer-readable control program for controlling an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a tracking step of measuring a position/posture of an object such as a performer or the like; and

an affecting CG data step of reflecting the position/posture obtained in the tracking step in CG (computer graphics) data to be superimposed on an image of the object.

42. A storage medium that stores a computer-readable control program for controlling an image process for measuring a position/posture of an object such as a performer or the like, and reflecting the measured position/posture in CG (computer graphics) data to be superimposed on an image of the object to display the CG data on display means in an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a image input step of image inputting the object using image input means;

a image generation step of generating a CG image from a viewpoint of the image input means on the basis of a image input parameter of the image input means and a display parameter of the display means;

a image composition step of compositing a real image of the object taken by the image input means with the CG image generated in the CG image generation step, and displaying a composite image on the display means; and

a prohibited region processing step of limiting in the image composition step a range in which the CG image is present.

43. The medium according to claim 42, further comprising the studio set step of forming a background, the display step of displaying an image using display means that a performer or a person associated with image input wears, and the scenario management step of managing scenario information used to generate the CG image.

44. A computer-readable control program for controlling an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a tracking step of measuring a position/posture of an object such as a performer or the like; and

an affecting CG data step of reflecting the position/posture obtained in the tracking step in CG (computer graphics) data to be superimposed on an image of the object.

45. A computer-readable control program for controlling an image process for measuring a position/posture of an object such as a performer or the like, and reflecting the measured position/posture in CG (computer graphics) data to be superimposed on an image of the object to display the CG data on display means in an image processing apparatus for processing a real image and a CG (computer graphics) image, comprising a program code for making a computer execute:

a image input step of image inputting the object using image input means;

a CG image generation step of generating a CG image from a viewpoint of the image input means on the basis of a image input parameter of the image input means and a display parameter of the display means;

a image composition step of compositing a real image of the object taken by the image input means with the CG image generated in the CG image generation step, and displaying a composite image on the display means; and

a prohibited region processing step of limiting in the image composition step a range in which the CG image is present.

46. The program according to claim 45, further comprising the studio set step of forming a background, the display step of displaying an image using display means that a performer or a person associated with image input wears, and the scenario management step of managing scenario information used to generate the CG image.