Method and apparatus for merging real and synthetic images

• US 6,151,009 A
• Filed: 08/21/1996
• Issued: 11/21/2000
• Est. Priority Date: 08/21/1996
• Status: Expired due to Fees

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Tanaka, Masaya, Kano, Hiroshi, Yoshida, Atsushi, Oda, Kazuo, Kanade, Takeo
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): TRINH, WILLIAM B
• Application Number:
  US08/697,213
• Time in Patent Office: 1,553 Days
• Field of Search: 345/113, 345/585, 345/114, 345/115, 348/591, 348/590
• US Class Current: 345/641
• CPC Class Codes: G06T 15/405 using Z-buffer
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method for merging real and synthetic images in real time is comprised of the steps of providing a first signal containing depth and image information per pixel about a real image. A second signal containing depth and image information per pixel about a synthetic image is provided. The depth information corresponding to the real image and the depth information corresponding to the synthetic image for each pixel are compared. Based on the comparison, either the image information corresponding to the real image or the image information corresponding to the synthetic image is selected and combined. Because the image information is compared based on depth, any interaction such as occluding, shadowing, reflecting, or colliding can be determined and an appropriate output generated

Claims

1. A method of merging a real image and a synthetic image in real time, comprising:

providing a first signal containing depth and image information per pixel about said real image;

providing a second signal containing depth and image information per pixel about said synthetic image;

comparing said depth information corresponding to said real image and said depth information corresponding to said synthetic image for each pixel;

selecting one of said image information corresponding to said real image and said image information corresponding to said synthetic image in response to said comparison step for each pixel;

combining the selected information to produce an output signal;

calculating an interaction between images in the first signal and the second signal, wherein said interaction is selected from the group consisting of occlusions, collisions, shadows, and reflections; and

generating output signals based on said calculated interaction.

2. The method of claim 1 wherein said generating step includes the step of generating an audible output signal.

3. The method of claim 1 wherein said generating step includes the step of generating a visible output signal.

4. The method of claim 1 additionally comprising the step of producing an image mask for an object in said first signal and wherein said comparing step includes the step of comparing said depth information corresponding to only said image mask and said depth information corresponding to said synthetic image for each pixel.

5. The method of claim 1 additionally comprising the step of producing an image mask for an object in said second signal, and wherein said comparing step includes the step of comparing depth information corresponding to said real image and said depth information corresponding to only said image mask for each pixel.

6. A method of merging images in real time, comprising:

providing first and second signals containing depth and image information per pixel about first and second images, wherein said first and second images are selected from the group consisting of real images and synthetic images;

producing an image mask for an object in said second signal;

comparing said depth information corresponding to said first image and said depth information corresponding to said second image for each pixel, said comparing includes comparing depth information corresponding to said first image and said depth information corresponding to only said image mask for each pixel;

selecting one of said image information corresponding to said first image and said image information corresponding to said second image in response to said comparison step for each pixel; and

combining the selected information to produce an output signal.

7. The method of claim 6 additionally comprising the steps of calculating an interaction between images in the first signal and the second signal and generating output signals based on said calculated interaction, wherein said interaction is selected from the group consisting of collisions, shadows, and reflections.

8. The method of claim 7 wherein said generating step includes the step of generating an audible output signal.

9. The method of claim 7 wherein said generating step includes the step of generating a visible output signal.

10. A method of determining interactions between objects in a real image and objects in a synthetic image, comprising:

providing a first signal containing depth and image information per pixel about said real image;

providing a second signal containing depth and image information per pixel about said synthetic image;

comparing said depth information corresponding to said real image and said depth information corresponding to said synthetic image for each pixel; and

calculating interaction based on said comparison step, wherein said interaction is selected from the group consisting of occlusions, collisions, shadows, and reflections.

11. A method of merging a real image and a synthetic image in real time, comprising:

providing a first signal containing depth and image information per pixel about said real image;

providing a second signal containing depth and image information per pixel about said synthetic image;

producing an image mask for an object in said first signal;

comparing said depth information corresponding to said real image and said depth information corresponding to said synthetic image for each pixel, said comparing includes comparing said depth information corresponding to only said image mask and said depth information corresponding to said synthetic image for each pixel;

selecting one of said image information corresponding to said real image and said image information corresponding to said synthetic image in response to said comparison step for each pixel; and

combining the selected information to produce an output signal.

12. The method of claim 11 additionally comprising the steps of calculating an interaction between images in the first signal and the second signal and generating output signals based on said calculated interaction, wherein said interaction is selected from the group consisting of occlusions, collisions, shadows, and reflections.

13. The method of claim 12 wherein said generating step includes the step of generating an audible output signal.

14. The method of claim 12 wherein said generating step includes the step of generating a visible output signal.

15. A method of merging a real image and a synthetic image in real time, comprising:

providing a first signal containing depth and image information per pixel about said real image;

providing a second signal containing depth and image information per pixel about said synthetic image;

producing an image mask for an object in said second signal;

comparing said depth information corresponding to said real image and said depth information corresponding to said synthetic image for each pixel, said comparing includes comparing depth information corresponding to said real image and said depth information corresponding to only said image mask for each pixel;

selecting one of said image information corresponding to said real image and said image information corresponding to said synthetic image in response to said comparison step for each pixel; and

combining the selected information to produce an output signal.

16. The method of claim 15 additionally comprising the steps of calculating an interaction between images in the first signal and the second signal and generating output signals based on said calculated interaction, wherein said interaction is selected from the group consisting of occlusions, collisions, shadows, and reflections.

17. The method of claim 16 wherein said generating step includes the step of generating an audible output signal.

18. The method of claim 16 wherein said generating step includes the step of generating a visible output signal.

19. A method of merging images in real time, comprising:

providing first and second signals containing depth and image information per pixel about first and second images, wherein said first and second images are selected from the group consisting of real images and synthetic images;

comparing said depth information corresponding to said first image and said depth information corresponding to said second image for each pixel;

selecting one of said image information corresponding to said first image and said image information corresponding to said second image in response to said comparison step for each pixel;

combining the selected information to produce an output signal;

calculating an interaction between images in said first and second signals wherein said interaction is selected from the group consisting of collisions, shadows, and reflections; and

generating output signals based on said calculated interaction.

20. The method of claim 19 wherein said generating step includes the step of generating an audible output signal.

21. The method of claim 19 wherein said generating step includes the step of generating a visible output signal.

22. The method of claim 19 additionally comprising the step of producing an image mask for an object in said first signal and wherein said comparing step includes the step of comparing said depth information corresponding to only said image mask and said depth information corresponding to said second image for each pixel.

23. The method of claim 19 additionally comprising the step of producing an image mask for an object in said second signal, and wherein said comparing step includes the step of comparing depth information corresponding to said first image and said depth information corresponding to only said image mask for each pixel.

24. A method of determining interactions between objects in a first image and objects in a second image, comprising:

providing first and second signals containing depth and image information per pixel about said first and second images, wherein said first and second images are selected from the group consisting of real images and synthetic images;

comparing said depth information corresponding to said first image and said depth information corresponding to said second image for each pixel; and

calculating an interaction based on said comparison step, wherein said interaction is selected from the group consisting of collisions, shadows, and reflections.

25. A method of merging images in real time, comprising:

providing first and second signals containing depth and image information per pixel about first and second images, wherein said images are selected from the group consisting of real images and synthetic images;

producing an image mask for an object in said first signal;

comparing said depth information corresponding to said first image and said depth information corresponding to said second image for each pixel, said comparing includes comparing said depth information corresponding to only said image mask and said depth information corresponding to said second image for each pixel;

selecting one of said image information corresponding to said first image and said image information corresponding to said second image in response to said comparison step for each pixel; and

combining the selected information to produce an output signal.

26. The method of claim 25 additionally comprising the steps of calculating an interaction between images in the first signal and the second signal and generating output signals based on said calculated interaction, wherein said interaction is selected from the group consisting of collisions, shadows, and reflections.

27. The method of claim 26 wherein said generating step includes the step of generating an audible output signal.

28. The method of claim 26 wherein said generating step includes the step of generating a visible output signal.