Image-display control apparatus

• US 6,822,643 B2
• Filed: 11/13/2001
• Issued: 11/23/2004
• Est. Priority Date: 11/17/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Morita, Kenji, Okuno, Yasuhiro, Matsui, Taichi
• Primary Examiner(s): Nguyen, Jimmy H.
• Application Number:
  US09/986,915
• Publication Number:
  US 20020060648A1
• Time in Patent Office: 1,106 Days
• Field of Search: 345/204, 345/157, 345/158, 345/8, 345/9, 382/103, 382/111
• US Class Current: 345/204
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

An image-display control apparatus includes a detection unit for detecting a predetermined object from a photographed image, a position calculation unit for obtaining a positional relationship between the predetermined object in the photographed image detected by the detection unit and an object in a virtual image, an image removing unit for removing a predetermined partial image in the object in the virtual image by referring to a result of calculation of the position calculation unit, a synthesis unit for synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed by the image removing unit, and a display control unit for displaying an image obtained as a result of synthesis of the synthesis unit on a display device.

Claims

1. An image-display control apparatus comprising:

detection means for detecting a predetermined object from a photographed image;

position calculation means for obtaining a positional relationship between the predetermined object in the photographed image detected by said detection means and an object in a virtual image;

image removing means for removing a predetermined partial image in the object in the virtual image by referring to a calculation result of calculation of said position calculation means;

synthesis means for synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed by said image removing means; and

display control means for displaying an image obtained as a synthesis result of of said synthesis means on display means.

2. An apparatus according to claim 1, further comprising:

image input means for inputting an image photographed by a camera for photographing a real space as the photographed image; and

assumption means for assuming a position and a direction of an object in the real space,

wherein said detection means detects the predetermined object from the photographed image input from said image input means, based on an assumption result of said assumption means.

3. An apparatus according to claim 2, wherein said assumption means assumes the position and a posture of the object from detection information of position/direction detection means for detecting the position and the direction of the object in the real space.

4. An apparatus according to claim 3, further comprising holding means for holding the detection information of the position/direction detection means when the object in the real space has a predetermined posture, wherein said assumption means assumes the position and the posture of the object in the real space based on the detection information in said holding means.

5. An apparatus according to claim 4, further comprising mask formation means for forming an external mask of the predetermined object in the photographed image obtained from said image input means, based on the position and the posture of the object in the real space assumed by said assumption means, wherein said image removing means removes the predetermined partial image in the virtual image using the external mask.

6. An apparatus according to claim 5, further comprising image formation means for forming the virtual image.

7. An image-display control apparatus comprising:

detection means for detecting a predetermined object from each of first and second photographed images obtained by photographing a real space;

position calculation means for obtaining a positional relationship between the predetermined objects in the first and second photographed images detected by said detection means and an object in a virtual image;

image removing means for removing predetermined partial images in the object in the virtual image by referring to a calculation result of calculation of said position calculation means;

synthesis means for forming a first synthesized image by synthesizing the virtual image and the first photographed image and forming a second synthesized image by synthesizing the virtual image and the second photographed image so that the predetermined objects in the first and second photographed images are present at respective positions of the partial images in the virtual image removed by said image removing means; and

display control means for displaying the first synthesized image obtained by said synthesis means on display means for a left eye, and displaying the second synthesized image on display means for a right eye.

8. An apparatus according to claim 7, further comprising:

image input means for inputting respective images photographed by first and second cameras for photographing a real space as the first and second photographed images; and

assumption means for assuming a position and a direction of an object in the real space,

wherein said detection means detects the predetermined objects from the first and second photographed images input from said image input means, based on an assumption result of said assumption means.

9. An apparatus according to claim 8, wherein said assumption means assumes the position and a posture of the object from detection information of position/direction detection means for detecting the position and the direction of each of the object in the real space.

10. An apparatus according to claim 9, further comprising holding means for holding the detection information of the position/direction detection means when the object in the real space has a predetermined posture, wherein said assumption means assumes the position and the posture of the object in the real space based on the detection information in said holding means.

11. An apparatus according to claim 10, further comprising mask formation means for forming an external mask of each of the predetermined objects in the first and second photographed images obtained from said image input means, based on the position and the posture of the object in the real space assumed by said assumption means, wherein said image removing means removes the predetermined partial image in the virtual image using the external mask.

12. An apparatus according to claim 11, further comprising image formation means for forming the virtual image.

13. An image processing system comprising:

a camera for photographing a real space;

a position/direction sensor for detecting a position and a direction of an object in the real space;

a computer for synthesizing a photographed image photographed by said camera and a virtually formed virtual image; and

display means for displaying an image obtained as a result of synthesis by said computer,

said computer comprising;

input means for inputting the photographed image photographed by said camera;

detection means for detecting a predetermined object from the photographed image;

position calculation means for obtaining a positional relationship between the predetermined object in the photographed image detected by said detection means and an object in a virtual image;

image removing means for removing a predetermined partial image in the object in the virtual image by referring to a calculation result of calculation of said position calculation means;

synthesis means for synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed by said image removing means; and

display control means for displaying an image obtained as a synthesis result of said synthesis means on said display means.

14. A system according to claim 13, wherein said computer further comprises assumption means for assuming a position and a direction of an object in the real space, and wherein said detection means detects the predetermined object from the photographed image input from said image input means, based on an assumption result of said assumption means.

15. A system according to claim 14, wherein said assumption means assumes the position and a posture of the object from detection information of position/direction detection means for detecting the position and the direction of the object in the real space.

16. A system according to claim 15, wherein said computer comprises holding means for holding the detection information of the position/direction detection means when the object in the real space has a predetermined posture, and wherein said assumption means assumes the position and the posture of the object in the real space based on the detection information in said holding means.

17. A system according to claim 16, wherein said computer comprises mask formation means for forming an external mask of the predetermined object in the photographed image obtained from said image input means, based on the position and the posture of the object in the real space assumed by said assumption means, and wherein said image removing means removes the predetermined partial image in the virtual image using the external mask.

18. A system according to claim 17, further comprising image formation means for forming the virtual image.

19. An image generation system comprising:

first and second cameras for photographing the same real space;

a position/direction sensor for detecting a position and a direction of an object in the real space;

a computer for outputting first and second synthesized images by synthesizing first and second photographed images photographed by said first and second cameras, respectively, and a virtually formed virtual image; and

display means for a right eye and display means for a left eye for displaying the first and second synthesized images, respectively,

said computer comprising;

image input means for inputting the first and second photographed images;

detection means for detecting respective predetermined objects from the first and second photographed images;

position calculation means for obtaining a positional relationship between the predetermined objects in the first and second photographed images detected by said detection means and an object in a virtual image;

image removing means for removing predetermined partial images in the object in the virtual image by referring to a calculation result of said position calculation means;

synthesis means for forming a first synthesized image by synthesizing the virtual image and the first photographed image and forming a second synthesized image by synthesizing the virtual image and the second photographed image so that the predetermined objects in the first and second photographed images are present at respective positions of the partial images of the virtual image removed by said image removing means; and

display control means for displaying the first synthesized image obtained by said synthesis means on display means for a left eye, and displaying the second synthesized image on display means for a right eye.

20. A system according to claim 19, wherein said computer comprises assumption means for assuming a position and a direction of an object in the real space, and wherein said detection means detects each of the predetermined objects from the first and second photographed images input from said image input means, based on an assumption result of said assumption means.

21. A system according to claim 20, wherein said assumption means assumes the position and a posture of the object from detection information of position/direction detection means for detecting the position and the direction of the object in the real space.

22. A system according to claim 21, wherein said computer further comprises holding means for holding the detection information of the position/direction detection means when the object in the real space has a predetermined posture, and wherein said assumption means assumes the position and the posture of the object in the real space based on the detection information in said holding means.

23. A system according to claim 22, wherein said computer further comprises mask formation means for forming an external mask of each of the predetermined objects in the first and second photographed images obtained from said image input means, based on the position and the posture of the object in the real space assumed by said assumption means, and wherein said image removing means removes the predetermined partial image in the virtual image using the external mask.

24. A system according to claim 23, wherein said computer further comprises image formation means for forming the virtual image.

25. An image-display control method comprising:

a detection step of detecting a predetermined object from a photographed image;

a position calculation step of obtaining a positional relationship between the predetermined object in the photographed image detected in said detection step and an object in a virtual image;

an image removing step of removing a predetermined partial image in the object in the virtual image by referring to a calculation result in said position calculation step;

a synthesis step of synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed in said image removing step; and

a display control step of displaying an image obtained as a synthesis result in said synthesis step on display means.

26. A method according to claim 25, further comprising:

an image input step of inputting an image photographed by a camera for photographing a real space as the photographed image; and

an assumption step of assuming a position and a direction of an object in the real space,

wherein in said detection step, the predetermined object is detected from the photographed image input in said image input step, based on an assumption result in said assumption step.

27. A method according to claim 26, wherein in said assumption step, the position and a posture of the object are assumed from detection information of position/direction detection means for detecting the position and the direction of the object in the real space.

28. A method according to claim 27, wherein the detection information of the position/direction detection means when the object in the real space has a predetermined posture is held, and wherein in said assumption step, the position and the posture of the object in the real space are assumed based on the held detection information in the position/direction detection means.

29. A method according to claim 28, wherein an external mask of the predetermined object in the photographed image obtained in said image input step is formed, based on the position and the posture of the object in the real space assumed in said assumption step, and wherein in said image removing step, the predetermined partial image in the virtual image is removed using the external mask.

30. A method according to claim 29, further comprising an image formation step of forming the virtual image.

31. A storage medium storing a processing program for said steps of said method according to claim 25.

32. An image-display control method comprising:

a detection step of detecting a predetermined object from each of first and second photographed images obtained by photographing the same real space;

a position calculation step of obtaining a positional relationship between the predetermined objects in the first and second photographed images detected in said detection step and an object in a virtual image;

an image removing step of removing predetermined partial images in the object in the virtual image by referring to a calculation result in said position calculation step;

a synthesis step of forming a first synthesized image by synthesizing the virtual image and the first photographed image and forming a second synthesized image by synthesizing the virtual image and the second photographed image so that the predetermined objects in the first and second photographed images are present at respective positions of the partial images of the virtual image removed in said image removing step; and

a display control step of displaying the first synthesized image obtained in said synthesis step on display means for a left eye, and displaying the second synthesized image on display means for a right eye.

33. A method according to claim 32, further comprising:

an image input step of inputting respective images photographed by first and second cameras for photographing the same real space as the first and second photographed images; and

an assumption step of assuming a position and a direction of an object in the real space,

wherein in said detection step, the predetermined objects are detected from the first and second photographed images input in said image input step, based on an assumption result assumption in said assumption step.

34. A method according to claim 33, wherein in said assumption step, the position and a posture of each of the objects are assumed from detection information of position/direction detection means for detecting the position and the direction of each of the objects in the real space.

35. A method according to claim 34, wherein the detection information of the position/direction detection means when each of the objects in the real space has a predetermined posture is held, and wherein in said assumption step, the position and the posture of each of the objects in the real space are assumed based on the held detection information.

36. A method according to claim 35, wherein an external mask of each of the predetermined objects in the first and second photographed images obtained in said image input step is formed, based on the position and the posture of each of the objects in the real space assumed in said assumption step, and wherein in said image removing step, the predetermined partial image in the virtual image is removed using the external mask.

37. A method according to claim 36, further comprising an image formation step of forming the virtual image.

38. A storage medium storing a processing program for said steps of said method according to claim 32.