Image generating system and method and storage medium

• US 20020075286A1
• Filed: 11/15/2001
• Published: 06/20/2002
• Est. Priority Date: 11/17/2000
• Status: Abandoned Application

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Morita, Kenji, Yonezawa, Hiroki
• Application Number:
  US10/000,668
• Publication Number:
  US 20020075286A1
• Time in Patent Office: Days
• US Class Current: 345/679
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/008 Aspects relating to glasses for viewing stereoscopic images

Abstract

An image generating system is provided which is capable of reducing the time difference between the real space image and the virtual space image to thereby provide a more real composite image for the observer. A video camera captures an image of a real space at an observer'"'"'s eye position and in the observer'"'"'s line-of-sight direction. A line-of-sight detecting device detects the observer'"'"'s eye position and line-of-sight direction. A virtual space image generator generates a virtual space image at the observer'"'"'s eye position and in the observer'"'"'s line-of-sight direction, the position and orientation being detected by the line-of-sight detecting device. A composite image generator generates a composite image by synthesizing the virtual space image generated by the virtual space image generator and a real space image outputted by the video camera. A display device displays the composite image generated by the composite image generator. A managing system collectively manages information on objects present in the real space and the virtual space as well as locations and orientations thereof.

Claims

1. An image generating system comprising:

image pickup means for capturing an image of a real space at an eye position of an observer and in a line-of-sight direction of the observer;

detecting means for detecting the eye position of the observer and the line-of-sight direction of the observer;

virtual space image generating means for generating an image of a virtual space at the eye position of the observer and in the line-of-sight direction of the observer, the eye position and the line-of-sight direction being detected by said detecting means;

composite image generating means for generating a composite image by synthesizing the image of the virtual space generated by said virtual space image generating means and the image of the real space outputted by said image pickup means;

display means for displaying the composite image generated by said composite image generating means; and

managing means for collectively managing information on objects present in the real space and the virtual space as well as locations and orientations thereof.

2. An image generating system according to claim 1, wherein said managing means can update the information on the objects present in the real space and the virtual space as well as the locations, orientations, and status thereof.

3. An image generating system according to claim 2, wherein said managing means notifies said composite image generating means of the information on the objects present in the real space and the virtual space as well as the locations, orientations, and status thereof, at predetermined time intervals.

4. An image generating system according to claim 3, wherein said virtual space image generating means is responsive to the updating of the information by said managing means, for generating the image of the virtual space based on the information on the objects present in the real space and the virtual space as well as the locations and orientations thereof.

5. An image generating system according to claim 3, wherein said composite image generating means is responsive to the updating of the information by said managing means, for starting drawing the image of the real space.

6. An image generating system according to claim 5, wherein said composite image generating means synthesizes the image of the real space and the image of the virtual space after said composite image generating means has completed drawing image of the real space and said virtual space image generating means has then generated the image of the virtual space.

7. An image generating system according to claim 6, wherein said composite image generating means regenerates the image of the virtual space based on the eye position of the observer and the line-of-sight direction of the observer detected by said detecting means, immediately before synthesizing the image of the real space and the image of the virtual space.

8. An image generating system according to claim 3, wherein said virtual space image generating means executes a process of generating an image of the virtual space based on the information on the objects present in the real space and the virtual space as well as the locations and orientations thereof, according to the information updated by said managing means, and said composite image generating means executes a process of starting drawing the image of the real space in parallel with the process of generating the image of the virtual space, executed by said virtual space image generating means.

9. An image generating system according to claim 1, wherein the observer comprises a plurality of observers.

10. An image generating system according to claim 9, further comprising operation detecting means for detecting an operation of the observer including a gesture and status thereof based on results of the detection by said detecting means.

11. An image generating system according to claim 10, wherein the operation of the observer detected by said operation detecting means can be used as an input that acts on a space in which the composite image is present and objects present in the space.

13. An image generating method according to claim 12, further comprising the step of updating the management information.

14. An image generating method according to claim 13, wherein the management information is notified to said step of generating a composite image, at predetermined time intervals.

15. An image generating method according to claim 14, wherein said step of generating the image of the virtual space is executed based on the objects present in the real space and the virtual space as well as the locations and orientations thereof, in response to the updating of the information.

16. An image generating method according to claim 15, wherein when the composite image is generated, drawing of the obtained image of the real space is started in response to the updating of the information.

17. An image generating method according to claim 16, wherein said step of generating the composite image is executed by synthesizing the image of the real space and the image of the virtual space after the drawing of the image of the real space has been completed and the image of the virtual space has been generated in said step of generating the image of the virtual space.

18. An image generating method according to claim 17, wherein regeneration of the image of the virtual 17, wherein regeneration of the image of the virtual space is executed based on the detected eye position and line-of-sight direction of the observer, immediately before the image of the real space and the image of the virtual space are synthesized.

19. An image generating method according to claim 14, wherein generation of the image of the virutal space is started in response to the updating of the management information, and drawing of the obtained image of the real space in connection with the generation of the composite image is started in response to the updating of the management information, and wherein the generation of the image of the virutal space and the drawing of the image of the real space are executed in parallel with each other.

20. An image generating method according to claim 19, wherein the observer comprises a plurality of observers.

21. An image generating method according to claim 20, further comprising the step of detecting an operation of the observer including a gesture and status thereof based on the eye position and line-of-sight direction of the observer.

22. An image generating method according to claim 21, wherein the detected operation of the observer can be used as an input that acts on a space in which the composite image is present and objects present present

24. A storage medium according to claim 23, wherein said managing module can update the information on the objects present in the real space and the virtual space as well as the locations, orientations, and status thereof.

25. A storage medium according to claim 24, wherein said managing module notifies said composite image generating module of the information on the objects present in the real space and the virtual space as well as the locations, orientations, and status thereof, at predetermined time intervals.

26. A storage medium according to claim 25, wherein said virtual space image generating module is responsive to the updating of the information by said managing module, for generating the image of the virtual space based on the information on the objects present in the real space and the virtual space as well as the locations and orientations thereof.

27. A storage medium according to claim 26, wherein said composite image generating module is responsive to the updating of the information by said managing module, for starting drawing the image of the real space.

28. A storage medium according to claim 27, wherein said composite image generating module synthesizes the image of the real space and the image of the virtual space after said composite image generating module has completed drawing image of the real space and said virtual space image generating module has then generated the image of the virtual space.

29. A storage medium according to claim 27, wherein said composite image generating module regenerates the image of the virtual space based on the eye position of the observer and the line-of-sight direction of the observer detected by said detecting module, immediately before synthesizing the image of the real space and the image of the virtual space.

30. A storage medium according to claim 25, wherein said virtual space image generating module executes a process of generating an image of the virtual space based on the information on the objects present in the real space and the virtual space as well as the locations and orientations thereof, according to the information updated by said managing module, and said composite image generating module executes a process of starting drawing the image of the real space in parallel with the process of generating the image of the virtual space, executed by said virtual space image generating module.

31. A storage medium according to claim 23, wherein the observer comprises a plurality of observers.

32. A storage medium according to claim 23, further comprising an operation detecting module for detecting an operation of the observer including a gesture and status thereof based on results of the detection by said detecting module.

33. A storage medium according to claim 23, wherein the operation of the observer detected by said operation detecting module can be used as an input that acts on a space in which the composite image is present and objects present in the space.

12. An image generating method of generating a composite image by synthesizing an image of a virtual space on an image of a real space obtained at an eye position of an observer and in the line-of-sight direction of the observer, the method comprising the steps of:

detecting the eye position and the line-of-sight direction of the observer;

obtaining the image of the real space at the eye position of the observer and in the line-of-sight direction of the observer;

obtaining management information containing objects present in the real space and the virtual space as well as locations and orientations thereof;

generating the image of the virtual space at the eye position of the observer and in the line-of-sight direction of the observer based on the management information; and

generating a composite image by synthesizing the image of the virtual space and the image of the real space based on the management information.

23. A computer-readable storage medium storing a program for generating a composite image, which is executed by an image generating system comprising image pickup means for capturing an image of a real space at an eye position of an observer and in a line-of-sight direction of the observer, detecting means for detecting the eye position of the observer and the line-of-sight direction of the observer, and display means for displaying a composite image obtained by synthesizing the image of the real space and an image of a virtual space generated at the eye position of the observer and in the line-of-sight direction of the observer, the program comprising:

a detecting module for causing the detecting means to detect an eye position and line-of-sight direction of an observer;

a virtual space image generating module for generating an image of a virtual space at the eye position of the observer and in the line-of-sight direction of the observer, the eye position and the line-of-sight direction being detected by said detecting module;

a composite image generating module for generating a composite image from the image of the virtual space generated by said virtual space image generating module and an image of a real space; and

a managing module for collectively managing objects present in the real space and the virtual space as well as locations and orientations thereof.