Image synthesis and display apparatus and simulation system using same

• US 5,796,991 A
• Filed: 11/19/1997
• Issued: 08/18/1998
• Est. Priority Date: 05/16/1994
• Status: Expired due to Term

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Shimizu, Seiya
• Primary Examiner(s): Teska, Kevin J.
• Assistant Examiner(s): MOHAMED, AYNI Y
• Application Number:
  US08/974,557
• Time in Patent Office: 272 Days
• Field of Search: 395/500, 364/578
• US Class Current: 703/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation

Abstract

An image synthesis and display apparatus. The apparatus includes a display device, an image pickup device, an extraction unit and a synthesis unit. The display device displays a simulation image to a user, and provides a field of view of the simulation image to the user. The image pickup device detects an image of a range of an actual space corresponding to the field of view provided to the user by the display device. The extraction unit extracts an image of an object from the image detected by the image pickup device. The synthesis unit inserts the image extracted by the extraction unit into a simulation of a virtual space, to produce the simulation image displayed by the display device. The image pickup device is aligned with the display device, and the image pickup device and the display device move with the user while maintaining the alignment between the image pickup device and the display device.

Claims

1. An image synthesis and display apparatus comprising:

image pickup means for picking up an image of a range of an actual space corresponding to a field of view presented to a user by a display;

extraction means for extracting an image from the image picked up by said pickup means, the extracted image representing a specified object;

synthesis means for inserting the image extracted by said extraction means into a simulation image of an exclusively virtual space, and for providing the simulation image with the inserted image to the display for displaying to the user;

a position sensor which measures a position and light axis direction of said image pickup means in a real space; and

a simulation image forming means which forms a simulated image which represents an object in the virtual space, based on a position and light axis direction in the virtual space corresponding to the position and light axis direction measured by said position sensor, wherein said simulation image forming means comprises

movement information forming means for forming virtual movement information related to movement of a virtual position of the user in the virtual space,

coordinate transforming means for transforming the position and light axis direction measured by said position sensor to a position and light axis direction in the virtual space, based on said virtual movement information, and

computer graphic image generating means for generating computer graphics having a projection reference point and normal direction to a picture plane, in accordance with the position and light axis direction in the virtual space.

2. An image synthesis and display apparatus as set forth in claim 1, wherein said extraction means comprises a mask image forming unit which forms a mask image from the image picked up by said image pickup means.

3. An image synthesis and display apparatus as set forth in claim 2, further comprising an image processing unit which performs gradation processing with respect to the image picked up by said image pickup means, to produce a gradation processed image,

wherein synthesis is performed on the gradation processed image of said image processing unit with the mask image from said mask image forming unit.

4. An image synthesis and display apparatus as set forth in claim 2, wherein said mask image forming unit comprising:

a color data converting unit which reads in the pixel color values of the image picked up by said image pickup means and converts said color values to color data which represents the brightness, saturation, and hue and

a color data comparing unit which compares said color data with pre-established color conditions, and which, when said color data satisfy said color conditions, outputs a respective logic value of first and second logic values, and which, when the color is any other color, outputs the other of said first and second logic values, to thereby produce a mask image in the form of a pattern of the first and second logic values.

5. An image synthesis and display apparatus comprising:

image pickup means for picking up an image of a range of an actual space corresponding to a field of view presented to a user by a display;

extraction means for extracting an image from the image picked up by said pickup means, the extracted image representing a specified object;

synthesis means for inserting the image extracted by said extraction means into a simulation image of an exclusively virtual space, and for providing the simulation image with the inserted image to the display for displaying to the user;

a position sensor which measures a position and light axis direction of said image pickup means in a real space; and

a simulation image forming means which forms a simulated image which represents an object in the virtual space, based on a position and light axis direction in the virtual space corresponding to the position and light axis direction measured by said position sensor, wherein

said display comprises first and second display devices which independently display images to the left eye and the right eye, respectively, of the user,

said image pickup means comprises first and second image pickup devices which pick up a range of a real space corresponding to the field of view presented to the user,

said extraction means being configured so as to perform independent extraction processing with respect to the individual images obtained by said first and second image pickup devices,

said simulation image forming means forms simulated images corresponding to the left eye and the right eye of the user, respectively, based on position and view orientation of the left eye and right eye, respectively, in the virtual space, in accordance with the position and light axis direction measured by said position sensor, and

said synthesis means synthesizes simulation images corresponding to the left eye and right eye, respectively, with a result of extraction with respect to images obtained by said first and second image pickup devices, synthesis being performed independently for the left and right.

6. An image synthesis and display apparatus as set forth in claim 5, wherein said extraction means comprises a mask image forming unit which forms a mask image from the image picked up by said image pickup means.

7. An image synthesis and display apparatus as set forth in claim 6, further comprising an image processing unit which performs gradation processing with respect to the image picked up by said image pickup means, to produce a gradation processed image,

wherein synthesis is performed on the gradation processed image of said image processing unit with the mask image from said mask image forming unit.

8. An image synthesis and display apparatus as set forth in claim 6, wherein said mask image forming unit comprising:

a color data converting unit which reads in the pixel color values of the image picked up by said image pickup means and converts said color values to color data which represents the brightness, saturation, and hue and

a color data comparing unit which compares said color data with pre-established color conditions, and which, when said color data satisfy said color conditions, outputs a respective logic value of first and second logic values, and which, when the color is any other color, outputs the other of said first and second logic values, to thereby produce a mask image in the form of a pattern of the first and second logic values.

9. An apparatus comprising:

a display device which displays a simulation image to a user, the display device providing a field of view of the simulation image to the user;

an image pickup device which detects an image of a range of an actual space corresponding to the field of view provided to the user by the display device, the detected image including an image of an object;

an extraction unit which extracts the image of the object from the image detected by the image pickup device;

a synthesis unit which inserts the image extracted by the extraction unit into a simulation of a virtual space, to produce the simulation image displayed by the display device;

a position sensor which measures a position and a light axis direction in real space of the image pickup device; and

a simulation image forming unit which forms a simulated image representing an object in the virtual space, based on a position and a light axis direction in the virtual space corresponding to the position and light axis direction measured by the position sensor, wherein

the display device comprises first and second display devices which independently display images to the left eye and the right eye, respectively, of the user,

the image pickup device comprises first and second image pickup devices which detect first and second images, respectively, the first and second images each being an image of a range of an actual space corresponding to the field of view presented to the user, the first and second images each including an image of the object,

the extraction unit extracts the image of the object from the first and second images of the first and second image pickup devices,

the simulation image forming unit forms simulated images corresponding to the left eye and the right eye of the user, respectively, based on position and view orientation of the left eye and right eye, respectively, in the virtual space, in accordance with the position and light axis direction measured by the position sensor, and

the synthesis unit inserts the image of the object extracted by the extraction unit from the first image into a simulation image corresponding to the left eye of the user, and inserts the image of the object extracted by the extraction unit from the second image into a simulation image corresponding to the right eye of the user, to produce the simulation image displayed to the user by the display device.

10. An apparatus as set forth in claim 9, wherein the extraction unit comprises a mask image forming unit which forms a mask image from the image detected by the image pickup device.

11. An apparatus as set forth in claim 10, further comprising an image processing unit which performs gradation processing with respect to the image detected by the image pickup device, to produce a gradation processed image,

wherein synthesis is performed on the gradation processed image of the image processing unit with the mask image formed by the mask image forming unit.

12. An apparatus as set forth in claim 10, wherein the mask image forming unit comprises:

a color data converting unit which reads in pixel color values of the image detected up by the image pickup device and converts the color values to color data representing brightness, saturation, and hue and

a color data comparing unit which compares the color data with pre-established color conditions, and which, when the color data satisfy the color conditions, outputs a respective logic value of first and second logic values, and which, when the color data does not satisfy the color conditions, outputs the other of the first and second logic values, to produce a mask image in the form of a pattern of the first and second logic values.

13. An apparatus as set forth in claim 9, wherein the image pickup device is aligned with the display device, and the image pickup device and the display device move with the user while maintaining the alignment between the image pickup device and the display device.