Apparatus for displaying three-dimensional virtual object and method of displaying the same

• US 6,020,891 A
• Filed: 07/29/1997
• Issued: 02/01/2000
• Est. Priority Date: 08/05/1996
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Rekimoto, Junichi
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): GOOD JOHNSON, MOTILEWA
• Application Number:
  US08/902,510
• Time in Patent Office: 917 Days
• Field of Search: 345/419, 345/435, 345/355, 345/433, 345/157
• US Class Current: 345/419
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A table and an apparatus of the other party as a picture in the actual world are displayed on a picture plane of a portable display, a three-dimensional virtual object generated by computer graphics and so on is displayed on the table, and the shade thereof is displayed on the table. With this, even in the midst of a work in the actual world, it is possible to switch the work simply to a work in the virtual world only by shifting eyes to a picture plane of a portable display. Further, it is possible to grasp the position in the space of a three-dimensional virtual object easily by displaying the shade. It is made so that the work in the virtual world may be performed easily as described even while the work in the actual world is continued.

Claims

1. An apparatus for displaying a three-dimensional virtual object comprising:

display means for displaying a predetermined picture;

image pickup means installed with an image pickup direction set to a back direction of a display plane of said display means for picking up an image of an actual picture of a predetermined subject;

coordinates measuring means for measuring three-dimensional space coordinates of said image pickup means with respect to a predetermined reference plane;

picture generating means for generating a picture of a predetermined virtual three-dimensional object based on three-dimensional space coordinates for said reference plane of said image pickup means measured by said coordinates measuring means; and

composite picture generating means for generating a composite picture in which a picture of a virtual three-dimensional object generated by said picture generating means with said actual picture the image of which has been picked up by said image pickup means as the background and based on said three-dimensional space coordinates for said reference plane of said image pickup means measured by said coordinates measuring means;

wherein;

said display means displays said composite picture.

2. An apparatus for displaying a three-dimensional virtual object according to claim 1, wherein said coordinates measuring means is composed of magnetic field generating means arranged on said reference plane, magnetic field detecting means arranged in said image pickup means, and arithmetic means for computing said three-dimensional space coordinates based on the result of detection by said magnetic field detecting means.

3. An apparatus for displaying a three-dimensional virtual object according to claim 1, wherein said picture generating means generates a picture of a shadow of said virtual three-dimensional object on said reference plane.

4. An apparatus for displaying a three-dimensional virtual object according to claim 1, further comprising operating means for grasping said three-dimensional virtual object virtually and applying predetermined operation to said three-dimensional virtual object;

wherein, when movement of said three-dimensional virtual object with respect to said reference plane is instructed by said operating means, said picture generating means generates a picture of a reference line which shows that said three-dimensional virtual object is grasped along the image pickup direction of said image pickup means, and generates a picture that the picture of said three-dimensional virtual object moves on said reference plane under a state that the point of said reference line is in contact therewith in accordance with the operation of said operating means.

5. An apparatus for displaying a three-dimensional virtual object according to claim 1, wherein a common picture of a three-dimensional virtual object generated by said picture generating means is displayed on a plurality sets of display apparatus each composed at least of said display means, said image pickup means, said coordinates measuring means and said operating means, and positional change on said reference plane of said three-dimensional virtual object caused by the operation performed by said operating means for said three-dimensional virtual object is reflected in respective display apparatus.

6. A method of displaying a three-dimensional virtual object in an apparatus for displaying a three-dimensional virtual object provided with display means for displaying a predetermined picture and an image pickup means installed with the image pickup direction set to a back direction of a display plane of said display means for picking up an image of an actual picture of a predetermined subject, comprising the steps of:

coordinates measurement processing for measuring three-dimensional space coordinates of said image pickup means for a predetermined reference plane;

picture generation processing for generating a picture of a predetermined virtual three-dimensional object based on measured three-dimensional space coordinates for said reference plane of said image pickup means; and

image composition processing for generating a composite picture in which a picture of a virtual three-dimensional object generated by said picture generation processing is superimposed on a predetermined position of said background is generated with said actual picture the image of which has been picked up as the background based on said three-dimensional space coordinates for said reference plane of said image pickup means measured by said coordinates measurement processing, and said composite picture is displayed on the display plane of said display means.