Mixed reality apparatus and mixed reality presentation method

• US 6,972,734 B1
• Filed: 02/22/2000
• Issued: 12/06/2005
• Est. Priority Date: 06/11/1999
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Sato, Kiyohide
• Primary Examiner(s): WU, XIAO MIN
• Application Number:
  US09/510,232
• Time in Patent Office: 2,114 Days
• Field of Search: 345/7, 345/8, 345/9, 345/419, 345/157, 345/158, 345/169, 348/51, 348/52, 348/53, 703/1, 434/307.R, 463/31, 463/32, 463/33, 463/34
• US Class Current: 345/8
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

This invention provides a mixed reality apparatus and mixed reality presentation method, which can present reality even when a plurality of players share a common mixed reality space. Virtual objects (41–47) are superposed to hide devices such as an HMD and the like that another player wears.

Claims

1. A mixed reality presentation method of displaying to a player a mixed reality space obtained by mixing real space and virtual space, comprising the steps of:

detecting a location of the player;

detecting a location of a controller operated by the player;

inputting an image of the real space;

generating a first image of the virtual space corresponding to the detected location of the player by using model information of the player;

generating a pointer display corresponding to the location of the controller as a second image of the virtual space;

generating an image of the mixed reality space by mixing the first and second images of the virtual space with the image of the real space; and

displaying the image of the mixed reality space to the player, wherein the pointer display is made up of not less than n (n is an integer not less than

     2) parallel lines each of which passes through vertices of a regular n-sided polygon.

2. The method according to claim 1, further comprising the steps of:

detecting a posture of the player; and

detecting a posture of the controller,

wherein said step of generating a pointer display generates a pointer display corresponding to the location and the posture of the controller as a second image of the virtual space.

3. The method according to claim 1, wherein a length of the n parallel lines and a spacing between two neighboring parallel lines are defined to be values determined by a size of the mixed reality space.

4. The method according to claim 1, wherein a length of the n parallel lines and a spacing between two neighboring parallel lines are defined to be visually recognized with a distance in the mixed reality space.

5. The method according to claim 1, further comprising:

generating a third image of the virtual space and mixing the third image with the image of the real space so that the third image covers the controller and a predetermined portion of the player in the image of the real space.

6. The method according to claim 5, wherein the n lines comprise the pointer display starting from substantially a distal end portion of the third image of the virtual space.

7. The method according to claim 1, wherein the position display further comprises an image representing scale marks.

8. A storage medium which stores the mixed reality presentation method of claim 1 as a program which can be executed by a computer.

9. A mixed reality apparatus for displaying to a player a mixed reality space obtained by mixing real space and virtual space, comprising:

a first detecting unit adapted to detect a location of the player;

a second detecting unit adapted to detect a location of a controller operated by the player;

an inputting unit adapted to input an image of the real space;

a first generating unit adapted to generate a first image of the virtual space corresponding to the detected location of the player by using model information of the player;

a second generating unit adapted to generate a pointer display corresponding to the location of the controller as a second image of the virtual space;

a third generating unit adapted to generate an image of the mixed reality space by mixing the first and second images of the virtual space with the image of the real space; and

a displaying unit adapted to display the image of the mixed reality space to the player;

wherein the pointer display is made up of not less that n (n is an integer not less than

     2) parallel lines each of which passes through vertices of a regular n-sided polygon.

10. A mixed reality presentation method comprising:

an input step of inputting a sensed image sensed by a camera of a first player;

a first detection step of detecting first location information representing locations of plural portions of the first player;

a second detection step of detecting second location information representing locations of plural portions of the second player;

a virtual image generating step of generating a virtual image to be superimposed on an image of the second player included in the sensed image based on the first and second location information and a player model; and

a mixed reality image generating step of generating a mixed reality image to be presented to the first player and representing the mixed reality space, by mixing the virtual image with the sensed image.

11. The method according to claim 10, wherein the plural portions including the head of the player on which a display device is worn through which the first or the second player experiences the mixed reality, respectively.

12. The method according to claim 10, wherein said second detection step also detects command information and said virtual image generating step varies the virtual image based on the command information.

13. The method according to claim 10, wherein the player model is a model that approximates a player using simple shapes.

14. The method according to claim 10, further comprising a visual axis detection step of detecting visual axes of the first and the second player, and wherein said virtual image generating step varies the virtual image based on a relation between the location and visual axis of the first player and the location and visual axis of the second player.

15. The method according to claim 10, wherein the first location information further represents postures of the plural portions of the first player and the second location information further represents postures of the plural portions of the second player.

16. A mixed reality presentation apparatus comprising:

an input unit adapted to input a sensed image sensed by a camera of a first player;

a first detection unit adapted to detect first location information representing locations of plural portions of the first player;

a second detection unit adapted to detect second location information representing locations of plural portions of the second player;

a virtual image generating unit adapted to generate a virtual image to be superimposed on an image of the second player included in the sensed image based on the first and second location information and a player model; and

a mixed reality image generating unit adapted to generate a mixed reality image to be presented to the first player and representing the mixed reality space, by mixing the virtual image with the sensed image.

17. A computer-readable storage medium storing codes for performing steps comprising:

an input program step of inputting a sensed image sensed by a camera of a first player;

a first detection program step of detecting first location information representing locations of plural portions of the first player;

a second detection program step of detecting second location information representing locations of plural portions of the second player;

a virtual image generating program step of generating a virtual image to be superimposed on an image of the second player included in the sensed images based on the first and second location information and a player model; and

a mixed reality image generating program step of generating a mixed reality image to be presented to the first player and representing the mixed reality space, by mixing the virtual image with the sensed image.

18. A mixed reality presentation method of three-dimensionally displaying a pointer for pointing at a three-dimensional point within a mixed reality space that is represented by mixing a real space and a virtual space, comprising the steps of:

obtaining location and posture information of a player;

obtaining location and posture information of a unit operated by the player;

generating a virtual image of a pointer display that comprises n parallel lines each of which passes through vertices of a regular n-sided polygon (n is an integer not less than

     2) based on the location and posture information of the player and of the unit; and

mixing the virtual image of the pointer display with a real space image to represent the mixed reality space to the player.

19. The method according to claim 18, wherein the virtual image further includes an image covering said unit, and wherein the pointer display starts from an edge of the image covering the unit.

20. A computer-readable storage medium which stores the mixed reality presentation method of claim 18 as a program which can be executed by a computer.