Marker layout method, mixed reality apparatus, and mixed reality space image generation method

• US 7,084,887 B1
• Filed: 02/18/2000
• Issued: 08/01/2006
• Est. Priority Date: 06/11/1999
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Sato, Kiyohide
• Primary Examiner(s): YANG, RYAN R
• Application Number:
  US09/506,382
• Time in Patent Office: 2,356 Days
• Field of Search: 345/633, 345/632, 345/641, 345/9, 345/8, 345/7, 348/116, 702150-153, 702/94, 702/95, 463/1, 463/31, 463/32, 463/33, 382/287
• US Class Current: 345/633
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/5372 for tagging characters, objects or locations in the game scene, e.g. displaying a circle under the character controlled by the player
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/306 for displaying a marker associated to an object or location in the game field
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

There is disclosed a marker layout method in a mixed reality space, which can reliably detect markers in units of players, even when a plurality of players share a common mixed reality space. According to this invention, markers to be used by only a given player are laid out at positions that cannot be seen from other players. Real objects used in an application that uses a mixed reality space may be used.

Claims

1. A mixed reality space image generation method for generating a mixed reality space image which makes a player experience mixed reality by mixing a real space in which markers serving as position indices are laid out, and a virtual space, comprising:

a marker detection step of detecting the markers from image data obtained by sensing the real space from a substantially viewpoint position of the player;

a correcting step of correcting location/orientation information of the player based on the markers detected by said marker detection step; and

a mixed reality space image generation step of generating a mixed reality space image to be observed by the player, so the player observes virtual object images that do not include any images of the markers in surrounding regions (marker regions) including the markers in the image data;

wherein the mixed reality space image generation step includes a step of substituting or overlaying images of the marker regions by predetermined virtual object images.

3. The method according to claim 1, wherein the predetermined virtual objects are images of plane patches with the same or similar texture, size, location and orientation of the marker regions before the markers are laid out.

2. A mixed reality space image generation method for generating a mixed reality space image which makes a player experience mixed reality by mixing a real space in which markers serving as position indices are laid out, and a virtual space, comprising:

a marker detection step of detecting the markers from image data obtained by sensing the real space from a substantially viewpoint position of the player;

a correcting step of correcting location/orientation information of the player based on the markers detected by the marker detection step; and

a mixed reality space image generation step of generating a mixed reality space image to be observed by the player, so the player observes virtual object images that do not include any images of the markers in surrounding regions (marker regions) including the markers from the image data,

wherein the mixed reality space image generation step includes a step of substituting or overlaying an image of a region including all the markers in the image data by a predetermined virtual object image.

4. A computer readable storage medium that stores a program which can be executed by a computer, making the computer which executes the program function as:

marker detection means for detecting the markers from image data obtained by sensing the real space from a substantially viewpoint position of the player;

correcting means for correcting location/orientation information of the player based on the markers detected by the marker detection means; and

mixed reality space image generation means for generating a mixed reality space image to be observed by the player, so the player observes virtual object images that do not include any images of the markers in surrounding regions (marker regions) including the markers in the image data;

wherein the mixed reality space image generation means includes means for substituting or overlaying images of the marker regions by predetermined virtual object images.

5. A computer readable storage medium that stores a program which can be executed by a computer, said program making the computer which executes the program function as:

a mixed reality apparatus for making a player experience mixed reality by making the player observe a mixed reality space image obtained by mixing a real space in which markers serving as position indices are laid out, and a virtual space, comprising;

marker detection means for detecting the markers from image data obtained by sensing the real space from a substantially viewpoint position of the player;

correcting means correcting location/orientation information of the player based on the markers detected by the marker detection means; and

mixed reality space image generation means for generating a mixed reality space image to be observed by the player, so the player observes virtual object images that do not include any images of the markers in surrounding regions (marker regions) including the markers in the image data;

wherein the mixed reality space image generation means includes means for substituting or overlaying an image of a region including all the markers in the image data by a predetermined virtual object image.

6. A mixed reality apparatus for providing a mixed reality space shared by a plurality of players, comprising:

an input unit adapted to input a real space image representing the real space;

a marker detecting unit adapted to extract pixels having a color that is predefined as a marker from the real space image, perform a labeling process to detect marker regions, and detect positions of markers from the marker regions;

a head location/orientation estimation unit to estimate a location/orientation of a viewpoint of a player by using the detected marker positions;

a mixed reality space image generation unit adapted to generate a virtual space image to be observed by the player based on the location/orientation of the viewpoint of the player and model information; and

an image combining unit adapted to generate a mixed reality space image by combining the real space image with the virtual space image,

wherein the markers are laid out in the real space so that the markers to be used by only the given player are laid out at positions hidden by real objects when the markers are observed from the other player.

7. A method for providing a mixed reality space shared by a plurality of players, comprising:

an input step of inputting a real space image representing the real space;

a marker detecting step of extracting pixels having a color that is predefined as a marker from the real space image, performing a labeling process to detect marker regions, and detecting positions of markers from the marker regions;

a head location/orientation estimation step of estimating a location/orientation of a viewpoint of a player by using the detected marker positions;

a mixed reality space image generation step of generating a virtual space image to be observed by the player based on the location/orientation of the viewpoint of the player and model information; and

an image combining step of generating a mixed reality space image by combining the real space image with the virtual space image;

wherein the markers are laid out in the real space so that the markers to be used by only a given player are laid out at positions hidden by real objects when the markers are observed from the other player.

8. A mixed reality apparatus having an image generating unit adapted to generate a virtual space image corresponding to a viewpoint position and orientation of a player, an input unit adapted to input a real space image obtained by sensing from a substantially viewpoint position of the player and an image combining unit adapted to generate a mixed reality space image by combining the real space image with the virtual space image, comprising:

a marker detection unit adapted to detect marker regions for the real space image input from said input unit and obtained by sensing the real space in which markers serving as position indices are laid out;

a correction unit adapted to correct location/orientation information of the player based on the marker regions detected by said marker detection unit;

a location/orientation estimation unit adapted to estimate a location/orientation of a viewpoint of a given player using the detected marker regions;

a storing unit adapted to store a real space image obtained by sensing the real space before the markers are laid out as a texture image; and

a replacing unit adapted to replace said marker regions in the real space image with corresponding regions in said texture image or superimpose said corresponding regions in said texture image on said marker regions in the real space image.

9. A mixed reality image generating method including an image generating step of generating a virtual image corresponding to a viewpoint position and orientation of a player, an input step of inputting a real space image obtained by sensing from a substantially viewpoint position of the player and an image combining step of generating a mixed reality space image by combining the real space image with the virtual space image, comprising:

a marker detection step of detecting marker regions for the real space image obtained by sensing the real space in which markers serving as position indices are laid out;

a correction step of correcting location/orientation information of the player based on the marker regions detected by said marker detection step;

a storing step of storing a real space image obtained by sensing the real space before the markers are laid out as a texture image; and

a replacing step of replacing said marker regions in the real space image with corresponding regions in said texture or superimposing said corresponding regions in said texture image on said marker regions in the real space image.

10. The mixed reality generating method according to claim 9, wherein the same texture image is used for replace or superimpose the marker regions corresponding to markers laid out on real objects made of an identical material.

11. The mixed reality image generating method according to claim 9 further comprising:

a location/orientation estimation step of estimating a location/orientation of a viewpoint of the player using the detected marker regions, the location information of the detected markers and output signal from a position/orientation sensor mounted on the player'"'"'s head.

12. A computer-readable storage medium that stores program code for controlling an apparatus to perform a mixed reality generating method, wherein the generating method includes an image generating step of generating a virtual image corresponding to a viewpoint position and orientation of a player, an input step of inputting a real space image obtained by sensing from substantially a viewpoint position of the player and an image combining step of generating a mixed reality space image by combining the real space image with the virtual space image, said storage medium comprising program code for controlling the apparatus to perform:

a marker detection step of detecting marker regions for the real space image obtained by sensing the real space in which markers serving as position indices are laid out;

a correction step of correcting location/orientation information of the player based on the marker regions detected by said marker detection step;

a storing step of storing a real space image obtained by sensing the real space before the markers are laid out as a texture image; and

a replacing step of replacing said marker regions in the real space image with corresponding regions in said texture or superimposing said corresponding regions in said texture image on said marker regions in the real space image.