Game apparatus for mixed reality space, image processing method thereof, and program storage medium

• US 20030032484A1
• Filed: 02/17/2000
• Published: 02/13/2003
• Est. Priority Date: 06/11/1999
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Satoh, Kiyohide
• Granted Patent: US 6,951,515 B2
• Time in Patent Office: Days
• US Class Current: 463/43
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/8076 Shooting
• CPC Class Codes: A63F 2300/8082 Virtual reality

Abstract

There is disclosed a game apparatus which allows a virtual object to act in a mixed reality space as if it had its own will. The game apparatus according to this invention allows a player to play a game with a virtual object in a mixed reality space. A rule for controlling the action of the virtual object on the basis of the objective of the game, and the relative positional relationship between the virtual object and real object is pre-stored. The locations of the player and real object are detected, and the relative positional relationship between the player or real object, and the virtual object is recognized. The next action of the virtual object is determined according to the stored rule, and a virtual image of the virtual object is generated on the basis of the location/posture of the virtual object after the determined action and the location/posture of the viewpoint position of the player, thus presenting the generated virtual image to the player.

Claims

1. A game apparatus with which a player plays a game with virtual object(s) in a mixed reality space comprising:

viewpoint detection means for detecting the location/posture of a viewpoint of the player;

geometric information acquisition means for acquiring geometric information of real object(s);

recognition means for recognizing a current, relative relationship between the virtual object(s) and real object(s);

a rule memory for storing rules for controlling the action of the virtual object(s);

computation means for determining the next action of the virtual object(s) in accordance with the rules stored in said rule memory and in correspondence with the location/posture of the real object(s), and computing thee location/posture of the virtual object(s) after the determined action; and

presentation means for generating at least one of image of the virtual object on the basis of the location/posture of the virtual object(s) after the action and the location/posture of the viewpoint position of the player, and for representing the mixed reality space to the player by superimposing the virtual object image(s) on the player'"'"'s view of the real space.

2. A game apparatus according to claim 1, wherein said presentation means further comprising,

image-capturing means for capturing real space images of said player'"'"'s view of the real space;

image generation means for generating mixed reality images representing of the mixed reality space by superimposing or overlaying said virtual object image(s) on said real space images; and

a video see-through type display means that the player wares wherein said mixed reality images are displayed.

3. A game apparatus according to claim 1, wherein said presentation means further comprising,

An optical see-through type display means that the player wares wherein said virtual object image(s) are displayed.

4. A game apparatus according to claim 1, further comprising,

status detecting means for detecting status of the player;

wherein said computation means determines a next action of the virtual object in accordance with the rule stored in said rule memory and in correspondence with the location/posture of the real object and/or the status, and computing a location/posture of the virtual object after the determined action.

5. The apparatus according to claim 1, wherein the current, relative relationship includes a layout relationship between the virtual object and real object at that time in the mixed reality space.

6. The apparatus according to claim 1, wherein the current, relative relationship includes a behavior of the real object with respect to the virtual object at that time in the mixed reality space.

7. The apparatus according to claim 1, wherein the real object includes the player himself or herself, and said recognition means recognizes a current, relative relationship between the virtual object and the player.

8. The apparatus according to claim 1, wherein the real object includes a plurality of players who operate said game apparatus, and the plurality of players share a single mixed reality space.

9. The apparatus according to claim 1, wherein the real object is an object which is fixed in position, and

said geometric information acquisition means comprises:

a predetermined memory for pre-storing location information and shape information of the real object; and

means for reading out the location information and shape information of the real object from said memory as needed.

10. The apparatus according to claim 1, wherein the real object is an object which is movable but does not deform, and

said geometric information acquisition means comprises:

a predetermined memory for pre-storing shape information of the real object;

a location/posture sensor for detecting a location/posture of the real object; and

means for setting a region the real object is expected to occupy in the mixed real space in accordance with the detected location/posture of the real object.

11. The apparatus according to claim 1, wherein the real object is a player, and

said geometric information acquisition means comprises:

a sensor for detecting a location/posture of a head of the player; and

means for setting a region having a fixed, known shape that approximates the player in the mixed reality space in accordance with the detected location/posture of the head.

12. The apparatus according to claim 1, wherein when the game is a battle game with the virtual object, an objective is to decrease an expected score of the player.

13. The apparatus according to claim 1, wherein when the game is a cooperative game with the virtual object, an objective is to increase an expected score of the player.

14. The apparatus according to claim 1, wherein the rule controls the action of the virtual object on the basis of an objective of the game and a relative relationship between the virtual object and real object.

15. The apparatus according to claim 1, wherein the rule stored in said rule memory expresses the action of the virtual object as an action pattern with a predetermined aim for achieving the objective.

16. The apparatus according to claim 15, wherein the pattern has a path disadvantageous to the player in consideration of a layout relationship between the virtual object and real object.

17. The apparatus according to claim 11, wherein when the player is one of real objects, an output from said viewpoint detection means for detecting the location/posture of the viewpoint of the player is also used as information which is to be acquired by said geometric information acquisition means and pertains to a location and shape of the player.

18. The apparatus according to claim 1, wherein said viewpoint detection means detects a location/posture of a head of the player, and

said apparatus further comprises detection means for detecting a location/posture of a hand of the player;

and

means for recognizing a relative location of the hand of the player with respect to the head as a command on the basis of an output from said detection means.

19. The apparatus according to claim 1, wherein said presentation means comprises:

means for aligning the location/posture of the real object to the location/posture of the virtual object after movement;

means for generating an image of the virtual object after alignment in correspondence with an occlusion relationship; and

a head-mounted display device.

20. An image processing method for a game apparatus with which a player plays a game with virtual object(s) in a mixed reality space comprising:

viewpoint detection step for detecting the location/posture of a viewpoint of the player;

geometric information acquisition step for acquiring geometric information of real object(s);

recognition step for recognizing a current, relative relationship between the virtual object(s) and real object(s);

a rule memory for storing rules for controlling the action of the virtual object(s);

computation step for determining the next action of the virtual object(s) in accordance with the rules stored in said rule memory and in correspondence with the location/posture of the real object(s), and computing thee location/posture of the virtual object(s) after the determined action; and

presentation step for generating at least one of image of the virtual object on the basis of the location/posture of the virtual object(s) after the action and the location/posture of the viewpoint position of the player, and for representing the mixed reality space to the player by superimposing the virtual object image(s) on the player'"'"'s view of the real space.

21. The method according to claim 20, wherein the player wares a video see-through type display, and said presentation step further comprising,

image-capturing step for capturing real space images of said player'"'"'s view of the real space;

image generation step for generating mixed reality images representing of the mixed reality space by superimposing or overlaying said virtual object image(s) on said real space images and for displaying said mixed reality images on the display.

22. The method according to claim 20, wherein the player wares an optical see-through type display and said presentation step representing the mixed reality space to the player by displaying the virtual object image(s) on the display.

23. The method according to claim 20, further comprising,

status detecting step for detecting status of the player;

wherein said computation step determines a next action of the virtual object in accordance with the rule stored in said rule memory and in correspondence with the location/posture of the real object and/or the status, and computing a location/posture of the virtual object after the determined action.

24. The method according to claim 20, wherein the current, relative relationship includes a layout relationship between the virtual object and real object at that time in the mixed reality space.

25. The method according to claim 20, wherein the current, relative relationship includes a behavior of the real object with respect to the virtual object at that time in the mixed reality space.

26. The method according to claim 20, wherein the real object includes the player himself or herself, and the recognition step includes the step of recognizing a current, relative relationship between the virtual object and the player.

27. The method according to claim 20, wherein the real object includes a plurality of players who operate the game apparatus, and the plurality of players share a single mixed reality space.

28. The method according to claim 20, wherein the real object is an object which is fixed in position, and

the geometric information acquisition step includes the steps of:

pre-storing location information and shape information of the real object in a predetermined memory; and

reading out the location information and shape information of the real object from the memory as needed.

29. The method according to claim 20, wherein the real object is an object which is movable but does not deform, and

the geometric information acquisition step includes the steps of:

pre-storing shape information of the real object in a predetermined memory;

detecting a location/posture of the real object by a location/posture sensor; and

setting a region the real object is expected to occupy in the mixed real space in accordance with the detected location/posture of the real object.

30. The method according to claim 20, wherein the real object is a player, and

the geometric information acquisition step includes the steps of:

detecting a location/posture of a head of the player; and

setting a region having a fixed, known shape that approximates the player in the mixed reality space in accordance with the detected location/posture of the head.

31. The method according to claim 20, wherein when the game is a battle game with the virtual object, an objective is to decrease an expected score of the player.

32. The method according to claim 20, wherein when the game is a cooperative game with the virtual object, an objective is to increase an expected score of the player.

33. The method according to claim 20, wherein the rule controls the action of the virtual object on the basis of an objective of the game and a relative relationship between the virtual object and real object.

34. The method according to claim 20, wherein the rule stored in the rule memory expresses the action of the virtual object as an action pattern with a predetermined aim for achieving the objective.

35. The method according to claim 34, wherein the pattern has a path disadvantageous to the player in consideration of a layout relationship between the virtual object and real object.

36. The method according to claim 30, wherein when the player is one of real objects, an output from said viewpoint detection step of detecting the location/posture of the viewpoint of the player is also used as information which is to be acquired in the geometric information acquisition step and pertains to a location and shape of the player.

37. The method according to claim 20, wherein the viewpoint detection step includes the step of detecting a location/posture of a head of the player, and

said method further comprises the detection step of detecting a location/posture of a hand of the player;

and

the step of recognizing a relative location of the hand of the player with respect to the head as a command on the basis of an output in the detection step.

38. The method according to claim 20, wherein the presentation step includes the steps of:

aligning the location/posture of the real object to the location/posture of the virtual object after movement; and

generating an image of the virtual object after alignment in correspondence with an occlusion relationship.

39. A storage medium which stores a program of an image processing method for a game apparatus with which a player plays a game with virtual object(s) in a mixed reality space comprising:

viewpoint detection program step for detecting the location/posture of a viewpoint of the player;

geometric information acquisition program step for acquiring geometric information of real object(s);

recognition program step for recognizing a current, relative relationship between the virtual object(s) and real object(s);

a rule memory for storing rules for controlling the action of the virtual object(s);

computation program step for determining the next action of the virtual object(s) in accordance with the rules stored in said rule memory and in correspondence with the location/posture of the real object(s), and computing thee location/posture of the virtual object(s) after the determined action; and

presentation program step for generating at least one of image of the virtual object on the basis of the location/posture of the virtual object(s) after the action and the location/posture of the viewpoint position of the player, and for representing the mixed reality space to the player by superimposing the virtual object image(s) on the player'"'"'s view of the real space.