Augmented reality presentation apparatus and method, and storage medium

• US 7,289,130 B1
• Filed: 09/08/2000
• Issued: 10/30/2007
• Est. Priority Date: 01/13/2000
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Satoh, Kiyohide
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US09/658,463
• Time in Patent Office: 2,608 Days
• Field of Search: 382/276, 345/633, 345/629, 345/435
• US Class Current: 345/629
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A game state manager (201) manages the state of an AR game (information that pertains to rendering of each virtual object (102), the score of a player (101), the AR game round count, and the like). An objective viewpoint video generator (202) generates a video of each virtual object (102) viewed from a camera (103). An objective viewpoint video composition unit (203) generates a composite video of the video of the virtual object (102) and an actually sensed video, and outputs it to a display (106). A subjective viewpoint video generator (212) generates a video of the virtual object (102) viewed from an HMD (107). A subjective viewpoint video composition unit (213) generates a composite video of the video of the virtual object (102) and an actually sensed video, and outputs it to the HMD (107).

Claims

1. An augmented reality presentation apparatus for superimposing a virtual object in a real space, characterized by comprising:

objective viewpoint augmented reality presentation means for presenting to an observer an augmented reality view viewed from an objective viewpoint position, which differs from a viewpoint position of any player,

wherein said objective viewpoint augmented reality presentation means includes

first video sensing means for sensing a video of the real space, where a player experiencing an augmented reality exists, viewed from the objective viewpoint position, wherein the objective viewpoint position is a fixed position so that the player appears in the video sensed by said first video sensing means;

first video generation means for generating a video of the virtual object viewed from the objective viewpoint position;

first video composition means for composing an augmented reality video viewed from the objective viewpoint position on the basis of the sensed video of the real space sensed by said first video sensing means and the generated video of the virtual object generated by said first video generation means; and

objective viewpoint video display means for displaying the composed augmented reality video composed by said first video composition means on a screen of an observer'"'"'s display apparatus, the observer'"'"'s display apparatus being separate from a head-mounted display worn by any player;

wherein said apparatus further comprises;

player'"'"'s viewpoint augmented reality presentation means for presenting an augmented reality view viewed from a player'"'"'s viewpoint;

wherein said player'"'"'s viewpoint augmented reality presentation means includes

a head-mounted display having a screen;

second video sensing means for sensing a video of the real space viewed from the player'"'"'s viewpoint position;

player'"'"'s viewpoint position acquiring means for acquiring information indicating the player'"'"'s viewpoint position;

second video generation means for generating a video of the virtual object viewed from the player'"'"'s viewpoint position using the information indicating the player'"'"'s viewpoint position;

second video composition means for composing an augmented reality video viewed from the player'"'"'s viewpoint position on the basis of the sensed video of the real space sensed by said second video sensing means and the generated video of the virtual object generated by said second video generation means; and

player'"'"'s viewpoint video display means for displaying the composed augmented reality video composed by said second video composition means on the screen of said head-mounted display, wherein said head-mounted display includes said second video sensing means and said player'"'"'s viewpoint video display means, and is worn by a player.

2. The apparatus according to claim 1, characterized by further comprising printing means,

in that said first video composition means outputs the augmented reality video to said printing means,

said printing means grabs one frame of the received video and prints out to the paper as a still image.

3. An augmented reality presentation method for superimposing a virtual object in a real space, characterized by comprising:

an objective viewpoint augmented reality presentation step of presenting to an observer an augmented reality view viewed from an objective viewpoint position, which differs from a viewpoint position of any player;

wherein said objective viewpoint augmented reality presentation step includes

a first video sensing step of sensing a video of the real space, where a player experiencing an augmented reality exists, viewed from the objective viewpoint position, wherein the objective viewpoint position is a fixed position so that the player appears in the video sensed in said first video sensing step;

a first video generation step of generating a video of the virtual object viewed from the objective viewpoint position;

a first video composition step of composing an augmented reality video viewed from the objective viewpoint position on the basis of the sensed video of the real space sensed in said first video sensing step and the generated video of the virtual object generated in said first video generation step, and

an objective viewpoint video display step of displaying the composed augmented reality video composed in said first video composition step on a screen of an observer'"'"'s display apparatus, the observer'"'"'s display apparatus being separate from a head-mounted display worn by any player;

wherein said method further comprises;

a player'"'"'s viewpoint augmented reality presentation step of presenting an augmented reality view viewed from a player'"'"'s viewpoint position;

wherein said player'"'"'s viewpoint augmented reality presentation step includes

a second video sensing step of sensing a video of the real space viewed from the player'"'"'s viewpoint position;

a player'"'"'s viewpoint position acquiring step of acquiring information indicating the player'"'"'s viewpoint position;

a second video generation step of generating a video of the virtual object viewed from the player'"'"'s viewpoint position using the information indicating the player'"'"'s viewpoint position;

a second video composition step of composing an augmented reality video viewed from the player'"'"'s viewpoint position on the basis of the sensed video of the real space sensed in said second video sensing step and the generated video of the virtual object generated in said second video generation step; and

a player'"'"'s viewpoint video display step for displaying the composed augmented reality video composed in said second video composition step on a screen of a head-mounted display worn by a player.

4. The method according to claim 3, characterized by further comprising printing step, in that in said first video composition step the augmented reality video is output to means for printing,

in said printing step one frame of the received video is grabbed and printed out to the paper as a still image.

5. A computer-readable storage medium storing a program code for superimposing a virtual object in a real space when said program code is loaded by a computer, characterized by comprising:

a program code of an objective viewpoint augmented reality presentation step of presenting to an observer an augmented reality view viewed from an objective viewpoint position, which differs from a viewpoint position of any player;

wherein said program code of the objective viewpoint augmented reality presentation step includes

a program code of a first video sensing step of sensing a video of the real space, where a player who is experiencing an augmented reality exists, viewed from the objective viewpoint position, wherein the objective viewpoint position is a fixed position so that the player appears in the video sensed in said first video sensing step;

a program code of a first video generation step of generating a video of the virtual object viewed from the objective viewpoint position;

a program code of a first video composition step of composing an augmented reality video viewed from the objective viewpoint position on the basis of the sensed video of the real space sensed in said first video sensing step and the generated video of the virtual object generated in said first video generation step; and

a program code for an objective viewpoint video display step of displaying the composed augmented reality video composed in said first video composition step on a screen of an observer'"'"'s display apparatus, the observer'"'"'s display apparatus being separate from a head-mounted display worn by any player,

wherein said storage medium further stores;

a program code for a player'"'"'s viewpoint augmented reality presentation step of presenting an augmented reality view viewed from a player'"'"'s viewpoint position;

wherein said program code for the player'"'"'s viewpoint augmented reality presentation step includes

a program code for a second video sensing step of sensing a video of the real space viewed from the player'"'"'s viewpoint position;

a program code for a player'"'"'s viewpoint position acquiring step of acquiring information indicating the player'"'"'s viewpoint position;

a program code for a second video generation step of generating a video of the virtual object viewed from the player'"'"'s viewpoint position using the information indicating the player'"'"'s viewpoint position;

a program code for a second video composition step of composing an augmented reality video viewed from the player'"'"'s viewpoint position on the basis of the sensed video of the real space sensed in said second video sensing step and the generated video of the virtual object generated in said second video generation step; and

a program code for a player'"'"'s viewpoint video display step of displaying the composed augmented reality video composed in said second video composition step on a screen of a head-mounted display worn by the player.

6. An augmented reality presentation apparatus for superimposing a virtual object in a real space, characterized by comprising:

an objective viewpoint augmented reality presentation unit adapted to present to an observer an augmented reality view viewed from an objective viewpoint position, which differs from a viewpoint position of any player,

wherein the objective viewpoint augmented reality presentation unit includes

a first video sensing unit adapted to sense a video of the real space, where a player experiencing an augmented reality, viewed from the objective viewpoint position, wherein the objective viewpoint position is a fixed position so that the player appears in the video sensed by said first video sensing unit;

a first video generation unit adapted to generate a video of the virtual object viewed from the objective viewpoint position;

a first video composition unit adapted to compose an augmented reality video viewed from the objective viewpoint on the basis of the sensed video of the real space sensed by said first video sensing unit and the generated video of the virtual object generated by said first video generation unit; and

an objective viewpoint video display unit adapted to display the composed augmented reality video composed by said first video composition unit on a screen of an observer'"'"'s display apparatus, the observer'"'"'s display apparatus being separate from a head-mounted display worn by any player;

wherein the apparatus further comprises;

a player'"'"'s viewpoint augmented reality presentation unit adapted to present an augmented reality view viewed from a player'"'"'s viewpoint position;

wherein the player'"'"'s viewpoint augmented reality presentation unit includes;

a head-mounted display having a screen;

a second video sensing unit adapted to sense a video of the real space viewed from the player'"'"'s viewpoint position;

a player'"'"'s viewpoint position acquiring unit adapted to acquire information indicating the player'"'"'s viewpoint position;

a second video generation unit adapted to generate a video of the virtual object viewed from the player'"'"'s viewpoint position using the information indicating the player'"'"'s viewpoint position;

a second video composition unit adapted to compose an augmented reality video viewed from the player'"'"'s viewpoint position on the basis of the sensed video of the real space sensed by said second video sensing unit and the generated video of the virtual object generated by said second video generation unit; and

a player'"'"'s viewpoint video display unit adapted to display the composed augmented reality video composed by said second video composition unit on the screen of said head-mounted display, wherein said head-mounted display includes said second video sensing unit and said player'"'"'s viewpoint video display unit, and is worn by a player.