DISPLAY CONTROL METHOD FOR HEAD MOUNTED DISPLAY (HMD) AND IMAGE GENERATION DEVICE

• US 20150352437A1
• Filed: 03/20/2015
• Published: 12/10/2015
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Bandai Namco Entertainment Inc. (BANDAI NAMCO Holdings Inc.)
• Original Assignee: Namco Bandai Games Inc.
• Inventors: KOSEKI, Kazumasa, ONDA, Akio
• Granted Patent: US 9,884,248 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

The position and the posture of an HMD, and the position of a game controller in the real space are detected, and tracked. An image of a virtual space is displayed on the HMD. A controller object is disposed and displayed at a position within the virtual space that corresponds to the position of the game controller in the real space viewed from the HMD. Therefore, the player can easily find and hold the game controller even when the player wears the HMD.

Claims

1. A method that causes a computer to control a display on a head-mounted display (HMD) that is worn on a head of a user, the method comprising:

causing the computer to receive data that represents a captured image from an imaging section, the imaging section being provided so as to capture a real space that includes the HMD, and a controller that is held and operated by the user;

causing the computer to calculate a virtual position of the controller within a virtual space using the captured image, the virtual space being a display image space of the HMD; and

causing the computer to display a first guide display on the HMD using the virtual position, the first guide display indicating a position within the virtual space that corresponds to a position of the controller within the real space.

2. The method as defined in claim 1, further comprising:

causing the computer to calculate a virtual position of the HMD within the virtual space using the captured image; and

causing the computer to display the first guide display on the HMD so that the first guide display indicates the virtual position of the controller viewed from the virtual position of the HMD.

3. The method as defined in claim 2, further comprising:

causing the computer to determine a position and/or a posture of the HMD in the real space;

causing the computer to generate an image of the virtual space that is displayed on the HMD, the image of the virtual space that is displayed on the HMD changing in field of view corresponding to the determined position and/or the determined posture of the HMD; and

causing the computer to change a state of the first guide display corresponding to whether or not the virtual position of the controller is within the field of view.

4. The method as defined in claim 3,

a marker being provided to the HMD,

the method further comprising:

causing the computer to determine the position and/or the posture of the HMD using imaging information about the marker within the captured image.

5. The method as defined in claim 3,

the HMD including a posture detection sensor that includes at least one of an acceleration sensor, a gyro sensor, and a camera for capturing a stationary marker that is provided at a given position within the real space,

the method further comprising:

causing the computer to determine the position and/or the posture of the HMD using detection results of the posture detection sensor.

6. The method as defined in claim 3, further comprising:

causing the computer to perform image processing that displays the first guide display when the first guide display is situated at a hidden position within the virtual space when viewed from the virtual position of the HMD.

7. The method as defined in claim 3, further comprising:

causing the computer to display an object that represents the controller as the first guide display when the virtual position of the controller is within the field of view.

8. The method as defined in claim 3, further comprising:

causing the computer to display an object that represents a direction from the virtual position of the HMD toward the virtual position of the controller as the first guide display when the virtual position of the controller is outside the field of view.

9. The method as defined in claim 1, further comprising:

causing the computer to calculate a virtual position of a hand of the user within the virtual space that corresponds to the real space using the captured image; and

causing the computer to display a second guide display on the HMD using the virtual position of the hand, the second guide display indicating a position within the virtual space that corresponds to a position of the hand within the real space.

10. The method as defined in claim 1, further comprising:

causing the computer to calculate a virtual position of a hand of the user within the virtual space that corresponds to the real space using the captured image; and

causing the computer to change the first guide display using a relative positional relationship between the virtual position of the controller and the virtual position of the hand.

11. The method as defined in claim 1, further comprising:

causing the computer to detect whether or not the controller is held; and

causing the computer to display the first guide display when the controller is not held.

12. The method as defined in claim 1, further comprising:

causing the computer to output a guide sound from a headphone that is worn by the user, the guide sound representing the virtual position of the controller.

13. The method as defined in claim 1, further comprising:

causing the computer to initialize a coordinate system of the virtual space so as to correspond to the real space.

14. The method as defined in claim 1,

a camera being provided to the HMD,

the method further comprising:

causing the computer to detect whether or not the controller is captured within an image captured by the camera; and

causing the computer to perform a given notification control process corresponding to a detection result as to whether or not the controller is captured within the image captured by the camera.

15. An image generation device that generates an image that is displayed on a head-mounted display (HMD) that is worn on a head of a user, the image generation device comprising:

a captured image-receiving section that receives data that represents a captured image from an imaging section, the imaging section being provided so as to capture a real space that includes the HMD, and a controller that is held and operated by the user;

a controller position calculation section that calculates a virtual position of the controller within a virtual space using the captured image, the virtual space being a display image space of the HMD; and

a first guide display control section that displays a first guide display on the HMD using the virtual position, the first guide display indicating a position within the virtual space that corresponds to a position of the controller within the real space.