Display control method for head-mounted display (HMD) and image generation device

• US 9,884,248 B2
• Filed: 03/20/2015
• Issued: 02/06/2018
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Bandai Namco Entertainment Inc. (BANDAI NAMCO Holdings Inc.)
• Original Assignee: Bandai Namco Entertainment Inc. (BANDAI NAMCO Holdings Inc.)
• Inventors: Koseki, Kazumasa, Onda, Akio
• Primary Examiner(s): Rowland, Steve
• Application Number:
  US14/663,984
• Publication Number:
  US 20150352437A1
• Time in Patent Office: 1,054 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

The position and the posture of an HMD, and the position of a game controller in the real space are detected, and tracked. An image of a virtual space is displayed on the HMD. A controller object is disposed and displayed at a position within the virtual space that corresponds to the position of the game controller in the real space viewed from the HMD. Therefore, the player can easily find and hold the game controller even when the player wears the HMD.

Claims

1. A method that causes a computer to control a display on a head-mounted display (HMD) that is worn on a head of a user, the method comprising:

causing the computer to receive data that represents a captured image from an imaging section, the imaging section being provided so as to capture a real space that includes the HMD, and a controller that is held and operated by the user, the captured image including the HMD and the controller;

causing the computer to calculate a virtual position of the controller within a virtual space using the captured image, the virtual space being a display image space of the HMD;

causing the computer to calculate a virtual position of the HMD within the virtual space using the captured image;

causing the computer to display a first guide display on the HMD so that the first guide display indicates the virtual position of the controller viewed from the virtual position of the HMD;

causing the computer to determine a position and/or a posture of the HMD in the real space;

causing the computer to generate an image of the virtual space that is displayed on the HMD, the image of the virtual space that is displayed on the HMD changing in field of view corresponding to the determined position and/or the determined posture of the HMD in the real space;

causing the computer to change a state of the first guide display corresponding to whether or not the virtual position of the controller is within the field of view; and

causing the computer to display an object that represents a direction from the virtual position of the HMD toward the virtual position of the controller as the first guide display when the virtual position of the controller is outside of the field of view, wherein the first guide display indicates a position within the virtual space that corresponds to a position of the controller within the real space.

2. The method as defined in claim 1, wherein

a marker is provided to the HMD, and

the method further comprises:

causing the computer to determine the position and/or the posture of the HMD using imaging information about the marker within the captured image.

3. The method as defined in claim 1, wherein

the HMD includes a posture detection sensor that includes at least one of an acceleration sensor, a gyro sensor, and a camera for capturing a stationary marker that is provided at a given position within the real space, and

the method further comprises:

causing the computer to determine the position and/or the posture of the HMD using detection results of the posture detection sensor.

4. The method as defined in claim 1, further comprising:

causing the computer to perform image processing that displays the first guide display when the first guide display is situated at a hidden position within the virtual space when viewed from the virtual position of the HMD.

5. The method as defined in claim 1, further comprising:

causing the computer to display an object that represents the controller as the first guide display when the virtual position of the controller is within the field of view.

6. The method as defined in claim 1, further comprising:

causing the computer to calculate a virtual position of a hand of the user within the virtual space that corresponds to a position of the hand of the user within the real space using the captured image; and

causing the computer to display a second guide display on the HMD using the virtual position of the hand of the user, the second guide display indicating a position within the virtual space that corresponds to the position of the hand of the user within the real space.

7. The method as defined in claim 1, further comprising:

causing the computer to calculate a virtual position of a hand of the user within the virtual space that corresponds to a position of the hand of the user within the real space using the captured image; and

causing the computer to change the first guide display using a relative positional relationship between the virtual position of the controller and the virtual position of the hand.

8. The method as defined in claim 1, further comprising:

causing the computer to output a guide sound from a headphone that is worn by the user, the guide sound representing the virtual position of the controller.

9. The method as defined in claim 1, further comprising:

causing the computer to initialize a coordinate system of the virtual space so as to correspond to the real space.

10. The method as defined in claim 1, wherein

a camera is provided to the HMD, and

the method further comprises:

causing the computer to detect whether or not the controller is captured within an image captured by the camera; and

causing the computer to perform a given notification control process corresponding to a detection result as to whether or not the controller is captured within the image captured by the camera.

11. A method that causes a computer to control a display on a head-mounted display (HMD) that is worn on a head of a user, the method comprising:

causing the computer to receive data that represents a captured image from an imaging section, the imaging section being provided so as to capture a real space that includes the HMD, and a controller that is held and operated by the user, the captured image including the HMD and the controller;

causing the computer to calculate a virtual position of the controller within a virtual space using the captured image, the virtual space being a display image space of the HMD;

causing the computer to detect whether or not the controller is held; and

causing the computer to display a first guide display on the HMD using the virtual position of the controller, when the controller is not held, the first guide display indicating a position within the virtual space that corresponds to a position of the controller within the real space.

12. An image generation device that generates an image that is displayed on a head-mounted display (HMD) that is worn on a head of a user, the image generation device comprising:

a captured image-receiving section that receives data that represents a captured image from an imaging section, the imaging section being provided so as to capture a real space that includes the HMD, and a controller that is held and operated by the user, the captured image including the HMD and the controller;

a controller position calculation section that calculates a virtual position of the controller within a virtual space using the captured image, the virtual space being a display image space of the HMD;

an HMD position-posture calculation section that calculates a virtual position of the HMD within the virtual space using the captured image, and determines a position and/or a posture of the HMD in the real space; and

a display control section that;

(i) generates an image of the virtual space that is displayed on the HMD, the image of the virtual space that is displayed on the HMD changing in field of view corresponding to the determined position and/or the determined posture of the HMD in the real space, and (ii) displays a first guide display on the HMD so that the first guide display indicates the virtual position of the controller viewed from the virtual position of the HMD, wherein the display control section;

changes a state of the first guide display corresponding to whether or not the virtual position of the controller is within the field of view,

displays an object that represents a direction from the virtual position of the HMD toward the virtual position of the controller as the first guide display when the virtual position of the controller is outside of the field of view, and

displays the first guide display to indicate a position within the virtual space that corresponds to a position of the controller within the real space.

13. An image generation device that generates an image that is displayed on a head-mounted display (HMD) that is worn on a head of a user, the image generation device comprising:

a captured image-receiving section that receives data that represents a captured image from an imaging section, the imaging section being provided so as to capture a real space that includes the HMD, and a controller that is held and operated by the user, the captured image including the HMD and the controller;

a controller position calculation section that calculates a virtual position of the controller within a virtual space using the captured image, the virtual space being a display image space of the HMD;

a controller hold determination section that detects whether or not the controller is held; and

a first guide display control section that displays a first guide display on the HMD using the virtual position of the controller, when the controller is not held, the first guide display indicating a position within the virtual space that corresponds to a position of the controller within the real space.