Spatially-correlated multi-display human-machine interface

• US 10,449,444 B2
• Filed: 09/05/2017
• Issued: 10/22/2019
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Takeda, Genyo, Cheng, Howard
• Primary Examiner(s): Renwick, Reginald A
• Application Number:
  US15/695,169
• Publication Number:
  US 20170361224A1
• Time in Patent Office: 777 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast

Abstract

A human-machine interface involves plural spatially-coherent visual presentation surfaces at least some of which are movable by a person. Plural windows or portholes into a virtual space, at least some of which are handheld and movable, are provided by using handheld and other display devices. Aspects of multi-dimensional spatiality of the moveable window (e.g., relative to another window) are determined and used to generate images. As one example, the moveable window can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable window'"'"'s spatiality in multi-dimensional space relative to a stationary window. A display can present an image of a virtual space, and an additional, moveable display can present an additional image of the same virtual space.

Claims

1. A display system to provide an immersive spatial human-machine interface, the display system comprising:

a handheld housing configured and dimensioned to be grasped and supported by the left and right hands at ends thereof, the handheld housing being configured and dimensioned to move in free space to change position and orientation;

an inertial sensor disposed within the housing, the inertial sensor sensing orientation of the handheld housing;

a wireless transceiver disposed within the handheld housing;

a touch display disposed on the handheld housing; and

a distributed graphics generator a first part of which is contained within the handheld housing and a second part of which is colocated with the further display device, the distributed graphics generator first and second parts communicating using the wireless transceiver, the distributed graphics generator spatially correlating at least some aspects of images displayed on the touch display with images displayed on the further display device to provide two spatially correlated windows into the same virtual world, the distributed graphics generator configured to display moving images of a part of the virtual 3D space to provide spatial correlation between images of parts of the virtual world displayed by the touch display and images of parts of the virtual world displayed by the further display device, to thereby provide immersive 3D spatiality,

wherein the system uses the touch display as a pointing device to control the position of a symbol displayed on the further display device.

2. The handheld of claim 1, wherein an image for the touch display represents at least a part of the virtual space which is not represented in an image for display by the further display device.

3. The handheld of claim 1, wherein the further display device displays images from a reference viewing direction, and the touch display displays images of a different/free/arbitrary viewing direction which alters according to attitude of the handheld housing, the different/free/arbitrary viewing direction being based on the reference direction.

4. The interface of claim 1 wherein images the touch display displays represent at least a part of the virtual 3D space which is not represented in an image display by the further display device when the determined attitude aspects are out of the range corresponding to the further display device.

5. The interface of claim 1 wherein images the touch display displays represent at least a part of the virtual 3D space which is represented in an image display by the further display device when the determined attitude aspects are in a range corresponding to the further display device.

6. The handheld of claim 1, wherein sensed handheld attitude defines a virtual camera, and wherein the images displayed by the touch display alter the viewing direction of the virtual camera according to the determined attitude aspects substantially without altering the viewing direction of a virtual camera of the further display device.

7. The interface of claim 6, wherein the position of the virtual camera for the touch display and the position of the virtual camera for the further display device are substantially the same.

8. The interface of claim 1 wherein the touch display displays based on a modified projection viewpoint in response to the determined attitude aspects of the handheld housing.

9. The interface of claim 1 wherein the touch display displays a movable patch of the virtual space that is spatially correlated with the image displayed by the further display device.

10. The handheld of claim 1 wherein the handheld housing orientation determines the position at which a pointing object is displayed on the further display device when the handheld housing is pointed/aimed at the further display device, and displays on the touch display an image of the virtual 3D space from a viewing direction responsive to at least some aspects of attitude of the handheld housing when the determining determines that the handheld housing is not pointed/aimed at the further display.