SPATIALLY-CORRELATED MULTI-DISPLAY HUMAN-MACHINE INTERFACE

• US 20130063350A1
• Filed: 11/09/2012
• Published: 03/14/2013
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: TAKEDA, Genyo, CHENG, Howard
• Granted Patent: US 8,896,534 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/327 using wireless networks, e.g. Wi-Fi® or piconet
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 13/98 Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/8082 : Virtual reality
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: H04N 21/42202 : environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/4854 : for modifying image parameters, e.g. image brightness, contrast

Abstract

A human-machine interface involves plural spatially-coherent visual presentation surfaces at least some of which are movable by a person. Plural windows or portholes into a virtual space, at least some of which are handheld and movable, are provided by using handheld and other display devices. Aspects of multi-dimensional spatiality of the moveable window (e.g., relative to another window) are determined and used to generate images. As one example, the moveable window can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable window'"'"'s spatiality in multi-dimensional space relative to a stationary window. A display can present an image of a virtual space, and an additional, moveable display can present an additional image of the same virtual space.

Claims

1. A handheld for use with an image of a virtual 3D space displayed on a stationary display to provide an immersive spatial human-machine interface, the handheld comprising:

a housing dimensioned to be grasped and supported by the left and right hands of a user at opposite ends thereof, the housing being configured and dimensioned to move with the user'"'"'s hands in free space to change position and orientation;

a gyroscope disposed within the housing, the gyroscope sensing angular rate of the housing in three axes;

an accelerometer disposed within the housing, the accelerometer sensing linear acceleration of the housing in three axes;

a magnetometer disposed within the housing, the magnetometer sensing direction of the earth'"'"'s magnetic north relative to the housing;

a wireless transceiving arrangement disposed within the housing and coupled to the gyroscope, the accelerometer and the magnetometer, the wireless transceiving arrangement (a) wirelessly transmitting sensed angular rate, sensed linear acceleration and sensed magnetic north direction and (b) wirelessly receiving a video signal generated in response to the transmitted sensed angular rate, sensed linear acceleration and sensed magnetic north direction; and

a display disposed on the housing, the display coupled to the wireless transceiving arrangement and configured to display moving images of a part of the virtual 3D space responsive to the received video signal to provide spatial correlation between the current attitude of the housing and images of part of the virtual world displayed by the display to thereby provide immersive 3D spatiality without requiring the user to be surrounded by displays.

2. The handheld of claim 1, wherein an image for the handheld display represents at least a part of the virtual space which is not represented in an image for display by the stationary display.

3. The handheld of claim 1, wherein the stationary display displayed images from a reference viewing direction, and the handheld display displays images of a different/free/arbitrary viewing direction which alters according to the determined attitude aspects of the handheld display, the different/free/arbitrary viewing direction being based on the reference direction.

4. The interface of claim 1 wherein images the handheld display displays represent at least a part of the virtual 3D space which is not represented in an image display by the stationary display when the determined attitude aspects are out of the range corresponding to the stationary display.

5. The interface of claim 1 wherein images the handheld display displays represent at least a part of the virtual 3D space which is represented in an image display by the stationary display when the determined attitude aspects are in a range corresponding to the stationary display.

6. The handheld of claim 1, wherein the sensed handheld attitude defines a virtual camera, and wherein the images displayed by the handheld display alter the viewing direction of the virtual camera according to the determined attitude aspects substantially without altering the viewing direction of the virtual camera of the stationary display.

7. The interface of claim 6, wherein the position of the virtual camera for the handheld display and the position of the virtual camera for the stationary display are substantially the same.

8. The interface of claim 1 wherein the handheld display displays based on a modified projection viewpoint in response to the determined attitude aspects of the handheld display.

9. The interface of claim 1 wherein the handheld display comprises a touch display.

10. The interface of claim 1 wherein the handheld display displays a movable patch of the virtual space that is spatially correlated with the image displayed by the stationary display.

11. The handheld of claim 1 wherein the handheld display displays moving images that are spatially correlated with the images the stationary display displays.

12. The handheld of claim 1 wherein the handheld orientation determines the position at which a pointing object is displayed on the stationary display when the handheld display is pointed/aimed at the additional display, and displays on the handheld display an image of the virtual 3D space from a viewing direction responsive to at least some aspects of attitude of the handheld display when the determining determines that the movable display is not pointed/aimed at the additional display.

20. The interface of claim 1 wherein the handheld display displays a movable patch of the virtual space that is spatially correlated with the image displayed by a stationary display.

13. A display device for use with a stationary display to provide an immersive spatial human-machine interface, the display device comprising:

a housing graspable and supportable by the hands of a user, the housing being configured and dimensioned to move with the user'"'"'s hands to change attitude in free space;

a MARG sensor array disposed within the housing, MARG sensor array sensing attitude of the housing;

a wireless radio disposed within the housing and coupled to the MARG sensor array, the wireless radio (a) wirelessly transmitting sensed attitude and (b) wirelessly receiving a video signal generated in response to the transmitted sensed attitude; and

a display disposed on the housing, the display coupled to the wireless radio and configured to display moving images of a part of a virtual 3D space responsive to the received video signal to provide spatial coherence between the current attitude of the housing and images of part of the virtual world displayed by the display to thereby provide immersive 3D spatiality.

14. The handheld of claim 13, wherein an image for the handheld display represents at least a part of the virtual space which is not represented in an image for display by a stationary display.

15. The handheld of claim 13, wherein a stationary display displays images from a reference viewing direction, and the handheld display displays images of a different/free/arbitrary viewing direction which alters according to the determined attitude aspects of the handheld display, the different/free/arbitrary viewing direction being based on the reference direction.

16. The interface of claim 13 wherein images the handheld display displays represent at least a part of the virtual 3D space which is not represented in an image display by a stationary display when the determined attitude aspects are out of the range corresponding to the stationary display.

17. The interface of claim 13 wherein images the handheld display displays represent at least a part of the virtual 3D space which is represented in an image display by a stationary display when the determined attitude aspects are in a range corresponding to the stationary display.

18. The interface of claim 13 wherein the handheld display displays based on a modified projection viewpoint in response to the determined attitude of the handheld display.

19. The interface of claim 13 wherein the handheld display comprises a touch display.