Spatially-correlated multi-display human-machine interface

• US 9,776,083 B2
• Filed: 11/10/2014
• Issued: 10/03/2017
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Takeda, Genyo, Cheng, Howard
• Primary Examiner(s): RENWICK, REGINALD A
• Application Number:
  US14/537,654
• Publication Number:
  US 20150062122A1
• Time in Patent Office: 1,058 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast

Abstract

A human-machine interface involves plural spatially-coherent visual presentation surfaces at least some of which are movable by a person. Plural windows or portholes into a virtual space, at least some of which are handheld and movable, are provided by using handheld and other display devices. Aspects of multi-dimensional spatiality of the moveable window (e.g., relative to another window) are determined and used to generate images. As one example, the moveable window can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable window'"'"'s spatiality in multi-dimensional space relative to a stationary window. A display can present an image of a virtual space, and an additional, moveable display can present an additional image of the same virtual space.

Claims

1. A display device for use with another display to provide an immersive spatial human-machine interface, the display device comprising:

a housing graspable and supportable by the hands of a user, the housing being configured and dimensioned to move with the user'"'"'s hands to change attitude in free space;

a MARG sensor array disposed within the housing, the MARG sensor array sensing attitude of the housing;

a wireless radio disposed within the housing and coupled to the MARG sensor array, the wireless radio (a) wirelessly transmitting sensed attitude and (b) wirelessly receiving a video signal generated in response to the transmitted sensed attitude; and

a display disposed on the housing, the display coupled to the wireless radio and configured to display moving images of a part of a virtual 3D space responsive to the received video signal to provide spatial coherence between the current attitude of the housing and images of part of the virtual world displayed by the display to thereby provide immersive 3D spatiality without requiring sensing of the position of the housing.

2. The handheld of claim 1, wherein an image displayed by the display disposed on the housing represents at least a part of the virtual space which is not represented in an image for display by the another display.

3. The handheld of claim 1, wherein the another display displays images from a reference viewing direction, and the display disposed on the housing displays images of a different/free/arbitrary viewing direction which alters according to the sensed attitude of the housing, the different/free/arbitrary viewing direction being based on the reference direction.

4. The interface of claim 1 wherein images the display disposed on the housing displays represent at least a part of the virtual 3D space which is not represented in an image display by the another display when the sensed attitude is out of the range corresponding to the another display.

5. The interface of claim 1 wherein the another display is stationary, and images the display disposed on the housing displays represent at least a part of the virtual 3D space which is represented in an image displayed by the stationary display when the sensed attitude is in a range corresponding to the stationary display.

6. The interface of claim 1 wherein the display disposed on the housing displays based on a modified projection viewpoint in response to the determined attitude of the housing.

7. The interface of claim 1 wherein the display disposed on the housing comprises a touch display.

8. The interface of claim 1 wherein the display disposed on the housing displays a movable patch of the virtual space that is spatially correlated with images displayed by the another display.

9. A system comprising:

a handheld display device configured to be supported by the hands of a user, the handheld display device being configured and dimensioned to move with the user'"'"'s hands to change orientations in free space;

at least one orientation sensor disposed within the handheld display device, the at least one orientation sensor sensing at least one parameter indicating changing orientations of the handheld display device;

a wireless radio coupled to the at least one orientation sensor, the wireless radio configured to wirelessly communicate information relating to the sensed at least one parameter indicating the changing orientations of the handheld display device; and

a display disposed on the handheld display device, the display configured to display moving images of a part of a virtual 3D space and provide spatial coherence between the sensed changing orientations of the handheld display device and the moving images of the part of the virtual 3D space displayed by the display,

wherein the system is not required to sense position of the handheld display device to provide said spatial coherence.

10. The system of claim 9 wherein the handheld display device is configured to present a first person perspective porthole view into the virtual space, the porthole view changing based on aspects of the sensed orientation of the handheld display device in multi-dimensional space.

11. The system of claim 9 wherein the display comprises a touch screen and the system is configured to simulate the user touching parts of the virtual 3D space by placing a finger or stylus on the touch screen display.

12. The system of claim 9 wherein the handheld display device correlates its orientation with the orientation of a further handheld display device so that the images displayed by the handheld display device and images displayed by the further handheld display device together represent different but spatially coherent views of the virtual 3D space.

13. The system of claim 9 wherein the handheld display device comprises a housing configured to be held and supported by the user'"'"'s two hands, the housing having arranged thereon manual input controls including a first directional control, a further directional control, a button array and at least one additional control, the housing being configured such that the thumb of the user'"'"'s left hand can operate some of the manual input controls including the first directional control while the thumb of the user'"'"'s right hand can simultaneously operate other of the manual input controls including the further directional control and the button array, and while the user'"'"'s pointer fingers can operate the at least one additional control.

14. The system of claim 9 wherein the handheld display device is configured to be wearable by being attached to a part of the user'"'"'s body.

15. The system of claim 9 wherein the handheld display device further comprises an infrared optical sensor.

16. The system of claim 9 wherein the virtual 3D space comprises a game world.

17. The system of claim 9 wherein the system further comprises an additional display providing a spatial reference that orients the user within the 3D virtual space and provides a perceptual context or anchor for the person using handheld display device, wherein the system is configured so the user can see both the display on the handheld display device and the additional display at the same time, so that the user'"'"'s brain can interpolate or interpret spatially-correlated images displayed on the handheld display device and on the further display as two windows or other views into the same 3D virtual space.

18. The system of claim 17 wherein the additional display is stationary.

19. The system of claim 17 wherein the system does not sense the orientation of the stationary display.

20. The system of claim 17 wherein the system automatically detects when the handheld display device is oriented relative to the additional display to show substantially the same information, and in response automatically switches to display different information on at least one of the displays.

21. A system comprising:

a handheld display device configured to be supported by the hands of a user, the handheld display device being configured and dimensioned to move with the user'"'"'s hands to change attitudes in free space;

at least one sensor disposed within the handheld display device, the at least one sensor sensing at least one parameter relating to the changing attitudes of the handheld display device;

a wireless radio coupled to the at least one sensor, the wireless radio being configured to wirelessly communicate the sensed at least one parameter; and

a display disposed on the handheld display device, the display configured to display moving images of a part of a virtual 3D space and provide spatial coherence between the changing attitudes of the handheld display device and the moving images of the part of the virtual 3D space displayed by the display,

wherein the at least one sensor is not required to sense position and comprises a magnetic angular rate gravity sensing array.

22. A system comprising:

a handheld display device configured to be supported by the hands of a user, the handheld display device being configured and dimensioned to move with the user'"'"'s hands to change orientations in free space;

at least one orientation sensor disposed within the handheld display device, the at least one orientation sensor sensing at least one parameter indicating changing orientations of the handheld display device; and

a display disposed on the handheld display device, the display configured to display moving images of a part of a virtual 3D space and provide spatial coherence between the sensed changing orientations of the handheld display device and the moving images of the part of the virtual 3D space displayed by the display,

wherein the system is not required to sense position of the handheld display device to provide said spatial coherence with respect to further images of another part of the virtual 3D space displayed by at least one further display that does not move with the handheld display device.