Spatially-correlated multi-display human-machine interface

• US 8,896,534 B2
• Filed: 11/09/2012
• Issued: 11/25/2014
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Takeda, Genyo, Cheng, Howard
• Primary Examiner(s): SHERMAN, STEPHEN G
• Application Number:
  US13/672,862
• Publication Number:
  US 20130063350A1
• Time in Patent Office: 746 Days
• Field of Search: 345156-178, 463 30- 32, 463 36- 38
• US Class Current: 345/158
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/323 between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines
• CPC Class Codes: A63F 13/327 using wireless networks, e.g. Wi-Fi® or piconet
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 13/98 Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/8082 : Virtual reality
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 : 3D [Three Dimensional] image rendering
• CPC Class Codes: H04N 21/42202 : environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/4854 : for modifying image parameters, e.g. image brightness, contrast

Abstract

A human-machine interface involves plural spatially-coherent visual presentation surfaces at least some of which are movable by a person. Plural windows or portholes into a virtual space, at least some of which are handheld and movable, are provided by using handheld and other display devices. Aspects of multi-dimensional spatiality of the moveable window (e.g., relative to another window) are determined and used to generate images. As one example, the moveable window can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable window'"'"'s spatiality in multi-dimensional space relative to a stationary window. A display can present an image of a virtual space, and an additional, moveable display can present an additional image of the same virtual space.

Claims

1. A handheld device for displaying images of a virtual 3D object from different viewing perspectives, the handheld device comprising:

a handheld housing freely movable in free space and dimensioned to be grasped and supported by the hands of a user, the handheld housing being configured and dimensioned to change attitude in free space with movement of the user'"'"'s hands;

a gyroscopic sensor disposed within the housing, the gyroscopic sensor being configured to sense rotation in multiple axes;

an acceleration sensor disposed within the housing, the acceleration sensor being configured to sense acceleration in multiple axes;

a direction sensor disposed within the housing, the direction sensor being configured to sense direction;

a wireless transceiver disposed within the housing and operatively coupled to the gyroscopic sensor, the acceleration sensor and the direction sensor, the wireless transceiver configured to (a) wirelessly transmit information related to the sensed rotation, the sensed acceleration and the sensed direction, and (b) wirelessly receive a video signal encoding images of the virtual 3D object from viewing perspectives that continually update in response to sensed rotation, acceleration and direction of the handheld housing; and

a handheld display disposed on the handheld housing, the handheld display operatively coupled to the wireless transceiver and configured to display the virtual 3D object with continually updating viewing perspectives to spatially correlate 3D viewing perspective of the displayed virtual 3D object with free space attitude of the housing.

2. The handheld device of claim 1, further including a further display that simultaneously displays an image of the virtual 3D object from a reference viewing perspective, and wherein the handheld display shows the virtual 3D object from viewing perspective which are different from the reference viewing perspective displayed by the further display.

3. The handheld device of claim 2 wherein images the handheld display displays represent at least a part of a virtual 3D space which is not represented in an image displayed by the further display when the attitude of the handheld housing is out of a range corresponding to an attitude of the further display.

4. The handheld device of claim 2 wherein images the handheld display displays represent at least a part of a virtual 3D space which is represented in an image displayed by the further display when the attitude of the handheld housing is in a range corresponding to an attitude of the further display.

5. The handheld device of claim 1, wherein the housing attitude is used to define the viewing direction of a first virtual camera, and wherein images displayed by the handheld display alter based on the viewing direction of the first virtual camera according to the attitude of the housing substantially without altering the reference viewing direction of a further virtual camera for the further display.

6. The handheld device of claim 5, wherein a virtual position of the first virtual camera and a virtual position of the further virtual camera are substantially the same.

7. The handheld device of claim 1 wherein the handheld display displays based on a modified projection viewpoint in response to the attitude of the handheld housing.

8. The handheld device of claim 1 wherein the handheld display comprises a touch display.

9. The handheld device of claim 1 wherein the handheld display displays a movable patch of a virtual 3D space that is spatially correlated with a scene simultaneously displayed by a further display.

10. The handheld device of claim 1 wherein the handheld display displays moving images spatially correlated with images a further display simultaneously displays.

11. The handheld device of claim 1 wherein the attitude determines the position at which a pointing object is displayed on a further display when the handheld device is pointed/aimed at the further display, and the handheld device displays, on the handheld display, an image of a virtual 3D space from a viewing direction responsive to the attitude of the handheld housing at least when the handheld device is not pointed/aimed at the further display.

12. A display device comprising:

a handheld housing graspable and supportable by the hands of a user, the housing being configured and dimensioned to move in free space;

a MARG sensor array disposed within the housing, the MARG sensor array sensing attitude of the housing;

a wireless radio disposed within the housing and coupled to the MARG sensor array, the wireless radio (a) wirelessly transmitting the sensed attitude of the housing and (b) wirelessly receiving a video signal encoding images of a virtual world viewed from a viewpoint responsive to the transmitted sensed attitude; and

a display disposed on the housing, the display coupled to the wireless radio and configured to display the images of the virtual world responsive to the received video signal to provide spatial coherence between the attitude of the housing in the real world and the viewpoint of the virtual world displayed by the display.

13. The display device of claim 12, wherein the viewpoint displayed by the display represents a view of the virtual world which is not represented in a view of the virtual world simultaneously displayed by a further display.

14. The display device of claim 12, wherein a further display simultaneously displays images of the virtual world from a reference viewing direction, and the display disposed on the housing displays images of the virtual world from a viewing direction which changes according to the sensed attitude of the handheld housing, the changing viewing direction being based on the reference viewing direction and the sensed handheld housing attitude.

15. The display device of claim 12 wherein images the display displays represent a view of the virtual world which is not represented in an image of the virtual world simultaneously displayed by a further display when the sensed attitude is out of range relative to an attitude of the further display.

16. The display device of claim 12 wherein images the display displays represent views of the virtual world which are represented in images of the virtual world simultaneously displayed by a further display when the sensed attitude of the housing is in a range corresponding to an attitude of the further display.

17. The display device of claim 12 wherein the display displays images generated based on a modified projection viewpoint in response to the sensed attitude of the housing.

18. The display device of claim 12 wherein the display comprises a touch display configured to receive touch input.

19. The display device of claim 12 wherein the display displays a movable patch of the virtual world that is spatially correlated with the displayed orientation of the virtual world displayed by a further display.