SPATIALLY-CORRELATED MULTI-DISPLAY HUMAN-MACHINE INTERFACE

• US 20110285704A1
• Filed: 06/03/2011
• Published: 11/24/2011
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: TAKEDA, Genyo, CHENG, Howard
• Granted Patent: US 8,913,009 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/4854 for modifying image parameters, e.g. image brightness, contrast

Abstract

A human-machine interface involves plural spatially-coherent visual presentation surfaces at least some of which are movable by a person. Plural windows or portholes into a virtual space, at least some of which are handheld and movable, are provided by using handheld and other display devices. Aspects of multi-dimensional spatiality of the moveable window (e.g., relative to another window) are determined and used to generate images. As one example, the moveable window can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable window'"'"'s spatiality in multi-dimensional space relative to a stationary window. A display can present an image of a virtual space, and an additional, moveable display can present an additional image of the same virtual space. Technology is used to determine aspects of the spatiality of the display devices in the physical world, and use the determined spatiality to affect the images displayed on the displays. For example, determined spatiality of the movable display relative to another display can be used to provide relative spatiality of images the two displays present to enhance the user'"'"'s immersive perception. The movable display can context-switch between pointer functionality and spatial functionality depending on whether it is pointing toward a stationary display.

Claims

1. An immersive spatial human-machine interface comprising:

at least one handheld display movable in free space;

an arrangement configured to determine at least some aspects of the attitude of the movable handheld display;

at least one additional display; and

at least one graphical image generator operatively coupled to the handheld display and the additional display, the at least one graphical image generator generating images of a virtual space for display by the handheld display and the additional display,

wherein the at least one graphical image generator generates images of different perspectives viewed from a same virtual location for display by the handheld display and the additional display, at least some of said images generated at least in part in response to the determined attitude aspects to provide spatial correlation between the handheld display and additional display images and thereby enhance the immersive spatiality of the human-machine interface.

2. The interface of claim 1 wherein during a mode when the handheld display and additional display images are practically similar, the at least one graphical image generator substitutes other images and/or information for display by the handheld display.

3. The interface of claim 1 wherein the additional display comprises a relatively large stationary display, and the at least one graphical image generator does not practically alter, except for possible marginal lookaround perspective shifts, the rendering perspective for images generated for the stationary display based on the determined attitude aspects.

4. The interface of claim 3 wherein the at least one graphical image generator alters the rendering perspective for images generated for the stationary display in cases of marginal look-about perspective shifts.

5. The interface of claim 2 wherein during a mode when the perspective for the handheld display and the perspective for the additional display is substantially the same or near, the at least one graphical image generator substitutes other images and/or information for display by the handheld display.

6. An immersive spatial human-machine interface comprising:

at least one handheld display movable in free space;

an arrangement configured to determine at least some aspects of the attitude of the movable handheld display;

at least one additional display; and

at least one graphical image generator operatively coupled to the handheld display and the additional display, the at least one graphical image generator generating different images from a common or nearby location within a virtual space for respective display by the handheld display and the additional display,

wherein the at least one graphical image generator generates images for different viewing directions from the common or nearby location within the virtual space for display by the handheld display and the additional display, at least the images generated for display by the movable display being responsive to the determined attitude aspects to provide spatial correlation between the images displayed by the movable and additional and thereby enhance the immersive spatiality of the human-machine interface.

7. The interface of claim 6, wherein an image for the handheld display represents at least a part of the virtual space which is not represented in an image for display by the additional display.

8. The interface of claim 6, wherein the graphical image generator generates images of a reference viewing directions for display by the additional display and image of a different/free/arbitrary viewing direction which alters according to the determined attitude aspects for display by the handheld display, the different/free/arbitrary viewing direction being based on the reference direction.

9. The interface of claim 8 wherein the graphical image generator generates images for display by each of the handheld display and the additional display so that an image for the handheld display represents at least a part of the virtual space which is not represented in an image for display by the additional display when the determined attitude aspects are out of the range corresponding to the additional display

10. The interface of claim 6 wherein the graphical image generator generates images for display by each of the handheld display and the additional display so that an image for the handheld display represents at least a part of the virtual space which is not represented in an image for display by the additional display when the determined attitude aspects correspond to the predetermined attitude.

11. The interface of claim 6, wherein the graphical image generator generates images for display by the handheld display using a virtual camera for the handheld display and images for display by the additional display using a virtual camera for the additional display, and wherein the graphical image generator alters the viewing direction of the virtual camera for the handheld display according to the determined attitude aspects substantially without altering the viewing direction of the virtual camera of the additional display.

12. The interface of claim 11, wherein the position of the virtual camera for the handheld display and the position of the virtual camera for the additional display are substantially the same.

13. The interface of claim 12, wherein the position of the virtual camera is determined based on the position of a player object, which is changed by a operation of player.

14. The interface of claim 6, wherein the graphical image generator communicates wirelessly with the handheld display.

15. The interface of claim 6, wherein the graphical image generator is a single information processing unit, and wherein the information processing unit generates substantially simultaneously both image for the handheld display and image for the additional display.

16. The interface of claim 6 wherein the additional display comprises a relatively large stationary display, and the graphical image generator does not alter the rendering viewpoint for images generated for the stationary display based on the determined attitude aspects.

17. The interface of claim 6 wherein the movable display comprises a MARG sensor arrangement to determine attitude.

18. The interface of claim 6 wherein the graphical image generator modifies projection viewpoint in response to the determined attitude aspects of the movable display.

19. The interface of claim 6 wherein the movable device is enclosed in a housing, and the arrangement is disposed within the housing.

20. The interface of claim 6 wherein the handheld display comprises a touch display.

21. The interface of claim 6 wherein the arrangement includes at least one magnetometer.

22. The interface of claim 6 wherein the arrangement includes at least one element selected from (a) accelerometer, (b) gyroscope, (c) magnetometer, (d) optical, (e) ultrasonic, (f) infrared.

23. The interface of claim 6 wherein the arrangement is outside-looking-in.

24. The interface of claim 6 wherein the arrangement is inside-looking-out.

25. The interface of claim 6 wherein the movable handheld display displays a 3D movable patch of the virtual space that is spatially correlated with the image displayed by the additional display.

26. The interface of claim 6 wherein the movable handheld display displays 3D images.

27. An immersive spatial human-machine interface comprising:

multiple handheld displays each movable in free space;

an arrangement configured to determine at least some aspects of respective attitudes of the movable handheld displays;

at least one additional display; and

at least one graphical image generator in communication with the handheld displays and the additional display, the at least one graphical image generator generating images of a virtual space for display by each of the multiple handheld displays and the additional display,

wherein the at least one graphical image generator generates respective different perspective images for simultaneous display by the multiple handheld displays and the additional display at least in part in response to the respective determined attitude aspects to provide spatial correlation between all of the images and thereby enhance the immersive spatiality of the human-machine interface.

28. A method for providing an immersive spatial human-machine interface comprising:

measuring at least some aspects of the attitude at least one handheld presentation surface movable in free space; and

generating images of a virtual space for display by the handheld presentation surface and for an additional presentation surface, including generating different images for concurrent display by the handheld presentation surface and the additional presentation surface at least in part in response to the measured attitude aspects to provide spatial coherence between the displayed images and thereby enhance the immersive spatiality of the human-machine interface.

29. A non-transitory storage medium storing instructions that when executed by a computer provide an immersive spatial human-machine interface, the non-transitory storage medium comprising:

first instructions that determine at least some aspects of the attitude at least one handheld presentation surface as the handheld surface moves in free space; and

second instructions that generate images of a virtual space for display by the handheld presentation surface and for an additional presentation surface, including instructions that generate different images for viewing at the same time by the handheld presentation surface and the additional presentation surface at least in part in response to the measured attitude aspects to provide spatial correlation between the displayed images and thereby enhance the immersiveness of the human-machine interface.

30. A user interface method for use with (1) a movable display and (2) an additional display that displays an image of a virtual space, said user interface method comprising:

determining when the movable display is pointed/aimed at the additional display;

when the determining determines that the movable display is pointed/aimed at the additional display, displaying a pointing object on the additional display and controlling the displayed position of the pointing symbol in response to at least some aspects of attitude of the movable display relative to the additional display; and

when the determining determines that the movable display is not pointed/aimed at the additional display, displaying on the movable display an image of the virtual space from a viewing direction responsive to at least some aspects of attitude of the movable display.

31. A user interface including:

a movable display,

an additional display that displays an image of a virtual space,

a determiner that determines when the movable display is pointed/aimed at the additional display; and

an image generator coupled to the movable display and additional display, the image generator displaying a pointing object on the additional display and controlling the displayed position of the pointing symbol in response to at least some aspects of attitude of the movable display relative to the additional display when the determining determines that the movable display is pointed/aimed at the additional display, and displays on the movable display an image of the virtual space from a viewing direction responsive to at least some aspects of attitude of the movable display when the determining determines that the movable display is not pointed/aimed at the additional display.

32. A user interface method for use with (1) a movable display and (2) an additional display that displays an image of a virtual space, said user interface method comprising:

determining at least some aspects of attitude of the movable display;

based on the determined at least some aspects, inferring the posture of the movable display relative to the additional display; and

conditioned on the inferred relative posture between the movable and additional displays, switching context of the movable display between (a) controlling displayed position of a pointing object on the additional display, and (b) displaying on the movable display an image of the virtual space from a viewing direction responsive to the determined at least some aspects of attitude.

33. A user interface method for use with (1) a handheld display movable in free space, and (2) a television that displays an image of a virtual space, said user interface method comprising:

determining at least some aspects of attitude of the handheld display;

based on the determined at least some aspects, inferring the posture of the handheld display relative to the television; and

conditioned on the inferred relative posture, switching context of the handheld display between (a) presenting a lookabout image of the virtual space from viewing directions responsive to the determined at least some aspects of attitude, and (b) presenting user interface control information.

34. The user interface of claim 33 further including detecting, based on the at least some aspects of attitude, whether the handheld display is pointed at a periphery of the television, and selectively shifting the television display in response to said detection.