Information processing system, information processing apparatus, storage medium having information processing program stored therein, and image display method for controlling virtual objects based on at least body state data and/or touch position data

• US 9,370,712 B2
• Filed: 01/17/2012
• Issued: 06/21/2016
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Hayashi, Yugo, Minato, Keiichi
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US13/352,091
• Publication Number:
  US 20120229510A1
• Time in Patent Office: 1,617 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/214 for locating contacts on a surface, e.g. floor mats or touch pads
• CPC Class Codes: A63F 13/218 using pressure sensors, e.g. generating a signal proportional to the pressure applied by the player
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/816 Athletics, e.g. track-and-field sports
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1068 being specially adapted to detect the point of contact of the player on a surface, e.g. floor mat, touch pad
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/6684 by dynamically adapting its position to keep a game object in its viewing frustrum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 2300/8005 Athletics
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/20 : Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 3/60 : Rotation of whole images or parts thereof

Abstract

An action of a first object placed in a virtual world is controlled on the basis of body state data output from a portable display apparatus. An action of a second object placed in the virtual world is controlled on the basis of touch position data based on a touch position on a touch panel provided on a surface of a display screen of the portable display apparatus. Then, a first image including at least a part of the first object and at least a part of the second object is generated, and the first image is displayed on the portable display apparatus.

Claims

1. A non-transitory computer-readable storage medium storing an information processing program to be executed by a computer of an information processing apparatus capable of displaying an image on a portable display apparatus that outputs at least body state data based on an attitude and/or a motion of the portable display apparatus body and touch position data based on a touch position on a touch panel provided in the portable display apparatus, the information processing program, when executed, causing the computer to provide comprising:

calculating the attitude and/or the motion of the portable display apparatus on the basis of the body state data;

controlling, on the basis of the calculated attitude and/or motion of the portable display apparatus, an action of a first object without causing a second object to perform an action, the first object and the second object placed in a virtual world;

controlling, on the basis of the touch position data, an action of the second object without causing the first object to perform an action, the first object and the second object placed in the virtual world;

generating a first image including at least a part of the first object and at least a part of the second object; and

displaying the first image on the portable display apparatus,

if a motion of rotation of the portable display apparatus is calculated, the action of the first object is controlled so that the first object rotates relative to a display screen of the portable display apparatus, and the action of the second object is controlled so that the second object does not rotate relative to the display screen of the portable display apparatus.

2. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 1, wherein

the action of the first object is controlled on the basis of the attitude and/or the motion of the portable display apparatus such that the first object rotates, and

the action of the second object is controlled on the basis of the touch position data such that the second object moves.

3. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 2, wherein

the action of the first object is controlled such that the first object rotates relative to the second object, and

the action of the second object is controlled such that the second object moves relative to the first object.

4. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 2, the information processing program further causing the computer to provide execution comprising:

acquiring data based on a load applied to a load detection device, wherein

the second object is controlled on the basis of the acquired data so as to move in a moving direction defined by a direction of the first object.

5. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 4, wherein

only when the touch position data indicates that a touch operation is not being performed on the touch panel, the second object is controlled on the basis of the acquired data so as to move.

6. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 2, the information processing program further causing the computer to provide execution comprising:

calculating, on the basis of the touch position data, a displacement of the touch position input to the touch panel, wherein

the second object is controlled on the basis of the displacement of the touch position so as to move.

7. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 6, wherein

a moving direction in which the second object is to be caused to move is calculated on the basis of a direction of the displacement of the touch position, and the second object is controlled so as to move in the moving direction.

8. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 7, wherein

a direction corresponding to the direction of the displacement and included in the virtual world displayed so as to overlap the displacement of the touch position is calculated as the moving direction, and the second object is controlled so as to move in the moving direction.

9. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 6, wherein

a moving distance at which the second object is to be caused to move is calculated on the basis of an amount of the displacement of the touch position, and the second object is controlled so as to move at the moving distance.

10. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 9, wherein

a distance corresponding to the amount of the displacement and included in the virtual world displayed so as to overlap the displacement of the touch position is calculated as the moving distance, and the second object is controlled so as to move at the moving distance.

11. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 1, wherein

at least an attitude and/or a motion of the portable display apparatus that are obtained by rotating the portable display apparatus about a perspective direction of, and perpendicular to, a display screen of the portable display apparatus are calculated, and

the action of the first object is controlled in accordance with the attitude and/or the motion of the portable display apparatus that are obtained by rotating the portable display apparatus about the perspective direction.

12. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 11, wherein

the action of the first object is controlled such that in accordance with a direction of rotation in which the portable display apparatus rotates about the perspective direction, the first object rotates or moves about a predetermined direction set in the virtual world and in a direction opposite to the direction of rotation.

13. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 11, wherein

the action of the first object is controlled such that the first object rotates or moves in accordance with an angle by which a facing direction of the portable display apparatus body changes about the perspective direction.

14. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 13, wherein

the action of the first object is controlled such that the first object rotates or moves by the same angle as the angle by which the facing direction of the portable display apparatus body changes.

15. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 11, wherein

the action of the first object is controlled such that in accordance with the attitude and/or the motion of the portable display apparatus that are obtained by rotating the portable display apparatus about the perspective direction, the first object rotates or moves about a predetermined direction set in the virtual world.

16. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 11, wherein

the action of the first object is controlled such that on the basis of the attitude and/or the motion of the portable display apparatus with respect to a direction of gravity in real space, and in accordance with the attitude and/or the motion of the portable display apparatus that are obtained by rotating the portable display apparatus about the perspective direction, the first object rotates or moves about a direction of gravity set in the virtual world.

17. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 11, wherein

the action of the first object is controlled such that in accordance with a direction of rotation and an angle of rotation in and by which the portable display apparatus rotates about the perspective direction, the first object rotates about a predetermined direction in the virtual world in a direction opposite to the direction of rotation and by the angle of rotation, and

when the touch position data indicates an operation of dragging the touch panel, the action of the second object is controlled such that in accordance with a moving direction and a moving distance of the touch position in the virtual world displayed on the display screen so as to overlap the touch position indicated by the touch position data, the second object moves along a plane perpendicular to the predetermined direction in the virtual world in the moving direction and at the moving distance.

18. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 2, the information processing program further causing the computer to provide execution comprising:

setting a first virtual camera for generating the first image, and control an attitude and/or a position of the first virtual camera on the basis of the calculated attitude and/or motion of the portable display apparatus, wherein

at least an attitude and/or a motion of the portable display apparatus that are obtained by rotating the portable display apparatus about a perspective direction of, and perpendicular to, a display screen of the portable display apparatus are calculated,

the action of the first object is controlled such that in accordance with a direction of rotation and an angle of rotation in and by which the portable display apparatus rotates about the perspective direction, the first object rotates about a direction of a line of sight of the first virtual camera in the direction of rotation and by the angle of rotation,

the attitude of the first virtual camera is controlled such that in accordance with the direction of rotation and the angle of rotation in and by which the portable display apparatus rotates about the perspective direction, the first virtual camera rotates about the direction of the line of sight in the direction of rotation and by the angle of rotation,

when the touch position data indicates an operation of dragging the touch panel, the action of the second object is controlled such that in accordance with a moving direction and a moving distance of the touch position in the virtual world displayed on the display screen so as to overlap the touch position indicated by the touch position data, the second object moves along a plane perpendicular to the direction of the line of sight in the virtual world in a direction opposite to the moving direction and at the moving distance, and

when the touch position data indicates the operation of dragging the touch panel, the position of the first virtual camera is controlled such that in accordance with the moving direction and the moving distance of the touch position in the virtual world displayed on the display screen so as to overlap the touch position indicated by the touch position data, the first virtual camera moves along the plane perpendicular to the direction of the line of sight in the virtual world in the direction opposite to the moving direction and at the moving distance.

19. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 1, wherein

the portable display apparatus includes at least one of a gyro sensor and an acceleration sensor, and

the attitude and/or the motion of the portable display apparatus are calculated on the basis of data output from the at least one of the gyro sensor and the acceleration sensor.

20. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 1, wherein

image data indicating the first image is output to the portable display apparatus,

the portable display apparatus includes an image data acquisition device that acquires the image data output from the information processing apparatus, and

a display screen of the portable display apparatus displays the first image indicated by the acquired image data.

21. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 20, the information processing program further causing the computer to provide execution comprising:

generating compression image data by compressing the image data indicating the first image, wherein

the generated compression image data is output to the portable display apparatus,

the image data acquisition device acquires the compression image data output from the information processing apparatus,

the portable display apparatus further includes a display image decompression device that decompresses the compression image data to obtain the image data indicating the first image, and

the display screen of the portable display apparatus displays the first image indicated by the image data that has been acquired by the image data acquisition device and has been decompressed by the display image decompression device.

22. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 1, wherein

besides the first image, a second image representing the virtual world is further displayed on another display apparatus connected to the information processing apparatus.

23. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 22, the information processing program further causing the computer to provide execution comprising:

generating compression image data by compressing the image data indicating the first image, wherein

the generated compression image data is output to the portable display apparatus, and, besides the compression image data, image data indicating the second image is output to said another display apparatus without being compressed, and

the portable display apparatus configured to;

acquire the compression image data output from the information processing apparatus; and

decompress the compression image data to obtain the image data indicating the first image, wherein

a display screen of the portable display apparatus displays the first image indicated by the acquired and decompressed image data.

24. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 22, wherein

an image including at least a part of the first object and at least a part of the second object in the virtual world viewed from a point of view different from a point of view toward the virtual world for generating the first image is displayed as the second image on said another display apparatus.

25. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 22, wherein

a point of view toward the virtual world for generating the second image is set at a position further away from the first object or the second object than a point of view toward the virtual world for generating the first image is from the first object or the second object, and

a range wider than a range of the virtual world represented by the first image is displayed as the second image on said another display apparatus.

26. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 22, wherein

a point of view for generating the second image is set at a position of viewing from a bird'"'"'s-eye view the first object and the second object in the virtual world, and

an image obtained by viewing from a bird'"'"'s-eye view at least a part of the first object placed in the virtual world and at least a part of the second object placed in the virtual world is displayed as the second image on said another display apparatus.

27. The non-transitory computer-readable storage medium having stored thereon the information processing program according to claim 22, the information processing program further causing the computer to provide execution comprising:

setting a second virtual camera for generating the second image, and control an attitude and/or a position of the second virtual camera, wherein

the attitude of the second virtual camera is controlled such that in accordance with a direction of rotation and an angle of rotation in and by which the portable display apparatus rotates about a perspective direction of, and perpendicular to, a display screen of the portable display apparatus and which are calculated on the basis of the body state data, the second virtual camera rotates about a direction of a line of sight in a direction opposite to the direction of rotation and by the angle of rotation.

28. An information processing apparatus comprising processing circuitry having at least one processor, the information processing apparatus capable of displaying an image on a portable display apparatus that outputs at least body state data based on an attitude and/or a motion of the portable display apparatus body and touch position data based on a touch position on a touch panel provided in the portable display apparatus, the processing circuitry controlling the information processing apparatus to:

calculate the attitude and/or the motion of the portable display apparatus on the basis of the body state data;

control, on the basis of the calculated attitude and/or motion of the portable display apparatus, an action of a first object without causing a second object to perform an action, the first object and the second object placed in a virtual world;

control, on the basis of the touch position data, an action of the second object without causing the first object to perform an action, the first object and the second object placed in the virtual world;

generate a first image including at least a part of the first object and at least a part of the second object; and

display the first image on the portable display apparatus,

if a motion of rotation of the portable display apparatus is calculated, the action of the first object is controlled so that the first object rotates relative to a display screen of the portable display apparatus, and the action of the second object is controlled so that the second object does not rotate relative to the display screen of the portable display apparatus.

29. An information processing system including a plurality of apparatuses configured to communicate with each other, the information processing system capable of displaying an image on a portable display apparatus that outputs at least body state data based on an attitude and/or a motion of the portable display apparatus body and touch position data based on a touch position on a touch panel provided in the portable display apparatus, the information processing system comprising:

a processing system having at least one processor, the processing system configured to;

calculate the attitude and/or the motion of the portable display apparatus on the basis of the body state data;

control, on the basis of the calculated attitude and/or motion of the portable display apparatus, an action of a first object without causing a second object to perform an action, the first object and the second object placed in a virtual world;

control, on the basis of the touch position data, an action of the second object without causing the first object to perform an action, the first object and the second object placed in the virtual world;

generate a first image including at least a part of the first object and at least a part of the second object; and

display the first image on the portable display apparatus,

if a motion of rotation of the portable display apparatus is calculated, the action of the first object is controlled so that the first object rotates relative to a display screen of the portable display apparatus, and the action of the second object is controlled so that the second object does not rotate relative to the display screen of the portable display apparatus.

30. An image processing method performed by a processor or a cooperation of a plurality of processors included in an image processing system including at least one information processing apparatus capable of displaying an image on a portable display apparatus that outputs at least body state data based on an attitude and/or a motion of the portable display apparatus body and touch position data based on a touch position on a touch panel provided in the portable display apparatus, the information processing method comprising:

calculating the attitude and/or the motion of the portable display apparatus on the basis of the body state data;

controlling, on the basis of the calculated attitude and/or motion of the portable display apparatus, an action of a first object without causing a second object to perform an action, the first object and the second object placed in a virtual world;

controlling, on the basis of the touch position data, an action of the second object without causing the first object to perform an action, the first object and the second object placed in the virtual world;

generating a first image including at least a part of the first object and at least a part of the second object; and

displaying the first image on the portable display apparatus,

if a motion of rotation of the portable display apparatus is calculated, the action of the first object is controlled so that the first object rotates relative to a display screen of the portable display apparatus, and the action of the second object is controlled so that the second object does not rotate relative to the display screen of the portable display apparatus.