Storage medium having stored thereon information processing program, and information processing device

• US 9,623,330 B2
• Filed: 09/15/2009
• Issued: 04/18/2017
• Est. Priority Date: 03/26/2009
• Status: Active Grant

• Current Assignee: Nintendo Company Limited, Tohoku University National Universities Corporation
• Original Assignee: Nintendo Company Limited, Tohoku University National Universities Corporation
• Inventors: Yoshizawa, Makoto, Yambe, Tomoyuki, Sugita, Norihiro, Furuta, Norikatsu
• Primary Examiner(s): JANG, CHRISTIAN YONGKYUN
• Application Number:
  US12/559,667
• Publication Number:
  US 20100249494A1
• Time in Patent Office: 2,772 Days
• Field of Search: 600481, 600483, 600484, 600583, 600500, 482 51- 78, 436 36
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/222 Electrical cables or leads therefor, e.g. coaxial cables or ribbon cables
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0059 using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/021 Measuring pressure in heart or blood vessels A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/026 Measuring blood flow A61B3/1233, A61B3/1241 take precedence
• CPC Class Codes: A61B 5/0261 using optical means, e.g. infrared light
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/14552 Details of sensors specially adapted therefor
• CPC Class Codes: A61B 5/24 Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
• CPC Class Codes: A61B 5/318 Heart-related electrical modalities, e.g. electrocardiography [ECG]
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/4035 Evaluating the autonomic nervous system
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6826 Finger
• CPC Class Codes: A61B 5/6838 : Clamps or clips
• CPC Class Codes: A61B 5/6897 : Computer input devices, e.g. mice or keyboards
• CPC Class Codes: A61B 5/7278 : Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/744 : Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: A61B 5/7455 : characterised by tactile indication, e.g. vibration or electrical stimulation
• CPC Class Codes: A61B 5/7475 : User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: A61M 2021/0044 : by the sight sense
• CPC Class Codes: A61M 21/02 : for inducing sleep or relaxation, e.g. by direct nerve stimulation, hypnosis, analgesia for massage A61H; electrotherapy A61N, e.g. applying alternating or intermittent electric currents for producing anaesthesia A61N1/36021
• CPC Class Codes: A61M 2205/3561 : local, e.g. within room or hospital
• CPC Class Codes: A61M 2205/3592 : using telemetric means, e.g. radio or optical transmission
• CPC Class Codes: A61M 2205/502 : User interfaces, e.g. screens or keyboards
• CPC Class Codes: A61M 2205/52 : with memories providing a history of measured variating parameters of apparatus or patient
• CPC Class Codes: A61M 2209/01 : Remote controllers for specific apparatus
• CPC Class Codes: A61M 2230/06 : Heartbeat rate only
• CPC Class Codes: A61M 2230/42 : Rate
• CPC Class Codes: A61M 2230/63 : Motion, e.g. physical activity
• CPC Class Codes: A63F 13/10 : Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/211 : using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 : using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/428 : involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/45 : Controlling the progress of the video game
• CPC Class Codes: A63F 2300/1012 : involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/105 : using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6045 : for mapping control signals received from the input arrangement into game commands

Abstract

A computer readable storage medium having stored thereon an information processing program executable by a computer of an information processing device which presents information corresponding to a biological signal acquired from a user, is provided. The information processing program causes the computer to function as biological signal acquiring means, motion/attitude information acquiring means, and presentation means. The biological signal acquiring means acquires the biological signal from the user. The motion/attitude information acquiring means acquires information about a motion or an attitude of the user from detecting means, in association with the biological signal acquired by the biological signal acquiring means. The presentation means performs predetermined presentation based on both the biological signal acquired by the biological signal acquiring means and the information acquired by the motion/attitude information acquiring means.

Claims

1. A non-transitory computer readable storage medium having stored thereon an information processing program executable by a computer of an information processing system which presents information, on a display device, corresponding to a biological signal acquired from a user, the information processing system having communication circuitry configured to receive the biological signal transmitted from a biological sensor, and configured to receive data related to aspects of motion and/or orientation transmitted from an input device, the information processing program causing the computer to perform operations comprising:

acquiring, via the communication circuitry, the biological signal transmitted from the biological sensor;

acquiring, via the communication circuitry, as information about aspects of motion and/or orientation of the user, the information about aspects of motion and/or orientation of the input device used by the user based on data output from an inertial sensor operatively coupled to the input device; and

presenting, on the display device, a presentation comprising an object that is individually controllable based on both the acquired biological signal and the acquired information about aspects of motion and/or orientation of the input device, the object is individually controllable in at least first and second directions based on, at least, the acquired biological signal.

2. The non-transitory computer readable storage medium according to claim 1, wherein

the information processing program causes the computer to perform further operations comprising:

instructing the user to take a specified motion and/or orientation.

3. The non-transitory computer readable storage medium according to claim 2, wherein

the object is controllable based on the biological signal and the information about aspects of motion and/or orientation of the input device acquired when the instructing is performed.

4. The non-transitory computer readable storage medium according to claim 2, wherein

the information processing program causes the computer to perform further operations comprising:

determining whether or not the user is in the motion and/or orientation which the instructing instructs the user to take, based on the acquired information about aspects of motion and/or orientation of the input device, and

the object is controllable based on the biological signal acquired during or after determining that the user is in the motion and/or orientation that the instructing instructs the user to take.

5. The non-transitory computer readable storage medium according to claim 2, wherein

the instructing instructs the user to take a motion and/or an orientation for causing a predetermined biological signal to be output.

6. The non-transitory computer readable storage medium according to claim 1, wherein

the presenting comprises indicating an operation moving the input device corresponding to a motion of the input device which can be determined by the motion aspect determining.

7. The non-transitory computer readable storage medium according to claim 1, wherein

the presenting comprises indicating a specified motion and/or orientation of the user, based on both the acquired biological signal and the acquired information about aspects of motion and/or orientation of the input device.

8. The non-transitory computer readable storage medium according to claim 7, wherein

the presenting comprises indicating a motion and/or an orientation for causing a change in the biological signal, based on the acquired biological signal.

9. The non-transitory computer readable storage medium according to claim 1, wherein

the biological signal acquiring acquires as the biological signal at least a signal relating to pulsation or heartbeat of the user.

10. The non-transitory computer readable storage medium according to claim 1, wherein

the biological signal acquiring acquires as the biological signal at least one selected from the group consisting of a pulse wave, a heart rate, an activity level of the sympathetic nervous system, an activity level of the parasympathetic nervous system, a heart rate variance coefficient, a cardiac cycle, a respiration frequency, and a pulse wave amplitude of the user.

11. The non-transitory computer readable storage medium according to claim 10, wherein

the biological signal acquiring acquires at least the pulse wave amplitude of the user, and

the presenting comprises determining a difficulty level of the user using a change in the pulse wave amplitude of the user, and presenting an indication of a motion based on a result of the difficulty level determining.

12. The non-transitory computer readable storage medium according to claim 1, wherein

the input device includes an acceleration sensor, and

the motion determining determines aspects of a motion of the input device using an acceleration indicated by acceleration-related data outputted from the acceleration sensor.

13. The non-transitory computer readable storage medium according to claim 12, wherein

the motion determining determines aspects of a motion of the input device based on an inclination of the input device with reference to a direction of gravity, where the direction of gravity is the acceleration indicated by the acceleration-related data outputted from the acceleration sensor.

14. The non-transitory computer readable storage medium according to claim 1, wherein

the input device includes a gyro-sensor, and

the motion determining determines aspects of a motion of the input device using an angular velocity indicated by angular-velocity-related data outputted from the gyro-sensor.

15. The non-transitory computer readable storage medium according to claim 1, wherein

the presenting comprises further displaying on the display device at least either of an image and a character object generated based on both the acquired biological signal and the acquired information about aspects of motion and/or orientation of the input device.

16. The non-transitory computer readable storage medium according to claim 1, wherein

the presenting comprises controlling a motion of the controllable object displayed on the display device, based on both the acquired biological signal and the acquired information about aspects of motion and/or orientation of the input device.

17. The non-transitory computer readable storage medium according to claim 16, wherein

the presenting is performed so that the controllable object performs a first motion in a virtual world based on the acquired biological signal, and the controllable object performs a second motion different from the first motion in the virtual world based on the acquired information about aspects of motion and/or orientation of the input device.

18. The non-transitory computer readable storage medium according to claim 17, wherein

the presenting moves at least a portion of the controllable object in the virtual world as one of the first and second motions, and changes an orientation of the controllable object in the virtual world as the other of the first and second motions.

19. The non-transitory computer readable storage medium according to claim 17, wherein

the presenting moves at least a portion of the controllable object in a first direction in the virtual world as one of the first and second motions, and moves at least a portion of the controllable object in a second direction in the virtual world as the other of the first and second motions.

20. The non-transitory computer readable storage medium according to claim 16, wherein

the information processing program causes the computer to perform further operations comprising:

determining a motion of the controllable object in the virtual world.

21. The non-transitory computer readable storage medium according to claim 20, wherein

the information processing program causes the computer to perform further operations comprising:

instructing the user to take a predetermined motion and/or orientation by causing a topographical object for designating a motion of the controllable object to appear in the virtual world, and

the object motion determining determines whether or not the controllable object contacts the topographical object.

22. The non-transitory computer readable storage medium according to claim 21, wherein

the instructing changes the topographical object, depending on the acquired biological signal.

23. The non-transitory computer readable storage medium according to claim 17, wherein

the presenting moves at least a portion of the controllable object in a specified direction as the first motion, depending on an acquired biological signal relating to respiration of the user.

24. The non-transitory computer readable storage medium according to claim 23, wherein

the presenting comprises further displaying on the display device an obstacle in the virtual world which limits a movement in the specified direction of the controllable object, the controllable object'"'"'s rising and falling based on an acquired respiration frequency of the user, and

the information processing program causes the computer to perform further operations comprising:

degrading assessment when the controllable object contacts or overlaps the rising and falling obstacle in the virtual world.

25. The non-transitory computer readable storage medium according to claim 17, wherein

the biological signal acquiring acquires at least a pulse wave amplitude of the user, and

the presenting comprises determining a difficulty level of the user using a change in the acquired pulse wave amplitude of the user, and when determining the user has difficulty, changing a way in which the controllable object is displayed.

26. The non-transitory computer readable storage medium according to claim 25, wherein

the presenting changes and inclines an orientation of the controllable object in the virtual world as the second motion based on the acquired information about aspects of motion and/or orientation of the input device,

the information processing program causes the computer to perform further operations comprising:

degrading assessment when the controllable object contacts or overlaps an obstacle in the virtual world, an inclination angle of the obstacle limiting an inclining motion of the controllable object in the virtual world, and

gradually increasing and changing the inclination angle of the obstacle with time, and when the user is determined to have difficulty, stopping changing the inclination angle of the obstacle.

27. The non-transitory computer readable storage medium according to claim 1, wherein

the presenting comprises outputting audio based on both the acquired biological signal and the acquired information about the aspects of motion and/or orientation of the input device.

28. The non-transitory computer readable storage medium according to claim 1, wherein

the object is individually controlled in the first direction based on an increasing value associated with the acquired biological signal, and

the object is individually controlled in the second direction based on a decreasing value associated with the acquired biological signal.

29. The non-transitory computer readable storage medium according to claim 1, wherein the information processing program further causing the computer to perform operations comprising:

storing, in a memory of the information processing system, biological data associated with the acquired biological signal and motion and/or orientation data associated with the acquired information about aspects of motion and/or orientation of the input device; and

presenting, on the display device, the presentation comprising the object that is individually controllable based on both the stored biological data and the stored motion and/or orientation data, the object is individually controllable in at least first and second directions based on, at least, the stored biological data.

30. An information processing device configured to present information corresponding to a biological signal acquired from a user, the information processing device comprising:

communication circuitry configured to receive the biological signal transmitted from a biological sensor, and configured to receive data related to aspects of motion and/or orientation transmitted from an input device; and

processing circuitry having at least one processor, the processing circuitry configured to;

acquire, via the communication circuitry, the biological signal transmitted from the biological sensor,

acquire, via the communication circuitry, as information about aspects of motion and/or orientation of the user, the information about aspects of motion and/or orientation of the input device used by the user based on data output from an inertial sensor operatively coupled to the input device, and

present, on a display device, a presentation comprising an object that is individually controllable based on both the acquired biological signal and the acquired information about aspects of motion and/or orientation of the input device, the object is individually controllable in at least first and second directions based on, at least, the acquired biological signal.

31. The information processing device according to claim 30, wherein the controllable object is in a virtual world.

32. The information processing device according to claim 30, wherein the presentation prompts the user to act so as to change the user'"'"'s biological signal.

33. A method implemented using at least one programmed computer, the method comprising:

acquiring, via communication circuitry, a biological signal transmitted from a biological sensor;

acquiring, via the communication circuitry, as information about aspects of motion and/or orientation of the user, information about aspects of motion and/or orientation of an input device used by the user based on data output from an inertial sensor operatively coupled to the input device; and

presenting, on a display, a presentation comprising an object that is individually controllable based on both the acquired biological signal and the acquired information, the object is individually controllable in at least first and second directions based on, at least, the acquired biological signal.

34. The method according to claim 33, wherein the controllable object is in a virtual world.

35. A system comprising:

a biological sensor for generating a biological signal associated with a biological function of a user;

a sensor for generating a signal representing aspects of motion and/or orientation of an input device; and

an information processing device, the information processing device comprising;

communication circuitry configured to receive the biological signal transmitted from the biological sensor, and configured to receive data related to aspects of motion and/or orientation transmitted from the input device, and

a processing system for generating, on a display, a presentation comprising an object that is individually controllable, based on both the biological signal and the signal representing aspects of motion and/or orientation of the input device, the object is individually controllable in at least first and second directions based on, at least, the generated biological signal associated with the biological function of the user.

36. The system according to claim 35, wherein the controllable object is in a virtual world.