Systems and methods for processing positional and sound input of user input to a touch panel

• US 9,498,708 B2
• Filed: 12/12/2008
• Issued: 11/22/2016
• Est. Priority Date: 12/25/2007
• Status: Expired due to Fees

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Masuda, Takumi, Kanata, Ryosuke, Nakano, Yoshinobu, Nishimura, Masahiko
• Primary Examiner(s): Bradford, Peter
• Application Number:
  US12/333,867
• Publication Number:
  US 20090163282A1
• Time in Patent Office: 2,902 Days
• Field of Search: 463/7, 463/36, 463/37, 345/177
• US Class Current: 1/1
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1068 being specially adapted to detect the point of contact of the player on a surface, e.g. floor mat, touch pad
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6072 of an input signal, e.g. pitch and rhythm extraction, voice recognition

Abstract

A game apparatus detects an input position, on an input plane, at which an input operation has been performed. The game apparatus also detects an operation sound that occurs due to the input operation performed on the input plane. The game apparatus evaluates the detected operation sound, and obtains a result of the evaluation (e.g., volume of the operation sound). The game apparatus performs game processing by using the detected input position and the result of the evaluation. This enables the game apparatus to perform more complex processing by using a touch operation as input information, and allows a player to provide more complex instructions based on the touch operation.

Claims

1. A non-transitory computer-readable storage medium storing computer readable instructions to be executed by a computing system that includes an input device having a touch screen display that is configured to detect a touch operation that is performed by a user, and a microphone, the stored instructions comprising instructions that are configured to:

output an image to the touch screen display;

obtain, from the input device, an input position which corresponds to an input operation performed by a user at a touched position on the touch screen display while the image is displayed on the touch screen display;

obtain, from the microphone, an operation sound based on the performed input operation; and

perform a game process based on the obtained input position and the obtained operation sound,

wherein the input position is a coordinate input position,

wherein the input position is obtained independently of the operation sound.

2. The medium of claim 1, wherein the instructions are further configured to calculate, by performing a calculation process that is based on the operation sound, feature information indicating a feature of the operation sound,

wherein the game process is further based on the feature information, and

the calculation process determines a detail in accordance with the obtained input position.

3. The medium of claim 2, wherein the instructions are further configured to:

determine a characteristic of a filter based on the input position; and

perform a filtering process on the operation sound, by using the determined characteristic of the filter, to obtain filtered operation sound,

wherein the calculation of the feature information from the operation sound is further based on the filtered operation sound.

4. The medium of claim 3, wherein the instructions are further configured to determine a frequency passband that is to be used for the filter based on the input position.

5. The medium of claim 2, wherein the instructions are further configured to:

determine an amplification rate for amplifying a signal of the operation sound based on the input position;

amplify the signal of the operation sound based on the determined amplification rate to obtain amplified operation sound data; and

calculate a volume of the operation sound as the feature information based on the amplified operation sound data.

6. The medium of claim 2, wherein the instructions are further configured to:

instruct a user to perform a second input operation on the touch screen display;

obtain, from the microphone, a second operation sound occurring due to the second input operation that is performed at a position within the predetermined area of the touch screen display;

determine, based on the second operation sound, the detail; and

generate and store association data that associates the predetermined area with the determined detail,

wherein the predetermined calculation process determines the detail based on the obtained input position being within the predetermined area of the stored association data.

7. The medium of claim 2, wherein the instructions are further configured to:

instruct a user to perform a second input operation on the touch screen display;

obtain, from the microphone, a second operation sound occurring due the second input operation that is performed on a second position within a predetermined area of the touch screen display;

generate and store association data that associates the predetermined area with the second operation sound;

select the second operation sound from the association data when the input position is within the predetermined area associated with the second operation sound; and

compare the second operation sound to the operation sound.

8. The medium of claim 1, wherein the instructions are further configured to:

determine whether or not the obtained operation sound satisfies a predetermined standard, the predetermined standard including using different reference values depending on the obtained input position; and

perform a game process using the obtained input position when the operation sound satisfies the predetermined standard.

9. The medium of claim 8, wherein the instructions are further configured to:

instruct a user to perform a second input operation on the touch screen display;

obtain, from the microphone, a second operation sound occurring due to the second input operation that is performed, in response to a provided instruction, at a second position within a predetermined area of the touch screen display;

set, based on the obtained second operation sound, a reference value used for when the input operation is performed on a position within the predetermined area; and

generate and store association data which associates the predetermined area with the set reference value,

wherein the game process uses the reference value which is associated with the predetermined area stored in the association data.

10. The medium of claim 1, wherein the instructions are further configured to:

calculate, by performing a predetermined calculation process on the operation sound, feature information indicating a feature of the operation sound; and

correct the feature information in accordance with the input position.

11. The medium of claim 10, wherein the instructions are further configured to calculate, as the feature information, volume of the operation sound.

12. The medium of claim 1, wherein the game process includes causing a controlled object to perform a predetermined action based on the input position and the operation sound.

13. The medium of claim 12, wherein

the image includes a first object,

wherein the game process includes:

determining, when the input position and a display position of the first object are in a predetermined relationship, whether or not to cause a second object, which is the same as or different from the first object, to perform the predetermined action; and

causing, when the second object is to perform the predetermined action, the second object to perform an action that corresponds to the operation sound.

14. The medium of claim 12, wherein

the image includes a first object,

wherein the game process includes:

specifying, when the input position and a display position of the first object are in a predetermined relationship, a second object which is the same as or different from the first object; and

causing the second object to perform an action corresponding to the operation sound.

15. The medium of claim 12, wherein

the image includes the controlled object,

wherein the game process includes:

determining, based on the input position, a value of a first game parameter relating to the predetermined action of the controlled object; and

determining, based on the operation sound, a value of a second game parameter that is different from the first game parameter and which relates to the predetermined action of the controlled object.

16. The medium of claim 13, wherein the instructions are further configured to determine, based on the volume of the operation sound, an amount of the action that corresponds to the operation sound of the second object.

17. The medium of claim 13, wherein the instructions are further configured to determine, based on the operation sound, a type of the action of the second object.

18. The medium of claim 1, wherein the instructions are further configured to:

specify, based on the input position, a position in a game space; and

determine, based on the operation sound, a detail of an effect to be provided to the game space in the specified position.

19. The medium of claim 18, wherein the instructions are further configured to determine a degree of the effect in accordance with volume of the operation sound.

20. The medium of claim 18, wherein the instructions are further configured to determine a type of the effect in accordance with the operation sound.

21. The medium of claim 1, wherein the instructions are further configured to:

detect an input performed on the surface; and

obtain, as the operation sound, a sound which is detected by the microphone at a timing that is set in relation to a point when the input is detected.

22. The medium of claim 1, wherein:

the instructions are further configured to process the operation sound based on the obtained input position to obtain changed sound data, and

the game process is further based on the changed sound data.

23. A game apparatus comprising:

an input position detecting unit which detects an input position that corresponds to an input operation performed by a user at a touched position on a touch screen display while an image is displayed thereon;

an operation sound detecting unit which detects an operation sound that occurs due to the input operation being performed by the user on the touch screen display; and

a processing system that includes at least one processor, the processing system being configured to perform a game process by using the input position and the operation sound obtained by the operation sound detecting unit,

wherein the input position is a coordinate input position,

wherein the input position detecting unit detects the input position independently of the operation sound.

24. The game apparatus of claim 23, wherein the processing system is further configured to calculate, by performing a calculation process that is based on the operation sound, feature information indicating a feature of the operation sound,

wherein the game process is further based on the feature information, and

wherein the calculation process determines a detail in accordance with the obtained input position.

25. A computing system comprising:

a processing system that includes at least one processor, the processing system configured to;

output an image to a touch screen display;

receive positional data based on an input operation that is detected on an input surface of the touch screen display, the input operation being performed at a touch position on the input surface while the image is output to the touch screen display for display thereon;

receive sound data that occurs due to the input operation being performed at the touch position on the input surface; and

perform a process based on the positional data and the sound data,

wherein the positional data is received independently of the received sound data.

26. The computing system of claim 25, wherein the processing system is further configured to process the sound data based on the positional data.

27. The computing system of claim 25, further comprising the touch screen display, wherein the processing system is further configured to create a plurality of areas associated with portions of the surface of the touch screen display, and the process is further based on identifying which one of the plurality of areas the input position is located within.

28. The computing system of claim 25, wherein the processing system is further configured to calculate feature information indicating a feature in the sound data by performing a predetermined calculation process that is based on the sound data,

wherein the process is further based on the feature information, and

wherein the predetermined calculation process determines a detail in accordance with the input position.

29. A handheld computing apparatus comprising:

a housing having a form factor to be held by a hand of a user;

an input touch surface that is disposed in the housing, the input touch surface configured to;

accept a first input operation that is performed by the user to a first position on the input touch surface; and

accept a second input operation that is performed by a user to a second position on the input touch surface, the second position being different from the first position;

a microphone that is disposed in the housing, the microphone configured to;

detect a first sound that is based on the first input operation performed by the user; and

detect a second sound that is based on the second input operation performed by the user, the second sound being different from the first sound; and

a processing system that includes at least one processor, the processing system disposed in the housing and configured to;

output a first image to a display that is generated based on the first position and the first sound; and

output a second image that is different from the first image to the display, the second image being based on the second position and the second sound,

wherein the first position is determined independently of the detected first sound.