Storage medium storing information processing program, information processing device, information processing method, and information processing system

• US 10,096,257 B2
• Filed: 08/24/2012
• Issued: 10/09/2018
• Est. Priority Date: 04/05/2012
• Status: Active Grant

• Current Assignee: Nintendo Company Limited, Tohoku University National Universities Corporation
• Original Assignee: Nintendo Company Limited, Tohoku University National Universities Corporation
• Inventors: Ito, Yuichiro, Jin, Toshikazu, Kawashima, Ryuta
• Primary Examiner(s): Yao, Sam
• Assistant Examiner(s): HUMPHREY, MICHAEL C
• Application Number:
  US13/593,779
• Publication Number:
  US 20130266920A1
• Time in Patent Office: 2,237 Days
• Field of Search: 434319, 434185, 434156, 434157, 434167, 434362, 434320
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/80 Special adaptations for executing a specific game genre or game mode
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G09B 5/04 with audible presentation of the material to be studied sound recording or reproducing G11B
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A new information processing device for presenting information and allowing a user to input sound in relation to the presented information is provided. An information processing device 10 includes: a display section 104 that displays a sentence that is a reading-aloud target; a sound input section 102 to which sound is input; a time measuring section 105 that measures a sound input time in which the sound is input to the sound input section 102; and a test execution section 103 that determines whether or not the reading-aloud target is read aloud based on the sound input time and a silent time after the sound is input to the sound input section.

Claims

1. A non-transitory storage medium having stored therein an information processing program, the information processing program when executed causing an information processing device to perform:

displaying, on a touch screen including a display panel and a touch sensor, a sentence including a target word that is displayed with a display mode different from a display mode of other words in the sentence;

receiving sound data corresponding to a sound captured by a sound input device configured to capture sound while the sentence is displayed; and

processing the received sound data to determine a silent time in which sound in the received sound data is absent or below a sound threshold and a sound input time in which sound is present in the received sound data;

determining whether or not the sentence is read aloud based on the sound input time and the silent time, wherein the sentence is determined to be read aloud when a continued amount of the silent time reaches a silent threshold value after an accumulated amount of the sound input time separated by one or more silent times reaches read-aloud completion time determined based on the number of words in the displayed sentence;

upon determining that the sentence is read aloud, controlling the display panel to stop displaying the sentence and display an answer input screen requesting a user to manually input the target word;

receiving, from the touch sensor, touch input data corresponding to input made to the answer input screen;

performing character recognition processing to determine whether the received touch input data represents characters corresponding to characters in the target word; and

upon determining that the received touch input data represents characters corresponding to characters in the target word, controlling the display panel to display another sentence including another target word.

2. The non-transitory storage medium according to claim 1, wherein the information processing program further causes the information processing device to perform:

accumulating the sound input time while the sound is present in the received sound data regardless of whether or not there is a silent time after the sound in the received sound data starts.

3. The non-transitory storage medium according to claim 1, wherein the read-aloud completion time is determined each time a new sentence is displayed based on the number of words in the displayed sentence.

4. The non-transitory storage medium according to claim 3, wherein the information processing program causes the information processing device to perform:

displaying an object for causing a user to instruct termination of displaying the sentence while continuing the display of the sentence when the determination is made that the sentence is read aloud.

5. The non-transitory storage medium according to claim 4, wherein when the silent time, during which there is no instruction for the object, reaches a third threshold value after the object is displayed, prompting an instruction of the user for the object.

6. The non-transitory storage medium according to claim 1, wherein the determination that the sentence is read aloud is made regardless of the silent time when the sound input time reaches a third threshold value.

7. The non-transitory storage medium according to claim 1, wherein the target word in the sentence is displayed with a mark.

18. The non-transitory storage medium according to claim 1 wherein the information processing device is a game device, and the sound input device is disposed inside the housing of the game device.

8. An information processing method, implemented at least in part by a computer processor coupled to a touch screen including a display panel and a touch sensor and a sound input device configured to capture sound, the method comprising:

displaying, on the display panel, a sentence including a target word that is displayed with a display mode different from a display mode of other words in the sentence;

receiving sound data corresponding to a sound captured by the sound input device while the sentence is displayed; and

processing the received sound data to determine a silent time in which sound in the received sound data is absent or below a sound threshold and a sound input time in which sound is present in the received sound data;

determining whether or not the sentence is read aloud based on the sound input time and the silent time, wherein the sentence is determined to be read aloud when a continued amount of the silent time reaches a silent threshold value after an accumulated amount of the sound input time separated by one or more silent times reaches read-aloud completion time determined based on the number of words in the displayed sentence;

upon determining that the sentence is read aloud, controlling the display panel to stop displaying the sentence and display an answer input screen requesting a user to manually input the target word;

receiving, from the touch sensor, touch input data corresponding to input made to the answer input screen;

performing character recognition processing to determine whether the received touch input data represents characters corresponding to characters in the target word; and

upon determining that the received touch input data represents characters corresponding to characters in the target word, controlling the display panel to display another sentence including another target word.

9. The information processing method according to claim 8,

wherein the sound input time is accumulated while the sound is present in the received sound signal regardless of whether or not there is a silent time after the sound in the received sound signal starts.

10. The information processing method according to claim 8, wherein the read-aloud completion time is determined each time a new sentence is displayed based on the number of words in the displayed sentence.

11. The information processing method according to claim 10, further comprising

displaying an object for causing a user to instruct termination of displaying the sentence.

12. The information processing method according to claim 11, further comprising

performing processing for prompting an instruction of the user for the object when the silent time, during which there is no instruction for the object, reaches a third threshold value after displaying the object.

13. The information processing method according to claim 8, wherein the sentence is determined to be read aloud regardless of the silent time when the sound input time reaches a third threshold value.

14. The information processing method according to claim 8,

wherein target words in the sentence is displayed with a mark.

15. The information processing method to claim 8, wherein the accumulated amount of the sound input time includes one or more silent times that do not reach the silent threshold value.

16. The information processing method to claim 8, wherein the plurality of words in the sentence are simultaneously displayed, and immediately after determining that the sentence is read aloud, remove the sentence from the display panel.

17. The information processing method according to claim 8, wherein the read-aloud completion time is determined by multiplying the number of words in the sentence by a factor representing a difficulty level of the sentence.

19. The information processing method according to claim 8, wherein the method further comprises, after it is determined that the sentence is read aloud, update the read-aloud completion time with a value determined based on a number of words in the next sentence.

20. The information processing method according to claim 8, wherein the silent threshold value is equal to the read-aloud completion time.

21. An information processing apparatus comprising:

a touch screen including a display panel and a touch sensor;

a sound input device configured to capture sound; and

a processing system, comprising a processor and memory, the processing system configured to;

control the display panel to display a sentence including a target word that is displayed with a display mode different from a display mode of other words in the sentence;

receive sound data corresponding to sound captured by the sound input device while the sentence is displayed;

process the received sound data to determine a silent time in which sound in the received sound data is absent or below a sound threshold and a sound input time in which sound is present in the received sound data;

determine whether or not the sentence is read aloud based on the sound input time and the silent time, wherein the sentence is determined to be read aloud when a continued amount of the silent time reaches a silent threshold value after an accumulated amount of the sound input time separated by one or more silent times reaches read-aloud completion time determined based on the number of words in the displayed sentence;

upon determining that the sentence is read aloud, control the display panel to stop displaying the sentence and display an answer input screen requesting a user to manually input the target word;

receive, from the touch sensor, touch input data corresponding to input made to the answer input screen;

perform character recognition processing to determine whether the received touch input data represents characters corresponding to characters in the target word; and

upon determining that the received touch input data represents characters corresponding to characters in the target word, control the display panel to display another sentence including another target word.

22. The information processing apparatus of claim 21, wherein the read-aloud completion time is determined by multiplying the number of words in the sentence by a factor representing a difficulty level of the sentence.

23. The information processing apparatus of claim 21, wherein the read-aloud completion time is determined each time a new sentence is displayed based on the number of words in the displayed sentence.

24. The information processing apparatus of claim 21, wherein the processing system is further configured to:

after it is determined that the sentence is read aloud, update the read-aloud completion time with a value determined based on a number of words in the next sentence.

25. The information processing apparatus of claim 21, wherein the silent threshold value is equal to the read-aloud completion time.

26. The information processing apparatus of claim 21, wherein the information processing apparatus is a game device, and the sound input device is disposed inside the housing of the game device.