Language learning apparatus and method therefor

• US 6,816,831 B1
• Filed: 10/27/2000
• Issued: 11/09/2004
• Est. Priority Date: 10/28/1999
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Iwahashi, Naoto
• Primary Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US09/699,251
• Time in Patent Office: 1,474 Days
• Field of Search: 704/9, 704/251, 704/257
• US Class Current: 704/9
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04

Abstract

A language learning apparatus includes a syntax-structure analyzing unit and a syntax-rule learning unit. The syntax-structure analyzing unit is provided with a speech recognition portion for recognizing a word string from input speech data O_i based on speech pattern information of a vocabulary-information storage unit. A semantic analyzer detects image patterns from input moving picture data V_i by using the information stored in the vocabulary-information storage unit, and analyzes the relevance of the detected image patterns and outputs a concept representation using the words corresponding to the individual image patterns. A relevance analyzer verifies the word string extracted by the speech recognition portion against the semantic representation extracted by the semantic analyzer so as to determine the syntax structure of the word string. Then, the syntax-rule learning unit learns the syntax rule by using the determined syntax structure.

Claims

1. A language learning apparatus comprising:

a recognition portion for receiving language information and for extracting a word string according to the input language information;

a semantic analyzing portion for receiving perceptual information related to the language information and for extracting concepts and a concept representation indicating the relevance of the concepts; and

a relevance analyzing portion for verifying the word string extracted by said recognition portion against the concept representation extracted by said semantic analyzing portion and for determining a syntax structure of the word string according to the relevance between the word string and the concept representation.

2. A language learning apparatus according to claim 1, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the language information and vocabulary information for recognizing the perceptual information,

wherein said recognition portion recognizes the language information based on the vocabulary information stored in said vocabulary-information storage unit, and said semantic analyzing portion analyzes the perceptual information based on the vocabulary information stored in said vocabulary-information storage unit.

3. A language learning apparatus according to claim 1, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the language information and vocabulary information for recognizing the perceptual information,

wherein said recognition portion recognizes the language information based on the vocabulary information stored in said vocabulary-information storage unit and reports a vocabulary checked for recognizing the language information to said semantic analyzing portion, and said semantic analyzing portion analyzes the perceptual information by searching the vocabulary information stored in said vocabulary-information storage unit concerning only the vocabulary reported from said recognition portion.

4. A language learning apparatus according to claim 1, wherein said semantic analyzing portion comprises a sensor for obtaining moving picture information as the perceptual information.

5. A language learning apparatus according to claim 1, wherein said semantic analyzing portion comprises a sensor for obtaining still image information as the perceptual information.

6. A language learning apparatus according to claim 1, wherein said semantic analyzing portion comprises a sensor for obtaining touch information as the perceptual information.

7. A language learning apparatus according to claim 1, wherein said semantic analyzing portion comprises a sensor for obtaining acceleration information as the perceptual information.

8. A language learning apparatus according to claim 1, wherein said semantic analyzing portion comprises a sensor for obtaining pressure information as the perceptual information.

9. A language learning apparatus comprising:

a speech recognition portion for receiving speech information and for extracting a word string according to the input speech information;

a semantic analyzing portion for receiving perceptual information related to the speech information and for extracting concepts and a concept representation indicating the relevance of the concepts; and

a relevance analyzing portion for verifying the word string extracted by said speech recognition portion against the concept representation extracted by said semantic analyzing portion and for determining a syntax structure of the word string according to the relevance between the word string and the concept representation.

10. A language learning apparatus according to claim 9, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the speech information and vocabulary information for recognizing the perceptual information,

wherein said speech recognition portion recognizes the speech information based on the vocabulary information stored in said vocabulary-information storage unit, and said semantic analyzing portion analyzes the perceptual information based on the vocabulary information stored in said vocabulary-information storage unit.

11. A language learning apparatus according to claim 9, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the speech information and vocabulary information for recognizing the perceptual information,

wherein said speech recognition portion recognizes the speech information based on the vocabulary information stored in said vocabulary-information storage unit and reports a vocabulary checked for recognizing the speech information to said semantic analyzing portion, and said semantic analyzing portion analyzes the perceptual information by searching the vocabulary information stored in said vocabulary-information storage unit concerning only the vocabulary reported from said speech recognition portion.

12. A language learning apparatus according to claim 9, wherein said semantic analyzing portion comprises a sensor for obtaining moving picture information as the perceptual information.

13. A language learning apparatus according to claim 9, wherein said semantic analyzing portion comprises a sensor for obtaining still image information as the perceptual information.

14. A language learning apparatus according to claim 9, wherein said semantic analyzing portion comprises a sensor for obtaining touch information as the perceptual information.

15. A language learning apparatus according to claim 9, wherein said semantic analyzing portion comprises a sensor for obtaining acceleration information as the perceptual information.

16. A language learning apparatus according to claim 9, wherein said semantic analyzing portion comprises a sensor for obtaining pressure information as the perceptual information.

17. A language learning apparatus comprising:

a syntax-structure analyzing unit including a recognition portion for receiving language information and for extracting a word string according to the input language information, a semantic analyzing portion for receiving perceptual information related to the language information and for extracting concepts and a concept representation indicating the relevance of the concepts, and a relevance analyzing portion for verifying the word string extracted by said recognition portion against the concept representation extracted by said semantic analyzing portion and for determining a syntax structure of the word string according to the relevance between the word string and the concept representation; and

a syntax-rule learning unit for receiving the syntax structure determined by said relevance analyzing portion of said syntax-structure analyzing unit and for learning a syntax rule.

18. A language learning apparatus according to claim 17, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the language information and vocabulary information for recognizing the perceptual information,

wherein said recognition portion recognizes the language information based on the vocabulary information stored in said vocabulary-information storage unit, and said semantic analyzing portion analyzes the perceptual information based on the vocabulary information stored in said vocabulary-information storage unit.

19. A language learning apparatus according to claim 18, wherein said syntax-rule learning unit outputs information of the learned syntax rule to said recognition portion of said syntax-structure analyzing unit, and said recognition portion recognizes the language information based on the vocabulary information stored in said vocabulary-information storage unit and the information of the syntax rule learned by said syntax-rule learning unit.

20. A language learning apparatus according to claim 17, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the language information and vocabulary information for recognizing the perceptual information, wherein said recognition portion recognizes the language information based on the vocabulary information stored in said vocabulary-information storage unit and reports a vocabulary checked for recognizing the language information to said semantic analyzing portion, and said semantic analyzing portion analyzes the perceptual information by searching for the vocabulary information stored in said vocabulary-information storage unit concerning only the vocabulary reported from said recognition portion.

21. A language learning apparatus according to claim 20, wherein said syntax-rule learning unit outputs information of the learned syntax rule to said recognition portion of said syntax-structure analyzing unit, and said recognition portion recognizes the language information based on the vocabulary information stored in said vocabulary-information storage unit and the information of the syntax rule learned by said syntax-rule learning unit.

22. A language learning apparatus according to claim 17, wherein said syntax-rule learning unit outputs information of the learned syntax rule to said recognition portion of said syntax-structure analyzing unit, and said recognition portion recognizes the language information based on the information of the syntax rule learned by said syntax-rule learning unit.

23. A language learning apparatus according to claim 17, wherein said semantic analyzing portion comprises a sensor for obtaining moving picture information as the perceptual information.

24. A language learning apparatus according to claim 17, wherein said semantic analyzing portion comprises a sensor for obtaining still image information as the perceptual information.

25. A language learning apparatus according to claim 17, wherein said semantic analyzing portion comprises a sensor for obtaining touch information as the perceptual information.

26. A language learning apparatus according to claim 17, wherein said semantic analyzing portion comprises a sensor for obtaining acceleration information as the perceptual information.

27. A language learning apparatus according to claim 17, wherein said semantic analyzing portion comprises a sensor for obtaining pressure information as the perceptual information.

28. A language learning apparatus according to claim 17, wherein said syntax-rule learning unit uses a probability context-free grammar as the syntax rule.

29. A language learning apparatus according to claim 17, wherein said syntax-rule learning unit uses a modification grammar as the syntax rule.

30. A language learning apparatus according to claim 17, wherein said syntax-rule learning unit uses a probability modification grammar as the syntax rule.

31. A language learning apparatus comprising:

a syntax-structure analyzing unit including a speech recognition portion for receiving speech information and for extracting a word string according to the input speech information, a semantic analyzing portion for receiving perceptual information related to the speech information and for extracting concepts and a concept representation indicating the relevance of the concepts, and a relevance analyzing portion for verifying the word string extracted by said speech recognition portion against the concept representation extracted by said semantic analyzing portion and for determining a syntax structure of the word string according to the relevance between the word string and the concept representation; and

a syntax-rule learning unit for receiving the syntax structure determined by said relevance analyzing portion of said syntax-structure analyzing unit and for learning a syntax rule.

32. A language learning apparatus according to claim 31, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the speech information and vocabulary information for recognizing the perceptual information,

wherein said speech recognition portion recognizes the speech information based on the vocabulary information stored in said vocabulary-information storage unit, and said semantic analyzing portion analyzes the perceptual information based on the vocabulary information stored in said vocabulary-information storage unit.

33. A language learning apparatus according to claim 32, wherein said syntax-rule learning unit outputs information of the learned syntax rule to said speech recognition portion of said syntax-structure analyzing unit, and said speech recognition portion recognizes the speech information based on the vocabulary information stored in said vocabulary-information storage unit and the information of the syntax rule learned by said syntax-rule learning unit.

34. A language learning apparatus according to claim 31, further comprising a vocabulary-information storage unit for storing vocabulary information for recognizing the speech information and vocabulary information for recognizing the perceptual information,

wherein said speech recognition portion recognizes the speech information based on the vocabulary information stored in said vocabulary-information storage unit and reports a vocabulary checked for recognizing the speech information to said semantic analyzing portion, and said semantic analyzing portion analyzes the perceptual information by searching the vocabulary information stored in said vocabulary-information storage unit concerning only the vocabulary reported from said speech recognition portion.

35. A language learning apparatus according to claim 34, wherein said syntax-rule learning unit outputs information of the learned syntax rule to said speech recognition portion of said syntax-structure analyzing unit, and said speech recognition portion recognizes the speech information based on the vocabulary information stored in said vocabulary-information storage unit and the information of the syntax rule learned by said syntax-rule learning unit.

36. A language learning apparatus according to claim 31, wherein said syntax-rule learning unit outputs information of the learned syntax rule to said speech recognition portion of said syntax-structure analyzing unit, and said speech recognition portion recognizes the speech information based on the information of the syntax rule learned by said syntax-rule learning unit.

37. A language learning apparatus according to claim 31, wherein said semantic analyzing portion comprises a sensor for obtaining moving picture information as the perceptual information.

38. A language learning apparatus according to claim 31, wherein said semantic analyzing portion comprises a sensor for obtaining still image information as the perceptual information.

39. A language learning apparatus according to claim 31, wherein said semantic analyzing portion comprises a sensor for obtaining touch information as the perceptual information.

40. A language learning apparatus according to claim 31, wherein said semantic analyzing portion comprises a sensor for obtaining acceleration information as the perceptual information.

41. A language learning apparatus according to claim 31, wherein said semantic analyzing portion comprises a sensor for obtaining pressure information as the perceptual information.

42. A language learning apparatus according to claim 31, wherein said syntax-rule learning unit uses a probability context-free grammar as the syntax rule.

43. A language learning apparatus according to claim 31, wherein said syntax-rule learning unit uses a modification grammar as the syntax rule.

44. A language learning apparatus according to claim 31, wherein said syntax-rule learning unit uses a probability modification grammar as the syntax rule.

45. A language learning method comprising:

a first extraction step of extracting a word string from language information;

a second extraction step of extracting concepts and a concept representation indicating the relevance of the concepts from perceptual information related to the language information; and

a determination step of determining a syntax structure of the word string according to the relevance between the extracted word string and the extracted concept representation by verifying the word string against the concept representation.

46. A language learning method according to claim 45, wherein said second extraction step searches only for the concepts of words contained in the word string and extracts the corresponding concept representation.

47. A language learning method according to claim 45, wherein said perceptual information comprises still image information.

48. A language learning method according to claim 45, wherein said perceptual information comprises moving picture information.

49. A language learning method according to claim 45, wherein said perceptual information comprises touch information.

50. A language learning method according to claim 45, wherein said perceptual information comprises acceleration information.

51. A language learning method according to claim 45, wherein said perceptual information comprises pressure information.

52. A language learning method comprising:

a first extraction step of extracting a word string from speech information;

a second extraction step of extracting concepts and a concept representation indicating the relevance of the concepts from perceptual information related to the speech information; and

a determination step of determining a syntax structure of the word string according to the relevance between the extracted word string and the extracted concept representation by verifying the word string against the concept representation.

53. A language learning method according to claim 52, wherein said second extraction step searches only for the concepts of words contained in the word string and extracts the corresponding concept representation.

54. A language learning method according to claim 52, wherein said perceptual information comprises still image information.

55. A language learning method according to claim 52, wherein said perceptual information comprises moving picture information.

56. A language learning method according to claim 52, wherein said perceptual information comprises touch information.

57. A language learning method according to claim 52, wherein said perceptual information comprises acceleration information.

58. A language learning method according to claim 52, wherein said perceptual information comprises pressure information.

59. A language learning method comprising:

a first extraction step of extracting a word string from language information;

a second extraction step of extracting concepts and a concept representation indicating the relevance of the concepts from perceptual information related to the language information;

a determination step of determining a syntax structure of the word string according to the relevance between the extracted word string and the extracted concept representation by verifying the word string against the concept representation; and

a learning step of learning a syntax rule based on the determined syntax structure.

60. A language learning method according to claim 59, wherein said second extraction step searches only for the concepts of words contained in the word string and extracts the corresponding concept representation.

61. A language learning method according to claim 60, wherein said first extraction step extracts the word string based on the learned syntax rule.

62. A language learning method according to claim 59, wherein said first extraction step extracts the word string based on the learned syntax rule.

63. A language learning method according to claim 59, wherein said perceptual information comprises still image information.

64. A language learning method according to claim 59, wherein said perceptual information comprises moving picture information.

65. A language learning method according to claim 59, wherein said perceptual information comprises touch information.

66. A language learning method according to claim 59, wherein said perceptual information comprises acceleration information.

67. A language learning method according to claim 59, wherein said perceptual information comprises pressure information.

68. A language learning method according to claim 59, wherein said syntax rule comprises a probability context-free grammar.

69. A language learning method according to claim 59, wherein said syntax rule comprises a modification grammar.

70. A language learning method according to claim 59, wherein said syntax rule comprises a probability modification grammar.

71. A language learning method comprising:

a first extraction step of extracting a word string from speech information;

a second extraction step of extracting concepts and a concept representation indicating the relevance of the concepts from perceptual information related to the speech information;

a determination step of determining a syntax structure of the word string according to the relevance between the extracted word string and the extracted concept representation by verifying the word string against the concept representation; and

a learning step of learning a syntax rule based on the determined syntax structure.

72. A language learning method according to claim 71, wherein said second extraction step searches only for the concepts of words contained in the word string and extracts the corresponding concept representation.

73. A language learning method according to claim 71, wherein said first extraction step extracts the word string based on the learned syntax rule.

74. A language learning method according to claim 71, wherein said first extraction step extracts the word string based on the learned syntax rule.

75. A language learning method according to claim 71, wherein said perceptual information comprises still image information.

76. A language learning method according to claim 71, wherein said perceptual information comprises moving picture information.

77. A language learning method according to claim 71, wherein said perceptual information comprises touch information.

78. A language learning method according to claim 71, wherein said perceptual information comprises acceleration information.

79. A language learning method according to claim 71, wherein said perceptual information comprises pressure information.

80. A language learning method according to claim 71, wherein said syntax rule comprises a probability context-free grammar.

81. A language learning method according to claim 71, wherein said syntax rule comprises a modification grammar.

82. A language learning method according to claim 71, wherein said syntax rule comprises a probability modification grammar.