Apparatus and method for determining user's mental state

• US 9,928,462 B2
• Filed: 11/08/2013
• Issued: 03/27/2018
• Est. Priority Date: 11/09/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Ho-Sub
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US14/075,391
• Publication Number:
  US 20140136450A1
• Time in Patent Office: 1,600 Days
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area

Abstract

An apparatus for determining a user'"'"'s mental state in a terminal is provided. The apparatus includes a data collector configured to collect sensor data; a data processor configured to extract feature data from the sensor data; and a mental state determiner configured to provide the feature data to an inference model to determine the user'"'"'s mental state.

Claims

1. An apparatus for determining a mental state of a user, comprising:

a data collector configured to collect sensor data generated as the user is composing a message using a terminal;

a mental state determiner configured to determine the mental state of the user based on extracted feature data from the sensor data; and

a measure executor configured to control an implementation of one or more predetermined measures based on the determined mental state,

wherein the feature data comprises text input information, and the text input information comprises one or more of keyboard input information, writing input information, and terminal state information.

2. The apparatus of claim 1, wherein the sensor data are measured through one or more of a keyboard, a stylus pen, a touch screen, an accelerometer sensor, a gyroscope sensor, a magnetometer sensor, an orientation sensor, a global positioning system (GPS), a barometer sensor, a fingerprint sensor, an illuminance sensor, a microphone, and a software sensor of the terminal.

3. The apparatus of claim 2, wherein the feature data further comprises context information;

and

the context information comprises one or more of a location, weather, a discomfort index, a time, an average illuminance, and one or more words of the composed message.

4. The apparatus of claim 1, wherein,

the mental state comprises at least one of an emotion, a feeling, or a stress, and

each mental state comprises one or more sub-categorized related mental states.

5. The apparatus of claim 1, wherein the apparatus is the terminal.

6. The apparatus of claim 1, wherein the one or more predetermined measures comprise providing information on the determined mental state of the user, controlling a user interface based on the mental state, and/or recommending content based on the mental state of the user.

7. The apparatus of claim 1, further comprising a measure database configured to store the one or more predetermined measures.

8. The apparatus of claim 1, wherein the measure executor is installed as an interactive software agent that is configured to provide a conversational interface.

9. The apparatus of claim 1, further comprising an authenticator configured to authenticate the user at the apparatus.

10. The apparatus of claim 9, wherein the user is authenticated based on at least one of logon information, fingerprint, or biometric information.

33. The apparatus of claim 1, wherein the one or more predetermined measures comprise controlling a user interface of the terminal based on the determined mental state, the measure executor being configured to provide the user interface within at least one of a text or messaging application, an email application, a social network service (SNS) application, a multimedia application, a gaming application, an internet browsing application, a shopping application, and a navigation application.

34. The apparatus of claim 33, wherein, to implement the controlling of the user interface of the terminal, the measure executor is configured to change at least one of themes of the user interface depending on the determined mental state, the at least one of the themes including a font, a font size, a color, a background, a brightness and corresponding to items of the text input information for the user interface.

35. The apparatus of claim 33, wherein, to implement the controlling of the user interface of the terminal, the measure executor is configured to change at least one of a font, a font size, a color, a background, a brightness and items corresponding to the text input information for the composed message depending on the determined mental state.

36. The apparatus of claim 1, wherein the one or more predetermined measures comprise providing information on the determined mental state of the user.

37. The apparatus of claim 36, wherein the providing of the information on the determined mental state of the user includes providing the user with the information on the determined mental state of the user.

38. The apparatus of claim 1, wherein the one or more predetermined measures comprise recommending content based on the mental state of the user.

39. The apparatus of claim 38, wherein the recommending of the content based on the mental state of the user includes recommending the content to the user based on the mental state of the user.

40. The apparatus of claim 1, wherein

the keyboard input information includes at least one of a typing speed, a length of the longest text, a length of the shortest and longest deleted text, a number of words typed per editing, a number of long touches per editing, a frequency of use of backspace key, a frequency of use of enter key, and a frequency of use of special sign,

the writing input information includes at least one of a writing input speed, a writing pressure, a character size, a straightness or curvature of a character stroke, a character interval, a row interval, and writing input regularity, and

the terminal state information includes at least one of a number of shakings of the terminal, a degree of inclination of the terminal, a remaining capacity of a battery, and a voice volume.

41. The apparatus of claim 6, wherein the providing of the information on the determined mental state of the user, the controlling of the user interface, and the recommending of the content are implemented within a social network services (SNS) task.

42. The apparatus of claim 41, wherein the SNS task is implemented by the terminal.

43. The apparatus of claim 1, wherein the data collector, the mental state determiner, and the measure executor are implemented by one or more processors of the apparatus.

11. A processor-implemented method of determining a mental state of a user, comprising:

collecting sensor data generated as the user is composing a message using a terminal;

determining the mental state of the user based on extracted feature data from the sensor data; and

controlling an implementation of one or more predetermined measures based on the determined mental state,

wherein the feature data comprises text input information, and the text input information comprises one or more of keyboard input information, writing input information, and terminal state information.

12. The method of claim 11, wherein the sensor data are measured through one or more of a keyboard, a stylus pen, a touch screen, an accelerometer sensor, a gyroscope sensor, a magnetometer sensor, an orientation sensor, a global positioning system (GPS), a barometer sensor, a fingerprint sensor, an illuminance sensor, a microphone, and a software sensor of the terminal.

13. The method of claim 11, wherein the feature data further comprises context information;

and

the context information comprises one or more of a location, weather, a discomfort index, a time, information on recipient of the text, an average illuminance, and one or more words of the composed message.

14. The method of claim 11, wherein the one or more predetermined measures comprise controlling a user interface of the terminal based on the determined mental state, the controlling of the implementation selectively enables the user interface to be implemented within at least one of a text or messaging application, an email application, a social network service (SNS) application, a multimedia application, a gaming application, an internet browsing application, a shopping application, and a navigation application.

15. The method of claim 14, wherein the controlling of the user interface of the terminal further comprises changing at least one of themes of the user interface depending on the determined mental state, the at least one of the themes including a font, a font size, a color, a background, a brightness and corresponding to items of the text input information for the user interface.

16. The method of claim 14, wherein the controlling of the user interface of the terminal further comprises changing at least one of a font, a font size, a color, a background, a brightness and items corresponding to the text input information for the composed message depending on the determined mental state.

17. The method of claim 11, wherein the one or more predetermined measures comprise providing information on the determine mental state of the user.

18. The method of claim 17, wherein the providing of the information on the determined mental state of the user includes providing the user with the information on the determined mental state of the user.

19. The method of claim 11, wherein,

the mental state comprises at least one of an emotion, a feeling, or a stress, and

each mental states comprises ones or more sub-categorized related mental states.

20. The method of claim 11, wherein the one or more predetermined measures comprise recommending content based on the mental state of the user.

21. The method of claim 11, wherein the one or more predetermined measures comprise providing information on the determined mental state, controlling a user interface of the terminal on the basis of the mental state, and/or recommending content based on the mental state of the user.

27. The method of claim 11, wherein the determining of the mental state of the user and the controlling of the implementation of the one or more predetermined measures are performed by the terminal.

28. The method of claim 20, wherein the recommending of the content based on the mental state of the user includes recommending the content to the user based on the mental state of the user.

29. The method of claim 21, wherein the providing of the information on the determined mental state of the user, the controlling of the user interface, and the recommending of the content are implemented within a social network services (SNS) task.

30. The method of claim 29, wherein the SNS task is implemented by the terminal.

31. The method of claim 11, wherein

the keyboard input information includes at least one of a typing speed, a length of the longest text, a length of the shortest and longest deleted text, a number of words typed per editing, a number of long touches per editing, a frequency of use of backspace key, a frequency of use of enter key, and a frequency of use of special sign,

the writing input information includes at least one of a writing input speed, a writing pressure, a character size, a straightness or curvature of a character stroke, a character interval, a row interval, and writing input regularity, and

the terminal state information includes at least one of a number of shakings of the terminal, a degree of inclination of the terminal, a remaining capacity of a battery, and a voice volume.

32. The method of claim 11, wherein the mental state of the user is determined using an algorithm.

22. A processor-implemented method of determining a mental state of a user, comprising:

extracting a speed at which a user inputs text to a terminal while the user is composing a message including the text;

determining the mental state of the user based on the text input speed; and

controlling an implementation of one or more predetermined measures on the basis of the determined mental state.

23. The method of claim 22, further comprising extracting other information, which is generated when the user is inputting the text, wherein the determining of the mental state of the user further includes determining the mental state of the user based on the extracted other information.

24. The method of claim 23, wherein the extracted other information comprises location information of the terminal.

25. The method of claim 23, wherein the extracted other information comprises at least one of weather information and state information of the terminal.

26. The method of claim 23, wherein the extracted other information comprises a number of shakings of the terminal.

44. A system determining a mental state of a user, comprising:

a first processor, of one or more processors, configured to collect sensor data generated as the user is composing a message using a terminal;

a second processor, of the one or more processors, configured to determine the mental state of the user based on extracted features from the sensor data; and

a third processor, of the one or more processors, configured to control an instruction of one or more predetermined measures based on the determined mental state,

wherein the feature data comprises text input information, and the text input information comprises one or more of keyboard input information, writing input information, and terminal state information.

45. The system of claim 44, wherein the second processor is included in the terminal.

46. The system of claim 44, wherein the sensor data are measured through one or more of a keyboard, a stylus pen, a touch screen, an accelerometer sensor, a gyroscope sensor, a magnetometer sensor, an orientation sensor, a global positioning system (GPS), a barometer sensor, a fingerprint sensor, an illuminance sensor, a microphone, and a software sensor of the terminal.

47. The system of claim 46, wherein the feature data further comprises context information, and the context information comprises one or more of a location, weather, a discomfort index, a time, an average illuminance, and one or more words of the composed message.

48. The system of claim 44, wherein the one or more predetermined measures comprise providing information on the determined mental state of the user, controlling a user interface based on the mental state, and/or recommending content based on the mental state of the user.

49. The system of claim 48, wherein the providing of the information on the determined mental state of the user, the controlling of the user interface, and the recommending of the content are implemented within a social network services (SNS) task.

50. The system of claim 49, wherein the SNS task is implemented by the terminal.

51. The system of claim 44, wherein the one or more predetermined measures comprise providing information on the determined mental state of the user.

52. The system of claim 51, wherein the providing of the information on the determined mental state of the user includes providing the information on the determined mental state of the user to the user.

53. The system of claim 44, wherein the one or more predetermined measures comprise controlling a user interface based on the mental state.

54. The system of claim 53, wherein the user interface is a user interface of the terminal.

55. The system of claim 44, wherein the one or more predetermined measures comprise recommending content based on the mental state of the user.

56. The system of claim 55, wherein the recommending of the content includes recommending the content to the user.

57. The system of claim 44, further comprising a measure database configured to store the one or more predetermined measures.

58. The system of claim 44, wherein the first processor is further configured to perform the feature extraction and configured to implement an interactive software agent that is configured to provide a conversational interface.

59. The system of claim 44, wherein the first processor, the second processor, and the third processor are one or more processors of the terminal.

60. The system of claim 44, wherein one of the one or more processors is further configured to determine and provide an emoticon based on the user'"'"'s composed message.

61. The system of claim 60, wherein the one of the one or more processors is further configured to determine and provide the emoticon based on the mental state of the user.