Empathetic user interface, systems, and methods for interfacing with empathetic computing device

• US 10,409,377 B2
• Filed: 03/08/2018
• Issued: 09/10/2019
• Est. Priority Date: 02/23/2015
• Status: Active Grant

• Current Assignee: SomniQ, Inc.
• Original Assignee: SomniQ, Inc.
• Inventors: Sakaguchi, Rikko, Ishikawa, Hidenori, Nishimoto, Takuya, Yano, Ken
• Primary Examiner(s): Haley, Joseph R
• Application Number:
  US15/916,155
• Publication Number:
  US 20180203512A1
• Time in Patent Office: 551 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L

Abstract

Devices, systems, and methods for empathetic computing are described. A method of interfacing with an empathetic computing system may include receiving sensor data from sensors of an empathetic computing device, wherein the sensor data is generated responsive, at least in part, to user interaction with the empathetic computing device, and wherein the user interaction comprises a plurality of interaction units. The method may include associating one or more individual interaction units of the plurality of interaction units with a corresponding interaction unit type in accordance with pre-determined definitions of types of interaction units, generating feedback responsive to the associating of an individual interaction unit with an interaction unit type, wherein the feedback is based on the type of the individual interaction unit, storing a sequence of interaction unit types of the individual interaction units associated with corresponding interaction unit types, and generating a descriptor of the sequence based at least on contextual information derived, at least in part, from the sensor data.

Claims

1. A method of interfacing with an empathetic computing system, the method comprising:

receiving sensor data from sensors of an empathetic computing device, wherein the sensor data is generated responsive, at least in part, to user interaction with the empathetic computing device, and wherein the user interaction comprises a plurality of interaction units;

using the sensor data, associating one or more individual interaction units of the plurality of interaction units with a corresponding interaction unit type in accordance with pre-determined definitions of types of interaction units;

generating feedback for each individual interaction unit from the plurality of interaction units, wherein the feedback is based on the type of the individual interaction unit;

storing a sequence of the interaction unit types of the plurality of interaction units; and

generating a descriptor of the sequence, wherein the descriptor is based at least on contextual information derived, at least in part, from the sensor data.

2. The method of claim 1, wherein the empathetic computing device is one of a plurality of empathetic computing devices and wherein the user interaction comprises one or more natural human actions including any of a vocal utterance, a touch or a movement of at least one of the plurality of empathetic computing devices.

3. The method of claim 1, wherein the receiving sensor data comprises receiving an indication of a detected touch, sound, or movement of the empathetic computing device, or of a detected ambient condition.

4. The method of claim 1, further comprising determining, from the sensor data, a length of a time between successive user interaction units and wherein the associating one or more individual interaction units is based further on the length of the time between successive user interaction units.

5. The method of claim 1, wherein the generating feedback comprises generating visual feedback in the form of an illumination of a plurality of light sources of the empathetic computing device, and wherein a color of the illumination is based on the interaction unit type of the interaction unit associated with the feedback.

6. The method of claim 1, wherein the associating one or more individual interaction units comprises associating at least three sequential individual interaction units to corresponding different interaction unit types.

7. The method of claim 1, wherein the generating feedback further comprises providing feedback of recorded sensor data including at least one detected sound segment by illuminating one or more light sources of the empathetic computing device in a manner representative of the recorded sensor data.

8. The method of claim 1, wherein the storing a sequence of interaction unit types comprises storing at least some of the sensor data associated with the interaction units included in the sequence of interaction unit types.

9. The method of claim 8, wherein the storing at least some of the sensor data comprises storing recorded sound associated with the interaction units included in the sequence of interaction unit types.

10. The method of claim 9, further comprising segmenting the recorded sound into a plurality of sound segments, and indexing individual ones of the plurality of sound segments, and wherein each indexed one of the plurality of sound segments is individually accessible for playback by the user.

11. The method of claim 10, further comprising receiving an indication of a tagged sound segment responsive to user input.

12. The method of claim 8, further comprising providing a user interface for retrieval and playback of recorded data including at least one of the stored sequence of interaction unit types and the stored sensor data associated with a given user interaction session.

13. The method of claim 12, wherein the providing a user interface includes providing a first user interface to a user associated with the user interaction and providing a second user interface to a different user for accessing or tagging the stored sequence of interaction unit types, the stored sensor data, or combinations thereof.

14. The method of claim 1, wherein the generating a descriptor of the sequence comprises one of:

measuring a length of the user interaction;

measuring a length of an individual user interaction;

measuring a level of ambient sound; and

comparing one or more parameters of one user interaction with a corresponding one or more parameters of another user interaction in a series of user interactions.

15. An empathetic computing system comprising:

a plurality of sensors configured to generate sensor data based, at least in part, on user interaction with one or more empathetic computing devices, the user interaction comprising a plurality of interaction units;

a plurality of light sources provided on at least one of the one or more empathetic computing devices;

a processor communicatively coupled to the plurality of sensors and the plurality of light sources, and wherein the processor is configured, by executable instructions stored in memory;

to;

receive sensor data from one or more of the plurality of sensors;

using the sensor data, associate one or more individual interaction units of the plurality of interaction units with a corresponding interaction unit type in accordance with the stored definitions of interaction unit types;

generate feedback for each individual interaction unit from the plurality of interaction units, wherein the feedback is based on the type of the individual interaction unit;

store a sequence of the interaction unit types of the plurality of interaction units; and

generating a descriptor of the sequence, wherein the descriptor is based at least on contextual information derived, at least in part, from the sensor data.

16. The empathetic computing system of claim 15, wherein the processor is configured to determine;

from the sensor data, a length of time between successive user interaction units and to associate the one or more individual interaction units with a corresponding interaction unit type based further on the length of the time.

17. The empathetic computing system of claim 15, wherein the processor is configured to generate visual feedback in the form of an illumination of the plurality, of light sources, wherein a color of the illumination is based on the interaction unit type of the interaction unit associated with the feedback.

18. The empathetic computing system of claim 15, wherein the processor is configured to perform feature recognition on the sensor data to extract the individual interaction units from the user interaction.

19. The empathetic computing system of claim 18, wherein the feature recognition includes speech recognition, and wherein the processor is configured to provide visual feedback of detected speech by illuminating the plurality of light sources in a pattern corresponding to the detected speech.

20. The empathetic computing system of claim 15, wherein the processor is further configured to provide a user interface for retrieval and playback of recorded data including at least one of the stored sequence of interaction unit types and stored sensor data associated with a given user interaction session.

21. The empathetic computing system of claim 20, wherein the stored sensor data includes recorded sound, and wherein the processor is configured to index one or more segments of the recorded sound such that each indexed segment of the recorded sound is individually accessible by the user for playback.

22. The empathetic computing system of claim 21, wherein the user interface is further configured to receive user input for tagging one or more of the individual segments of the recorded sound.

23. The empathetic computing system of claim 15, wherein the processor is configured to generate the descriptor of the sequence based, at least in part on, a length of the user interaction, a length of an individual user interaction unit, a level of ambient sound, a comparison of one or more parameters of one user interaction with a corresponding one or more parameters of another user interaction in a series of user interactions, or any combination thereof.

24. The empathetic computing system of claim 15, wherein the processor and the memory are components of the at least one empathetic computing device.

25. The empathetic computing system of claim 15, wherein the empathetic computing system includes at least two empathetic computing devices, each of which comprises at least one feedback generation unit, at least one sensor unit, or a combination thereof.

26. The empathetic computing system of claim 15, wherein at least some of the plurality of sensors are associated with a first empathetic computing device comprising a first feedback generation unit, the system further comprising a second empathetic computing device including a second feedback generation unit, and wherein the processor is communicatively coupled to at least one of the first empathetic computing device and the second empathetic computing device to cause the first empathetic computing device, the second empathetic computing device, or both, to generate feedback based on the sensor data.