REQUESTING ASSISTANCE BASED ON USER STATE

• US 20170340256A1
• Filed: 05/27/2016
• Published: 11/30/2017
• Est. Priority Date: 05/27/2016
• Status: Abandoned Application

• Current Assignee: The Affinity Project, Inc.
• Original Assignee: The Affinity Project, Inc.
• Inventors: Suskind, Ronald Steven, Patterson, Stuart R., Nguyen, John, Fanty, Mark Alan, Springer, Stephen R., Gregory, Erik Martin
• Application Number:
  US15/166,507
• Publication Number:
  US 20170340256A1
• Time in Patent Office: Days
• CPC Class Codes: A61B 2562/0204 Acoustic sensors
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/168 Evaluating attention deficit, hyperactivity
• CPC Class Codes: A61B 5/4088 Diagnosing of monitoring cognitive diseases, e.g. Alzheimer, prion diseases or dementia
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/744 Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: A61F 4/00 Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body operating or control means for prostheses A61F2/48, A61F2/68
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

Assistance may be provided to a first user of a first device by monitoring sensors of the first device and providing assistance via a character presented by the device. Sensor data, such as audio, video, or biometric data, may be transmitted to a server, and a server may process the sensor data to determine a state of the first user. The server may determine to request assistance based on the state of the first user. The server may then send a request to a second device of a second user to guide the character presented by the first device. The second user may then provide assistance to the first user by guiding the character presented by the first device.

Claims

1. A system for providing assistance to a first user of a first device by a communications session using a guided character wherein the communication session comprises the first device streaming audio and video of the first user to be presented to a second user of a second device and the second device transmitting a phrase to the first device to be played as audio by the first device using a text-to-speech voice of the guided character, the system comprising the first device, a server computer, and the second device, wherein:

the first device is configured to;

before the communications session has started, obtain a first audio signal corresponding to audio of the first user captured by a microphone of the first device, and

transmit the first audio signal to the server computer;

the server computer is configured to;

before the communication session has started;

determine a state of the first user of the first device by processing the first audio signal,

determine to request assistance based on the state of the first user,

send, to the second device of the second user, a request for the second user to guide the character of the first device in the communications session, and

receive, from the second device, an acceptance of the request, and

after the acceptance of the request has been received, start the communications session wherein, during the communications session, the server computer (i) streams an audio signal from the first device to the second device in real time, (ii) streams a video signal from the first device to the second device in real time, (iii) receives the phrase to be spoken by the character from the second device, and (iv) causes the first device to present audio of the character speaking the phrase using a text-to-speech voice of the character, and wherein the communications session comprises multiple audio communications of the first user and multiple phrases spoken by the character;

the first device is configured to join the communications session after the communications session has been started by the server, wherein during the communications session, the first device (i) streams the audio signal corresponding to audio captured by a microphone of the first device to the server computer, (ii) streams the video signal corresponding to video captured by a camera of the first device to the server computer, (iii) receives information corresponding to the phrase from the server computer, and (iv) presents audio using the information corresponding to the phrase and using the text-to-speech voice of the character presented by the first device;

the second device is configured to join the communications session after the communications session has been started by the server, wherein during the communications session, the second device (i) presents audio corresponding to the streamed audio signal, (ii) presents video corresponding to the streamed video signal, (iii) receives the information corresponding to the phrase from the second user, and (iv) transmits the information corresponding to the phrase to the server computer.

24. The system of claim 1, wherein the first audio signal is not part of a communication between the first device and another device.

25. The system of claim 1, wherein the first device is configured to obtain the first audio signal while the first user is not interacting with the first device.

26. The system of claim 1, wherein the server computer is configured to cause the first device to get the attention of the first user by at least one of causing the first device to play a sound, activating a camera flash of the first device, or causing the first device to vibrate.

27. The system of claim 1, wherein the first device is configured to allow the first user to communicate with the character in an automated mode of operation, wherein phrases spoken by the character are determined by a computer.

28. The system of claim 27, wherein the first device is configured to:

transmit the first audio signal while the first user is interacting with the character in the automated mode of operation; and

present the audio using the information corresponding to the phrase and indicating to the first user that the phrase was not generated automatically.

29. The system of claim 1, wherein the server computer is configured to transmit the first audio signal to the second device, and wherein the second device is configured to present the first audio signal to the second user to allow the second user to prepare for the communications session.

30. The system of claim 1, wherein:

the second device is configured to present, during the communications session, a list of video clips to the second user, receive a selection of a video clip by the second user, and transmit information indicative of the selected video clip to the server computer; and

the server computer is configured to receive, during the communications session, the information indicative of the selected video clip and cause the video clip to be presented by the first device and the second device.

2-10. -10. (canceled)

11. A computer-implemented method for providing assistance to a first user of a first device by a communications session using a guided character wherein the communication session comprises the first device streaming audio and video of the first user to be presented to a second user of a second device and the second device transmitting a phrase to the first device to be played as audio by the first device using a text-to-speech voice of the guided character, the method comprising:

before the communications session has started;

receiving from the first device a first audio signal corresponding to audio captured by a microphone of the first device,

determining a state of the first user of the first device by processing the first audio signal,

determining to request assistance based on the state of the first user,

sending, to the second device of the second user, a request for the second user to guide the character of the first device in the communications session, and

receiving, from the second device, an acceptance of the request; and

after the acceptance of the request has been received, starting the communications session between the first device and the second device wherein, during the communications session, the computer (i) streams an audio signal from the first device to the second device in real time, (ii) streams a video signal from the first device to the second device in real time, (iii) receives the phrase to be spoken by the character from the second device, and (iv) causes the first device to present audio of the character speaking the phrase using the text-to-speech voice of the character, and wherein the communications session comprises multiple audio communications of the first user and multiple phrases spoken by the character.

37. The computer-implemented method of claim 11, comprising, after determining to request assistance and prior to starting the communications session, causing the first device to take an action to assist the first user of the first device.

38. The computer-implemented method of claim 37, wherein the action comprises the first device doing at least one of:

presenting audio using the text-to-speech voice of the character, playing a video, or playing music.

39. The computer-implemented method of claim 11, wherein the second user is a family member of the first user or a medical professional who is treating the first user.

40. The computer-implemented method of claim 11, comprising:

determining a severity level of the state of the first user using the first audio signal received from the first device; and

determining to request assistance using the severity level.

41. The computer-implemented method of claim 11, comprising determining the state of the first user by processing the first audio signal using a classifier that detects at least one of yelling or self-stimulation.

12-17. -17. (canceled)

18. One or more non-transitory computer-readable media comprising computer executable instructions that, when executed, cause at least one processor to perform actions comprising:

before a communications session has started;

receiving from a first device a first audio signal corresponding to audio captured by a microphone of the first device,

determining a state of a first user of the first device by processing the first audio signal,

determining to request assistance based on the state of the first user,

sending, to a second device of a second user, a request for the second user to guide the character of the first device in the communications session, and

receiving, from the second device, an acceptance of the request; and

after the acceptance of the request has been received, starting the communications session between the first device and the second device wherein, during the communications session, a computer (i) streams an audio signal from the first device to the second device in real time, (ii) streams a video signal from the first device and to the second device in real time, (iii) receives a phrase to be spoken by the character from the second device, and (iv) causes the first device to present audio of the character speaking the phrase using a text-to-speech voice of the character, and wherein the communications session comprises multiple audio communications of the first user and multiple phrases spoken by the character.

42. The one or more non-transitory computer-readable media of claim 18 in which the actions comprise, before the communications session has started, receiving from the first device at least one of (i) a video signal corresponding to video captured by a camera of the first device, (ii) location data corresponding to a location of the first device, or (iii) biometric data corresponding to readings of a biometric sensor of the first device.

43. The one or more non-transitory computer-readable media of claim 42, the actions comprising determining the state of the first user using the video signal, the location data, or the biometric data.

44. The one or more non-transitory computer-readable media of claim 18, the actions comprising:

determining a first state of the first user by processing first information, wherein the first information does not include an audio signal; and

causing the first device to transmit the first audio signal based on the first state of the first user.

45. The one or more non-transitory computer-readable media of claim 44, wherein the first information comprises biometric data or location data.

46. The one or more non-transitory computer-readable media of claim 18, the actions comprising identifying the second user using a data store of guiding users, wherein (i) the data store comprises an availability indicator for each of the guiding users, (ii) the data store comprises profile information for each of the guiding users, and (iii) the second user is identified by comparing profile information of the first user with profile information of the second user.

19-23. -23. (canceled)