SENSOR INPUT RECORDING AND TRANSLATION INTO HUMAN LINGUISTIC FORM

• US 20140044307A1
• Filed: 08/10/2012
• Published: 02/13/2014
• Est. Priority Date: 08/10/2012
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm Labs, Inc. (Qualcomm, Inc.)
• Inventors: Kenagy, Jason B., Kaskoun, Kenneth, Hannan, John J.
• Application Number:
  US13/572,324
• Publication Number:
  US 20140044307A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: H04M 1/7243 with interactive means for internal management of messages
• CPC Class Codes: H04M 1/72436 for text messaging, e.g. short messaging services [SMS] or e-mails
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions

Abstract

Systems, methods, and devices use a mobile device'"'"'s sensor inputs to automatically draft natural language messages, such as text messages or email messages. In the various embodiments, sensor inputs may be obtained and analyzed to identify subject matter which a processor of the mobile device may reflect in words included in a communication generated for the user. In an embodiment, subject matter associated with a sensor data stream may be associated with a word, and the word may be used to assemble a natural language narrative communication for the user, such as a written message.

Claims

1. A method for communicating using a computing device, comprising:

obtaining sensor data from one or more sensors within the computing device;

analyzing the sensor data to determine whether the data includes a cue that a written communication should be generated;

analyzing the sensor data to identify subject matter for inclusion in a written communication;

identifying a word associated with the identified subject matter; and

automatically assembling a communication including the identified word based on the identified subject matter.

2. The method of claim 1, wherein a cue is one of a button press, a recognized sound, a gesture, a touch screen press, a zoom-in event, a zoom-out event, a stream of video including substantially the same image, a loud sound, a spoken command, an utterance, a squeeze, a tilt, or an eye capture indication.

3. The method of claim 1, wherein the sensor data includes one or more of an image, sound, movement/acceleration measurement, temperature, location, speed, time, date, heart rate, and/or respiration.

4. The method of claim 1, wherein analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

analyzing a segment of the sensor data to identify a characteristic of the data segment;

comparing the characteristic of the data segment to a threshold value;

determining whether the characteristic of the segment matches or exceeds the threshold value; and

identifying the segment as the cue in response to the characteristic of the segment exceeding the threshold value.

5. The method of claim 4, wherein the sensor data is a video data stream and the data segment is a video frame.

6. The method of claim 1, wherein the sensor data is a video data stream and comprises a plurality of video frames, and

wherein analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

analyzing each video frame of the plurality of video frames to identify a characteristic of each video frame;

comparing the characteristics of each video frame of the plurality of video frames to the characteristic of another video frame of the plurality of video frames;

determining if the characteristics of any video frames of the plurality of video frames are similar; and

identifying the video frames with similar characteristics as the cue in response to the characteristics of any video frames of the plurality of video frames being determined as similar.

7. The method of claim 1, further comprising:

recording a portion of the sensor data; and

analyzing the recorded portion of the sensor data to identify other subject matter,

wherein automatically assembling a communication including the identified word based on the identified subject matter comprises using the identified other subject matter.

8. The method of claim 1, wherein assembling a communication including the identified word based on the identified subject matter is performed, at least in part, using natural language processing.

9. The method of claim 1, wherein automatically assembling a communication including the identified word based on the identified subject matter further comprises assembling the communication including at least a portion of the sensor data.

10. The method of claim 1, wherein identifying a word associated with the subject matter includes using a word associated with a typical speech pattern of a user of the computing device.

11. The method of claim 1, wherein identifying a word associated with the subject matter and automatically assembling a communication including the identified word based on the identified subject matter are accomplished on a mobile device.

12. The method of claim 1, further comprising transmitting a portion of the sensor data to a server in response to recognizing a cue,

wherein identifying a word associated with the subject matter and automatically assembling a communication including the identified word based on the identified subject matter are done on the server.

13. The method of claim 12, further comprising transmitting the assembled communication to the computing device.

14. A computing device, comprising:

means for obtaining sensor data from one or more sensors within the computing device;

means for analyzing the sensor data to determine whether the data includes a cue that a written communication should be generated;

means for analyzing the sensor data to identify subject matter for inclusion in a written communication;

means for identifying a word associated with the identified subject matter; and

means for automatically assembling a communication including the identified word based on the identified subject matter.

15. The computing device of claim 14, wherein a cue is one of a button press, a recognized sound, a gesture, a touch screen press, a zoom-in event, a zoom-out event, a stream of video including substantially the same image, a loud sound, a spoken command, an utterance, a squeeze, a tilt, or an eye capture indication.

16. The computing device of claim 14, wherein the sensor data includes one or more of an image, sound, movement/acceleration measurement, temperature, location, speed, time, date, heart rate, and/or respiration.

17. The computing device of claim 14, wherein means for analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

means for analyzing a segment of the sensor data to identify a characteristic of the data segment;

means for comparing the characteristic of the data segment to a threshold value;

means for determining whether the characteristic of the segment matches or exceeds the threshold value; and

means for identifying the segment as the cue in response to the characteristic of the segment exceeding the threshold value.

18. The computing device of claim 17, wherein the sensor data is a video data stream and the data segment is a video frame.

19. The computing device of claim 14, wherein the sensor data is a video data stream and comprises a plurality of video frames, and

wherein means for analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

means for analyzing each video frame of the plurality of video frames to identify a characteristic of each video frame;

means for comparing the characteristics of each video frame of the plurality of video frames to the characteristic of another video frame of the plurality of video frames;

means for determining if the characteristics of any video frames of the plurality of video frames are similar; and

means for identifying the video frames with similar characteristics as the cue in response to the characteristics of any video frames of the plurality of video frames being determined as similar.

20. The computing device of claim 14, further comprising:

means for recording a portion of the sensor data; and

means for analyzing the recorded portion of the sensor data to identify other subject matter,

wherein means for automatically assembling a communication including the identified word based on the identified subject matter comprises means for using the identified other subject matter.

21. The computing device of claim 14, wherein means for assembling a communication including the identified word based on the identified subject matter comprises means for assembling a communication using natural language processing.

22. The computing device of claim 14, wherein means for assembling a communication including the identified word based on the identified subject matter further comprises means for assembling the communication including at least a portion of the sensor data.

23. The computing device of claim 14, wherein means for identifying a word associated with the subject matter comprises means for using a word associated with a typical speech pattern of a user of the computing device.

24. The computing device of claim 14, wherein computing device is a mobile device.

25. A computing device, comprising:

a memory;

one or more sensors; and

a processor coupled to the memory and the one or more sensors, wherein the processor is configured with processor-executable instructions to perform operations comprising;

obtaining sensor data from the one or more sensors within the computing device;

analyzing the sensor data to determine whether the data includes a cue that a written communication should be generated;

analyzing the sensor data to identify subject matter for inclusion in a written communication;

identifying a word associated with the identified subject matter; and

automatically assembling a communication including the identified word based on the identified subject matter.

26. The computing device of claim 25, wherein a cue is one of a button press, a recognized sound, a gesture, a touch screen press, a zoom-in event, a zoom-out event, a stream of video including substantially the same image, a loud sound, a spoken command, an utterance, a squeeze, a tilt, or an eye capture indication.

27. The computing device of claim 25, wherein the sensor data includes one or more of an image, sound, movement/acceleration measurement, temperature, location, speed, time, date, heart rate, and/or respiration.

28. The computing device of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

analyzing a segment of the sensor data to identify a characteristic of the data segment;

comparing the characteristic of the data segment to a threshold value;

determining whether the characteristic of the segment matches or exceeds the threshold value; and

identifying the segment as the cue in response to the characteristic of the segment exceeding the threshold value.

29. The computing device of claim 28, wherein the sensor data is a video data stream and the data segment is a video frame.

30. The computing device of claim 25, wherein the sensor data is a video data stream and comprises a plurality of video frames, and

wherein the processor is configured with processor-executable instructions to perform operations such that analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

analyzing each video frame of the plurality of video frames to identify a characteristic of each video frame;

comparing the characteristics of each video frame of the plurality of video frames to the characteristic of another video frame of the plurality of video frames;

determining if the characteristics of any video frames of the plurality of video frames are similar; and

identifying the video frames with similar characteristics as the cue in response to the characteristics of any video frames of the plurality of video frames being determined as similar.

31. The computing device of claim 25, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

recording a portion of the sensor data; and

analyzing the recorded portion of the sensor data stream to identify other subject matter,

wherein the processor is configured with processor-executable instructions to perform operations such that automatically assembling a communication including the identified word based on the identified subject matter comprises using the identified other subject matter.

32. The computing device of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that assembling a communication including the identified word based on the identified subject matter is performed, at least in part, using natural language processing.

33. The computing device of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that assembling a communication including the identified word based on the identified subject matter further comprises assembling the communication including at least a portion of the sensor data.

34. The computing device of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that identifying a word associated with the subject matter includes using a word associated with a typical speech pattern of a user of the computing device.

35. The computing device of claim 25, wherein the computing device is a mobile device.

36. A non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor to perform operations;

comprising;

obtaining sensor data from one or more sensors within a computing device;

analyzing the sensor data to determine whether the data includes a cue that a written communication should be generated;

analyzing the sensor data to identify subject matter for inclusion in a written communication;

identifying a word associated with the identified subject matter; and

automatically assembling a communication including the identified word based on the identified subject matter.

37. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that a cue is one of a button press, a recognized sound, a gesture, a touch screen press, a zoom-in event, a zoom-out event, a stream of video including substantially the same image, a loud sound, a spoken command, an utterance, a squeeze, a tilt, or an eye capture indication.

38. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that the sensor data includes one or more of an image, sound, movement/acceleration measurement, temperature, location, speed, time, date, heart rate, and/or respiration.

39. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises:

analyzing a segment of the sensor data to identify a characteristic of the data segment;

comparing the characteristic of the data segment to a threshold value;

determining whether the characteristic of the segment matches or exceeds the threshold value; and

identifying the segment as the cue in response to the characteristic of the segment exceeding the threshold value.

40. The non-transitory processor-readable storage medium of claim 39, wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that the sensor data is a video data stream and the data segment is a video frame.

41. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that:

the sensor data is a video data stream and comprises a plurality of video frames; and

analyzing the sensor data to determine whether it includes a cue that a written communication should be generated comprises;

analyzing each video frame of the plurality of video frames to identify a characteristic of each video frame;

comparing the characteristics of each video frame of the plurality of video frames to the characteristic of another video frame of the plurality of video frames;

determining if the characteristics of any video frames of the plurality of video frames are similar; and

identifying the video frames with similar characteristics as the cue in response to the characteristics of any video frames of the plurality of video frames being determined as similar.

42. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause a processor to perform operations further comprising:

recording a portion of the sensor data stream; and

analyzing the recorded portion of the sensor data to identify other subject matter,

wherein the stored processor-executable instructions are configured to cause a processor to perform operations such that automatically assembling a communication including the identified word based on the identified subject matter comprises using the identified other subject matter.

43. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that assembling a communication including the identified word based on the identified subject matter is performed, at least in part, using natural language processing.

44. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that automatically assembling a communication including the identified word based on the identified subject matter further comprises assembling the communication including at least a portion of the sensor data.

45. The non-transitory processor-readable storage medium of claim 36, wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that identifying a word associated with the subject matter includes using a word associated with a typical speech pattern of a user of the computing device.

46. A system for communicating using a computing device, comprising:

means for obtaining sensor data from one or more sensors within the computing device;

means for analyzing the sensor data to determine whether the data includes a cue that a written communication should be generated;

means for transmitting a portion of the sensor data to a server in response to recognizing a cue;

means for analyzing, on the server, the sensor data to identify subject matter for inclusion in a written communication;

means for identifying, on the server, a word associated with the identified subject matter; and

means for automatically assembling, on the server, a communication including the identified word based on the identified subject matter.

47. The system of claim 46, further comprising means for transmitting the assembled communication to the computing device.

48. The system of claim 46, wherein the computing device is a mobile device.

49. A system, comprising:

a computing device, comprising;

a memory;

one or more sensors; and

a device processor coupled to the memory and the one or more sensors; and

a server comprising a server processor,

wherein the device processor is configured with processor-executable instructions to perform operations comprising;

obtaining sensor data from the one or more sensors within the computing device;

analyzing the sensor data to determine whether the data includes a cue that a written communication should be generated; and

transmitting a portion of the sensor data to the server in response to recognizing a cue,

wherein the server processor is configured with processor-executable instructions to perform operations comprising;

analyzing the sensor data to identify subject matter for inclusion in a written communication;

identifying a word associated with the identified subject matter; and

automatically assembling a communication including the identified word based on the identified subject matter.

50. The system of claim 49, wherein the server processor is configured with processor-executable instructions to perform operations further comprising transmitting the assembled communication to the computing device.

51. The system of claim 49, wherein the computing device is a mobile device.