Computer-based method and system for providing active and automatic personal assistance using an automobile or a portable electronic device

• US 9,223,837 B2
• Filed: 03/14/2013
• Issued: 12/29/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Djugash, Joseph M. A.
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Brown, Jr., Nathan
• Application Number:
  US13/830,888
• Publication Number:
  US 20140280051A1
• Time in Patent Office: 1,020 Days
• Field of Search: 706/12, 706/20, 706/21, 382/118
• US Class Current: 1/1
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2556/55 using telemetry
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 50/0098 Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

A method and a system for providing personal assistance in daily activities. A method and a system for automatically and actively providing personal assistance, using a vehicle or a portable electronic device, based on detected data regarding a user and the user'"'"'s environment. The method and system may include a processor, at least one sensor, an output device, a communications unit, and a database. The database may further include a memory and cloud-based database and computing. The method and system may actively and automatically provide personal assistance regarding health, exercise, diet, or nutrition. The method and system may assist the user or a health professional in health diagnosis and treatment.

Claims

1. A computer-based method comprising the steps of:

detecting, using at least one sensor, a data associated with at least one person, an object, a living being, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

selectively storing, using a processor, a learned data in a database based on the detected data and a pre-programmed algorithm stored in the database;

passively outputting, using a communications unit or an output device coupled to or in communication with the processor, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database; and

actively and automatically outputting, using the communications unit or the output device, a second output data based on the detected data, the learned data, and the pre-programmed algorithm,

wherein the step of actively and automatically outputting, using the communications unit or the output device, the second output data includes at least one of;

displaying, using a display, an output image or an output video, the display being coupled to or in communication with the communications unit or the output device, or

generating, using a speaker, an output speech, the speaker being coupled to or in communication with the communications unit or the output device.

2. The method of claim 1, wherein the detected data corresponds to at least one of:

a visual characteristic, a speech, tactile information, a motion, an interaction, an action, a behavior, a conversation, a location, an identity, an attribute, or combinations thereof associated with the at least one person,

a motion, a sound, an interaction, an action, tactile information, a visual characteristic, a location, an identity, or combinations thereof associated with the object, the living being, the place, the environment, or combinations thereof within a proximity of at least one of the at least one sensor, or

a visual characteristic, a type, a location, a temperature, a moisture content, a weather condition, a sound, an attribute, or combinations thereof associated with the place or the environment within a proximity of at least one of the at least one sensor.

3. The method of claim 1, further comprising:

analyzing, using the processor, the detected data based on the learned data corresponding to the at least one person, the object, the living being, the event, the place, the environment, or combinations thereof,

wherein the steps of selectively storing the database and actively and automatically outputting the second output data are further based on the analyzed data.

4. The method of claim 1, wherein the method is implemented in a vehicle.

5. The method of claim 1, wherein the method is implemented in a portable electronic device.

6. The method of claim 1, further comprising:

analyzing, using the processor, a conversation or an interaction based on the detected data,

wherein the step of actively and automatically outputting the second output data further includes at least one of;

actively and automatically generating, using the communications unit or the output device, an output speech, an output image, an output video, or combinations thereof for actively and automatically initiating or participating in the conversation or the interaction with the at least one person, or

actively and automatically generating, using the communications unit or the output device, the output speech, the output image, the output video, or combinations thereof for actively communicating a reminder to a user.

7. The method of claim 1, further comprising:

determining, using the processor, a relationship, a sequence, or a correlation between at least two of the at the at least one person, the object, the living being, the event, the place, or the environment based on the learned data and the pre-programmed algorithm,

wherein the step of actively and automatically outputting the second output data is further based on the determined relationship, sequence, or correlation.

8. The method of claim 1, further comprising:

actively and automatically improving a quality of or increasing a quantity of the detected data by performing at least one of;

actively and automatically outputting, using the communications unit or the output device, the second output data in form of a question or an inquiry for eliciting a response or an input from the at least one person, or

actively and automatically outputting, using the communications unit or the output device, the second output data for participating in a conversation or an interaction with the at least one person.

9. The method of claim 1, wherein:

the at least one sensor is at least one of an image sensor, a vibration sensor, a microphone, a proximity detector, a position detector, a force sensor, a distance sensor, a laser scanner, or a radar sensor, and

the at least one sensor is integrated in or coupled to at least a vehicle or a portable electronic device.

10. The method of claim 1, wherein the database includes or is in communication with a cloud-based database, the processor being further configured to:

transmit, using the cloud-based database, the detected data, the first output data, the second output data, or combinations thereof to an external device coupled to or in communication with the cloud-based database, and

receive, using the cloud-based database, an input data from the external device.

11. The method of claim 1, wherein the at least one person includes a user with disabilities or suffering from health or memory complications, and wherein the step of actively and automatically outputting the second output data further includes:

associating the detected data with the learned data corresponding to the person, the object, the living being, the event, the place, the environment, or combinations thereof,

wherein the second output data corresponds to a reminder data directed to communicating information to the user with respect to identification, characteristics, location, actions, or interactions of the at least one person, the object, the living being, the event, the place, the environment, or combinations thereof.

12. The method of claim 1, wherein the method further comprises:

analyzing, using the processor, health, exercise, or diet activities of the at least one person based on the detected data,

wherein the step of actively and automatically outputting the second output data includes at least one of;

outputting, using the communications unit or the output device, the second output data to a health professional for assisting the health professional in evaluation of the health, exercise, or diet activities of the at least one person, or

outputting, using the communications unit or the output device, the second output data to the at least one person for outputting a recommendation associated with the health, exercise, or diet activities.

13. The method of claim 1, wherein the pre-programmed algorithm corresponds to appropriateness algorithm rules for determining:

information content of the second output data, privacy concerns of a user, context of a detected event, or combinations thereof, the method further comprising;

analyzing, using the processor, the detected data based on the appropriateness algorithm rules and the learned data,

wherein the step of actively and automatically outputting the second output data is based on the analyzed data.

14. A vehicle comprising:

at least one sensor configured to detect a data associated with a driver, a passenger, an object, a living being, an event, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

a database storing a pre-programmed algorithm including a speech recognition and processing algorithm;

a communications unit including a display, a microphone, and a speaker; and

a processor coupled to or in communication with the at least one sensor, the database, and the communications unit, the processor being configured to;

selectively store a learned data in the database based on the detected data and the pre-programmed algorithm,

analyze the detected data based on the learned data and the pre-programmed algorithm,

detect, using the speech recognition and processing algorithm, a speech of at least one of the driver or the passenger,

analyze, using the speech recognition and processing algorithm, the detected speech based on the learned data,

determine a relationship, a sequence, or a correlation between the detected speech and the driver, the passenger, the object, the living being, the event, the place, the environment, or combinations thereof,

passively output, using the communications unit, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database, and

actively and automatically output, using the communications unit, a second output data based on the analyzed data, the learned data, and the pre-programmed algorithm, by performing at least one of;

generating, using the speaker, an output speech based on the determined relationship, sequence, or correlation, or

displaying, using the display, an output image or an output video based on the determined relationship, sequence, or correlation.

15. The vehicle of Claim 14, wherein the processor is further configured to:

determine a current expected level of driving attention based on the detected data and based on a current operation of the vehicle, and

determine an output timing and an information content of the generated output speech or the displayed output image or output video based on the determined current expected level of driving attention.

16. The vehicle of claim 15, wherein the database includes or is in communication with a cloud-based database, the processor being further configured to:

transmit, using the cloud-based database, the detected data, the first output data, the second output data, or combinations thereof to an external device, and

receive, using the cloud-based database, an input data from the external device.

17. A portable electronic device comprising:

at least one sensor configured to detect a data associated with at least one person or a user, an object, a living being, an event, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

a database storing a pre-programmed algorithm including a speech recognition and processing algorithm;

a communications unit including a display, a microphone, and a speaker; and

a processor coupled to or in communication with the at least one sensor, the database, and the communications unit, the processor being configured to;

selectively store a learned data in the database based on the detected data and the pre-programmed algorithm,

analyze the detected data based on the learned data and the pre-programmed algorithm,

detect, using the speech recognition and processing algorithm, a speech of the at least one person or the user,

analyze, using the speech recognition and processing algorithm, the detected speech based on the learned data,

determine a relationship, a sequence, or a correlation between the detected speech and the at least one person, the user, the object, the living being, the event, the place, the environment, or combinations thereof,

passively output, using the communications unit, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database, and

actively and automatically output, using the communications unit, a second output data based on the analyzed data, the learned data, and the pre-programmed algorithm by performing at least one of;

generating, using the speaker, an output speech based on the determined relationship, sequence, or correlation, or

displaying, using the display, an output image or an output video based on the determined relationship, sequence, or correlation.

18. A computer-based method comprising the steps of:

detecting, using at least one sensor, a data associated with at least one person, a user, an object, a living being, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

selectively storing, using a processor, a learned data in a database based on the detected data and a pre-programmed algorithm stored in the database;

analyzing, using the processor, a conversation or an interaction based on the detected data;

passively outputting, using a communications unit or an output device coupled to or in communication with the processor, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database; and

actively and automatically outputting, using the communications unit or the output device, a second output data based on the detected data, the learned data, and the pre-programmed algorithm,

wherein the step of actively and automatically outputting the second output data further includes at least one of;

actively and automatically generating, using the communications unit or the output device, an output speech, an output image, an output video, or combinations thereof for actively and automatically initiating or participating in the conversation or the interaction with the at least one person or the user, or

actively and automatically generating, using the communications unit or the output device, the output speech, the output image, the output video, or combinations thereof for actively communicating a reminder to the user.

19. The computer-based method of claim 18, wherein the detected data corresponds to at least one of:

a visual characteristic, a speech, tactile information, a motion, an interaction, an action, a behavior, a conversation, a location, an identity, an attribute, or combinations thereof associated with the at least one person,

a motion, a sound, an interaction, an action, tactile information, a visual characteristic, a location, an identity, or combinations thereof associated with the object, the living being, the place, the environment, or combinations thereof within a proximity of at least one of the at least one sensor, or

a visual characteristic, a type, a location, a temperature, a moisture content, a weather condition, a sound, an attribute, or combinations thereof associated with the place or the environment within a proximity of at least one of the at least one sensor.

20. The computer-based method of claim 18, wherein the method is implemented in a vehicle.

21. A computer-based method comprising the steps of:

detecting, using at least one sensor, a data associated with at least one person, an object, a living being, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

selectively storing, using a processor, a learned data in a database based on the detected data and a pre-programmed algorithm stored in the database;

passively outputting, using a communications unit or an output device coupled to or in communication with the processor, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database;

actively and automatically outputting, using the communications unit or the output device, a second output data based on the detected data, the learned data, and the pre-programmed algorithm; and

actively and automatically improving a quality of or increasing a quantity of the detected data by performing at least one of;

actively and automatically outputting, using the communications unit or the output device, the second output data in form of a question or an inquiry for eliciting a response or an input from the at least one person, or

actively and automatically outputting, using the communications unit or the output device, the second output data for participating in a conversation or an interaction with the at least one person.

22. The computer-based method of claim 21, wherein the method is implemented in a vehicle.

23. The computer-based method of claim 21, wherein the at least one sensor is at least one of an image sensor, a vibration sensor, a microphone, a proximity detector, a position detector, a force sensor, a distance sensor, a laser scanner, or a radar sensor, and

the at least one sensor is integrated in or coupled to at least a vehicle or a portable electronic device.

24. A computer-based method comprising the steps of:

detecting, using at least one sensor, a data associated with at least one person, an object, a living being, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

selectively storing, using a processor, a learned data in a database based on the detected data and a pre-programmed algorithm stored in the database, the database including or being in communication with a cloud-based database;

passively outputting, using a communications unit or an output device coupled to or in communication with the processor, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database;

actively and automatically outputting, using the communications unit or the output device, a second output data based on the detected data, the learned data, and the pre-programmed algorithm;

transmitting, using the cloud-based database, the detected data, the first output data, the second output data, or combinations thereof to an external device coupled to or in communication with the cloud-based database; and

receiving, using the cloud-based database, an input data from the external device.

25. The computer-based method of claim 24, wherein the detected data corresponds to at least one of:

a visual characteristic, a speech, tactile information, a motion, an interaction, an action, a behavior, a conversation, a location, an identity, an attribute, or combinations thereof associated with the at least one person,

a motion, a sound, an interaction, an action, tactile information, a visual characteristic, a location, an identity, or combinations thereof associated with the object, the living being, the place, the environment, or combinations thereof within a proximity of at least one of the at least one sensor, or

a visual characteristic, a type, a location, a temperature, a moisture content, a weather condition, a sound, an attribute, or combinations thereof associated with the place or the environment within a proximity of at least one of the at least one sensor.

26. The computer-based method of claim 24, wherein the step of actively and automatically outputting the second output data includes at least one of:

displaying, using a display coupled to or in communication with the communications unit or the output device, an output image or an output video, or

generating, using a speaker coupled to or in communication with the communications unit or the output device, an output speech.

27. A computer-based method comprising the steps of:

detecting, using at least one sensor, a data associated with at least one person, an object, a living being, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor;

selectively storing, using a processor, a learned data in a database based on the detected data and a pre-programmed algorithm stored in the database;

analyzing, using the processor, the detected data based on the appropriateness algorithm rules and the learned data;

passively outputting, using a communications unit or an output device coupled to or in communication with the processor, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database; and

actively and automatically outputting, using the communications unit or the output device, a second output data based on the analyzed data, the detected data, the learned data, and the pre-programmed algorithm,

wherein the pre-programmed algorithm corresponds to appropriateness algorithm rules for determining an information content of the second output data, a privacy concern of a user, a context of a detected event, or combinations thereof.

28. The computer-based method of claim 27, wherein the method is implemented in a vehicle.

29. The computer-based method of claim 27, wherein the step of actively and automatically outputting the second output data includes at least one of:

displaying, using a display coupled to or in communication with the communications unit or the output device, an output image or an output video, or

generating, using a speaker coupled to or in communication with the communications unit or the output device, an output speech.