Information presenting apparatus and control method therefor

• US 10,691,197 B2
• Filed: 09/27/2016
• Issued: 06/23/2020
• Est. Priority Date: 10/08/2015
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Yoneda, Aki, Kusukame, Koichi, Shikii, Shinichi, Silawan, Nawatt, Uchida, Shinji, Itoh, Tatsuo
• Primary Examiner(s): Merkoulova, Olga V
• Application Number:
  US15/277,450
• Publication Number:
  US 20170102765A1
• Time in Patent Office: 1,365 Days
• CPC Class Codes: G01C 21/3484 Personalized, e.g. from learned user behaviour or user-defined profiles
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/30 Writer recognition; Reading and verifying signatures

Abstract

A control method for an information presenting apparatus includes: estimating an emotion felt by a user, based on a physical quantity acquired by a sensor; generating identification information that identifies a cause that causes the user to feel the estimated emotion, based on the physical quantity acquired by the sensor or information that a presenter included in the information presenting apparatus presents to the user; and controlling presentation performed by the information presenting apparatus, by using information obtained by executing predetermined processing using the estimated emotion and the generated identification information.

Claims

1. A control method for an information presenting apparatus mounted in a vehicle, the method comprising:

estimating with a processor of the information presenting apparatus whether an emotion felt by a driver of the vehicle is a positive emotion or a negative emotion, based on a physical quantity acquired by a sensor mounted in the vehicle;

identifying with the processor based on information from the sensor whether a line-of-sight of the driver is directed to a frontward direction or a sideward direction, wherein the frontward direction indicates that the line-of-sight of the driver is directed to a front of a road on which the vehicle travels, and wherein the sideward direction indicates that the line-of-sight of the driver is directed to a side of the road;

acquiring with the processor a current road situation on one or more roads outside the vehicle related to the direction of the line-of-sight of the driver when it is identified that the direction of the line-of-sight of the driver is the frontward direction or the sideward direction;

estimating with the processor a sentiment of the driver based on

a) the estimated emotion of the driver being the positive emotion or the negative emotion,

b) the identified direction of the line-of-sight of the driver being the frontward direction or the sideward direction, and

c) the acquired current road situation;

determining with the processor an operation to obtain information for maintaining or modifying the estimated emotion felt by the driver, based on

a) the estimated emotion felt by the driver being the positive emotion or the negative emotion,

b) the identified direction of the line-of-sight of the driver being the frontward direction or the sideward direction, and

c) the acquired current road situation; and

controlling with the processor an output interface of the information presenting apparatus, to present the information for maintaining or modifying the estimated emotion felt by the driver, wherein the information for maintaining or modifying the estimated emotion felt by the driver is obtained by the processor executing the determined operation.

2. The control method according to claim 1,

wherein, the information presenting apparatus further includes a memory that stores a table including sentiment information pieces indicating sentiments of the driver, and each of the sentiment information pieces is associated with a corresponding combination of (i) an emotion information piece indicating a positive emotion or a negative emotion, (ii) a direction information piece indicating a frontward direction, or a sideward direction and (iii) a road situation information piece indicating one of predetermined road situations,

wherein, the table further includes operation information pieces indicating operations to be executed by the processor, and each of the operation information pieces is associated with a corresponding combination of (i) the emotion information piece, (ii) the direction information piece and (iii) the road situation information piece,

wherein, in the estimating of the sentiment of the driver, the estimated sentiment of the driver is a sentiment indicated by one of the sentiment information pieces included in the table and the one of the sentiment information pieces is associated with a corresponding combination of (i) an emotion information piece indicating the same emotion as the estimated emotion of the driver, (ii) a direction information piece indicating the same direction as the identified direction of the line-of-sight of the driver and (iii) a road situation information piece indicating the same road situation as the acquired current road situation, and

wherein, in the determining, the determined operation is an operation indicated by one of the operation information pieces included in the table and the one of the operation information pieces is associated with a corresponding combination of (i) the emotion information piece indicating the same emotion as the estimated emotion of the driver, (ii) the direction information piece indicating the same direction as the identified direction of the line-of-sight of the driver, and (iii) the road situation information piece indicating the same road situation as the acquired current road situation.

3. The control method according to claim 1,

wherein the sensor is a camera that acquires an image of the driver as the physical quantity;

the information presenting apparatus is a navigation apparatus;

road navigation or sound output is performed by the navigation apparatus;

in the identifying,

the direction of the line-of-sight of the driver is identified based on information that is obtained from the image acquired by the camera.

4. The control method according to claim 3,

wherein, road information regarding the road situation is obtained by a processor of the navigation apparatus executing the determined operation.

5. The control method according to claim 1,

wherein the output interface includes at least one of a display panel and a speaker.

6. The control method according to claim 2, wherein

when the estimated emotion of the driver is the negative emotion, the identified direction of the line-of-sight of the driver is the frontward direction, and the acquired current road situation indicates that congestion occurs on the road ahead of the vehicle,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver is irritated by the congestion, and

the controlling controls the information presenting apparatus to present to the driver information on how long it takes for the congestion to end.

7. The control method according to claim 2, wherein

when the estimated emotion of the driver is the negative emotion, the identified direction of the line-of-sight of the driver is the frontward direction, and the acquired current road situation indicates that congestion does not occur on the road ahead of the vehicle,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver is annoyed by the vehicle ahead, and

the controlling controls the information presenting apparatus to present to the driver relaxing music.

8. The control method according to claim 2, wherein

when the estimated emotion of the driver is the positive emotion, the identified direction of the line-of-sight of the driver is the frontward direction, and the acquired current road situation indicates that congestion occurs on the road ahead of the vehicle,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver cannot wait to arrive at a destination of the vehicle, and

the controlling controls the information presenting apparatus to present to the driver information about the surroundings of the destination.

9. The control method according to claim 2, wherein

when the estimated emotion of the driver is the positive emotion, the identified direction of the line-of-sight of the driver is the frontward direction, and the acquired current road situation indicates that congestion does not occur on the road ahead of the vehicle,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver is having fun driving, and

the controlling controls the information presenting apparatus to learn the driver'"'"'s preferred roads and to use the preferred roads for next-route searching.

10. The control method according to claim 2, wherein

when the estimated emotion of the driver is the negative emotion, the identified direction of the line-of-sight of the driver is the sideward direction, and the acquired current road situation indicates that the road on which the vehicle travels is a five-lane road,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver is irritated that a lane change cannot be made, and

the controlling controls the information presenting apparatus to present to the driver an alert.

11. The control method according to claim 2, wherein

when the estimated emotion of the driver is the negative emotion, the identified direction of the line-of-sight of the driver is the sideward direction, and the acquired current road situation indicates that the vehicle is travelling through a restaurant district,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver is irritated because the driver is hungry, and

the controlling controls the information presenting apparatus to present to the driver navigation information to a restaurant.

12. The control method according to claim 2, wherein

when the estimated emotion of the driver is the negative emotion, the identified direction of the line-of-sight of the driver is the sideward direction, and the acquired current road situation indicates that the vehicle is not travelling through a restaurant district and the road on which the vehicle travels is not a five-lane road,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver is irritated for no particular reason, and

the controlling controls the information presenting apparatus to present to the driver relaxing music.

13. The control method according to claim 2, wherein

when the estimated emotion of the driver is the positive emotion, the identified direction of the line-of-sight of the driver is the sideward direction, and the acquired current road situation indicates that a scenic place is nearby,

in the estimating of the sentiment of the driver, the sentiment of the driver estimated using the table is that the driver would have fun to see the scenic place, and

the controlling controls the information presenting apparatus to present to the driver information about the scenic place and to learn the driver'"'"'s preferred scenic places.

14. A control method for an information presenting apparatus, the method comprising:

estimating an emotion felt by a user, based on a physical quantity acquired by a sensor;

generating, by a processor, identification information that identifies a cause that causes the user to feel the estimated emotion, based on the physical quantity acquired by the sensor or information presented to the user by an output interface of the information presenting apparatus; and

controlling, by the processor, the output interface of the information presenting apparatus, to present information for maintaining or modifying the estimated emotion felt by the user, the information for maintaining or modifying the estimated emotion felt by the user being obtained by executing a predetermined processing using the estimated emotion and the cause identified by the generated identification information,

wherein the information for maintaining or modifying the estimated emotion felt by the user includes content different from the information forming the basis for generating the identification information,

wherein the cause is a piece of speech data using a first language, the piece of speech data being presented to the user by the output interface through speech output;

in the estimating,

a degree of understanding indicated by the user about the piece of speech data using the first language presented as the information forming the basis for generating the identification information by the output interface is estimated as the emotion;

in the generating,

information that identifies the first language of the piece of speech data presented by the output interface is generated as the identification information;

in the executing of the predetermined processing,

when the estimated degree of understanding is smaller than a predetermined degree, a piece of speech data using a second language different from the first language is obtained as the information for modifying the estimated emotion felt by the user; and

in the controlling,

the output interface is controlled to present the obtained piece of speech data using the second language through speech output.

15. The control method according to claim 14,

wherein the piece of speech data using the first language is a piece of speech guidance data using the first language, and the piece of speech data using the second language is a piece of speech guidance data using the second language,

wherein the information presenting apparatus has a memory in which the piece of speech guidance data using the first language, and the piece of speech guidance data using the second language are stored;

and

in the executing of the predetermined processing,

(a) when the estimated degree of understanding about the piece of speech guidance data using the first language is smaller than a predetermined degree, the piece of speech guidance data using the second language is selected and obtained as the information for modifying the estimated emotion felt by the user from the memory, and

(b) when the estimated degree of understanding about the piece of speech guidance data using the first language is larger than or equal to the predetermined degree, it is determined that a speech guidance data using the first language as the speech output of the output interface can be understood by the user.

16. An information presenting apparatus mounted in a vehicle comprising:

a processor; and

a memory that stores a program,

wherein the program causes the processor to execute;

estimating whether an emotion felt by a driver of the vehicle is a positive emotion or a negative emotion, based on a physical quantity acquired by a sensor mounted in the vehicle;

identifying, based on information from the sensor, whether a line-of-sight of the driver is directed to a frontward direction or a sideward direction, wherein the frontward direction indicates that the line-of-sight of the driver is directed to a front of a road on which the vehicle travels, and wherein the sideward direction indicates that the line-of-sight of the driver directs to a side of the road and wherein the downward direction indicates that the line-of-sight of the driver directs to a display panel of the information presenting apparatus;

acquiring a current road situation on one or more roads outside the vehicle related to the direction of the line-of-sight of the driver when it is identified that the direction of the line-of-sight of the driver is the frontward direction or the sideward direction;

estimating a sentiment of the driver based on

a) the estimated emotion of the driver being the positive emotion or the negative emotion,

b) the identified direction of the line-of-sight of the driver being the frontward direction or the sideward direction, and

c) the acquired current road situation;

determining an operation to obtain information for maintaining or modifying the estimated emotion felt by the driver, based on

a) the estimated emotion felt by the driver being the positive emotion or the negative emotion,

b) the identified direction of the line-of-sight of the driver being the frontward direction or the sideward direction, and

c) the acquired current road situation; and

controlling an output interface of the information presenting apparatus, to present the information for maintaining or modifying the estimated emotion felt by the driver, wherein the information for maintaining or modifying the estimated emotion felt by the driver is obtained by the processor executing the determined operation.