Empathetic image selection

• US 10,394,511 B2
• Filed: 09/01/2017
• Issued: 08/27/2019
• Est. Priority Date: 10/20/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chun, Travis W., Flores, Romelia H., Johnson, Marlentae A., Lau, Michael A., Rodriguez, Roberto R., Rutkowski, Ronald J.
• Primary Examiner(s): Shih, Haoshian
• Application Number:
  US15/693,640
• Publication Number:
  US 20180113662A1
• Time in Patent Office: 725 Days
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: G09G 2380/16 Digital picture frames

Abstract

An approach is provided for selecting and displaying image(s). A user and user profile information corresponding to the user are identified. A sentiment of the user is identified by determining that user is in proximity to a digital picture frame in a room, receiving a measurement of ambient lighting of the room from a light sensor coupled to the digital picture frame, and determining an emotional state of the user based on the ambient lighting. Based on the user profile, an association is determined between the sentiment of the user and sentiment(s), which are determined to be conveyed by image(s). Based on the ambient lighting, the association between the sentiment of the user and the sentiment(s), and the image(s) conveying the sentiment(s), the image(s) are selected from multiple images. The selected image(s) are displayed on a display included in the digital picture frame.

Claims

1. A method of selecting and displaying one or more images, the method comprising the steps of:

a computer identifying a user and user profile information corresponding to the identified user;

the computer identifying a sentiment of the user;

based on the user profile information, the computer determining an association between the identified sentiment of the user and one or more sentiments included in a plurality of sentiments conveyed by a plurality of images;

the computer determining that one or more images included in the plurality of images convey the one or more sentiments;

based on the association between the identified sentiment of the user and the one or more sentiments and the one or more images conveying the one or more sentiments, the computer selecting the one or more images from the plurality of images;

the computer displaying the selected one or more images,

wherein the computer is included in a digital picture frame, wherein the step of displaying the selected one or more images includes displaying the one or more images on a display included in the digital picture frame, and wherein the step of identifying the sentiment of the user includes the steps of;

the computer determining that the user is in a proximity to the digital picture frame;

in response to the step of determining that the user in the proximity to the digital picture frame, the computer receiving lighting information from a light sensor coupled to the digital picture frame, wherein the lighting information includes a measurement of ambient lighting of a room in which the digital picture frame is located; and

in response to the step of determining that the user is in the proximity to the digital picture frame, and based on the measurement of the ambient lighting of the room in which the digital picture frame is located, the computer determining an emotional state of the user, the emotional state being included in the sentiment of the user, wherein the step of selecting the one or more images is based in part on the measurement of the ambient lighting of the room in which the digital picture frame is located;

the computer determining that one or more additional users are in the proximity to the digital picture frame;

in response to the one or more additional users being in the proximity to the digital picture frame, the computer determining one or more emotional states of the one or more additional users, respectively;

based on the emotional state of the user and the one or more emotional states of the one or more additional users, the computer determining one emotional state of a majority of users in a group of users consisting of the user and the one or more additional users;

the computer determining an association between the one emotional state of the majority of the users and one sentiment included in the plurality of sentiments conveyed by the plurality of images;

the computer determining that an image included in the plurality of images conveys the one sentiment;

based on the association between the one emotional state of the majority of users and the one sentiment and based on the image conveying the one sentiment, the computer selecting the image from the plurality of images; and

the computer displaying the image on the display included in the digital picture frame.

2. The method of claim 1, further comprising the step of the computer collecting schedules of events of interest to the user from a calendar service or a calendar application, wherein the step of identifying the sentiment of the user is based on the schedules of events of interest to the user collected from the calendar service or the calendar application.

3. The method of claim 1, further comprising the step of the computer automatically generating metadata tags for the plurality of images, the metadata tags specifying respective contents of corresponding images included in the plurality of images, wherein the step of determining that the one or more images convey the one or more sentiments includes:

identifying contents of the one or more images based on one or more metadata tags generated for the one or more images; and

identifying the one or more sentiments based on the identified contents of the one or more images.

4. The method of claim 1, further comprising the step of the computer automatically generating metadata tags for the plurality of images, the metadata tags specifying respective sentiments of corresponding images included in the plurality of images, wherein the step of determining that the one or more images convey the one or more sentiments includes identifying the one or more sentiments based on one or more metadata tags generated for the one or more images.

5. The method of claim 1, further comprising the step of the computer identifying a new post authored by the user via a social media service, wherein the step of identifying the sentiment of the user includes determining the sentiment based in part on contents of the new post, and wherein the step of selecting the one or more images is based in part on the sentiment based in part on the contents of the new post.

6. The method of claim 1, further comprising the steps of:

based on the lighting information received from the light sensor coupled to the digital picture frame, the computer determining that the ambient lighting in the room is dim;

based on the ambient lighting in the room being dim, the computer determining a mood of an environment in proximity to the digital picture frame is somber;

based on the mood of the environment in proximity to the digital picture frame being somber, the computer determining an association between the mood of the environment in proximity to the digital picture frame and a sentiment included in the plurality of sentiments conveyed by the plurality of images;

the computer determining that an image included in the plurality of images conveys the sentiment;

based on the association between the mood of the environment in proximity to the digital picture frame and the sentiment and based on the image conveying the sentiment, the computer selecting the image from the plurality of images; and

the computer displaying the selected image on the display included in the digital picture frame.

7. The method of claim 1, further comprising the step of:

providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable program code in the computer, the program code being executed by a processor of the computer to implement the steps of identifying the user and the user profile information, identifying the sentiment of the user, determining the association between the identified sentiment of the user and the one or more sentiments, determining that the one or more images convey the one or more sentiments, selecting the one or more images, displaying the selected one or more images, displaying the one or more images, determining that the user is in the proximity to the digital picture frame, receiving the lighting information, determining the emotional state of the user, determining that the one or more additional users are in the proximity to the digital picture frame, determining the one or more emotional states of the one or more additional users, determining the one emotional state of the majority of users in the group of users, determining the association between the one emotional state and the one sentiment, determining that the image conveys the one sentiment, selecting the image from the plurality of images, and displaying the image on the display.

8. A computer program product, comprising:

a computer-readable, storage device; and

a computer-readable program code stored in the computer-readable, storage device, the computer-readable program code containing instructions that are executed by a central processing unit (CPU) of a computer system to implement a method of selecting and displaying one or more images, the method comprising the steps of;

the computer system identifying a user and user profile information corresponding to the identified user;

the computer system identifying a sentiment of the user;

based on the user profile information, the computer system determining an association between the identified sentiment of the user and one or more sentiments included in a plurality of sentiments conveyed by a plurality of images;

the computer system determining that one or more images included in the plurality of images convey the one or more sentiments;

based on the association between the identified sentiment of the user and the one or more sentiments and the one or more images conveying the one or more sentiments, the computer system selecting the one or more images from the plurality of images;

the computer system displaying the selected one or more images,

wherein the computer system is included in a digital picture frame, wherein the step of displaying the selected one or more images includes displaying the one or more images on a display included in the digital picture frame, and wherein the step of identifying the sentiment of the user includes the steps of;

the computer system determining that the user is in a proximity to the digital picture frame;

in response to the step of determining that the user in the proximity to the digital picture frame, the computer system receiving lighting information from a light sensor coupled to the digital picture frame, wherein the lighting information includes a measurement of ambient lighting of a room in which the digital picture frame is located; and

in response to the step of determining that the user is in the proximity to the digital picture frame, and based on the measurement of the ambient lighting of the room in which the digital picture frame is located, the computer system determining an emotional state of the user, the emotional state being included in the sentiment of the user, wherein the step of selecting the one or more images is based in part on the measurement of the ambient lighting of the room in which the digital picture frame is located;

the computer system determining that one or more additional users are in the proximity to the digital picture frame;

in response to the one or more additional users being in the proximity to the digital picture frame, the computer system determining one or more emotional states of the one or more additional users, respectively;

based on the emotional state of the user and the one or more emotional states of the one or more additional users, the computer system determining one emotional state of a majority of users in a group of users consisting of the user and the one or more additional users;

the computer system determining an association between the one emotional state of the majority of the users and one sentiment included in the plurality of sentiments conveyed by the plurality of images;

the computer system determining that an image included in the plurality of images conveys the one sentiment;

based on the association between the one emotional state of the majority of users and the one sentiment and based on the image conveying the one sentiment, the computer system selecting the image from the plurality of images; and

the computer system displaying the image on the display included in the digital picture frame.

9. The computer program product of claim 8, wherein the method further comprises the step of the computer system collecting schedules of events of interest to the user from a calendar service or a calendar application, wherein the step of identifying the sentiment of the user is based on the schedules of interest to the user collected from the calendar service or the calendar application.

10. The computer program product of claim 8, wherein the method further comprises the step of the computer system automatically generating metadata tags for the plurality of images, the metadata tags specifying respective contents of corresponding images included in the plurality of images, wherein the step of determining that the one or more images convey the one or more sentiments includes:

identifying contents of the one or more images based on one or more metadata tags generated for the one or more images; and

identifying the one or more sentiments based on the identified contents of the one or more images.

11. The computer program product of claim 8, wherein the method further comprises the step of the computer system automatically generating metadata tags for the plurality of images, the metadata tags specifying respective sentiments of corresponding images included in the plurality of images, wherein the step of determining that the one or more images convey the one or more sentiments includes identifying the one or more sentiments based on one or more metadata tags generated for the one or more images.

12. The computer program product of claim 8, wherein the method further comprises the steps of:

based on the lighting information received from the light sensor coupled to the digital picture frame, the computer system determining that the ambient lighting in the room is dim;

based on the ambient lighting in the room being dim, the computer system determining a mood of an environment in proximity to the digital picture frame is somber;

based on the mood of the environment in proximity to the digital picture frame being somber, the computer system determining an association between the mood of the environment in proximity to the digital picture frame and a sentiment included in the plurality of sentiments conveyed by the plurality of images;

the computer system determining that an image included in the plurality of images conveys the sentiment;

based on the association between the mood of the environment in proximity to the digital picture frame and the sentiment and based on the image conveying the sentiment, the computer system selecting the image from the plurality of images; and

the computer system displaying the selected image on the display included in the digital picture frame.

13. A computer system comprising:

a central processing unit (CPU);

a memory coupled to the CPU; and

a computer readable storage device coupled to the CPU, the computer readable storage device containing instructions that are executed by the CPU via the memory to implement a method of selecting and displaying one or more images, the method comprising the steps of;

the computer system identifying a user and user profile information corresponding to the identified user;

the computer system identifying a sentiment of the user;

based on the user profile information, the computer system determining an association between the identified sentiment of the user and one or more sentiments included in a plurality of sentiments conveyed by a plurality of images;

the computer system determining that one or more images included in the plurality of images convey the one or more sentiments;

based on the association between the identified sentiment of the user and the one or more sentiments and the one or more images conveying the one or more sentiments, the computer system selecting the one or more images from the plurality of images;

the computer system displaying the selected one or more images,

wherein the computer system is included in a digital picture frame, wherein the step of displaying the selected one or more images includes displaying the one or more images on a display included in the digital picture frame, and wherein the step of identifying the sentiment of the user includes the steps of;

the computer system determining that the user is in a proximity to the digital picture frame;

in response to the step of determining that the user in the proximity to the digital picture frame, the computer system receiving lighting information from a light sensor coupled to the digital picture frame, wherein the lighting information includes a measurement of ambient lighting of a room in which the digital picture frame is located; and

in response to the step of determining that the user is in the proximity to the digital picture frame, and based on the measurement of the ambient lighting of the room in which the digital picture frame is located, the computer system determining an emotional state of the user, the emotional state being included in the sentiment of the user, wherein the step of selecting the one or more images is based in part on the measurement of the ambient lighting of the room in which the digital picture frame is located;

the computer system determining that one or more additional users are in the proximity to the digital picture frame;

in response to the one or more additional users being in the proximity to the digital picture frame, the computer system determining one or more emotional states of the one or more additional users, respectively;

based on the emotional state of the user and the one or more emotional states of the one or more additional users, the computer determining one emotional state of a majority of users in a group consisting of the user and the one or more additional users;

the computer system determining an association between the one emotional state of the majority of the users and one sentiment included in the plurality of sentiments conveyed by the plurality of images;

the computer system determining that an image included in the plurality of images conveys the one sentiment;

based on the association between the one emotional state of the majority of users and the one sentiment and based on the image conveying the one sentiment, the computer system selecting the image from the plurality of images; and

the computer system displaying the image on the display included in the digital picture frame.

14. The computer system of claim 13, wherein the method further comprises the step of the computer system collecting schedules of events of interest to the user from a calendar service or a calendar application, wherein the step of identifying the sentiment of the user is based on the schedules of events of interest to the user collected from the calendar service or the calendar application.

15. The computer system of claim 13, wherein the method further comprises the step of the computer system automatically generating metadata tags for the plurality of images, the metadata tags specifying respective contents of corresponding images included in the plurality of images, wherein the step of determining that the one or more images convey the one or more sentiments includes:

identifying contents of the one or more images based on one or more metadata tags generated for the one or more images; and

identifying the one or more sentiments based on the identified contents of the one or more images.

16. The computer system of claim 13, wherein the method further comprises the step of the computer system automatically generating metadata tags for the plurality of images, the metadata tags specifying respective sentiments of corresponding images included in the plurality of images, wherein the step of determining that the one or more images convey the one or more sentiments includes identifying the one or more sentiments based on one or more metadata tags generated for the one or more images.

17. The computer system of claim 13, wherein the method further comprises the steps of:

based on the lighting information received from the light sensor coupled to the digital picture frame, the computer system determining that the ambient lighting in the room is dim;

based on the ambient lighting in the room being dim, the computer system determining a mood of an environment in proximity to the digital picture frame is somber;

based on the mood of the environment in proximity to the digital picture frame being somber, the computer system determining an association between the mood of the environment in proximity to the digital picture frame and a sentiment included in the plurality of sentiments conveyed by the plurality of images;

the computer system determining that an image included in the plurality of images conveys the sentiment;

based on the association between the mood of the environment in proximity to the digital picture frame and the sentiment and based on the image conveying the sentiment, the computer system selecting the image from the plurality of images; and

the computer system displaying the selected image on the display included in the digital picture frame.