System and method for user-behavior based content recommendations

• US 10,664,500 B2
• Filed: 12/29/2015
• Issued: 05/26/2020
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Chang, E-Cheng, Huang, Yuning
• Primary Examiner(s): Gurmu, Muluemebet
• Application Number:
  US14/982,842
• Publication Number:
  US 20170185669A1
• Time in Patent Office: 1,610 Days
• Field of Search: 707740
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/287 Visualization; Browsing
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/438 Presentation of query results
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/90324 using system suggestions

Abstract

A system and method of predicting a user'"'"'s most meaningful multimedia content includes enabling a sensing device on a user device in response to a user requesting a multimedia operation, performing the multimedia operation for a multimedia content, in response to the multimedia operation, identifying behavioral and interaction cues of the user with the sensing device substantially when the multimedia operation is being performed, updating a recommendation from a set of multimedia content including the multimedia content represented by the behavioral and interaction cues identified, and presenting the updated recommendation to the user.

Claims

1. A method comprising:

enabling a sensing device on a user device in response to a user requesting a multimedia operation, the multimedia operation comprising displaying a multimedia content on a display of the user device, the multimedia content being a photograph or a video, the multimedia content being stored in a multimedia library;

performing the multimedia operation;

identifying behavioral cues of the user according to signals received from the sensing device substantially when the multimedia operation is being performed, wherein identifying the behavioral cues of the user comprises;

searching for multimedia sub-cues in the signals captured from the sensing device; and

classifying the multimedia sub-cues as positive reactions or negative reactions to produce the behavioral cues;

identifying interaction cues of the user according to whether the user interacts with an input device of the user device substantially when the multimedia operation is being performed, the interaction cues indicating whether, during the displaying of the multimedia content on the display of the user device, the user viewed the multimedia content for a predetermined amount of time, shared the multimedia content, edited the multimedia content, zoomed into the multimedia content, and rotated the user device to match an original orientation of the multimedia content;

selecting a subset of the multimedia library according to the behavioral cues and further according to the interaction cues;

displaying the subset of the multimedia library on the display of the user device;

requesting feedback from the user in response to presenting the subset of the multimedia library;

classifying the feedback from the user as a positive reaction or a negative reaction to produce feedback cues; and

selecting a further subset of the multimedia library in accordance with the feedback cues.

2. The method of claim 1, wherein searching for multimedia sub-cues comprises producing the multimedia sub-cues in response to a view time for the multimedia content.

3. The method of claim 1, wherein searching for multimedia sub-cues comprises producing the multimedia sub-cues in response to the user sharing the multimedia content.

4. The method of claim 1, wherein searching for multimedia sub-cues comprises producing the multimedia sub-cues in response to the user editing the multimedia content.

5. The method of claim 1, wherein searching for multimedia sub-cues comprises producing the multimedia sub-cues in response to the user zooming into the multimedia content.

6. The method of claim 1, wherein searching for multimedia sub-cues comprises producing the multimedia sub-cues in response to the user rotating the user device in order to match an orientation of the user device with an orientation of the multimedia content.

7. The method of claim 1, wherein identifying the behavioral cues of the user comprises:

capturing facial expressions of the user with the sensing device; and

recognizing the facial expressions of the user to produce the behavioral cues.

8. The method of claim 7, wherein recognizing the facial expressions comprises classifying the facial expressions of the user as a positive facial reaction, a negative facial reaction, or a lack of facial reaction.

9. The method of claim 7, wherein recognizing the facial expressions comprises recognizing a gaze of the user as the user stares at a hotspot on a display of the user device.

10. The method of claim 1, wherein identifying the behavioral cues of the user comprises:

capturing ambient activity with the sensing device; and

analyzing the ambient activity to produce the behavioral cues.

11. The method of claim 10, wherein capturing the ambient activity comprises capturing sound with the sensing device while performing the multimedia operation.

12. The method of claim 10, wherein capturing the ambient activity comprises:

capturing video with the sensing device;

performing facial recognition to detect subjects in the video; and

correlating subjects in the video with entries in a contact list on the user device.

13. A method comprising:

providing a plurality of multimedia content on a user device;

displaying a first subset of the multimedia content on a display of the user device;

capturing behavioral cues of a user with a sensing device when displaying the first subset of the multimedia content, wherein capturing the behavioral cues of the user comprises;

capturing signals from the sensing device;

searching for multimedia sub-cues in the signals captured from the sensing device; and

classifying the multimedia sub-cues as positive reactions or negative reactions to produce the behavioral cues;

identifying interaction cues of a user with an input device when displaying the first subset of the multimedia content, the interaction cues indicating the user manipulated the first subset of the multimedia content using the input device during the displaying, wherein the interaction cues indicate the user manipulated the first subset of the multimedia content when the user shared the multimedia content, edited the multimedia content, zoomed into the multimedia content, and rotated the user device to match an original orientation of the multimedia content;

analyzing the behavioral cues and the interaction cues to produce multimedia cues;

assigning weights to each of the multimedia cues;

generating a score for each of the plurality of multimedia content in accordance with the weights and the multimedia cues;

displaying a second subset of the multimedia content on the display in accordance with the scores;

requesting feedback from the user in response to displaying the second subset of the multimedia content;

classifying the feedback from the user as a positive reaction or a negative reaction to produce feedback cues; and

displaying a third subset of the multimedia content on the display in accordance with the feedback cues.

14. The method of claim 13, further comprising:

adjusting the weights for each of the multimedia cues in accordance with the behavioral cues and further according to the interaction cues of the user; and

normalizing the weights.

15. The method of claim 13, wherein generating the scores for each of the plurality of multimedia content comprises:

computing and summing raw scores for each of the multimedia cues in accordance with the weights;

computing and applying a correlation factor to the raw scores; and

applying a learning factor to the raw scores to produce the scores.

16. The method of claim 15, wherein producing the scores comprises computing the score according to:

17. A device comprising:

a sensing device;

an input device different from the sensing device;

a display;

a processor; and

a computer-readable storage medium storing a program to be executed by the processor, the program including instructions to perform a method comprising;

enabling the sensing device in response to a user requesting a multimedia operation, the multimedia operation comprising displaying a multimedia content on the display, the multimedia content being a photograph or a video, the multimedia content being stored in a multimedia library;

identifying behavioral cues of the user according to signals captured from the sensing device while performing the multimedia operation, wherein identifying the behavioral cues of the user comprises;

searching for multimedia sub-cues in the signals captured from the sensing device; and

classifying the multimedia sub-cues as positive reactions or negative reactions to produce the behavioral cues;

identifying interaction cues of the user according to whether the user interacts with the input device while performing the multimedia operation, the interaction cues indicating whether, during the displaying of the multimedia content on the display, the user viewed the multimedia content for a predetermined amount of time, shared the multimedia content, edited the multimedia content, zoomed into the multimedia content, and rotated the display to match an original orientation of the multimedia content;

associating the behavioral cues and the interaction cues with the multimedia operation;

ranking multimedia content with a model according to the behavioral cues and further according to the interaction cues;

selecting a subset of the multimedia content in accordance with the ranks of the multimedia content;

displaying the subset of the multimedia content on the display;

requesting feedback from the user in response to displaying the subset of the multimedia content;

classifying the feedback from the user as a positive reaction or a negative reaction to produce feedback cues; and

selecting a further subset of the multimedia content in accordance with the feedback cues.

18. The device of claim 17, wherein the instructions for identifying the behavioral cues comprise instructions for capturing and analyzing ambient sounds to produce the behavioral cues.

19. The device of claim 17, wherein the instructions for identifying the behavioral cues comprise instructions for capturing and recognizing facial expressions of the user to produce the behavioral cues.

20. The device of claim 17, wherein the instructions for identifying the interaction cues comprise instructions for searching for photo sub-cues in the multimedia content and classifying the photo sub-cues as positive reactions or negative reactions to produce the interaction cues.

21. The device of claim 17, wherein the program further includes instructions for:

ranking the multimedia content with the model further in accordance with the feedback cues.

22. A device comprising:

a sensing device;

an input device;

a memory storing a plurality of multimedia content, the multimedia content being a photograph or a video;

a display;

a processor; and

a computer-readable storage medium storing a program to be executed by the processor, the program including instructions for;

receiving a request from a user to perform a multimedia operation, the multimedia operation comprising displaying a first subset of the multimedia content on the display;

performing the multimedia operation while receiving signals from the sensing device and the input device, and while determining whether the user views the multimedia content for a predetermined amount of time, shares the multimedia content, edits the multimedia content, zooms into the multimedia content, and rotates the display to match an original orientation of the multimedia content during the displaying of the first subset of the multimedia content on the display;

updating a recommendation of a second subset of the multimedia content in accordance with the request, the signals from the sensing device, and whether the user manipulated the first subset of the multimedia content, wherein updating the recommendation comprises;

searching for multimedia sub-cues in the signals received from the sensing device; and

classifying the multimedia sub-cues as positive reactions or negative reactions to produce interaction cues, the recommendation updated according to the interaction cues;

displaying the updated recommendation on the display;

requesting feedback from the user in response to displaying the updated recommendation;

classifying the feedback from the user as a positive reaction or a negative reaction to produce feedback cues; and

further updating the recommendation of the second subset of the multimedia content in accordance with the feedback cues.

23. The device of claim 22, wherein the instruction for retrieving data with the sensing device comprises instructions for sensing behavior of the user with the sensing device and receiving device input behavior of the user with the input device.

24. The device of claim 22, wherein the sensing device is one of a microphone or a front camera, and wherein the input device is a touchscreen.