Attention estimation to control the delivery of data and audio/video content

• US 9,729,920 B2
• Filed: 03/15/2013
• Issued: 08/08/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Inventors: Baum, Kevin L., Mohapatra, Shivajit, Needham, Michael L., Li, Renxiang, Ishtiaq, Faisal
• Primary Examiner(s): Bruckart, Benjamin R
• Assistant Examiner(s): Huang, Jen-Shi
• Application Number:
  US13/840,342
• Publication Number:
  US 20140282642A1
• Time in Patent Office: 1,607 Days
• Field of Search: 725 9, 725 10, 725 14, 725 21, 725 40, 725 42, 725 43, 725 46, 725 48
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/262 Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/4112 having fewer capabilities than the client, e.g. thin client having less processing power or no tuning capabilities
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/44213 Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

A method implemented in a computer system for controlling the delivery of data and audio/video content. The method delivers primary content to the subscriber device for viewing by a subscriber. The method also delivers secondary content to the companion device for viewing by the subscriber in parallel with the subscriber viewing the primary content, where the secondary content relates to the primary content. The method extracts attention estimation features from the primary content, and monitors the companion device to determine an interaction measurement for the subscriber viewing the secondary content on the companion device. The method calculates an attention measurement for the subscriber viewing the primary content based on the attention estimation features, and the interaction measurement, and controls the delivery of the secondary content to the companion device based on the attention measurement.

Claims

1. A computing device for controlling the delivery of data and audio/video content, comprising:

a communication interface that connects the computing device to a network that connects to a subscriber device, and a companion device;

a memory device resident in the computing device; and

a processor disposed in communication with the communication interface and the memory device, the processor configured to;

deliver primary content to the subscriber device for viewing by a subscriber;

deliver secondary content to the companion device for viewing by the subscriber in parallel with the subscriber viewing the primary content, the secondary content relating to the primary content;

extract at least one attention estimation feature from the primary content;

monitor the companion device to determine an interaction measurement for the subscriber viewing the secondary content on the companion device;

calculate an attention measurement for the subscriber viewing the primary content based on said at least one attention estimation feature, and the interaction measurement; and

control the delivery of the secondary content to the companion device based on the attention measurement.

2. The computing device of claim 1, wherein each of said at least one attention estimation feature is at least one of a visual feature in the primary content, an audio feature in the primary content, and a textual feature in the primary content.

3. The computing device of claim 1, wherein to extract said at least one attention estimation feature from the primary content, the processor is further configured to:

retrieve a personal profile and viewing preferences for the subscriber;

compute a weight for each of said at least one attention estimation feature based on the personal profile and viewing preferences for the subscriber,

wherein the calculation of the attention measurement for the subscriber includes the weight for each said at least one attention estimation feature.

4. The computing device of claim 1, wherein to extract said at least one attention estimation feature from the primary content, the processor is further configured to:

retrieve a past viewing history for the subscriber;

compute a weight for each of said at least one attention estimation feature based on the past viewing history for the subscriber,

wherein the calculation of the attention measurement for the subscriber includes the weight for each of said at least one attention estimation feature.

5. The computing device of claim 1, wherein to monitor the companion device, the processor is further configured to:

receive a measurement from the companion device,

wherein the measurement is at least one of an ability of the subscriber to interact with the companion device, and an operation of the companion device by the subscriber.

6. The computing device of claim 1, wherein to calculate the attention measurement, the processor is further configured to:

calculate a weight for each of said at least one attention estimation feature; and

adjust the weight for each of said at least one attention estimation feature based on the interaction measurement.

7. The computing device of claim 1, wherein to calculate the attention measurement, the processor is further configured to:

calculate a weight for each of said at least one attention estimation feature; and

calculate a weight for the interaction measurement,

wherein the attention measurement is a fusion of the weighted said at least one attention estimation feature, and the weighted interaction measurement.

8. The computing device of claim 1, wherein the attention measurement is based on said at least one attention estimation feature, and the interaction measurement as a function of time.

9. The computing device of claim 1, wherein to control the delivery of the secondary content, the processor is further configured to:

adjust presentation or pacing of delivery of the secondary content to the companion device based on the attention measurement.

10. The computing device of claim 1, wherein the processor is further configured to:

control the delivery of the primary content to the subscriber device based on the attention measurement.

11. A method implemented in a computer system for controlling the delivery of data and audio/video content, comprising:

delivering primary content to a subscriber device for viewing by a subscriber;

delivering secondary content to a companion device for viewing by the subscriber in parallel with the subscriber viewing the primary content, the secondary content relating to the primary content;

extracting at least one attention estimation feature from the primary content;

monitoring the companion device to determine an interaction measurement for the subscriber viewing the secondary content on the companion device;

calculating an attention measurement for the subscriber viewing the primary content based on said at least one attention estimation feature, and the interaction measurement; and

controlling the delivery of the secondary content to the companion device based on the attention measurement.

12. The method of claim 11, wherein each of said at least one attention estimation feature is at least one of a visual feature in the primary content, an audio feature in the primary content, and a textual feature in the primary content.

13. The method of claim 11, wherein the extracting of said at least one attention estimation feature from the primary content further comprises:

retrieving a personal profile and viewing preferences for the subscriber;

computing a weight for each of said at least one attention estimation feature based on the personal profile and viewing preferences for the subscriber,

wherein the calculation of the attention measurement for the subscriber includes the weight for each of said at least one attention estimation feature.

14. The method of claim 11, wherein the extracting of said at least one attention estimation feature from the primary content further comprises:

retrieving a past viewing history for the subscriber;

computing a weight for each of said at least one attention estimation feature based on the past viewing history for the subscriber,

wherein the calculation of the attention measurement for the subscriber includes the weight for each of said at least one attention estimation feature.

15. The method of claim 11, wherein the monitoring of the companion device further comprises:

receiving a measurement from the companion device,

wherein the measurement is at least one of an ability of the subscriber to interact with the companion device, and an operation of the companion device by the subscriber.

16. The method of claim 11, wherein the calculating of the attention measurement further comprises:

calculating a weight for each of said at least one attention estimation feature; and

adjusting the weight for each of said at least one attention estimation feature based on the interaction measurement.

17. The method of claim 11, wherein the calculating of the attention measurement further comprises:

calculating a weight for each of said at least one attention estimation feature; and

calculating a weight for the interaction measurement,

wherein the attention measurement is a fusion of the weighted said at least one attention estimation feature, and the weighted interaction measurement.

18. The method of claim 11, wherein the attention measurement is based on said at least one attention estimation feature, and the interaction measurement as a function of time.

19. The method of claim 11, wherein the controlling of the delivery of the secondary content further comprises:

adjusting presentation or pacing of delivery of the secondary content to the companion device based on the attention measurement.

20. The method of claim 11, further comprising:

controlling the delivery of the primary content to the subscriber device based on the attention measurement.

21. A non-transitory computer-readable medium, comprising computer-executable instructions that, when executed on a computing device, perform steps of:

delivering primary content to a subscriber device for viewing by a subscriber;

delivering secondary content to a companion device for viewing by the subscriber in parallel with the subscriber viewing the primary content, the secondary content relating to the primary content;

extracting at least one attention estimation feature from the primary content;

monitoring the companion device to determine an interaction measurement for the subscriber viewing the secondary content on the companion device;

calculating an attention measurement for the subscriber viewing the primary content based on said at least one attention estimation feature, and the interaction measurement; and

controlling the delivery of the secondary content to the companion device based on the attention measurement.