Systems and methods for automatically enabling subtitles based on detecting an accent

• US 9,854,324 B1
• Filed: 01/30/2017
• Issued: 12/26/2017
• Est. Priority Date: 01/30/2017
• Status: Active Grant

• Current Assignee: ROVI Product Corporation (Xperi Inc.)
• Original Assignee: Rovi Guides, Inc. (Adeia Inc.)
• Inventors: Panchaksharaiah, Vishwas Sharadanagar, Gupta, Vikram Makam
• Primary Examiner(s): Saltarelli, Dominic D
• Application Number:
  US15/419,284
• Time in Patent Office: 330 Days
• Field of Search: 725 34
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: H04N 21/25883 being end-user demographical data, e.g. age, family status or address arrangements for identifying locations of users in broadcast systems H04H60/52
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44226 on social networks
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages

Abstract

Systems and methods are described for automatically enabling subtitles based on a user profile when a language is spoken with an accent a user has difficulty understanding. For example, a media guidance application may detect a first plurality of user interactions of the user while the given language is being spoken with the accent. Based on the first plurality of interactions, the media guidance application may calculate a first value associated with a user specific level of difficulty indicating how difficult it is for the user to understand the language when spoken with the accent. If the first plurality of user interactions are not being performed again, the media guidance application may update the user specific difficulty with a second value that is lower than the first value. The media guidance application may automatically generate for display subtitles for a media asset based on the user specific level of difficulty.

Claims

1. A method for automatically enabling subtitles based on a user profile when a language is spoken with an accent, the method comprising:

storing, in a user profile associated with a user, a first data structure indicating a list of one or more languages that the user understands;

determining, at a first point in time, that a language of the one or more languages in the list is being spoken with an accent by retrieving the first data structure, extracting the list, and comparing the language to the one or more languages;

detecting a first plurality of user interactions of the user while the given language is being spoken with the accent;

storing, in the user profile, a data log indicating the first point in time and the first plurality of user interactions;

retrieving, from a remote source, an information table associating user interactions with values, wherein the values represent a general level of difficulty, the general level of difficulty being indicative of a measure of difficulty a plurality of users have in understanding accents in audio content;

comparing the first plurality of user interactions with the information table to determine a first plurality of values, wherein each value of the first plurality of values is associated with a respective one of the first plurality of user interactions;

calculating a first value based on the first plurality of values;

creating a second data structure, wherein the second data structure associates the first value with a user specific level of difficulty, the user specific level of difficulty being indicative of a measure of difficulty the user encounters in understanding the given language when spoken with the accent;

storing the second data structure in the user profile;

detecting that the given language is being spoken with the accent at a second point in time later than the first point in time;

based on detecting that the given language is being spoken with the accent at the second point in time, retrieving, from the user profile, the data log;

monitoring user interactions of the user while the given language is being spoken with the accent at the second point in time to determine whether the first plurality of user interactions are being performed again while the given language is being spoken with the accent;

based on determining that the first plurality of user interactions are not being performed again, updating the second data structure, the second data structure associating a second value that is lower than the first value with the user specific level of difficulty;

detecting that a media asset includes the given language spoken with the accent;

retrieving, from the user profile, the second data structure;

extracting, from the second data structure, the user specific level of difficulty; and

automatically generating for display subtitles for the media asset based on the extracted user specific level of difficulty.

2. The method of claim 1, wherein the respective one of the first plurality of user interactions is at least one of a facial expression, rewinding a previous media asset, pausing the previous media asset, enabling subtitles, increasing the volume, a head movement, a user gesture, a vocal utterance, a user setting input, a user geographic input, a user demographic information input, and a social media post.

3. The method of claim 1 further comprising calculating the second value, wherein calculating the second value comprises:

extracting, from the data log, the first point in time;

comparing the first point in time to the second point in time to determine an elapsed time between the first point in time and the second point in time;

retrieving, from the remote source, a decay function, the decay function associating the elapsed time to the user specific level of difficulty; and

inputting the elapsed time and the first value into the decay function to determine the second value.

4. The method of claim 1 further comprising calculating the second value, wherein calculating the second value comprises:

monitoring user usage of user equipment to determine a plurality of media assets that the user has watched, wherein the plurality of media assets each contain audio with the given language spoken with the accent;

storing a usage log indicating the plurality of media assets and when the user watched each of the plurality of media assets;

determining the number of media assets in the usage log that the user watched between the first point in time and the second point in time;

retrieving, from the remote source, a decay function, the decay function relating the number of media assets to the user specific level of difficulty; and

inputting the number of media assets and the first value into the decay function to determine the second value.

5. The method of claim 4, wherein the user is a first user, wherein the data log is a first data log, wherein the usage log is a first usage log, wherein the plurality of media assets is a first plurality of media assets, wherein the number of media assets is a first number of media assets, and wherein the decay function is determined by:

receiving, at the remote source, a second data log and a third data log from user equipment associated with a second user, wherein the second data log indicates a third point in time and a second plurality of user interactions, and wherein the third data log indicates a fourth point in time and a third plurality of interactions

comparing the second plurality of user interactions with the information table to determine a second plurality of values;

comparing the third plurality of user interactions with the information table to determine a third plurality of values;

calculating a third value based on the second plurality of values and a fourth value based on the third plurality of values;

receiving, at the remote source, a second usage log indicating a second plurality of media assets, wherein the second plurality of media assets each contain audio with the given language spoken with the accent;

determining a second number of media assets in the second usage log the second user watched between the third point in time and the fourth point in time; and

based on the calculated third value, the calculated fourth value, and the second number of media assets, determining the decay function.

6. The method of claim 1 further comprising:

based on monitoring user interactions of the user while the given language is being spoken with the accent at the second point in time, determining a second plurality of user interactions of the user while the given language is being spoken with the accent; and

calculating the second value, wherein calculating the second value comprises;

comparing the second plurality of user interactions with the information table to determine a second plurality of values; and

calculating the second value based on the second plurality of values; and

wherein determining that the first plurality of user interactions are not being performed again comprises comparing the second plurality of user interaction to the first plurality of user interactions.

7. The method of claim 1, wherein the information table is a first information table, and wherein calculating the first value further comprises:

determining the geographic location of the user;

retrieving, from a remote source, a second information table associating the geographic location and the accent with an augmenting value, wherein the augmenting value is calculated by;

receiving, from a plurality of users located in the geographic location, a plurality of data logs indicating a second plurality of user interactions of the plurality of users detected while the given language was being spoken with the accent on user equipment associated with the plurality of users;

comparing the second plurality of user interactions with the information table to determine a second plurality of values; and

calculating the augmenting value based on the second plurality of values; and

recalculating the first value by combining the augmenting value and the first value.

8. The method of claim 1, wherein detecting that a media asset includes the given language spoken with the accent includes determining an accent level, the accent level indicating how slight the accent is, the method further comprising:

comparing the accent level to the user specific level of difficulty to determine if the user specific level of difficulty is greater than the accent level; and

wherein automatically generating for display subtitles for the media asset is further based on determining that the user specific level of difficulty is greater than the accent level.

9. The method of claim 8, wherein detecting that a media asset includes the given language spoken with the accent comprises:

receiving, from a media source, the media asset and a third data structure indicating a first portion of the media asset during which a specific actor appears;

receiving, from the remote source, a fourth data structure associating the specific actor with the accent and with the accent level; and

wherein automatically generating for display subtitles for the media asset comprises automatically generating for display subtitles for the media asset during the first portion of the media asset.

10. The method of claim 1, wherein the user is a first user, and wherein the user specific level of difficulty is a first user specific level of difficulty, the method further comprising:

detecting that the first user is watching the media asset;

detecting that a second user is watching the media asset;

retrieving a third data structure from a user profile of the second user based on detecting that the second user is watching the media asset; and

extracting, from the third data structure, a second user specific level of difficulty for the second user;

wherein retrieving the second data structure is based on detecting that the first user is watching the media asset, and wherein automatically generating for display the subtitles for the media asset comprises;

determining a threshold level of difficulty for generating for display the subtitles;

determining that at least one of the first user specific level of difficulty and the second user specific level of difficulty exceeds the threshold level of difficulty; and

automatically generating for display subtitles for the media asset based on determining that at least one of the first user specific level of difficulty and the second user specific level of difficulty exceeds the threshold level of difficulty.

11. A system for automatically enabling subtitles based on a user profile when a language is spoken with an accent, the system comprising:

control circuitry configured to;

store, in a user profile associated with a user, a first data structure indicating a list of one or more languages that the user understands;

determine, at a first point in time, that a language of the one or more languages in the list is being spoken with an accent by retrieving the first data structure, extracting the list, and comparing the language to the one or more languages;

detect a first plurality of user interactions of the user while the given language is being spoken with the accent;

store, in the user profile, a data log indicating the first point in time and the first plurality of user interactions;

retrieve, from a remote source, an information table associating user interactions with values, wherein the values represent a general level of difficulty, the general level of difficulty being indicative of a measure of difficulty a plurality of users have in understanding accents in audio content;

compare the first plurality of user interactions with the information table to determine a first plurality of values, wherein each value of the first plurality of values is associated with a respective one of the first plurality of user interactions;

calculate a first value based on the first plurality of values;

create a second data structure, wherein the second data structure associates the first value with a user specific level of difficulty, the user specific level of difficulty being indicative of a measure of difficulty the user encounters in understanding the given language when spoken with the accent;

store the second data structure in the user profile;

detect that the given language is being spoken with the accent at a second point in time later than the first point in time;

retrieve, based on detecting that the given language is being spoken with the accent at the second point in time, from the user profile, the data log;

monitor user interactions of the user while the given language is being spoken with the accent at the second point in time to determine whether the first plurality of user interactions are being performed again while the given language is being spoken with the accent;

update, based on determining that the first plurality of user interactions are not being performed again, the second data structure, the second data structure associating a second value that is lower than the first value with the user specific level of difficulty;

detect that a media asset includes the given language spoken with the accent;

retrieve, from the user profile, the second data structure;

extract, from the second data structure, the user specific level of difficulty; and

automatically generate for display subtitles for the media asset based on the extracted user specific level of difficulty.

12. The system of claim 11, wherein the respective one of the first plurality of user interactions is at least one of a facial expression, rewinding a previous media asset, pausing the previous media asset, enabling subtitles, increasing the volume, a head movement, a user gesture, a vocal utterance, a user setting input, a user geographic input, a user demographic information input, and a social media post.

13. The system of claim 11, wherein the control circuitry is further configured to calculate the second value, and wherein the control circuitry is configured to calculate the second value by:

extracting, from the data log, the first point in time;

comparing the first point in time to the second point in time to determine an elapsed time between the first point in time and the second point in time;

retrieving, from the remote source, a decay function, the decay function associating the elapsed time to the user specific level of difficulty; and

inputting the elapsed time into the decay function to determine the second value.

14. The system of claim 11, wherein the control circuitry is further configured to calculate the second value, and wherein the control circuitry is configured to calculate the second value by:

monitoring user usage of user equipment to determine a plurality of media assets that the user has watched, wherein the plurality of media assets each contain audio with the given language spoken with the accent;

storing a usage log indicating the plurality of media assets;

determining the number of media assets in the usage log;

retrieving, from the remote source, a decay function, the decay function relating the number of media assets to the user specific level of difficulty; and

inputting the number of media assets into the decay function to determine the second value.

15. The system of claim 14, wherein the user is a first user, wherein the data log is a first data log, wherein the usage log is a first usage log, wherein the plurality of media assets is a first plurality of media assets, wherein the number of media assets is a first number of media assets, and wherein the decay function is determined by:

receiving, at the remote source, a second data log and a third data log from user equipment associated with a second user, wherein the second data log indicates a third point in time and a second plurality of user interactions, and wherein the third data log indicates a fourth point in time and a third plurality of interactions comparing the second plurality of user interactions with the information table to determine a second plurality of values;

comparing the third plurality of user interactions with the information table to determine a third plurality of values;

calculating a third value based on the second plurality of values and a fourth value based on the third plurality of values;

receiving, at the remote source, a second usage log indicating a second plurality of media assets, wherein the second plurality of media assets each contain audio with the given language spoken with the accent;

determining a second number of media assets in the second usage log the second user watched between the third point in time and the fourth point in time; and

based on the calculated third value, the calculated fourth value, and the second number of media assets, determining the decay function.

16. The system of claim 11, wherein the control circuitry is further configured to:

determine, based on monitoring user interactions of the user while the given language is being spoken with the accent at the second point in time, a second plurality of user interactions of the user while the given language is being spoken with the accent; and

calculate the second value, wherein the control circuitry is configured to calculate the second value by;

comparing the second plurality of user interactions with the information table to determine a second plurality of values; and

calculating the second value based on the second plurality of values; and

wherein the control circuitry is configured to determine that the first plurality of user interactions are not being performed again by comparing the second plurality of user interaction to the first plurality of user interactions.

17. The system of claim 11, wherein the information table is a first information table, and wherein the control circuitry is configured to calculate the first value by:

determining the geographic location of the user;

retrieving, from a remote source, a second information table associating the geographic location and the accent with an augmenting value, wherein the augmenting value is calculated by;

receiving, from a plurality of users located in the geographic location, a plurality of data logs indicating a second plurality of user interactions of the plurality of users detected while the given language was being spoken with the accent on user equipment associated with the plurality of users;

comparing the second plurality of user interactions with the information table to determine a second plurality of values; and

calculating the augmenting value based on the second plurality of values; and

recalculating the first value by combining the augmenting value and the first value.

18. The system of claim 11, wherein the control circuitry is configured to detect that a media asset includes the given language spoken with the accent by determining an accent level, the accent level indicating how slight the accent is, and wherein the control circuitry is further configured to:

compare the accent level to the user specific level of difficulty to determine if the user specific level of difficulty is greater than the accent level; and

wherein the control circuitry is further configured to automatically generate for display subtitles for the media asset based on determining that the user specific level of difficulty is greater than the accent level.

19. The system of claim 18, wherein the control circuitry is configured to detect that a media asset includes the given language spoken with the accent by:

receiving, from a media source, the media asset and a third data structure indicating a first portion of the media asset during which a specific actor appears;

receiving, from the remote source, a fourth data structure associating the specific actor with the accent and with the accent level; and

wherein the control circuitry is configured to automatically generate for display subtitles for the media asset by automatically generating for display subtitles for the media asset during the first portion of the media asset.

20. The system of claim 11, wherein the user is a first user, wherein the user specific level of difficulty is a first user specific level of difficulty, and wherein the control circuitry is further configured to:

detect that the first user is watching the media asset;

detect that a second user is watching the media asset;

retrieve a third data structure from a user profile of the second user based on detecting that the second user is watching the media asset; and

extract, from the third data structure, a second user specific level of difficulty for the second user;

wherein the control circuitry is configured to retrieve the second data structure based on detecting that the first user is watching the media asset, and wherein the control circuitry is configured to automatically generate for display the subtitles for the media asset by;

determining a threshold level of difficulty for generating for display the subtitles;

determining that at least one of the first user specific level of difficulty and the second user specific level of difficulty exceeds the threshold level of difficulty; and

automatically generating for display subtitles for the media asset based on determining that at least one of the first user specific level of difficulty and the second user specific level of difficulty exceeds the threshold level of difficulty.