Method and system for processing multimedia content to dynamically generate text transcript

• US 10,056,083 B2
• Filed: 10/18/2016
• Issued: 08/21/2018
• Est. Priority Date: 10/18/2016
• Status: Active Grant

• Current Assignee: Yen4Ken Inc. (VideoKen, Inc.)
• Original Assignee: Yen4Ken Inc. (VideoKen, Inc.)
• Inventors: Negi, Sumit, Patil, Sonal S, Biswas, Arijit, Gandhi, Ankit, Deshmukh, Om D
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US15/296,140
• Publication Number:
  US 20180108354A1
• Time in Patent Office: 672 Days
• Field of Search: 704 9, 704205, 704235
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting

Abstract

The disclosed embodiments illustrate method and system of processing multimedia content to generate a text transcript. The method includes segmenting each of a set of text frames to determine spatial regions. The method further includes extracting one or more keywords from each of the determined spatial regions. The method further includes determining the first set of keywords from the extracted one or more keywords based on filtering of one or more off-topic keywords from the extracted one or more keywords. The method further includes extracting a second set of keywords based on the determined first set of keywords. The method further includes generating a graph between each of a first set of keywords and one or more of a second set of keywords. The method further includes dynamically generating the text transcript of audio content in the multimedia content based on the generated graph.

Claims

1. A method for dynamically generating a text transcript, comprising:

segmenting, by an application server comprising a processor, each identified set of text frames to determine one or more spatial regions, wherein the one or more spatial regions comprise at least one or more portions of text content;

extracting, by an application server comprising a processor, one or more keywords from the one or more spatial regions, wherein the one or more keywords are extracted from one or more available portions of the text content in the one or more spatial regions;

determining, by an application server comprising a processor, a first set of keywords from the one or more extracted keywords by filtering one or more off-topic keywords from the one or more extracted keywords;

extracting, by an application server comprising a processor, a second set of keywords similar to or related with the one or more second set of keywords, wherein the second set of keywords being retrieved from one or more knowledge databases;

generating, by an application server comprising a processor, a graph from a semantic relationship between the first set of keywords and the second set of keywords, wherein the graph comprises one or more first nodes and one or more second nodes with each node in the one or more first nodes corresponding with a keyword in the first set of keywords and each node in the one or more second nodes corresponding with a keyword in the second set of keywords; and

generating, by an application server comprising a processor, the text transcript of the audio content in the multimedia content using the generated graph, wherein the generating of the text transcript comprises utilizing at least one of an updated language module and an updated dictionary module to generate the text transcript of the audio content in the multimedia content.

2. The method of claim 1, further comprising:

determining one or more frames of the multimedia content; and

identifying the set of text frames from the one or more frames, wherein

the set of text frames are identified from one or more available portions of the text content within the one or more frames.

3. The method of claim 1, further comprising:

identifying the one or more off-topic keywords, wherein

the identifying the one or more off-topic keywords comprises clustering a distributional representation of the one or more keywords in the identified set of text frames, and

the one or more off-topic keywords comprising one or more repeated keywords in the set of text frames, numerals or keywords with one or more special characters, or both.

4. The method of claim 1, wherein the graph further comprises one or more edges, wherein

the one or more edges corresponding to a semantic relationship between a first node in the one or more first nodes and a second node in the one or more second nodes.

5. The method of claim 4, wherein each of the one or more first nodes that correspond to a keyword in the generated graph is associated with a predefined weight corresponding to a probability of occurrence of the keyword, and

each of the one or more second nodes that correspond to a keyword in the generated graph is associated with a predefined weight corresponding to a probability of occurrence of the keyword.

6. The method of claim 5, further comprising:

updating the predefined weight of each of the one or more first nodes and the predefined weight of each of the one or more second nodes in the generated graph.

7. The method of claim 6, wherein the updating of the predefined weight comprises determining a weight for each node of the one or more first nodes and the one or more second nodes using

minΣ

(i,j)∈

E|a_i−

a_j|

where E corresponds to an extended keyword graph, a_i corresponds to a weight by which a unigram probability estimate for keyword I is incremented, and a_j is defined for keywords in set M=L∪

D, L and D correspond to nodes and edges.

8. The method of claim 7, wherein the determining of the weight is constrained by one or more of the following constraints

9. The method of claim 6, further comprising:

updating at least one of a language unit, a dictionary unit, or both, based on the updated predefined weight of each of the one or more first nodes and each of the one or more second nodes in the generated graph.

10. The method of claim 9, wherein the updating of the language unit corresponds to an update of a probability of occurrence of a keyword being spoken within the video content.

11. The method of claim 10, wherein the updating of the dictionary unit corresponds to an update of the one or more keywords in the dictionary unit.