Methods, computer program, computer program product and indexing systems for indexing or updating index

• US 10,311,038 B2
• Filed: 01/13/2014
• Issued: 06/04/2019
• Est. Priority Date: 08/29/2013
• Status: Active Grant

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Telefonaktiebolaget LM Ericsson
• Inventors: Arngren, Tommy, Kornhammar, Tim
• Primary Examiner(s): Ortiz Ditren, Belix M
• Application Number:
  US14/915,408
• Publication Number:
  US 20160217171A1
• Time in Patent Office: 1,968 Days
• Field of Search: None
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings

Abstract

Methods, indexing systems, computer program, computer program product for indexing and searching content are disclosed. For example, in one embodiment a method performed by an indexing system (104) for indexing or updating an index (106) for content is disclosed. The method includes for each one of a plurality of keywords (204) in a data storage system (1206), wherein each keyword is associated with a modality (206) and at least a first time stamp (208): retrieving (1002) from the data storage system a keyword (204a) and its associated first time stamp (208a); defining a time range for the keyword; determining which other keywords (214a) in the data storage system are associated with a time stamp within the time range; and updating the index with said other keywords that were determined to be associated with a time stamp within the time range.

Claims

1. A method performed by an indexing system for updating an index based on a video, the method comprising:

automatically detecting a first feature within a first video frame of the video or within a first audio component of the video;

identifying a first keyword associated with the identified first feature;

defining a first time range for the first keyword based on a time stamp identifying the first video frame or the first audio component;

automatically detecting a second feature within a second video frame of the video or within a second audio component of the video;

identifying a second keyword associated with the identified second feature;

defining a second time range for the second keyword based on a time stamp identifying the second video frame or the second audio component;

determining that the second keyword overlaps with the first keyword, wherein the determining comprises determining that there is an overlap between the first time range and the second time range; and

updating the index, wherein updating the index comprises adding to the index information indicating that the second keyword overlaps with the first keyword.

2. The method of claim 1, further comprising:

updating the index with a rank value and modality of each one said determined other keywords with time stamps within the time range.

3. The method of claim 1 further comprising, before the step of retrieving the keyword from the data storage system, on at least a plurality of frames of a video:

identifying one or more features present in the frames, wherein each one of the features are of at least one modality;

identifying, for each identified feature, a keyword associated with the identified feature;

generating the at least one time stamp for each one of the identified keywords; and

storing the keywords and their respective time stamps in the data storage system.

4. The method of claim 1, wherein the time range is defined as starting a certain time before the time of the first time stamp and ending a certain time after the first time stamp.

5. The method of claim 1, wherein the keyword is further associated with a second time stamp and the time range starts a certain time before the time of the first time stamp and ends a certain time after the second time stamp.

6. An indexing system, comprising:

a computer system; and

a data storage system, said data storage system containing computer readable instructions executable by said computer system whereby said indexing system is operative to;

detect a first feature within a first video frame of a video or within a first audio component of the video;

identify a first keyword associated with the identified first feature;

define a first time range for the first keyword based on a time stamp identifying the first video frame or the first audio component;

detect a second feature within a second video frame of the video or within a second audio component of the video;

identify a second keyword associated with the identified second feature;

define a second time range for the second keyword based on a time stamp identifying the second video frame or the second audio component;

determine that the second keyword overlaps with the first keyword, wherein the determining comprises determining that there is an overlap between the first time range and the second time range; and

add to an index information indicating that the second keyword overlaps with the first keyword.

7. The indexing system of claim 6, wherein said apparatus is further operative to update the index with a rank value and modality of each one the other keywords with time stamps within the time range.

8. The indexing system of claim 6, wherein said indexing system is further operative to, before the step of retrieving the keyword from the data storage system, on at least a plurality of frames of a video:

identify one or more features present in the frames, wherein each one of the features are of at least one modality;

identify, for each identified feature, one or more respective keywords associated with the identified feature;

generate the at least one time stamp for each one of the identified keywords; and

store the keywords and their respective time stamps in the data storage system.

9. The indexing system of claim 6, wherein the time range is defined as starting a certain time before the time of the first time stamp and ending the certain time after the first time stamp.

10. The indexing system of claim 6, wherein the keyword is further associated with a second time stamp and the time range starts a certain time before the time of the first time stamp and ends a certain time after the second time stamp.

11. A computer program product comprising a non-transitory computer readable medium storing a computer program for indexing content, the computer program comprising computer readable instructions which when run on a computer system causes the computer system to:

detect a first feature within a first video frame of a video or within a first audio component of the video;

identify a first keyword associated with the identified first feature;

define a first time range for the first keyword based on a time stamp identifying the first video frame or the first audio component;

detect a second feature within a second video frame of the video or within a second audio component of the video;

identify a second keyword associated with the identified second feature;

define a second time range for the second keyword based on a time stamp identifying the second video frame or the second audio component;

determine that the second keyword overlaps with the first keyword, wherein the determining comprises determining that there is an overlap between the first time range and the second time range; and

add to an index information indicating that the second keyword overlaps with the first keyword.

12. The computer program product of claim 11, wherein said computer readable instructions are further configured, when run, to cause the computer system to update the index with a rank value and modality of each one the other keywords with time stamps within the time range.

13. The computer program product of claim 11, wherein said computer readable instructions are further configured, when run, to cause the computer system to, before the step of retrieving the keyword from the data storage system, on at least a plurality of frames of a video:

identify one or more features present in the frames, wherein each one of the features are of at least one modality;

identify, for each identified feature, one or more respective keywords associated with the identified feature;

generate the at least one time stamp for each one of the identified keywords;

determine, for each identified keyword associated with an identified feature, a respective relevance of the keyword, wherein the respective relevance is dependent on a weight associated with the respective modality of the identified feature; and

store the keywords and their respective relevances and time stamps in the data storage system.

14. The computer program product of claim 11, wherein the time range is defined as starting a certain time before the time of the first time stamp and ending the certain time after the first time stamp.

15. The computer program product of claim 11, wherein the keyword is further associated with a second time stamp and the time range starts a certain time before the time of the first time stamp and ends a certain time after the second time stamp.

16. An indexing system for indexing or updating an index for content, comprising:

a data storage system; and

a data processor coupled to the data storage system, the data processor being configured to;

detect a first feature within a first video frame of a video or within a first audio component of the video;

identify a first keyword associated with the identified first feature;

define a first time range for the first keyword based on a time stamp identifying the first video frame or the first audio component;

detect a second feature within a second video frame of the video or within a second audio component of the video;

identify a second keyword associated with the identified second feature;

define a second time range for the second keyword based on a time stamp identifying the second video frame or the second audio component;

determine that the second keyword overlaps with the first keyword, wherein the determining comprises determining that there is an overlap between the first time range and the second time range; and

add to an index information indicating that the second keyword overlaps with the first keyword.

17. A content indexing method performed by an indexing and search system, the method comprising:

automatically detecting a first feature within a first video frame of a video or within a first audio component of the video;

identifying a first keyword associated with the identified first feature;

automatically detecting a second feature within a second video frame of the video or within a second audio component of the video;

identifying a second keyword associated with the identified second feature;

storing in a data storage system the first and second keywords;

defining a time range for the first keyword;

after defining the time range for the first keyword, determining that the second keyword falls within the defined time range; and

in response to determining that the second keyword falls within the defined time range, updating a first set of keywords by adding the second keyword to the first set of keywords, wherein the first set of keywords is associated with the first keyword.

18. The method of claim 17, further comprising:

further in response to determining that the second keyword falls within the defined time range, updating a second set of keywords by adding the first keyword to the second set of keywords, wherein the second set of keywords is associated with the second keyword.

19. The method of claim 17, wherein the method further comprises:

receiving a query comprising a keyword; and

determining that the keyword included in the query matches the first keyword;

in response to determining that the keyword included in the query matches the first keyword, using the first keyword to obtain the first set of keywords;

generating first search result information, wherein the first search result information comprises the first keyword and one or more keywords from the first set of keywords; and

adding the generated first search result information to a search results page.

20. The method of claim 17, wherein

the method further comprises:

receiving a query comprising a keyword;

determining that the keyword included in the query matches the first keyword;

in response to determining that the keyword included in the query matches the first keyword, using the first keyword to obtain the second keyword;

generating first search result information using the first keyword and the obtained second keyword, wherein the first search result information comprises a first locator identifying a first image file associated with the first feature and a second locator identifying a second image file associated with the second feature; and

adding the generated first search result information to a search results page.

21. The method of claim 20, wherein the first image files contains an image of the first feature and the second image files contains an image of the second feature.

22. The method of claim 17, wherein defining a time range for the first keyword comprises:

determining a sequence of frames of the video in which a feature associated with the first keyword appears in each frame of the sequence of frames;

determining a first time stamp assigned to the first frame of the sequence of frames;

determining a second time stamp assigned to the last frame of the sequence of frames;

using the first time stamp and the second time stamp to define the time range.

23. The method of claim 22, wherein

the time range is defined by a calculated starting time value and a calculated ending time value, and

using the first time stamp and the second time stamp to define the time range comprises calculating the starting time value by summing the first time stamp and a first offset value and calculating the ending time value by summing the second and a second offset value.

24. The method of claim 23, wherein the first index value is a first frame number or a first time index and the second index value is a second frame number or a second time index.

25. The method of claim 17, furtherer comprising:

receiving a query comprising a keyword and an offset value;

determining that the keyword included in the query matches the first keyword;

defining the time range for the first keyword in response to determining that the keyword included in the query matches the first keyword;

determining a set of other keywords that are located within the defined time range; and

generating a search result using the first keyword and the determined set of other keywords, wherein

defining the time range for the first keyword comprises obtaining a first index value associated with the first keyword and using the first index value associated with the first keyword and the offset value to define the time range.

26. The method of claim 25, wherein using the first index value, FIV, and the offset value, OV, to define the time range comprises computing (FIV+OV), wherein FIV defines the beginning of the time range and (FIV+OV) defines the end of the time range.

27. The method of claim 25, wherein

defining the time range for the first keyword further comprises obtaining a second index value associated with the first keyword and using the first index value, the second index value, and the offset value to define the time range.

28. The method of claim 27, wherein using the first index value, FIV, the second index value, SIV, and the offset value, OV, to define the time range comprises:

computing (FIV−

OV); and

computing (SIV+OV), wherein

(FIV−

OV) defines the beginning of the time range, and

(SIV+OV) defines the end of the time range.

29. An indexing and search system adapted to perform the method of claim 17.