Classification, search, and retrieval of complex video events

• US 9,244,924 B2
• Filed: 01/09/2013
• Issued: 01/26/2016
• Est. Priority Date: 04/23/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Cheng, Hui, Sawhney, Harpreet Singh, Divakaran, Ajay, Yu, Qian, Liu, Jingen, Tamrakar, Amir, Ali, Saad, Javed, Omar
• Primary Examiner(s): Stevens, R.
• Application Number:
  US13/737,607
• Publication Number:
  US 20130282747A1
• Time in Patent Office: 1,112 Days
• Field of Search: 707/758
• US Class Current: 1/1
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/787 using geographical or spatial information, e.g. location

Abstract

A complex video event classification, search and retrieval system can generate a semantic representation of a video or of segments within the video, based on one or more complex events that are depicted in the video, without the need for manual tagging. The system can use the semantic representations to, among other things, provide enhanced video search and retrieval capabilities.

Claims

1. A video search assistant embodied in one or more machine readable storage media and accessible by a computing system to assist a user with a video search by:

receiving a user-specified search request;

determining a higher level complex event of interest, based on the user-specified search request;

accessing a video event model, the video event model comprising;

(i) a plurality of semantic elements associated with a plurality of higher level complex events depicted in the plurality of videos, each higher level complex event evidenced by at least two different lower level complex events, each of the semantic elements describing one or more of a scene, an action, an actor, and an object depicted in one or more of the videos, and (ii) data indicative of evidentiary relationships of different combinations of semantic elements forming the lower level complex events, wherein the video event model is derived by;

(i) executing one or more event classifiers on one or more dynamic low level features of the videos to identify one or more semantic elements associated with the dynamic low level features, (ii) computing a strength of association of the one or more semantic elements with ones of the lower level complex events, (iii) and computing a strength of association of the associated ones of the lower level complex events with the higher level complex event;

determining, based on the video event model, one or more semantic elements of interest associated with the higher level complex event of interest; and

formulating a search for one or more videos depicting the higher level complex event of interest, the search comprising one or more of the semantic elements of interest.

2. The video search assistant of claim 1, wherein the user-specified search request comprises one or more of a textual element, a non-textual visual element, and an audio element.

3. The video search assistant of claim 1, wherein the user-specified search request comprises one or more spatio-temporal regions of interest within a video segment.

4. The video search assistant of claim 1, accessible by a computing system to present one or more of the semantic elements of interest to a user, receive user input relating to the presented semantic elements, and modify the search based on the user input.

5. The video search assistant of claim 1, accessible by a computing system to identify a video likely depicting the higher level complex event of interest and present a natural language description of the video to a user.

6. The video search assistant of claim 1, accessible by a computing system to identify a video likely depicting the higher level complex event of interest and present to a user a natural language description of one or more segments of the video that relate to the higher level complex event.

7. The video search assistant of claim 1, accessible by a computing system to identify a video likely depicting the higher level complex event of interest and locate, in the identified video, one or more video segments likely depicting the higher level complex event of interest.

8. The video search assistant of claim 1, accessible by a computing system to determine the higher level complex event of interest based on data stored in a knowledge base comprising one or more of a non-publicly accessible knowledge base and a publicly accessible knowledge base.

9. The video search assistant of claim 1, wherein the video event model comprises data relating each of a plurality of complex event types to an associated set of semantic elements.

10. The video search assistant of claim 9, wherein the video event model comprises data relating to a relative evidentiary value associated with one or more of the semantic elements in each of the sets of semantic elements associated with each of the complex event types, and the video search assistant is accessible by a computing system to recognize the higher level complex event based on one or more of the relative evidentiary values.

11. The video search assistant of claim 1, wherein at least a portion of the video event model is derived using a machine learning technique.

12. The video search assistant of claim 1, wherein at least a portion of the video event model is generated based on user input.

13. The video search assistant of claim 1, wherein the video event model comprises ontological information relating to the lower level and the higher level complex events and the semantic elements.

14. The video search assistant of claim 1, accessible by a computing system to determine the higher level complex event of interest by algorithmically comparing the user-specified search request to one or more of the semantic elements in the video event model.

15. The video search assistant of claim 1, wherein the user-specified search request comprises one or more videos, and the video search assistant determines a higher level complex event of interest likely depicted in the one or more videos without the use of example training videos.

16. A video search assistant embodied in one or more machine readable storage media and accessible by a computing system to generate a description of a video to assist with a video search, by:

accessing a set of semantic elements associated with the video, each of the semantic elements describing one or more of a scene, an action, an actor, and an object depicted in the video, wherein at least a portion of the set is, in combination, indicative of at least one lower level complex event;

recognizing a higher level complex event as being likely depicted in the video, as evidenced by a combination of the at least one lower level complex event;

algorithmically generating a human-intelligible representation of the higher level complex event based on the semantic elements evidencing the higher level complex event, the human-intelligible representation comprising one or more of;

a natural language description, one or more non-textual visual elements representative of the natural language description, and one or more audio elements representative of the natural language description; and

associating the human-intelligible representation with the video.

17. The video search assistant of claim 16, accessible by a computing system to omit from the human-intelligible representation semantic elements that do not evidence the higher level complex event.

18. The video search assistant of claim 16, accessible by a computing system to present the human-intelligible representation in response to a user-specified search request.

19. The video search assistant of claim 16, accessible by a computing system to receive user input relating to the human-intelligible representation and modify the video search based on the user input.

20. The video search assistant of claim 16, accessible by a computing system to determine a relative evidentiary value associated with each of the semantic elements evidencing the higher level complex event and formulate the human-intelligible representation based on the relative evidentiary values.

21. The video search assistant of claim 16, accessible by a computing system to determine a temporal, spatial, causal, or other relationship between at least two of the semantic elements evidencing the higher level complex event and include an indication of the relationship in the human-intelligible representation.

22. The video search assistant of claim 16, wherein one or more of the semantic elements evidencing the higher level complex event relates to audio included in the video, and the video search assistant is accessible by a computing system to include a description of the audio-related semantic element in the human-intelligible representation.

23. The video search assistant of claim 16, accessible by a computing system to associate an interactive hyperlink with a semantic element included in the human-intelligible representation and display an indication of the hyperlink in the human-intelligible representation, wherein the hyperlink may be user-activated to locate a portion of the video corresponding to the associated semantic element.

24. The video search assistant of claim 16, accessible by a computing system to associate an interactive hyperlink with a semantic element included in the human-intelligible representation and display an indication of the hyperlink in the human-intelligible representation, wherein the hyperlink may be user-activated to expand or contract the human-intelligible representation to a different level of detail.

25. The video search assistant of claim 16, wherein the human-intelligible representation describes the higher level complex event and the semantic elements evidencing the higher level complex event in an essay-like manner.

26. The video search assistant of claim 16, wherein the human-intelligible representation describes the complex event and the semantic elements evidencing the higher level complex event in a dialog-like manner.

27. The video search assistant of claim 16, accessible by a computing system to recognize the higher level complex event by referencing a video event model comprising data relating each of a plurality of complex event types to an associated set of semantic elements.

28. The video search assistant of claim 27, accessible by a computing system to assign a relative evidentiary value to one or more of the semantic elements in each of the sets of semantic elements associated with each of the complex event types, and recognize the higher level complex event based on one or more of the relative evidentiary values.

29. The video search assistant of claim 16, accessible by a computing system to determine the set of semantic elements associated with the video by algorithmically extracting a plurality of features from the video and classifying each of the extracted features using a set of classifiers.

30. A video search assistant embodied in one or more machine readable storage media and accessible by a computing system to recognize a video as likely depicting a complex event, by, algorithmically:

identifying a plurality of visual and non-visual features included in the video;

deriving a plurality of semantic elements from the visual and non-visual features, each of the semantic elements describing one or more of a scene, an action, an actor, and an object depicted in the video;

deriving a plurality of lower level complex events comprising combinations of the semantic elements;

recognizing a higher level complex event as being likely depicted in the video based on a combination of the lower level complex events; and

associating the higher level complex event with the video.

31. The video search assistant of claim 30, accessible by a computing system to identify the visual and non-visual features by one or more algorithmic feature extraction techniques.

32. The video search assistant of claim 30, accessible by a computing system to recognize a geographic location of one or more of the scenes depicted in the video and associate the geographic location with one or more of the semantic elements.

33. The video search assistant of claim 30, accessible by a computing system to recognize audio occurring in the video and associate the audio with one or more of the semantic elements.

34. The video search assistant of claim 33, accessible by a computing system to recognize speech included in the recognized audio and associate the speech with one or more of the semantic elements.

35. The video search assistant of claim 30, accessible by a computing system to recognize text depicted in the video and associate the text with one or more of the semantic elements.

36. The video search assistant of claim 30, accessible by a computing system to derive one or more of the semantic elements algorithmically using a set of classifiers.

37. The video search assistant of claim 36, accessible by a computing system to derive one or more of the classifiers using a machine learning technique.

38. The video search assistant of claim 30, accessible by a computing system to generate one or more of the semantic elements based on user input.

39. The video search assistant of claim 30, accessible by a computing system to recognize the higher level complex event by referencing a video event model comprising data relating each of a plurality of complex event types to an associated set of semantic elements.

40. The video search assistant of claim 39, accessible by a computing system to assign a relative evidentiary value to one or more of the semantic elements in each of the sets of semantic elements associated with each of the complex event types, and recognize each complex event based on one or more of the relative evidentiary values.

41. The video search assistant of claim 39, accessible by a computing system to create at least a portion of the video event model using a machine learning technique.

42. The video search assistant of claim 39, wherein the video event model comprises ontological information relating to the higher level complex events and the semantic elements.

43. The video search assistant of claim 30, accessible by a computing system to recognize another video as likely depicting the higher level complex event without the use of example training videos.

44. The video search assistant of claim 43, accessible by a computing system to recognize the other video as likely depicting the higher level complex event by comparing a semantic description of the other video with the plurality of semantic elements.

45. A system for creating a highlight reel by algorithmically recognizing and extracting complex events from a video, the system comprising a plurality of instructions embodied in one or more non-transitory machine readable storage media, the instructions executable by a processor to cause an electronic device to:

detect a plurality of visual features in the video;

identify a plurality of semantic elements associated with the detected visual features;

select a subset of the semantic elements, the subset comprising semantic elements evidencing at least two different actions associated with a plurality of events;

select a subset of the associated plurality of events indicating the complex event;

extract, from the video, a plurality of images corresponding to the actions associated with the complex event; and

by a display of the electronic device, present the extracted images as a highlight reel of the complex event.