System and method for contexually interpreting image sequences

• US 9,355,318 B2
• Filed: 10/23/2014
• Issued: 05/31/2016
• Est. Priority Date: 06/28/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hariharan, Rajaraman, Ramanathan, Sri, Subbian, Karthik, Trevathan, Matthew B.
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US14/522,078
• Publication Number:
  US 20150043778A1
• Time in Patent Office: 586 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 12/00 Accessing, addressing or allocating within memory systems or architectures digital input from, or digital output to record carriers, e.g. to disk storage units, G06F3/06
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 8/43 Checking; Contextual analysis
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04H 20/65 Arrangements characterised by transmission systems for broadcast
• CPC Class Codes: H04H 60/48 for recognising items expressed in broadcast information
• CPC Class Codes: H04H 60/76 Arrangements characterised by transmission systems other than for broadcast, e.g. the Internet for broadcast H04H20/65
• CPC Class Codes: H04L 41/0618 based on the physical or logical position
• CPC Class Codes: H04L 9/10 with particular housing, physical features or manual controls
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 21/4758 for providing answers, e.g. voting
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A system and method for contextually interpreting image sequences are provided. The method comprises receiving video from one or more video sources, and generating one or more questions associated with one or more portions of the video based on at least one user-defined objective. The method further comprises sending the one or more portions of the video and the one or more questions to one or more assistants, receiving one or more answers to the one or more questions from the one or more assistants, and determining a contextual interpretation of the video based on the one or more answers and the video.

Claims

1. A method, comprising:

receiving, by a computer, video from one or more video sources;

generating, by the computer, one or more questions associated with one or more portions of the video;

sending, by the computer, the one or more portions of the video and the one or more questions to one or more assistants;

receiving, by the computer, one or more answers to the one or more questions from the one or more assistants; and

determining, by the computer, a contextual interpretation of the video based on the one or more answers and the video,

wherein;

the video comprises recorded image sequences recorded by a surveillance system;

the one or more assistants are one or more persons; and

the determining the contextual interpretation by the computer uses interpretations of both the one or more persons and the computer;

further comprising identifying, by the computer, the one or more portions of the video for which the determining the contextual interpretation requires assistance based on at least one user-defined objective for the contextual interpretation;

wherein the at least one user-defined objective comprises identifying, based on one or more criterion defined by the user, one of;

(i) one or more items possessed by a person in the video, and (ii) a person in the video having difficulty breathing.

2. The method of claim 1, wherein the at least one user-defined objective for the contextual interpretation comprises quantifying one or more objects in the video.

3. The method of claim 1, further comprising changing an order of the one or more portions of the video before the sending of the one or more portions of the video such that the one or more assistants receiving the one or more portions of the video do not receive consecutive portions of the video in a same video sequence.

4. The method of claim 1, further comprising sending the one or more portions of the video and the one or more questions to the one or more assistants which are specifically trained to answer the one or more questions based on the one or more portions of the video.

5. The method of claim 1, wherein a service provider at least one of creates, maintains, deploys and supports a computer infrastructure operable to perform the steps of claim 1.

6. The method of claim 1, wherein the steps of claim 1 are provided by a service provider on a subscription, advertising, and/or fee basis.

7. A system comprising:

a processor;

a computer readable hardware storage device; and

program instructions recorded on the computer readable hardware storage device for execution by the processor, the program instructions comprising;

program instructions to receive video from one or more video sources;

program instructions to determine whether one or more Application Video Packets (AVPs) of the video require user assistance for analyzing based on user-defined criteria;

program instructions to determine one or more questions associated with the one or more AVPs of the video that require user assistance based on at least one of the user-defined criteria and predetermined criteria;

program instructions to send the one or more AVPs and the one or more questions to one or more processor stations;

program instructions to collate one or more answers to the one or more questions from the one or more processor stations;

program instructions to determine a contextual interpretation of the video based on the one or more answers, the predetermined criteria, and the video; and

program instructions to identify one or more portions of the video for which the system requires assistance in the determining the contextual interpretation based on at least one user-defined objective for the contextual interpretation;

wherein the at least one user-defined objective comprises identifying, based on one or more criterion defined by the user, one of;

(i) one or more items possessed by a person in the video; and

(ii) a person having difficulty breathing in the video.

8. The system of claim 7, further comprising program instructions to parse the video based on the user-defined criteria.

9. The system of claim 8, further comprising:

program instructions to determine whether the parsing of the video requires user assistance based on the user-defined criteria; and

program instructions to determine the one or more AVPs based on the user-defined criteria when the parsing of the video requires user assistance.

10. The system of claim 8, further comprising program instructions to change an order of the one or more AVPs before the sending of the one or more AVPs such that the one or more processor stations receiving the one or more AVPs do not receive consecutive AVPs in a same video sequence.

11. The system of claim 7, further comprising program instructions to send the one or more AVPs and the one or more questions to the one or more processor stations which are operated by one or more users specifically-trained to answer the one or more questions based on the one or more AVPs.

12. The system of claim 7, wherein the at least one user-defined objective for the contextual interpretation comprises an answer that quantifies one or more objects in the video.

13. A computer program product comprising a computer readable hardware storage device having program instructions recorded on the computer readable hardware storage device, the program instructions comprising:

program instructions to receive video from one or more video sources;

program instructions to generate one or more questions associated with one or more Application Video Packets (AVPs) of the video based on at least one user-defined objective;

program instructions to send the one or more AVPs of the video and the one or more questions to one or more assistants;

program instructions to receive one or more answers to the one or more questions from the one or more assistants;

program instructions to determine a contextual interpretation of the video based on the one or more answers and the video; and

program instructions to identify one or more portions of the video for which the determining the contextual interpretation requires assistance based on at least one user-defined objective for the contextual interpretation.

14. The computer program product of claim 13, wherein the at least one user-defined objective comprises identifying, based on one or more criterion defined by the user, one of:

(i) one or more items possessed by a person in the video; and

(ii) a person in the video.