System and method for contexually interpreting image sequences

• US 9,959,470 B2
• Filed: 04/29/2016
• Issued: 05/01/2018
• Est. Priority Date: 06/28/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hariharan, Rajaraman, Ramanathan, Sri, Subbian, Karthik, Trevathan, Matthew B.
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US15/142,345
• Publication Number:
  US 20160247028A1
• Time in Patent Office: 732 Days
• Field of Search: None
• CPC Class Codes: G06F 12/00 Accessing, addressing or allocating within memory systems or architectures digital input from, or digital output to record carriers, e.g. to disk storage units, G06F3/06
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 8/43 Checking; Contextual analysis
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04H 20/65 Arrangements characterised by transmission systems for broadcast
• CPC Class Codes: H04H 60/48 for recognising items expressed in broadcast information
• CPC Class Codes: H04H 60/76 Arrangements characterised by transmission systems other than for broadcast, e.g. the Internet for broadcast H04H20/65
• CPC Class Codes: H04L 41/0618 based on the physical or logical position
• CPC Class Codes: H04L 9/10 with particular housing, physical features or manual controls
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 21/4758 for providing answers, e.g. voting
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A system and method for contextually interpreting image sequences are provided. The method comprises receiving video from one or more video sources, and generating one or more questions associated with one or more portions of the video based on at least one user-defined objective. The method further comprises sending the one or more portions of the video and the one or more questions to one or more assistants, receiving one or more answers to the one or more questions from the one or more assistants, and determining a contextual interpretation of the video based on the one or more answers and the video.

Claims

1. A system comprising:

a processor;

a computer readable hardware storage device; and

program instructions recorded on the computer readable hardware storage device for execution by the processor, the program instructions comprising;

program instructions to receive video from one or more video sources;

program instructions to determine that one or more Application Video Packets (AVPs) of the video require user assistance for analyzing based on user-defined criteria;

program instructions to determine one or more questions associated with the one or more AVPs of the video that require user assistance based on at least one of the user-defined criteria and predetermined criteria;

program instructions to append each of the one or more questions to respective ones of the one or more AVPs;

program instructions to send the one or more AVPs and the one or more questions to one or more processor stations;

program instructions to receive one or more answers to the one or more questions from the one or more processor stations;

program instructions to collate the one or more answers;

program instructions to determine a contextual interpretation of the video based on the one or more answers, the predetermined criteria, and the video;

program instructions to parse the video based on the user-defined criteria;

program instructions to determine whether the parsing of the video requires user assistance based on the user-defined criteria; and

program instructions to determine the one or more AVPs based on the user-defined criteria when the parsing of the video requires user assistance.

2. The system of claim 1, further comprising program instructions to change an order of the one or more AVPs before the sending of the one or more AVPs such that the one or more processor stations receiving the one or more AVPs do not receive consecutive AVPs in a same video sequence.

3. The system of claim 1, further comprising program instructions to send the one or more AVPs and the one or more questions to the one or more processor stations which are operated by one or more users specifically-trained to answer the one or more questions based on the one or more AVPs.

4. The system of claim 1, further comprising program instructions to identify one or more portions of the video for which the system requires assistance in the determining the contextual interpretation based on at least one user-defined objective for the contextual interpretation.

5. The system of claim 4, wherein the at least one user-defined objective for the contextual interpretation comprises an answer that quantifies one or more objects in the video.

6. A system comprising:

a processor;

a computer readable hardware storage device; and

program instructions recorded on the computer readable hardware storage device for execution by the processor, the program instructions comprising;

program instructions to receive video from one or more video sources;

program instructions to determine that one or more Application Video Packets (AVPs) of the video require user assistance for analyzing based on user-defined criteria;

program instructions to determine one or more questions associated with the one or more AVPs of the video that require user assistance based on at least one of the user-defined criteria and predetermined criteria;

program instructions to append each of the one or more questions to respective ones of the one or more AVPs;

program instructions to send the one or more AVPs and the one or more questions to one or more processor stations;

program instructions to receive one or more answers to the one or more questions from the one or more processor stations;

program instructions to collate the one or more answers;

program instructions to determine a contextual interpretation of the video based on the one or more answers, the predetermined criteria, and the video; and

program instructions to identify one or more portions of the video for which the system requires assistance in the determining the contextual interpretation based on at least one user-defined objective for the contextual interpretation,

wherein the at least one user-defined objective comprising identifying one or more items possessed by a person in the video based on one or more criterion defined by the user.

7. A system comprising:

a processor;

a computer readable hardware storage device; and

program instructions recorded on the computer readable hardware storage device for execution by the processor, the program instructions comprising;

program instructions to receive video from one or more video sources;

program instructions to determine that one or more Application Video Packets (AVPs) of the video require user assistance for analyzing based on user-defined criteria;

program instructions to determine one or more questions associated with the one or more AVPs of the video that require user assistance based on at least one of the user-defined criteria and predetermined criteria;

program instructions to append each of the one or more questions to respective ones of the one or more AVPs;

program instructions to send the one or more AVPs and the one or more questions to one or more processor stations;

program instructions to receive one or more answers to the one or more questions from the one or more processor stations;

program instructions to collate the one or more answers;

program instructions to determine a contextual interpretation of the video based on the one or more answers, the predetermined criteria, and the video; and

program instructions to identify one or more portions of the video for which the system requires assistance in the determining the contextual interpretation based on at least one user-defined objective for the contextual interpretation,

wherein the at least one user-defined objective comprises identifying a person having difficulty breathing in the video based on one or more criterion defined by the user.