×

SYSTEMS AND METHODS FOR ATTENTION-BASED CONFIGURABLE CONVOLUTIONAL NEURAL NETWORKS (ABC-CNN) FOR VISUAL QUESTION ANSWERING

  • US 20170124432A1
  • Filed: 06/16/2016
  • Published: 05/04/2017
  • Est. Priority Date: 11/03/2015
  • Status: Active Grant
First Claim
Patent Images

1. A computer-implemented method of improving accuracy in generating an answer to a question input related to an image input, the method comprising:

  • receiving an image input;

    receiving a question input related to the image input;

    inputting the question input and the image input into an Attention-Based Configurable Convolutional Neural Networks (ABC-CNN) framework to generate an answer, the ABC-CNN framework comprising;

    an image feature map extraction component comprising a CNN that extracts an image feature map from the image input;

    a semantic question embedding component that obtains question embeddings from the question input;

    a question-guided attention map generation component that receives the image feature map and the question embeddings and that obtains a question-guided attention map focusing on a region or regions asked by question input; and

    an answer generation component that obtains an attention weighted image feature map by weighting image feature map using the question-guided attention map and generates answers based on a fusion of the image feature map, the question embeddings, and the attention weighted image feature map.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×