System for optimal rapid serial visual presentation (RSVP) from user-specific neural brain signals

• US 8,699,767 B1
• Filed: 12/21/2010
• Issued: 04/15/2014
• Est. Priority Date: 10/06/2006
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Khosla, Deepak, Huber, David J., Bhattacharyya, Rajan
• Primary Examiner(s): DRENNAN, BARRY T
• Application Number:
  US12/975,352
• Time in Patent Office: 1,211 Days
• Field of Search: None
• US Class Current: 382/128
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures

Abstract

Described is a system for optimizing rapid serial visual presentation (RSVP). A similarity metric is computed for RSVP images, and the images are sequenced according to the similarity metrics. The sequenced images are presented to a user, and neural signals are received to detect a P300 signal. A neural score for each image is computed, and the system is optimized to model the neural scores. The images are resequenced according a predictive model to output a sequence prediction which does not cause a false P300 signal. Additionally, the present invention describes computing a set of motion surprise maps from image chips. The image chips are labeled as static or moving and prepared into RSVP datasets. Neural signals are recorded in response to the RSVP datasets, and an EEG score is computed from the neural signals. Each image chip is then classified as containing or not containing an item of interest.

Claims

1. A system for optimizing rapid serial visual presentation, the system comprising:

one or more processors and a memory having instructions such that when the instructions are executed, the one or more processors perform operations of;

extracting a set of image features from a pair of images in a rapid serial visual presentation (RSVP) image set;

computing a predicted similarity metric for the pair of images using the set of image features to detect at least one similarity in the pair of images, wherein predicted similarity metrics are computed for all pairs of images in the RSVP image set;

sequencing the images in the RSVP image set according to the predicted similarity metrics, resulting in a sequenced set of images;

presenting the sequenced set of images and receiving neural brain signals during visualization of the sequenced set of images to detect a P300 signal;

computing a neural score for each image in the sequenced set of images based on the existence and strength of the P300 signal, wherein the neural score represents a dissimilarity between at least two images in the RSVP image set;

optimizing the system through a predictive model which models the neural scores computed for the sequenced set of images;

resequencing the images in the RSVP image set according to the predictive model and presenting the resequenced images; and

outputting an image sequence prediction which minimizes a false P300 signal.

2. A system for optimizing rapid serial visual presentation as set forth in claim 1, wherein the one or more processors perform operations of:

generating a sequence of non-target images for presentation; and

receiving the neural brain signals during visualization of the sequence of non-target images to detect a P300 signal;

wherein the sequence of non-target images is used to eliminate a false P300 signal from the neural brain signals.

3. The system for optimizing rapid serial visual presentation as set forth in claim 2, wherein the one or more processors perform an operation of determining the predicted similarity metric between images based on a weighted combination of a set of distance metrics.

4. The system for optimizing rapid serial visual presentation as set forth in claim 3, wherein the act of optimizing the system, the one or more processors perform an operation of:

adjusting a weighting vector of the predictive model, such that a difference metric between the predicted similarity metric and the neural score is minimized for each consecutive image pairing in the RSVP image set.

5. A computer-implemented method for optimizing rapid serial visual presentation, comprising an act of:

causing a data processor to perform operations of;

extracting a set of image features from a pair of images in a rapid serial visual presentation (RSVP) image set;

computing a predicted similarity metric for the pair of images using the set of image features to detect at least one similarity in the pair of images, wherein predicted similarity metrics are computed for all pairs of images in the RSVP image set;

sequencing the images in the RSVP image set according to the predicted similarity metrics, resulting in a sequenced set of images;

presenting the sequenced set of images and receiving neural brain signals during visualization of the sequenced set of images to detect a P300 signal;

computing a neural score for each image in the sequenced set of images based on the existence and strength of the P300 signal, wherein the neural score represents a dissimilarity between at least two images in the RSVP image set;

optimizing the system through a predictive model which models the neural scores computed for the sequenced set of images;

resequencing the images in the RSVP image set according to the predictive model and presenting the resequenced images; and

outputting an image sequence prediction which minimizes a false P300 signal.

6. The method for optimizing rapid serial visual presentation as set forth in claim 5, further comprising acts of:

generating a sequence of non-target images for presentation; and

receiving the neural brain signals during visualization of the sequence of non-target images to detect a P300 signal;

wherein the sequence of non-target images is used to eliminate a false P300 signal from the neural brain signals.

7. The method for optimizing rapid serial visual presentation as set forth in claim 6, further comprising an act of determining the predicted similarity metric between images based on a weighted combination of a set of distance metrics.

8. The method for optimizing rapid serial visual presentation as set forth in claim 7, further comprising an act of:

adjusting a weighting vector of the predictive model, such that a difference metric between the predicted similarity metric and the neural score is minimized for each consecutive image pairing in the RSVP image set.

9. A computer program product for optimizing rapid serial visual presentation, the computer program product comprising:

computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of;

extracting a set of image features from a pair of images in a rapid serial visual presentation (RSVP) image set;

computing a predicted similarity metric for the pair of images using the set of image features to detect at least one similarity in the pair of images, wherein predicted similarity metrics are computed for all pairs of images in the RSVP image set;

sequencing the images in the RSVP image set according to the predicted similarity metrics, resulting in a sequenced set of images;

presenting the sequenced set of images and receiving neural brain signals during visualization of the sequenced set of images to detect a P300 signal;

computing a neural score for each image in the sequenced set of images based on the existence and strength of the P300 signal, wherein the neural score represents a dissimilarity between at least two images in the RSVP image set;

optimizing the system through a predictive model which models the neural scores computed for the sequenced set of images;

resequencing the images in the RSVP image set according to the predictive model and presenting the resequenced images; and

outputting an image sequence prediction which minimizes a false P300 signal.

10. The computer program product for optimizing rapid serial visual presentation as set forth in claim 9, further comprising instruction means for causing the processor to perform operations of:

generating a sequence of non-target images for presentation; and

receiving the neural brain signals during visualization of the sequence of non-target images to detect a P300 signal;

wherein the sequence of non-target images is used to eliminate a false P300 signal from the neural brain signals.

11. The computer program product for optimizing rapid serial visual presentation as set forth in claim 10, further comprising instruction means for causing the processor to perform an operation of determining the predicted similarity metric between images based on a weighted combination of a set of distance metrics.

12. The computer program product for optimizing rapid serial visual presentation as set forth in claim 11, further comprising instruction means for causing the processor to perform an operation of:

adjusting a weighting vector of the predictive model, such that a difference metric between the predicted similarity metric and the neural score is minimized for each consecutive image pairing in the RSVP image set.