System for optimal rapid serial visual presentation (RSVP) from user-specific neural brain signals

• US 9,269,027 B1
• Filed: 02/20/2014
• Issued: 02/23/2016
• Est. Priority Date: 10/06/2006
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Khosla, Deepak, Huber, David J., Bhattacharyya, Rajan
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US14/185,908
• Time in Patent Office: 733 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures

Abstract

Described is a system for optimizing rapid serial visual presentation (RSVP). A similarity metric is computed for RSVP images, and the images are sequenced according to the similarity metrics. The sequenced images are presented to a user, and neural signals are received to detect a P300 signal. A neural score for each image is computed, and the system is optimized to model the neural scores. The images are resequenced according a predictive model to output a sequence prediction which does not cause a false P300 signal. Additionally, the present invention describes computing a set of motion surprise maps from image chips. The image chips are labeled as static or moving and prepared into RSVP datasets. Neural signals are recorded in response to the RSVP datasets, and an EEG score is computed from the neural signals. Each image chip is then classified as containing or not containing an item of interest.

Claims

1. A system for optimizing rapid serial visual presentation, the system comprising:

one or more processors and a memory having instructions such that when the instructions are executed, the one or more processors perform operations of;

dividing an input image into a plurality of image chips;

computing a set of motion channel feature maps from the image chips;

computing a set of motion channel surprise maps from the set of motion channel feature maps;

labeling the image chips as static or moving;

preparing rapid serial visual presentation (RSVP) datasets of the image chips;

presenting RSVP datasets for visualization and recording of neural brain signals;

computing an electroencephalography (EEG) score from the neural brain signals based on a response to the image chips in the RSVP datasets; and

classifying each image chip as containing an item of interest or not containing an item of interest based on the EEG score.

2. The system for optimizing rapid serial visual presentation as set forth in claim 1, wherein the one or more processors perform operations of:

creating at least one video clip by taking an image chip labeled as moving from a set of multiple consecutive image frames; and

collecting all video clips into a RSVP video dataset.

3. The system for optimizing rapid serial visual presentation as set forth in claim 2, wherein the one or more processors perform operations of:

recording a neural brain signal upon an onset of presentation of a static image or a video clip;

creating an EEG data segment from the neural brain signal, wherein the EEG data segment is a neural signature of a desired visual response for the presentation of the static image or the video clip; and

classifying the data segment as a data segment containing an item of interest or a data segment not containing an item of interest.

4. The system for optimizing rapid serial visual presentation as set forth in claim 3, wherein the one or more processors perform operations of:

receiving a set of known images of targets and a set of known images of non-targets for training;

learning of a classifier in a set of data segments which correspond to the set of known images of targets and the set of known images of non-targets;

wherein two distinct classifiers are learned for a static type RSVP dataset and a video type RSVP dataset; and

employing the two distinct classifiers to determine an EEG score for each type of RSVP dataset presentation.

5. A computer-implemented method for optimizing rapid serial visual presentation comprising an act of:

causing a data processor to perform operations of;

dividing an input image into a plurality of image chips;

computing a set of motion channel feature maps from the image chips;

computing a set of motion channel surprise maps from the set of motion channel feature maps;

labeling the image chips as static or moving;

preparing rapid serial visual presentation (RSVP) datasets of the image chips;

presenting RSVP datasets for visualization and recording of neural brain signals;

computing an electroencephalography (EEG) score from the neural brain signals based on a response to the image chips in the RSVP datasets; and

classifying each image chip as containing an item of interest or not containing an item of interest based on the EEG score.

6. The method for optimizing rapid serial visual presentation as set forth in claim 5, further comprising acts of:

creating at least one video clip by taking an image chip labeled as moving from a set of multiple consecutive image frames; and

collecting all video clips into a RSVP video dataset.

7. The method for optimizing rapid serial visual presentation as set forth in claim 6, further comprising acts of:

recording a neural brain signal upon an onset of presentation of a static image or a video clip;

creating an EEG data segment from the neural brain signal, wherein the EEG data segment is a neural signature of a desired visual response for the presentation of the static image or the video clip; and

classifying the data segment as a data segment containing an item of interest or a data segment not containing an item of interest.

8. The method for optimizing rapid serial visual presentation as set forth in claim 7, further comprising acts of:

receiving a set of known images of targets and a set of known images of non-targets for training;

learning of a classifier in a set of data segments which correspond to the set of known images of targets and the set of known images of non-targets;

wherein two distinct classifiers are learned for a static type RSVP dataset and a video type RSVP dataset; and

employing the two distinct classifiers to determine an EEG score for each type of RSVP dataset presentation.

9. A computer program product for optimizing rapid serial visual presentation, the computer program product comprising:

computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of;

dividing an input image into a plurality of image chips;

computing a set of motion channel feature maps from the image chips;

computing a set of motion channel surprise maps from the set of motion channel feature maps;

labeling the image chips as static or moving;

preparing rapid serial visual presentation (RSVP) datasets of the image chips;

presenting prepared RSVP datasets for visualization and recording of neural brain signals;

computing an electroencephalography (EEG) score from the neural brain signals based on a response to the image chips in the RSVP datasets; and

classifying each image chip as containing an item of interest or not containing an item of interest based on the EEG score.

10. The computer program product for optimizing rapid serial visual presentation as set forth in claim 9, further comprising instructions means for causing the processor to perform operations of:

creating at least one video clip by taking an image chip labeled as moving from a set of multiple consecutive image frames; and

collecting all video clips into a RSVP video dataset.

11. The computer program product for optimizing rapid serial visual presentation as set forth in claim 10, further comprising instructions means for causing the processor to perform operations of:

recording a neural brain signal upon an onset of presentation of a static image or a video clip;

creating an EEG data segment from the neural brain signal, wherein the EEG data segment is a neural signature of a desired visual response for the presentation of the static image or the video clip; and

classifying the data segment as a data segment containing an item of interest or a data segment not containing an item of interest.

12. The computer program product for optimizing rapid serial visual presentation as set forth in claim 11, further comprising instructions means for causing the processor to perform operations of:

receiving a set of known images of targets and a set of known images of non-targets for training;

learning of a classifier in a set of data segments which correspond to the set of known images of targets and the set of known images of non-targets;

wherein two distinct classifiers are learned for a static type RSVP dataset and a video type RSVP dataset; and

employing the two distinct classifiers to determine an EEG score for each type of RSVP dataset presentation.