Determination of train presence and motion state in railway environments

• US 9,495,599 B2
• Filed: 05/14/2015
• Issued: 11/15/2016
• Est. Priority Date: 08/21/2012
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Bobbitt, Russell P., Feris, Rogerio S., Zhai, Yun
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Schwartz, Raphael
• Application Number:
  US14/711,871
• Publication Number:
  US 20150242692A1
• Time in Patent Office: 551 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Foreground feature data and motion feature data is determined for frames of video data acquired from a train track area region of interest. The frames are labeled as “train present” if the determined foreground feature data value meets a threshold value, else as “train absent; and as “motion present” if the motion feature data meets a motion threshold, else as “static.” The labels are used to classify segments of the video data comprising groups of consecutive video frames, namely as within a “no train present” segment for groups with “train absent” and “static” labels; within a “train present and in transition” segment for groups “train present” and “motion present” labels; and within a “train present and stopped” segment for groups with “train present” and “static” labels. The presence or motion state of a train at a time of inquiry is thereby determined from the respective segment classification.

Claims

1. A computer-implemented method for applying computer vision techniques to automatically detect and classify the presence or absence of a train within a railway track area, the method comprising executing on a processor the steps of:

classifying a segment of input video image data comprising a group of consecutive video frames that each comprise “

train absent”

label and “

static”

labels as a “

no train present”

segment, wherein the input video image data is acquired from a region of interest defined around a train track area within an image scene of a stream of the video image data that comprises a portion of a railway track area;

classifying a segment of the input video data comprising a group of consecutive video frames that each comprise the “

train present”

label and a “

motion present”

label as a “

train present and in transition”

segment;

classifying a segment of the input video data comprising a group of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as a “

train present and stopped”

segment;

determining a presence or a motion state of a train within the region of interest of the video scene at a time of inquiry as the “

no train present”

, the “

train present and in transition”

or the “

train present and stopped”

classification of the segment comprising the video frame of the time of inquiry;

determining a distribution of the “

train present” and

“

train absent”

labels of the plurality of the video segments within an auto-correction time period;

in response to the determined distribution of the “

train present” and

“

train absent”

labels not meeting an expected distribution for the auto-correction time period, exchanging the “

train present” and

the “

train absent”

labels that are determined for each of the video frames; and

repeating, as a function of the exchanged “

train present” and

“

train absent”

labels, the steps of classifying the segments of the input video data of consecutive video frames that each comprise the “

train absent”

label and the “

static”

label as the “

no train present”

segment groups, classifying the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

motion present”

label as the “

train present and in transition”

segments, and classifying the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as the “

train present and stopped”

segment groups.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processor, a computer readable memory and a computer readable hardware storage device, wherein the computer readable program code is embodied on the computer readable hardware storage device and comprises instructions for execution by the processor via the computer readable memory that cause the processor to perform the steps of classifying the segment of the input video image data video frames that each comprise the “

train absent”

label and the “

static”

labels as the “

no train present”

segment, classifying the segment of the input video image data video frames that each comprise the “

train present”

label and the “

motion present”

label as the “

train present and in transition”

segment, classifying the segment of the input video image data video frames that each comprise the “

train present”

label and the “

static”

label as the “

train present and stopped”

segment, determining the presence or the motion state of the train within the region of interest of the video scene at the time of inquiry as the “

no train present”

, the “

train present and in transition”

or the “

train present and stopped”

classification of the segment comprising the video frame of the time of inquiry, the determining the distribution of the “

train present” and

“

train absent”

labels of the plurality of the video segments within the auto-correction time period, the exchanging the “

train present” and

the “

train absent”

labels that are determined for each of the video frames in response to the determined distribution of the “

train present” and

“

train absent”

labels not meeting the expected distribution for the auto-correction time period; and

repeating as the function of the exchanged “

train present” and

“

train absent”

labels the steps of classifying the segments of the input video data of consecutive video frames that each comprise the “

train absent”

label and the “

static”

label into the “

no train present”

segment groups, the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

motion present”

label as the “

train present and in transition”

segments, and the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as the “

train present and stopped”

segment groups, as the function of the exchanged “

train present” and

“

train absent”

labels.

3. The method of claim 1, further comprising:

determining foreground feature data for each of the plurality of frames of the video input data via applying a background subtraction mask to each of the frames;

determining for each of the plurality of frames a frame differencing ratio of a value of motion feature data determined for the each frame to a value of motion feature data determined for an immediately previous frame of the plurality of frames;

labeling each of the frames as “

train present”

in response to the determined foreground feature data value meeting a background subtraction threshold value that is selected as indicative of the presence of a foreground object having the size of a train car, or as “

train absent”

in response to the determined foreground feature data value not meeting the background subtraction threshold value; and

labeling each of the frames as “

motion present”

in response to the determined frame differencing data value meeting a motion threshold value that is selected as indicative of a change in the visual features consistent with the movement of a train car within the frame data, or as “

static”

in response to the determined frame differencing data value not meeting the motion threshold value.

4. The method of claim 1, further comprising, prior to the repeating the steps of classifying segments of the input video data into the “

no train present”

segment groups, the “

train present and in transition”

segment groups and the “

train present and stopped”

segment groups of the consecutive video frames as the function of the exchanged “

train present” and

“

train absent”

labels;

accumulating appearance features extracted from each of the frames into histograms for each of the classified segments of the input video data;

comparing the accumulated histogram for a current one of the classified segments to the histograms of each of a plurality of segments that are previous in time to the current segment, wherein each of the compared plurality of segments occurred within a comparison time period of the current motion state segment, and wherein the comparison time period is selected to encompass each of the “

train present and stopped”

segment, the “

train present and in transition”

segment and the “

no train present”

segment under anticipated or historical train movement timings for the region of interest; and

changing a current classification of the current segment to the “

train present and stopped”

classification, the “

train present and in transition”

classification or the “

no train present”

classification of a one of the compared plurality of segments that has an accumulated histogram that best matches the accumulated histogram for the current segment.

5. The method of claim 4, further comprising:

selecting the auto-correction time period to encompass a plurality of train stopping events over a time frame longer than an anticipated or dynamically observed time that a train is stopped within the region of interest for one stopping event.

6. The method of claim 5, wherein the extracted, accumulated features are color features, and the accumulated histograms are color histograms.

7. The method of claim 5, wherein the step of accumulating the appearance features extracted from each the frames into the histograms for each of the classified segments of the input video data further comprises:

constructing the accumulated appearance histogram after elapse of a latency delay time period from a beginning of the classified segment.

8. The method of claim 7, wherein the latency delay time period ranges from one to two seconds.

9. The method of claim 7, further comprising:

defining the region of interest to comprise a train track area within an image scene of a stream of the video image data, and to exclude a pedestrian platform within the image scene that is adjacent to and visually distinguishable from the train track area.

10. An article of manufacture, comprising:

a computer readable hardware storage device having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a computer processing unit that cause the computer processing unit to;

classify a segment of input video image data comprising a group of consecutive video frames that each comprise “

train absent”

label and “

static”

labels as a “

no train present”

segment, wherein the input video image data is acquired from a region of interest defined around a train track area within an image scene of a stream of the video image data that comprises a portion of a railway track area;

classify a segment of the input video data comprising a group of consecutive video frames that each comprise the “

train present”

label and a “

motion present”

label as a “

train present and in transition”

segment;

classify a segment of the input video data comprising a group of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as a “

train present and stopped”

segment;

determine a presence or a motion state of a train within the region of interest of the video scene at a time of inquiry as the “

no train present”

, the “

train present and in transition”

or the “

train present and stopped”

classification of the segment comprising the video frame of the time of inquiry;

determine a distribution of the “

train present” and

“

train absent”

labels of the plurality of the video segments within an auto-correction time period;

exchange the “

train present” and

the “

train absent”

labels that are determined for each of the video frames, in response to the determined distribution of the “

train present” and

“

train absent”

labels not meeting an expected distribution for the auto-correction time period; and

repeat, as a function of the exchanged “

train present” and

“

train absent”

labels, the classification of the segments of the input video data of consecutive video frames that each comprise the “

train absent”

label and the “

static”

label as the “

no train present”

segment groups, the classification of the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

motion present”

label as the “

train present and in transition”

segments, and the classification of the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as the “

train present and stopped”

segment groups.

11. The article of manufacture of claim 10, wherein the computer readable program code instructions for execution by the computer processing unit, further cause the computer processing unit to:

determine foreground feature data for each of the plurality of frames of the video input data via applying a background subtraction mask to each of the frames;

determine for each of the plurality of frames a frame differencing ratio of a value of motion feature data determined for the each frame to a value of motion feature data determined for an immediately previous frame of the plurality of frames;

label each of the frames as “

train present”

in response to the determined foreground feature data value meeting a background subtraction threshold value that is selected as indicative of the presence of a foreground object having the size of a train car, or as “

train absent”

in response to the determined foreground feature data value not meeting the background subtraction threshold value; and

label each of the frames as “

motion present”

in response to the determined frame differencing data value meeting a motion threshold value that is selected as indicative of a change in the visual features consistent with the movement of a train car within the frame data, or as “

static”

in response to the determined frame differencing data value not meeting the motion threshold value.

12. The article of manufacture of claim 10, wherein the computer readable program code instructions for execution by the computer processing unit, further cause the computer processing unit to, prior to the repetition of the classification of the segments of the input video data into the “

no train present”

segment groups, the “

train present and in transition”

segment groups and the “

train present and stopped”

segment groups of the consecutive video frames as the function of the exchanged “

train present” and

“

train absent”

labels;

accumulate appearance features extracted from each of the frames into histograms for each of the classified segments of the input video data;

compare the accumulated histogram for a current one of the classified segments to the histograms of each of a plurality of segments that are previous in time to the current segment, wherein each of the compared plurality of segments occurred within a comparison time period of the current motion state segment, and wherein the comparison time period is selected to encompass each of the “

train present and stopped”

segment, the “

train present and in transition”

segment and the “

no train present”

segment under anticipated or historical train movement timings for the region of interest; and

change a current classification of the current segment to the “

train present and stopped”

classification, the “

train present and in transition”

classification or the “

no train present”

classification of a one of the compared plurality of segments that has an accumulated histogram that best matches the accumulated histogram for the current segment.

13. The article of manufacture of claim 12, wherein the computer readable program code instructions for execution by the computer processing unit, further cause the computer processing unit to select the auto-correction time period to encompass a plurality of train stopping events over a time frame longer than an anticipated or dynamically observed time that a train is stopped within the region of interest for one stopping event.

14. The article of manufacture of claim 12, wherein the computer readable program code instructions for execution by the computer processing unit, further cause the computer processing unit to accumulate the appearance features extracted from each the frames into the histograms for each of the classified segments of the input video data further by constructing the accumulated appearance histogram after elapse of a latency delay time period from a beginning of the classified segment.

15. The article of manufacture of claim 12, wherein the computer readable program code instructions for execution by the computer processing unit, further cause the computer processing unit to define the region of interest to comprise a train track area within an image scene of a stream of the video image data, and to exclude a pedestrian platform within the image scene that is adjacent to and visually distinguishable from the train track area.

16. A system, comprising:

a processing unit;

a computer readable memory in communication with the processing unit; and

a computer-readable storage medium in communication with the processing unit;

wherein the processing unit executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

classifies a segment of input video image data comprising a group of consecutive video frames that each comprise “

train absent”

label and “

static”

labels as a “

no train present”

segment, wherein the input video image data is acquired from a region of interest defined around a train track area within an image scene of a stream of the video image data that comprises a portion of a railway track area;

classifies a segment of the input video data comprising a group of consecutive video frames that each comprise the “

train present”

label and a “

motion present”

label as a “

train present and in transition”

segment;

classifies a segment of the input video data comprising a group of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as a “

train present and stopped”

segment;

determines a presence or a motion state of a train within the region of interest of the video scene at a time of inquiry as the “

no train present”

, the “

train present and in transition”

or the “

train present and stopped”

classification of the segment comprising the video frame of the time of inquiry;

determines a distribution of the “

train present” and

“

train absent”

labels of the plurality of the video segments within an auto-correction time period;

exchanges the “

train present” and

the “

train absent”

labels that are determined for each of the video frames, in response to the determined distribution of the “

train present” and

“

train absent”

labels not meeting an expected distribution for the auto-correction time period; and

repeats, as a function of the exchanged “

train present” and

“

train absent”

labels, the classification of the segments of the input video data of consecutive video frames that each comprise the “

train absent”

label and the “

static”

label as the “

no train present”

segment groups, the classification of the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

motion present”

label as the “

train present and in transition”

segments, and the classification of the segments of the input video data of consecutive video frames that each comprise the “

train present”

label and the “

static”

label as the “

train present and stopped”

segment groups.

17. The system of claim 16, wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory, and thereby further:

determines foreground feature data for each of the plurality of frames of the video input data via applying a background subtraction mask to each of the frames;

determines for each of the plurality of frames a frame differencing ratio of a value of motion feature data determined for the each frame to a value of motion feature data determined for an immediately previous frame of the plurality of frames;

labels each of the frames as “

train present”

in response to the determined foreground feature data value meeting a background subtraction threshold value that is selected as indicative of the presence of a foreground object having the size of a train car, or as “

train absent”

in response to the determined foreground feature data value not meeting the background subtraction threshold value; and

labels each of the frames as “

motion present”

in response to the determined frame differencing data value meeting a motion threshold value that is selected as indicative of a change in the visual features consistent with the movement of a train car within the frame data, or as “

static”

in response to the determined frame differencing data value not meeting the motion threshold value.

18. The system of claim 16, wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory, and thereby further, prior to the repetition of the classification of the segments of the input video data into the “

no train present”

segment groups, the “

train present and in transition”

segment groups and the “

train present and stopped”

segment groups of the consecutive video frames as the function of the exchanged “

train present” and

“

train absent”

labels;

accumulates appearance features extracted from each of the frames into histograms for each of the classified segments of the input video data;

compares the accumulated histogram for a current one of the classified segments to the histograms of each of a plurality of segments that are previous in time to the current segment, wherein each of the compared plurality of segments occurred within a comparison time period of the current motion state segment, and wherein the comparison time period is selected to encompass each of the “

train present and stopped”

segment, the “

train present and in transition”

segment and the “

no train present”

segment under anticipated or historical train movement timings for the region of interest; and

changes a current classification of the current segment to the “

train present and stopped”

classification, the “

train present and in transition”

classification or the “

no train present”

classification of a one of the compared plurality of segments that has an accumulated histogram that best matches the accumulated histogram for the current segment.

19. The system of claim 18, wherein the extracted, accumulated features are color features;

wherein the accumulated histograms are color histograms;

wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory, and thereby accumulates the appearance features extracted from each of the frames into the histograms for each of the classified segments of the input video data by constructing the accumulated appearance histogram after elapse of a latency delay time period from a beginning of the classified segment; and

wherein the latency delay time period ranges from one to two seconds.

20. The system of claim 18, wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory, and thereby defines the region of interest to comprise a train track area within an image scene of a stream of the video image data, and to exclude a pedestrian platform within the image scene that is adjacent to and visually distinguishable from the train track area.