Multi-stage vehicle detection in side-by-side drive-thru configurations

• US 10,044,988 B2
• Filed: 05/19/2015
• Issued: 08/07/2018
• Est. Priority Date: 05/19/2015
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Bulan, Orhan, Burry, Aaron M.
• Primary Examiner(s): Le, Peter D
• Application Number:
  US14/715,787
• Publication Number:
  US 20160342841A1
• Time in Patent Office: 1,176 Days
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 50/12 Hotels or restaurants
• CPC Class Codes: G06V 10/7788 the supervisor being a human, e.g. interactive learning with a human teacher
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Multi-stage vehicle detection systems and methods for side-by-side drive-thru configurations. One or more video cameras (or an image-capturing unit) can be employed for capturing video of a drive-thru of interest in a monitored area. A group of modules can be provided, which define multiple virtual detection loops in the video and sequentially perform classification with respect to each virtual detection loops among the multiple virtual detection loops, starting from a virtual detection loop closest to an order point, and when a vehicle having a car ID is sitting in a drive-thru queue, so as to improve vehicle detection performance in automated post-merge sequencing.

Claims

1. A multi-stage vehicle detection system for side-by-side drive-thru configurations, said system comprising:

a plurality of classifiers associated with at least one video camera for capturing video of a drive-thru of interest in a monitored area monitored by said at least one video camera;

a plurality of modules that defines multiple virtual detection loops in said video and sequentially performs classification for an automated post-merge sequencing application via at least one classifier among said plurality of classifiers with respect to each virtual detection loop among said multiple virtual detection loops, starting from a virtual detection loop closest to an order point, and when a vehicle having a car ID is sitting in a drive-thru queue, so as to improve vehicle detection performance in automated post-merge sequencing of said automated post-merge sequencing application and wherein said multiple virtual detection loops are defined for said automated post-merge sequencing application for detecting vehicles coming from a first order point, wherein a first virtual detection loop among said multiple virtual detection loops is defined at first said first order point and wherein a second virtual detection loop among said multiple virtual detection loops is defined at a merge point or after said merge point, or wherein said virtual detection loops are defined for detecting said vehicles from a second order point;

wherein at least one classifier among said plurality of classifiers performs said classification, and wherein for each defined virtual detection loop among said multiple virtual detection loops, said at least one classifier is trained using a set of positive samples and a set of negative samples extracted from at least one virtual detection loop among said virtual detection loops, wherein said positive samples among said set of positive samples comprise vehicle data extracted from said at least one virtual detection loop and said negative samples among said set of negative samples comprise non-vehicle data extracted from said at least one virtual detection loop; and

wherein said plurality of modules include;

a video acquisition module comprising said at least one video camera that acquires said video of said drive-thru of interest;

a vehicle detection module that defines said multiple virtual detection loops in said video and sequential classification in said virtual detection loop to detect vehicles as each vehicle among said vehicles is assigned to said car ID at said order point;

a vehicle tracker that tracks detected vehicles until said detected vehicles leave said monitored area, wherein said vehicle tracker comprises at least one of a point tracker and a region-based tracker and wherein a new tracker is started only if a number of currently tracked features in a motion-detected region is less than a specified threshold, and wherein if said point tracker is used for said tracking, extracted features from said tracking include at least one of SIFT (Scale Invariant Feature Transform) features, SURF (Speeded Up Robust Feature) features, Harris corner features, FAST (Fast Accelerated Segment Test) features, Eigen value algorithm features, and MSER (Maximally Stable Extremal Region) features, and if said region-based tracker is used for said tracking extracted attributes from said tracking include color attributes of a detected region, geometric attributes of a detected blob and shape attributes extracted from said detected region, wherein said geometric attributes comprise an orientation, an aspect ratio and an eccentricity; and

a sequence reporting module that reports an assigned car ID and originating lanes as vehicles proceed through a cash out point at said drive-thru and wherein based on outputs from said vehicle detection module and said vehicle tracking module, said sequence reporting module reports a temporal sequence of merging vehicles among said vehicles.

2. The system of claim 1 wherein said at least one camera comprises a point and shoot RGB camera and wherein said drive-thru of interest comprises a plurality of regions of interest, wherein each region of interest among said regions of interests corresponds to an order point of said drive-thru in a scene captured by said at least one camera wherein said each region of interest is defined per an installation of said at least one camera.

3. The system of claim 1 further comprising an offline stage wherein features of manually labeled samples of vehicles crossing said multiple virtual detection loops are extracted and fed into said plurality of classifiers for training and wherein said plurality of classifiers after training is employed within said system and wherein at least two classifiers among said plurality of classifiers comprise linear SVM (Support Vector Machine) classifiers trained using both said positive samples and said negative samples and wherein said negative samples with respect to said order point are extracted from positive samples for said merge point and said negative samples for said merge point are extracted from positive samples with respect to said order point.

4. The system of claim 3 wherein said at least one classifier comprises an order point classifier with respect to an order point at said drive-thru.

5. The system of claim 4 wherein said at least one classifier further comprises a merge point classifier with respect to said merge point at said drive-thru.

6. The system of claim 1 further comprising:

an offline stage wherein features of manually labeled samples of vehicles crossing said virtual detection loops are extracted and fed into said at least one classifier for training and wherein said at least one classifier after training is employed within said system; and

wherein said new tracker further comprises a point tracker that tracks a number of currently tracked features in said motion-detected region.

7. The system of claim 6 wherein plurality of classifiers includes an order point classifier with respect to said order point including said first and second order points at said drive-thru and a merge point classifier with respect to a merge point at said drive-thru and wherein said order point classifier and said merge point classifier are trained on HOG features calculated from pixel training images.

8. A multi-stage vehicle detection system for side-by-side drive-thru configurations, said system comprising:

at least one video camera and a plurality of classifiers;

a processor that communicates with said at least one video camera and said at least one classifier; and

a computer-usable medium embodying computer program code, said computer-usable medium capable of communicating with the processor, said computer program code comprising instructions executable by said processor and configured for;

monitoring a monitored area with said at least one video camera in association with said plurality of classifiers;

capturing via said at least one video camera, video of a drive-thru of interest in said monitored area;

defining multiple virtual detection loops in said video;

sequentially performing a classification for an automated post-merge sequencing application via at least one classifier among said plurality of classifiers with respect to each virtual detection loop among said multiple virtual detection loops, starting from a virtual detection loop closest to an order point, and when a vehicle having a car ID is sitting in a drive-thru queue, so as to improve vehicle detection performance in automated post-merge sequencing of said automated post-merge sequencing application, wherein said multiple virtual detection loops are defined for said automated post-merge sequencing application for detecting vehicles coming from a first order point, wherein a first virtual detection loop among said multiple virtual detection loops is defined at first said first order point and wherein a second virtual detection loop among said multiple virtual detection loops is defined at a merge point or after said merge point, or wherein said virtual detection loops are defined for detecting said vehicles from a second order point;

wherein at least one classifier among said plurality of classifiers performs said classification, and wherein for each defined virtual detection loop among said multiple virtual detection loops, said at least one classifier is trained using a set of positive samples and a set of negative samples extracted from at least one virtual detection loop among said virtual detection loops, wherein said positive samples among said set of positive samples comprise vehicle data extracted from said at least one virtual detection loop and said negative samples among said set of negative samples comprise non-vehicle data extracted from said at least one virtual detection loop; and

wherein said plurality of modules comprises;

a video acquisition module comprising said at least one video camera that acquires said video of said drive-thru of interest;

a vehicle detection module that defines said multiple virtual detection loops in said video and sequential classification in said virtual detection loop to detect vehicles as each vehicle among said vehicles is assigned to said car ID at said order point;

a vehicle tracker that tracks detected vehicles until said detected vehicles leave said monitored area, wherein said vehicle tracker comprises at least one of a point tracker and a region-based tracker and wherein a new tracker is started only if a number of currently tracked features in a motion-detected region is less than a specified threshold, and wherein if said point tracker is used for said tracking, extracted features from said tracking include at least one of SIFT (Scale Invariant Feature Transform) features, SURF (Speeded Up Robust Feature) features, Harris corner features, FAST (Fast Accelerated Segment Test) features, Eigen value algorithm features, and MSER (Maximally Stable Extremal Region) features, and if said region-based tracker is used for said tracking extracted attributes from said tracking include color attributes of a detected region, geometric attributes of a detected blob and shape attributes extracted from said detected region, wherein said geometric attributes comprise an orientation, an aspect ratio and an eccentricity; and

a sequence reporting module that reports an assigned car ID and originating lanes as vehicles proceed through a cash out point at said drive-thru and wherein based on outputs from said vehicle detection module and said vehicle tracking module, said sequence reporting module reports a temporal sequence of merging vehicles among said vehicles.

9. The system of claim 8 wherein said at least one camera comprises a point and shoot RGB camera and wherein said drive-thru of interest comprises a plurality of regions of interest, wherein each region of interest among said regions of interests corresponds to an order point of said drive-thru in a scene captured by said at least one camera wherein said each region of interest is defined per an installation of said at least one camera.

10. The system of claim 9 wherein said instructions are further configured for in an offline stage extracting and feeding manually labeled samples of vehicles crossing said virtual detection loops into said at least one classifier for training and wherein said at least one classifier after training is employed within said system;

and

wherein said new tracker further comprises a point tracker that tracks a number of currently tracked features in said motion-detected region.

11. The system of claim 10 wherein said at least one classifier comprises an order point classifier with respect to an order point at said drive-thru.

12. The system of claim 11 wherein said at least one classifier comprises a merge point classifier with respect to said merge point at said drive-thru and wherein said order point classifier and said merge point classifier are trained on HOG features calculated from pixel training images.

13. The system of claim 10 further comprising:

an offline stage wherein features of manually labeled samples of vehicles crossing said virtual detection loops are extracted and fed into said plurality of classifiers for training and wherein said plurality of classifiers after training is employed within said system and wherein at least two classifiers among said plurality of classifiers comprise linear SVM (Support Vector Machine) classifiers trained using both said positive samples and said negative samples and wherein said negative samples with respect to said order point are extracted from positive samples for said merge point and said negative samples for said merge point are extracted from positive samples with respect to said order point.

14. A multi-stage vehicle detection method for side-by-side drive-thru configurations, said method comprising:

capturing video of a drive-thru of interest in a monitored area utilizing at least one video camera associated with a plurality of classifiers; and

providing a plurality of modules that defines multiple virtual detection loops in said video and sequentially performs classification for an automated post-merge sequencing application via at least one classifier among said plurality of classifiers with respect to each virtual detection loop among said multiple virtual detection loops, starting from a virtual detection loop closest to an order point, and when a vehicle having a car ID is sitting in a drive-thru queue, so as to improve vehicle detection performance in automated post-merge sequencing of said automated post-merge sequencing application, wherein said multiple virtual detection loops are defined for said automated post-merge sequencing application for detecting vehicles coming from a first order point, wherein a first virtual detection loop among said multiple virtual detection loops is defined at first said first order point and wherein a second virtual detection loop among said multiple virtual detection loops is defined at a merge point or after said merge point, or wherein said virtual detection loops are defined for detecting said vehicles from a second order point; and

wherein at least one classifier among said plurality of classifiers performs said classification, and wherein for each defined virtual detection loop among said multiple virtual detection loops, said at least one classifier is trained using a set of positive samples and a set of negative samples extracted from at least one virtual detection loop among said virtual detection loops, wherein said positive samples among said set of positive samples comprise vehicle data extracted from said at least one virtual detection loop and said negative samples among said set of negative samples comprise non-vehicle data extracted from said at least one virtual detection loop; and

wherein said plurality of modules comprises;

a video acquisition module comprising said at least one video camera that acquires said video of said drive-thru of interest;

a vehicle detection module that defines said multiple virtual detection loops in said video and sequential classification in said virtual detection loop to detect vehicles as each vehicle among said vehicles is assigned to said car ID at said order point;

a vehicle tracker that tracks detected vehicles until said detected vehicles leave said monitored area, wherein said vehicle tracker comprises a-at least one of a point tracker and a region-based tracker and wherein a new tracker is started only if a number of currently tracked features in a motion-detected region is less than a specified threshold, and wherein if said point tracker is used for said tracking, extracted features from said tracking include at least one of SIFT (Scale Invariant Feature Transform) features, SURF (Speeded Up Robust Feature) features, Harris corner features, FAST (Fast Accelerated Segment Test) features, Eigen value algorithm features, and MSER (Maximally Stable Extremal Region) features, and if said region-based tracker is used for said tracking extracted attributes from said tracking include color attributes of a detected region, geometric attributes of a detected blob and shape attributes extracted from said detected region, wherein said geometric attributes comprise an orientation, an aspect ratio and an eccentricity; and

a sequence reporting module that reports an assigned car ID and originating lanes as vehicles proceed through a cash out point at said drive-thru and wherein based on outputs from said vehicle detection module and said vehicle tracking module, said sequence reporting module reports a temporal sequence of merging vehicles among said vehicles.

15. The method of claim 14 wherein said at least one camera comprises a point and shoot RGB camera and wherein said drive-thru of interest comprises a plurality of regions of interest, wherein each region of interest among said regions of interests corresponds to an order point of said drive-thru in a scene captured by said at least one camera wherein said each region of interest is defined per an installation of said at least one camera.

16. The method of claim 14 further comprising in an offline stage extracting and feeding said features of manually labeled samples of vehicles crossing said virtual detection loops into said plurality of classifiers for training and wherein said plurality of classifiers after training is employed within said system, and wherein said new tracker further comprises a point tracker that tracks a number of currently tracked features in said motion-detected region based on an assumption that when an already tracked vehicle is moving, said motion-detected region includes said tracked vehicle and a set of feature already being tracked, and at least two classifiers among said plurality of classifiers comprise linear SVM (Support Vector Machine) classifiers trained using both said positive samples and said negative samples and wherein said negative samples with respect to said order point are extracted from positive samples for said merge point and said negative samples for said merge point are extracted from positive samples with respect to said order point.

17. The method of claim 16 further comprising configuring said at least one classifier among said at least two classifiers to function as an order point classifier with respect to said order point and at least one other classifier among said at least two classifiers to comprises a merge point classifier with respect to said merge point wherein said order point classifier and said merge point classifier are trained on HOG features calculated from pixel training images.