System and method for door detection for corridor exploration

• US 9,978,149 B1
• Filed: 12/28/2015
• Issued: 05/22/2018
• Est. Priority Date: 04/20/2015
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Zhang, Lei, Kim, Kyungnam, Khosla, Deepak
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Shin, Soo
• Application Number:
  US14/981,131
• Time in Patent Office: 876 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components

Abstract

Described is a system for door detection for use with an unmanned aerial vehicle (UAV). The system receive a video input image from a single monocular camera. Edge points are detected in the video, with the edge points connected to form long edge lines. Orientations of the long edge lines are determined, such that long edge lines having a substantially vertical orientation are designated as initial door line candidates and long edge lines having a non-vertical, non-horizontal orientation are designated for use in detecting a vanishing point. A vanishing point is then detected in the video frame. Thereafter, intensity profile and line properties of the door line candidates are calculated. Finally, it is verified if the door line candidates are real world door lines and, if so, an area between the real world door lines is designated as an open door.

Claims

1. A system for door detection, the system comprising:

one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of;

receiving a video input image, the video input image having at least one video frame;

detecting edge points in the video frame;

connecting the edge points to form long edge lines;

determining orientations of the long edge lines, such that long edge lines having a substantially vertical orientation are designated as initial door line candidates and long edge lines having a non-vertical, non-horizontal orientation are designated for use in detecting a vanishing point;

determining if a vanishing point in the video frame is detectable, such that if a vanishing point is detected, the system performs operations of;

calculating intensity profile and line properties of the door line candidates; and

verifying if the door line candidates are real world door lines based on the intensity profile and line properties and, if so, then designating an area between the real world door lines as an open door; and

wherein in determining if a vanishing point in the video frame is detectable, if a vanishing point is not detected, the system performs an operation of comparing the door line candidates against a template of previously detected door lines to verify if the door line candidates are real world door lines.

2. The system as set forth in claim 1, further performing an operation of temporally filtering door line candidates previously verified as real world door lines.

3. The system as set forth in claim 1, wherein in detecting edge points in the video frame, a Canny edge detector detects edge pixels whose intensity gradient is within a preselected range, with said edge pixels being designated as edge points.

4. The system as set forth in claim 1, wherein connecting the edge points to form long edge lines, gradient directions at all edge pixels are calculated and discretized into N number of bins, such that edge pixels with gradient orientations belonging to a same bin are linked to form a long edge line.

5. The system as set forth in claim 1, wherein in detecting a vanishing point, the system performs operations of:

filtering detected long edge lines to remove long edge lines that are horizontal leaving remaining long edge lines;

splitting the video frame into an M×

M grid having a plurality of grid cells within the video frame;

calculating intersections of the remaining long edge lines such that a number of line intersections within each grid cell is calculated;

designating a cell with a maximum number of line intersections as containing the vanishing point; and

calculating a centroid of the line intersections within the designated cell as the vanishing point.

6. The system as set forth in claim 1, wherein verifying if the door line candidates are real world door lines, the system performs operations of:

determining a distance from the vanishing point to a door line candidate, such that if the distance is less than a predetermined threshold, the door line candidate is filtered out from further processing, leaving remaining door line candidates;

pairing each remaining door line candidate with one another to form candidate doors formed by a pair of door line candidates, and determining a distance between each pair of door line candidates such that if the distance is outside of a predetermined threshold, the candidate door formed by the pair of door line candidates is filtered out as a candidate door from further processing, and if the distance is within a predetermined threshold, the candidate door formed by the pair of door line candidates maintains designation as a candidate door for further processing and the corresponding pair of door line candidates are designated are remaining pairs of door lines;

designating three areas in the image frame around the remaining pair of door lines, with a first part at a left of the candidate door, a second part encompassing an area within the candidate door, and a third part at a right of the candidate door;

calculating an intensity profile for the three parts and comparing the intensity profile against a prior knowledge to filter out remaining pairs of door lines that fall outside a predetermined range, with at least one remaining pair of door lines that fall within the predetermined range being designated as real world door lines.

7. The system as set forth in claim 1, wherein the system is embedded in a mobile platform and further comprising instructions for causing the one or more processors to perform an operation of actuating motors within the mobile platform to cause the mobile platform to navigate between the real world door lines.

8. A computer implemented method for door detection, the method comprising an act of causing one or more processors to execute instructions encoded on a non-transitory computer readable medium, such that upon execution of the instructions, the one or more processors perform operations of:

receiving a video input image, the video input image having at least one video frame;

detecting edge points in the video frame;

connecting the edge points to form long edge lines;

determining orientations of the long edge lines, such that long edge lines having a substantially vertical orientation are designated as initial door line candidates and long edge lines having a non-vertical, non-horizontal orientation are designated for use in detecting a vanishing point;

determining if a vanishing point in the video frame is detectable, such that if a vanishing point is detected, the system performs operations of;

calculating intensity profile and line properties of the door line candidates; and

verifying if the door line candidates are real world door lines based on the intensity profile and line properties and, if so, then designating an area between the real world door lines as an open door; and

wherein in determining if a vanishing point in the video frame is detectable, if a vanishing point is not detected, comparing the door line candidates against a template of previously detected door lines to verify if the door line candidates are real world door lines.

9. The computer implemented method as set forth in claim 8, further comprising an act of temporally filtering door line candidates previously verified as real world door lines.

10. The computer implemented method as set forth in claim 8, wherein in detecting edge points in the video frame, a Canny edge detector detects edge pixels whose intensity gradient is within a preselected range, with said edge pixels being designated as edge points.

11. The computer implemented method as set forth in claim 8, wherein connecting the edge points to form long edge lines, gradient directions at all edge pixels are calculated and discretized into N number of bins, such that edge pixels with gradient orientations belonging to a same bin are linked to form a long edge line.

12. The computer implemented method as set forth in claim 8, wherein in detecting a vanishing point, further comprises acts of:

filtering detected long edge lines to remove long edge lines that are horizontal leaving remaining long edge lines;

splitting the video frame into an M×

M grid having a plurality of grid cells within the video frame;

calculating intersections of the remaining long edge lines such that a number of line intersections within each grid cell is calculated;

designating a cell with a maximum number of line intersections as containing the vanishing point; and

calculating a centroid of the line intersections within the designated cell as the vanishing point.

13. The computer implemented method as set forth in claim 8, wherein verifying if the door line candidates are real world door lines, further comprises acts of:

determining a distance from the vanishing point to a door line candidate, such that if the distance is less than a predetermined threshold, the door line candidate is filtered out from further processing, leaving remaining door line candidates;

pairing each remaining door line candidate with one another to form candidate doors formed by a pair of door line candidates, and determining a distance between each pair of door line candidates such that if the distance is outside of a predetermined threshold, the candidate door formed by the pair of door line candidates is filtered out as a candidate door from further processing, and if the distance is within a predetermined threshold, the candidate door formed by the pair of door line candidates maintains designation as a candidate door for further processing and the corresponding pair of door line candidates are designated are remaining pairs of door lines;

designating three areas in the image frame around the remaining pair of door lines, with a first part at a left of the candidate door, a second part encompassing an area within the candidate door, and a third part at a right of the candidate door;

calculating an intensity profile for the three parts and comparing the intensity profile against a prior knowledge to filter out remaining pairs of door lines that fall outside a predetermined range, with at least one remaining pair of door lines that fall within the predetermined range being designated as real world door lines.

14. The computer implemented method as set forth in claim 8, further comprising an act of causing the one or more processors to perform an operation of actuating motors within a mobile platform to cause the mobile platform to navigate between the real world door lines.

15. A computer program product for door detection, the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of;

receiving a video input image, the video input image having at least one video frame;

detecting edge points in the video frame;

connecting the edge points to form long edge lines;

determining orientations of the long edge lines, such that long edge lines having a substantially vertical orientation are designated as initial door line candidates and long edge lines having a non-vertical, non-horizontal orientation are designated for use in detecting a vanishing point;

determining if a vanishing point in the video frame is detectable, such that if a vanishing point is detected, the system performs operations of;

calculating intensity profile and line properties of the door line candidates; and

verifying if the door line candidates are real world door lines based on the intensity profile and line properties and, if so, then designating an area between the real world door lines as an open door; and

wherein in determining if a vanishing point in the video frame is detectable, if a vanishing point is not detected, comparing the door line candidates against a template of previously detected door lines to verify if the door line candidates are real world door lines.

16. The computer program product as set forth in claim 15, further performing an operation of temporally filtering door line candidates previously verified as real world door lines.

17. The computer program product as set forth in claim 15, wherein in detecting edge points in the video frame, a Canny edge detector detects edge pixels whose intensity gradient is within a preselected range, with said edge pixels being designated as edge points.

18. The computer program product as set forth in claim 15, wherein connecting the edge points to form long edge lines, gradient directions at all edge pixels are calculated and discretized into N number of bins, such that edge pixels with gradient orientations belonging to a same bin are linked to form a long edge line.

19. The computer program product as set forth in claim 15, wherein detecting a vanishing point, the computer program product further comprises instructions for causing the one or more processors to perform operations of:

filtering detected long edge lines to remove long edge lines that are horizontal leaving remaining long edge lines;

splitting the video frame into an M×

M grid having a plurality of grid cells within the video frame;

calculating intersections of the remaining long edge lines such that a number of line intersections within each grid cell is calculated;

designating a cell with a maximum number of line intersections as containing the vanishing point; and

calculating a centroid of the line intersections within the designated cell as the vanishing point.

20. The computer program product as set forth in claim 15, wherein verifying if the door line candidates are real world door lines, the computer program product further comprises instructions for causing the one or more processors to perform operations of:

determining a distance from the vanishing point to a door line candidate, such that if the distance is less than a predetermined threshold, the door line candidate is filtered out from further processing, leaving remaining door line candidates;

pairing each remaining door line candidate with one another to form candidate doors formed by a pair of door line candidates, and determining a distance between each pair of door line candidates such that if the distance is outside of a predetermined threshold, the candidate door formed by the pair of door line candidates is filtered out as a candidate door from further processing, and if the distance is within a predetermined threshold, the candidate door formed by the pair of door line candidates maintains designation as a candidate door for further processing and the corresponding pair of door line candidates are designated are remaining pairs of door lines;

designating three areas in the image frame around the remaining pair of door lines, with a first part at a left of the candidate door, a second part encompassing an area within the candidate door, and a third part at a right of the candidate door;

calculating an intensity profile for the three parts and comparing the intensity profile against a prior knowledge to filter out remaining pairs of door lines that fall outside a predetermined range, with at least one remaining pair of door lines that fall within the predetermined range being designated as real world door lines.

21. The computer program product as set forth in claim 15, further comprising instructions for causing the one or more processors to perform an operation of actuating motors within a mobile platform to cause the mobile platform to navigate between the real world door lines.