Generic frontal and side doorway detection with line tracking and vanishing point based verification

• US 10,068,336 B1
• Filed: 08/29/2016
• Issued: 09/04/2018
• Est. Priority Date: 04/20/2015
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Zhang, Lei, Monterroza, Fredy, Kim, Kyungnam, Xu, Jiejun, Chen, Yang, Khosla, Deepak
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Shin, Soo
• Application Number:
  US15/250,494
• Time in Patent Office: 736 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G01C 21/383 Indoor data
• CPC Class Codes: G01C 21/3852 Data derived from aerial or satellite images
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Described is a system for frontal and side doorway detection. Salient line segments are extracted from an image frame captured of an indoor environment. Existence of a vanishing point in the image frame is determined. If a vanishing point is detected with a confidence score that meets or exceeds a predetermined confidence score, then the system performs side doorway detection via a side doorway detection module. If a vanishing point is detected with a confidence score below the predetermined confidence score, then the system performs frontal doorway detection via a frontal doorway detection module. A description of detected doorways is output and used by a mobile robot (Unmanned Aerial Vehicle) to autonomously navigate the indoor environment.

Claims

1. A system for doorway detection, the system comprising:

an Unmanned Aerial Vehicle (UAV),

wherein the UAV further comprises one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform an operation of;

extracting salient line segments from an image frame captured of an indoor environment;

determining existence of a vanishing point in the image frame;

if the vanishing point is detected with a confidence score that meets or exceeds a predetermined confidence score, then performing side doorway detection via a side doorway detection module;

if the vanishing point is detected with a confidence score below the predetermined confidence score, then performing frontal doorway detection via a frontal doorway detection module; and

outputting a description of detected doorways,

wherein the description of detected doorways is used by the UAV to autonomously navigate the indoor environment.

2. The system as set forth in claim 1, wherein each doorway detection module has a corresponding edge tracker for tracking vertical edges of a door.

3. The system as set forth in claim 2, wherein the one or more processors further perform an operation of validating the detection of the frontal doorway and the side doorway using image intensity variance and a candidate door'"'"'s maximum width.

4. The system as set forth in claim 3, wherein if the vanishing point cannot be detected but the corresponding edge tracker tracks vertical edges of the candidate door, then the one or more processors perform the operation of performing the side doorway detection using a relaxed door validation process, such that there is no validation of the candidate door'"'"'s maximum width.

5. The system as set forth in claim 1, wherein the one or more processors further perform an operation of using distance from a candidate door to the vanishing point to switch between the side doorway detection module and the frontal doorway detection module.

6. The system as set forth in claim 1, wherein the one or more processors further perform an operation of using distance from a candidate door to the vanishing point to find parameters for the doorway detection.

7. A computer implemented method for doorway detection, the method comprising an act of:

causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of;

extracting salient line segments from an image frame captured of an indoor environment;

determining existence of a vanishing point in the image frame;

if the vanishing point is detected with a confidence score that meets or exceeds a predetermined confidence score, then performing side doorway detection via a side doorway detection module;

if the vanishing point is detected with a confidence score below the predetermined confidence score, then performing frontal doorway detection via a frontal doorway detection module; and

outputting a description of detected doorways,

wherein the description of detected doorways is used by the UAV to autonomously navigate the indoor environment.

8. The method as set forth in claim 7, wherein each doorway detection module has a corresponding edge tracker for tracking vertical edges of a door.

9. The method as set forth in claim 8, wherein the one or more processors further perform an operation of validating the detection of the frontal doorway and the side doorway using image intensity variance and a candidate door'"'"'s maximum width.

10. The method as set forth in claim 9, wherein if the vanishing point cannot be detected but the corresponding edge tracker tracks vertical edges of the candidate door, then the one or more processors perform the operation of performing the side doorway detection using a relaxed door validation process, such that there is no validation of the candidate door'"'"'s maximum width.

11. The method as set forth in claim 7, wherein the one or more processors further perform an operation of using distance from a candidate door to the vanishing point to switch between the side doorway detection module and the frontal doorway detection module.

12. The method as set forth in claim 7, wherein the one or more processors further perform an operation of using distance from a candidate door to the vanishing point to find parameters for the doorway detection.

13. A computer program product for doorway detection, the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of;

extracting salient line segments from an image frame captured of an indoor environment;

determining existence of a vanishing point in the image frame;

if the vanishing point is detected with a confidence score that meets or exceeds a predetermined confidence score, then performing side doorway detection via a side doorway detection module;

if the vanishing point is detected with a confidence score below the predetermined confidence score, then performing frontal doorway detection via a frontal doorway detection module; and

outputting a description of detected doorways,

wherein the description of detected doorways is used by the UAV to autonomously navigate the indoor environment.

14. The computer program product as set forth in claim 13, wherein each doorway detection module has a corresponding edge tracker for tracking vertical edges of a door.

15. The computer program product as set forth in claim 14, further comprising instructions for causing the one or more processors to further perform an operation of validating the detection of the frontal doorway and the side doorway using image intensity variance and a candidate door'"'"'s maximum width.

16. The computer program product as set forth in claim 15, wherein if the vanishing point cannot be detected but the corresponding edge tracker tracks vertical edges of the candidate door, then the one or more processors perform the operation of performing the side doorway detection using a relaxed door validation process, such that there is no validation of the candidate door'"'"'s maximum width.

17. The computer program product as set forth in claim 13, further comprising instructions for causing the one or more processors to further perform an operation of using distance from a candidate door to the vanishing point to switch between the side doorway detection module and the frontal doorway detection module.

18. The computer program product as set forth in claim 13, further comprising instructions for causing the one or more processors further perform an operation of using distance from a candidate door to the vanishing point to find parameters for the doorway detection.