Systems and methods for surveillance with a visual marker

• US 10,691,141 B2
• Filed: 10/10/2016
• Issued: 06/23/2020
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Liu, Ang, Hu, Xiao, Zhou, Guyue
• Primary Examiner(s): Cass, Jean Paul
• Application Number:
  US15/289,384
• Publication Number:
  US 20170031369A1
• Time in Patent Office: 1,352 Days
• Field of Search: 701 3
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06K 19/06037 multi-dimensional coding
• CPC Class Codes: G06K 7/1417 2D bar codes
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/045 Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933

Abstract

Systems and methods are provided for positing an unmanned aerial vehicle (UAV) in an environment. The UAV may be able to identify visual markers and patterns in the environment. The UAV can distinguish between visual markers and other features in the environment. The visual markers can be analyzed with visual sensors to determine the position of the UAV in the environment. The visual markers can encode instructions to the UAV.

Claims

1. A method of positioning an unmanned aerial vehicle (UAV) within an environment, said method comprising:

capturing, with aid of one or more vision sensors of the UAV while the UAV is in flight, an image including a visual marker having a unique visual pattern within the environment and affixed to a surface within the environment;

identifying and distinguishing the visual marker from a plurality of different markers based on the unique visual pattern of the visual marker in the image, wherein the identified visual marker has a predefined location that is different from locations of the plurality of different markers;

determining, with aid of one or more processors, a location of the UAV within a three- dimensional coordinate system, based on the predefined location of the identified visual marker; and

identifying the unique visual pattern and effecting a sequence of actions by the UAV in response to a plurality of instructions encoded in the identified visual pattern and communicated by the visual marker, the plurality of instructions including at least one of adjusting position of a payload or swapping the payload;

wherein effecting the sequence of actions by the UAV in response to the plurality of instructions includes causing the UAV to perform at least one of adjusting the position of the payload or swapping the payload.

2. The method of claim 1, wherein the environment is an indoor environment.

3. The method of claim 1, wherein:

vertices of shapes in the visual pattern are detected, and

a distance and an angle between two of the vertices are determined and used as inputs to calculate a relative position between the UAV and the visual marker.

4. The method of claim 1, wherein the visual marker is a ribbon having a unique visual pattern that includes squares.

5. The method of claim 1, wherein the unique visual pattern includes squares of different colors.

6. The method of claim 1, wherein the unique visual pattern includes AprilTags, QR codes, or bar codes.

7. The method of claim 1, further comprising calculating a dimension of at least a portion of the unique visual pattern of the visual marker in the image;

and

determining, with aid of the one or more processors, the location of a UAV within the three-dimensional coordinate system, based on the dimension of the portion of the unique visual pattern.

8. The method of claim 1, further comprising positioning the UAV within the environment by calculating a rotation and a position of the UAV relative to the visual marker using one or more intrinsic properties of the one or more vision sensors on board the UAV, a predefined position of the visual marker, and a set of points in the image including the visual marker taken by the vision sensor on board the UAV.

9. The method of claim 8, wherein the UAV is patrolling the environment autonomously when the one or more vision sensors capture the image.

10. The method of claim 1, wherein the one or more processors are on-board the UAV.

11. The method of claim 1, wherein the positioning is also determined with aid of an infrared or radar detector.

19. The method of claim 1, wherein the effecting the sequence of actions by the UAV includes effecting the sequence of actions by the UAV according to at least one of a specified time interval or a predetermined distance included in the instructions.

21. The method of claim 1, wherein the visual marker includes a ribbon having a plurality of markers or a series of discrete markers, and the effecting the sequence of actions by the UAV includes causing the UAV to follow the ribbon or the series of discrete markers along a path.

22. The method of claim 1, wherein:

the plurality of instructions include multiple sets of instructions associated with the visual marker, and

effecting the sequence of actions by the UAV includes;

selecting one set of instructions from the multiple sets of instructions based on one or more parameters; and

effecting the sequence of actions by the UAV according to the one set of instructions.

23. The method of claim 1,

wherein the effecting the sequence of actions by the UAV includes effecting the sequence of actions by the UAV according to a specified time interval indicating a wait time before the UAV effects the sequence of actions.

12. A system for positioning an unmanned aerial vehicle (UAV) within an environment, said system comprising:

one or more vision sensors of the UAV configured to capture an image including a visual marker having a unique visual pattern within the environment and affixed to a surface within the environment, while the UAV is flight; and

one or more processors, individually or collectively configured to;

identify and distinguish the visual marker from a plurality of different markers based on the unique visual pattern of the visual marker in the image, wherein the identified visual marker has a predefined location that is different from locations of the plurality of different markers;

determine a location of the UAV within a three-dimensional coordinate system, based on the predefined location of the identified visual marker; and

identify the unique visual pattern and effect a sequence of actions by the UAV in response to a plurality of instructions encoded in the identified visual pattern and communicated by the visual marker, the plurality of instructions including at least one of adjusting position of a payload or swapping the payload;

wherein effecting the sequence of actions by the UAV in response to the plurality of instructions includes causing the UAV to perform at least one of adjusting the position of the payload or swapping the payload.

13. The system of claim 12, wherein the environment is an environment in which global position system (GPS) signals are not reliable.

14. The system of claim 12, wherein:

vertices of shapes in the visual pattern are detected, and

a distance and an angle between two of the vertices are determined and used as inputs to calculate a relative position between the UAV and the visual marker.

15. The system of claim 12, wherein the visual marker is a ribbon having a unique visual pattern that includes squares.

16. The system of claim 12, wherein the one or more processors are further configured to calculate a dimension of at least a portion of the unique visual pattern of the visual marker in the image;

and

determine the location of a UAV within the three-dimensional coordinate system, based on the dimension of the portion of the unique visual pattern.

17. The system of claim 12, wherein the one or more processors are configured to calculate a rotation and a position of the UAV relative to the visual marker using one or more intrinsic properties of the one or more vision sensors on board the UAV, a predefined position of the visual marker, and a set of points in the image including the visual marker taken by the vision sensor on board the UAV.

18. The system of claim 17, wherein at least one of the one or more vision sensors is a camera.

20. A method of positioning an unmanned aerial vehicle (UAV) within an environment, said method comprising:

capturing, with aid of one or more vision sensors of the UAV while the UAV is in flight, an image including a visual marker having a unique visual pattern within the environment and affixed to a surface within the environment;

identifying and distinguishing the visual marker from a plurality of different markers based on the unique visual pattern of the visual marker in the image, wherein the identified visual marker has a predefined location that is different from locations of the plurality of different markers;

determining, with aid of one or more processors, a location of the UAV within a three- dimensional coordinate system, based on the predefined location of the identified visual marker; and

identifying the unique visual pattern and effecting a sequence of actions by the UAV in response to a plurality of instructions encoded in the identified visual pattern and communicated by the visual marker;

wherein;

effecting the sequence of actions by the UAV in response to the plurality of instructions includes causing the UAV to perform at least one of adjusting position of a payload or swapping a payload,

effecting the sequence of actions by the UAV further includes effecting the sequence of actions by the UAV according to at least one of a specified time interval or a predetermined distance included in the instructions,

the specified time interval indicates a wait time before the UAV effects the sequence of actions, and

the predetermined distance indicates a distance from the visual marker at which the UAV effects the sequence of actions.