Marker aided three-dimensional mapping and object labeling

• US 10,769,844 B1
• Filed: 05/14/2018
• Issued: 09/08/2020
• Est. Priority Date: 05/12/2017
• Status: Active Grant

• Current Assignee: Alarm.com Inc. (Alarm.com Holdings, Inc.)
• Original Assignee: Alarm.com Inc. (Alarm.com Holdings, Inc.)
• Inventors: Rezvani, Babak, Seyfi, Ahmad, Madden, Donald, Qian, Gang
• Primary Examiner(s): Garcia, Santiago
• Application Number:
  US15/979,096
• Time in Patent Office: 848 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Methods, systems, and apparatus, including computer programs encoded on a storage device, for generating a three-dimensional map of a property. The method includes storing at least a portion of an initial map modification data structure, receiving an instruction that instructs the robotic device to initiate property mapping, obtaining image data describing a portion of the property that is mounted to the robotic device, analyzing the obtained image data to determine whether the image data depicts a marker, in response to a determination that the obtained image data depicts a marker, determining the pose of the camera that is mounted to the robotic device, updating the initial map modification data structure using the pose of the camera that is mounted to the robotic device, obtaining an initial three-dimensional map of the property; and modifying the initial three-dimensional map of the property based on the updated map modification data structure.

Claims

1. A robotic device comprising:

one or more processors; and

one or more computer readable storage media storing instructions that, when executed by the one or more processors, causes the robotic device to perform operations, the operations comprising;

storing at least a portion of an initial map modification data structure;

receiving an instruction that is configured to instruct the robotic device to initiate property mapping;

determining that image data describing a portion of the property depicts one or more markers, wherein the image data is collected by a camera that is mounted to the robotic device;

in response to determining that the image data depicts one or more markers, determining a pose of the camera that is mounted to the robotic device;

updating the initial map modification data structure using the pose of the camera that is mounted to the robotic device;

determining, based on the initial map modification data structure that was updated, that an initial three-dimensional map of the property should be adjusted to include a label identifying one or more features of the property that is undetectable (i) by the camera that is mounted to the robotic device or (ii) by mapping sensors of the robotic device; and

labeling a particular portion of the initial three-dimensional map of the property using data stored in the updated map modification data structure.

2. The robotic device of claim 1, wherein the pose of the camera that is mounted to the robotic device includes a position and orientation of the camera.

3. The robotic device of claim 1, wherein the initial three-dimensional map of the property was previously generated by the robotic device using mapping sensors mounted to the robotic device.

4. The robotic device of claim 1, wherein storing at least a portion of an initial map modification data structure comprises:

storing a map modification data structure that includes one or more marker identifiers that each have a corresponding object identifier and marker location.

5. The robotic device of claim 1, wherein receiving an instruction that is configured to instruct the robotic device to initiate property mapping comprises:

receiving, by the robotic device and from a user device, an instruction that is configured to instruct the robotic device to initiate property mapping.

6. The robotic device of claim 1, wherein receiving an instruction that is configured to instruct the robotic device to initiate property mapping comprises:

determining, by the robotic device, that a current time and date satisfies a predetermined time and date that is stored in the one or more computer readable storage media as a condition that, when detected, triggers an instruction to be provided to the robotic device from the one or more computer readable storage media that instructs the robotic device to initiate property mapping.

7. The robotic device of claim 1, wherein the operations further comprise:

in response to determining that the image data depicts one or more markers, obtaining encoded data from the image data; and

decoding the encoded data obtained from the image data to obtain data that identifies an entry in the initial map modification data structure.

8. The robotic device of claim 7, wherein the operations further comprise:

determining, based on the entry in the initial map modification data structure, that the initial three-dimensional map should be adjusted to identify a glass window;

determining, based on the entry in the initial map modification data structure, dimensions of the glass window; and

labeling a particular portion of the initial three-dimensional map of the property comprises;

labeling a portion of the initial three-dimensional map where the glass window is located based on the determined dimensions of the glass window.

9. The robotic device of claim 7, wherein the operations further comprise:

determining, based on the entry in the initial map modification data structure, that the initial three-dimensional map should be adjusted to identify a glass wall;

determining, based on the entry in the initial map modification data structure, dimensions of the glass wall; and

labeling a particular portion of the initial three-dimensional map of the property comprises;

labeling a portion of the initial three-dimensional map where the glass wall is located based on the dimensions of the glass wall.

10. The robotic device of claim 7, wherein the operations further comprise:

determining, based on the entry in the initial map modification data structure, that the initial three-dimensional map should be adjusted to identify a doorway;

determining, based on the entry in the initial map modification data structure, dimensions of the doorway; and

labeling a particular portion of the initial three-dimensional map of the property comprises;

labeling a portion of the initial three-dimensional map where the doorway is located based on the dimensions of the doorway.

11. The robotic device of claim 1, wherein the operations further comprise:

obtaining a plurality of images of a portion of the property;

determining that each image of the plurality of images depicts a first marker in a different location of the property;

identifying a volumetric radius that envelopes each of the different locations of an object that includes the first marker; and

labeling a particular portion of the initial three-dimensional map of the property comprises;

labeling a portion of the initial three-dimensional map where the volumetric radius is located.

12. The robotic device of claim 1, wherein the operations further comprise:

obtaining a plurality of images of a portion of the property;

determining that each image of the plurality of images depicts a first marker of a blade of a ceiling fan in multiple different location;

identifying a volumetric radius that envelopes each of the different locations of the blade of the ceiling fan that includes the first marker; and

labeling a particular portion of the initial three-dimensional map of the property comprises;

labeling a portion of the initial three-dimensional map where volumetric radius is located.

13. The robotic device of claim 1, wherein the robotic device includes a quad-copter helicopter drone.

14. A method for generating a three-dimensional map of a property, the method comprising:

storing at least a portion of an initial map modification data structure;

receiving an instruction that is configured to instruct a robotic device to initiate property mapping;

determining that image data describing a portion of the property depicts one or more markers, wherein the image data is collected by a camera that is mounted to the robotic device;

in response to determining that the image data depicts one or more markers, determining a pose of the camera that is mounted to the robotic device;

updating the initial map modification data structure using the pose of the camera that is mounted to the robotic device;

determining, based on the initial map modification data structure that was updated, that an initial three-dimensional map of the property should be adjusted to include a label identifying one or more features of the property that is undetectable (i) by the camera that is mounted to the robotic device or (ii) by mapping sensors of the robotic device; and

labeling a particular portion of the initial three-dimensional map of the property using data stored in the updated map modification data structure.

15. The method of claim 14, the method further comprising:

determining, based on the entry in the initial map modification data structure, that the initial three-dimensional map should be adjusted to identify a glass wall;

determining, based on the entry in the initial map modification data structure, dimensions of the glass wall; and

wherein labeling a particular portion of the initial three-dimensional map comprises;

labeling a portion of the three-dimensional map where the glass wall is located based on the dimensions of the glass wall.

16. The method of claim 14, the method further comprising:

determining, based on the entry in the initial map modification data structure, that the initial three-dimensional map should be adjusted to identify a doorway;

determining, based on the entry in the initial map modification data structure, dimensions of the doorway; and

wherein labeling a particular portion of the initial three-dimensional map comprises;

labeling a portion of the three-dimensional map where the doorway is located based on dimensions of the doorway.

17. The method of claim 14, the method further comprising:

obtaining a plurality of images of a portion of the property;

determining that each image of the plurality of images depicts a first marker in a different location of the property;

identifying a volumetric radius that envelopes each of the different locations of an object that includes the first marker; and

wherein labeling a particular portion of the initial three-dimensional map comprises;

labeling a portion of the initial three-dimensional map where the volumetric radius is located.

18. At least one non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

storing at least a portion of an initial map modification data structure;

receiving an instruction that is configured to instruct the robotic device to initiate property mapping;

determining that image data describing a portion of the property depicts one or more markers, wherein the image data is collected by a camera that is mounted to the robotic device;

in response to determining that the image data depicts one or more markers, determining a pose of the camera that is mounted to the robotic device;

updating the initial map modification data structure using the pose of the camera that is mounted to the robotic device;

determining, based on the initial map modification data structure that was updated, that an initial three-dimensional map of the property should be adjusted to include a label identifying one or more features of the property that is undetectable (i) by the camera that is mounted to the robotic device or (ii) by mapping sensors of the robotic device; and

labeling a particular portion of the initial three-dimensional map of the property using data stored in the updated map modification data structure.

19. The non-transitory computer-readable storage media of claim 18, wherein the pose of the camera that is mounted to the robotic device includes a position and orientation of the camera.

20. The non-transitory computer-readable storage media of claim 18, wherein the initial three-dimensional map of the property was previously generated by the robotic device using mapping sensors mounted to the robotic device.