Directed image capture

• US 10,681,264 B2
• Filed: 07/20/2018
• Issued: 06/09/2020
• Est. Priority Date: 05/29/2015
• Status: Active Grant

• Current Assignee: Hover Inc. (Tucows Incorporated)
• Original Assignee: Hover Inc. (Tucows Incorporated)
• Inventors: Castillo, William, Altman, Adam J., Pavlidis, Ioannis, Sahu, Sarthak, Upendran, Manish
• Primary Examiner(s): Broome, Said
• Application Number:
  US16/040,663
• Publication Number:
  US 20180332217A1
• Time in Patent Office: 690 Days
• Field of Search: None
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 1/2104 for one or a few pictures
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A process is provided for graphically guiding a user of a capture device (e.g., smartphone) to more accurately capture a series of images of a building. Images are captured as the picture taker moves around the building—taking a plurality (e.g., 4-16) of images from multiple angles and distances. Before capturing an image, a quality of the image may be determined to prevent low quality images from being captured or to provide instructions on how to improve the quality of the image capture. The series of captured images are uploaded to an image processing system to generate a 3D building model that is returned to the user. The returned 3D building model may incorporate scaled measurements of building architectural elements and may include a dataset of measurements for one or more architectural elements such as siding (e.g., aluminum, vinyl, wood, brick and/or paint), windows, doors or roofing.

Claims

1. A non-transitory computer readable medium storing instructions that, upon execution, cause a computer system to perform operations including:

retrieving, from an image capture device, an image, the image including at least a portion of a subject building;

determining if the image is usable in creating a multi-dimensional building model by;

calculating a percentage of faç

ade pixels present within the image, wherein the faç

ade includes any side of the subject building;

determining a distance of the faç

ade pixels from a centroid of the image;

determining obfuscated faç

ade areas;

outputting the image to a quality classifier, wherein the quality classifier calculates a ranked quality of the image based on or more of;

the percentage of faç

ade pixels present within the image, the distance of the faç

ade pixels from a centroid of the image, or the obfuscated faç

ade areas; and

feeding an indication of the ranked quality to the image capture device along with instructions based on the indication to direct one or more additional image captures.

2. The non-transitory computer readable medium of claim 1, wherein the image is displayed on a viewfinder of the image capture device.

3. The non-transitory computer readable medium of claim 2, wherein the determining if the image is usable in creating a multi-dimensional building model is at least partially performed:

locally on the image capture device or remotely on an image processing system.

4. The non-transitory computer readable medium of claim 3, wherein the locally on the image capture device includes at least partial processing using an image capture device app.

5. The non-transitory computer readable medium of claim 1, wherein the distance of the faç

ade pixels from a centroid of the image is recognized by any of;

planes of pixels, vanishing lines, or identification of one or more architectural features.

6. The non-transitory computer readable medium of claim 5, wherein the architectural features include one or more of:

roofs windows, siding, or doors.

7. The non-transitory computer readable medium of claim 1, further comprises setting the percentage of faç

ade pixels to be above a threshold for a minimum quality image determination.

8. The non-transitory computer readable medium of claim 1, wherein the image comprises one or more of:

other visible structures, parts of visible structures, or building interior images.

9. The non-transitory computer readable medium of claim 1, wherein one or more training sessions are aggregated for the quality classifier using a series of sample building model images to properly assesses each image for quality based on previously defined quality levels.

10. The non-transitory computer readable medium of claim 9, wherein during the assessing of each image for quality includes weighting low-level issues including lighting or obfuscation less than a partial building capture or off-center issues.

11. The non-transitory computer readable medium of claim 9, wherein the quality classifier includes one or more of:

machine learning, neural learning, hierarchical learning, or deep learning.

12. The non-transitory computer readable medium of claim 9, wherein the training sessions include aggregated weighted ratings producing an overall ranking of quality.

13. The non-transitory computer readable medium of claim 1, wherein a low score of a number of pixels a far distance from the centroid is an indication of a higher quality image.

14. The non-transitory computer readable medium of claim 1, wherein the determining if the image is usable in creating a multi-dimensional building model further comprises determining a number of times a sky touches a ground, wherein a higher quality image occurs when the sky touches a ground twice.

15. The non-transitory computer readable medium of claim 14, wherein the determining if the image is usable in creating a multi-dimensional building model further comprises determining a number of sky pixels toughing a roof of the centroid of the image;

wherein a higher quality image occurs when all pixels of a roof located within the image touch the sky without any of;

obstruction, obfuscation or taller unassociated buildings in a background of the image.

16. The non-transitory computer readable medium of claim 1, wherein the indication includes any of:

a graphic, a sound or color indicator.

17. The non-transitory computer readable medium of claim 1, wherein the instructions include feedback to move the image capture device to a new location.

18. A non-transitory computer readable medium storing instructions that, upon execution, cause a computer system to perform operations including:

retrieving, on an image capture device, a potential building model image;

determining if the potential building model image is usable in creating a multi-dimensional building model by outputting the potential building model image to a quality classifier that is configured to;

calculate a percentage of faç

ade pixels identified as architectural features present within the potential building model image;

determine a distance of faç

ade pixels identified as architectural features from a centroid of the potential building model image;

determine obfuscated faç

ade areas;

calculate an image quality classification of the potential building model image based on or more of;

the percentage of faç

ade pixels identified as architectural features, the distance of faç

ade pixels identified as architectural features from the centroid of the potential building model image or the obfuscated faç

ade areas; and

feeding an indication of the image quality classification to the image capture device along with instructions based on the indication to direct one or more additional image captures.

19. A non-transitory computer readable medium storing instructions that, upon execution, cause a computer system to perform operations including:

retrieving, on an image capture device, a potential building model image;

determining if the potential building model image is usable in creating a multi- dimensional building model by outputting the potential building model image to a quality classifier that is configured to;

calculate a percentage of faç

ade pixels identified as an architectural feature present within the potential building model image;

determine a distance of the faç

ade pixels identified as an architectural feature from a centroid of the potential building model image;

determine a number of the faç

ade pixels identified as an architectural feature and pixels not identified as architectural feature pixels;

determine obfuscated faç

ade areas;

calculate an image quality classification of the potential building model image based on or more of;

the percentage of faç

ade pixels identified as architectural features, distance of faç

ade pixels identified as architectural features from the centroid of the centroid of the potential building model image or the obfuscated faç

ade areas; and

feeding an indication of the image quality classification to the image capture device along with instructions based on the indication to direct one or more additional image captures.

20. The non-transitory computer readable medium of claim 19, wherein the determining a number of the pixels identified as an architectural feature and pixels not identified as architectural feature pixels includes distinguishing between any of:

faç

ade pixels, window pixels, door pixels, roof pixels, sky pixels, or ground pixels.