System, method, and apparatus for detection of damages on surfaces

• US 10,255,521 B2
• Filed: 12/12/2016
• Issued: 04/09/2019
• Est. Priority Date: 12/12/2016
• Status: Active Grant

• Current Assignee: Advanced Lab For Automotive Ventures, LLC
• Original Assignee: Jack Cooper Logistics, LLC (Jack Cooper Enterprises, Inc.)
• Inventors: Amico, Andrea, Prabhushankar, Mohit
• Primary Examiner(s): Jiang, Zaihan
• Application Number:
  US15/375,895
• Publication Number:
  US 20180165541A1
• Time in Patent Office: 848 Days
• Field of Search: 348128
• CPC Class Codes: G01N 2021/8893 providing a video image and a processed signal for helping visual decision
• CPC Class Codes: G01N 21/8851 Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges G01N21/8806 and G01N21/93 - G01N21/95692 take precedence; optical measurement of dimensions G01B11/00; optical scanning G02B26/10; image transformation G06T3/00; computerised image enhancement G06T5/00; image processing per se for flaw detection G06T7/0002
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30156 Vehicle coating
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: H04N 1/6002 Corrections within particular colour systems
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/841 : to modify gamut
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 9/67 : for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

A portable computing device equipped with at least one image capture device and/or a light source captures an image (or a video) of a portion of a surface of interest having the damage that is exposed to a light from the light source. The portable computing device converts the image to an output image that highlights the damage. If the damage is a dent, the image is converted to a false color image using a saliency algorithm. If the damage is a scratch, the image is converted to a colorspace stretched image using color stretching algorithms. The size of the damage is determined by capturing an image of a ruler placed adjacent to the damage and the portion of surface of interest having the damage. The ruler is then removed from the image. The resulting image is converted to the output image. The ruler is added to the output image.

Claims

1. A system comprising:

a handheld portable computing device that comprises at least one image capture device, wherein the portable computing device is configured to;

capture, using the at least one image capture device, one or more images of a portion of a surface of interest that has a damage;

receive a user input identifying a type of the damage;

responsive to identifying the type of damage as a dent based on the user input,

generate, using the one or more captured images, a final image having an optimal reflection pattern;

blur the final image by removing high frequency edge components of the final image; and

generate, using the blurred image and a saliency algorithm, an output image in which less salient features of the blurred image are damped and more salient features of the blurred image are enhanced, wherein the less salient features include reflections from the surface of interest and the more salient features include the dent on the surface of interest; and

present the output image via a display of the portable computing device.

2. The system of claim 1, wherein the surface of interest is a reflective surface.

3. The system of claim 1, wherein the portable computing device further comprises an assistive light source, wherein light from the assistive light source is directed towards the portion of the surface that has the damage prior to capturing the one or more images.

4. The system of claim 1, wherein to capture the one or more images of the portion of the surface of interest that has the damage, the portable computing device is configured to:

process an image feed received by the image capture device to compensate for variable lighting conditions including natural light and artificial light;

render the processed image feed on the display of the portable computing device; and

generate and overlay one or more graphical cues on the rendered image feed in the display, wherein the one or more graphical cues are configured to provide guidance on how to position the image capture device to optimize reflection patterns in the one or more images.

5. The system of claim 1, wherein the output image is a false color image that highlights the dent.

6. The system of claim 1, wherein to generate the output image using the blurred image and the saliency algorithm, the portable computing device is configured to:

generate a saliency map comprising saliency values of each pixel of the blurred image by applying the saliency algorithm on the blurred image;

normalize the saliency values of each pixel in the saliency map; and

multiply the normalized saliency values of each pixel that exceed a threshold saliency value with corresponding pixels of the blurred image to generate the output image,

wherein the output image is a false color image.

7. The system of claim 1, wherein when the type of damage is a deep scratch, the portable computing device is configured to transform the one or more captured images to a colorspace stretched image that highlights the deep scratch.

8. A system comprising:

a handheld portable computing device that comprises at least one image capture device, wherein the portable computing device is configured to;

capture, using the at least one image capture device, one or more images of a portion of a surface of interest that has a damage;

receive a user input identifying a type of the damage;

responsive to identifying the type of damage as a deep scratch based on the user input, a deep scratch detection engine of the handheld portable computing device is configured to;

transform the one or more captured images to a colorspace stretched image, wherein to transform the one or more capture images to a colorspace stretched image, a pixel density enhancing engine of the handheld portable computing device is configured to generate a final image having an enhanced resolution using the one or more captured images; and

a colorspace stretching engine of the handheld portable device is configured to stretch a colorspace of the final image to define an edge of the deep scratch in more detail within the final image;

process, by a color gradient detection engine, the colorspace stretched image to determine a color gradient in the colorspace stretched image;

determine that pixels of the colorspace stretched image that correspond to the color gradient are within a threshold narrow band of pixels; and

responsively, identify the pixels corresponding to the color gradient as the deep scratch; and

present the colorspace stretched image via a display of the portable computing device.

24. The system of claim 8:

wherein to generate the final image having the enhanced resolution, the pixel density enhancing engine is configured to;

determine if a pixel density of the one or more captured images is less than a threshold pixel density value, and

responsive to a determining that the pixel density of the one or more captured images is less than the threshold pixel density value, superimpose the one or more captured images to generate the final image, and

wherein to identify the pixels corresponding to the color gradient as the deep scratch,

the deep scratch detection engine is configured to;

determine that the pixels of the colorspace stretched image that correspond to the color gradient are aligned in a line.

9. A system for detecting and sizing a damage on a reflective surface comprising:

a portable computing device that comprises a video capture device, an assistive light source, a processor, a display, and a wireless transceiver, wherein the portable computing device is configured to;

capture, using the video capture device, a video of a portion of the reflective surface having the damage and a ruler positioned adjacent the damage, wherein the ruler and the portion of the reflective surface having the damage are exposed to a light from the assistive light source;

receive a user input identifying a type of the damage; and

transmit the user input and the video; and

a server that is communicably coupled to the portable computing device via a network, wherein the server configured to;

receive the user input and the video from the wireless transceiver of the portable computing device;

responsive to receiving the video and based on the type of the damage, generate a set of final images from the video;

identify the ruler in the set of final images based on a geometry and/or a color of the ruler;

remove pixels corresponding to the ruler from the set of final images to form a modified set of final images;

based on the type of the damage, transform the modified set of final images to an output image that highlights the damage on the reflective surface;

add the pixels corresponding to the ruler to the output image;

calculate the size of the damage highlighted in the output image comprising the ruler using the ruler as reference; and

transmitting the output image comprising the ruler and the calculated size of the damage to the portable computing device for visual representation on the display.

10. The system of claim 9, wherein to capture the video of the portion of the reflective surface having the damage, the portable computing device is configured to:

process an image feed received by the at least one video capture device to compensate for variable lighting conditions including an intensity of a natural light and an intensity of an artificial light;

render the processed video feed on the display of the portable computing device; and

generate and overlay one or more graphical cues on the rendered video feed in the display, wherein the one or more graphical cues are configured to provide guidance on how to position the video capture device to optimize reflection patterns in the video.

11. The system of claim 9, wherein when the type of damage is a dent, a dent detection engine of the server is configured to transform the modified set of final images to a false color image that highlights the dent.

12. The system of claim 11, wherein to transform the modified set of final images to a false color image that highlights the dent, the dent detection engine is configured to:

blur the modified set of final images by removing high frequency edge components;

generate a saliency map comprising saliency values of each pixel of the modified set of final images that are blurred by applying a saliency algorithm on the blurred set of final images;

normalize the saliency values of each pixel in the saliency map;

multiply the normalized saliency values of each pixel of each final image of the modified set of final images that are blurred that exceeds a threshold saliency value with the corresponding pixels of the respective final image to generate a set of false color images that highlights the dent; and

select the false color image from the set of false color images for presentation.

13. The system of claim 12, wherein in each false color image, less salient features are damped and more salient features are enhanced, and wherein the less salient features include reflections from the surface of interest and the more salient features include the dent on the surface of interest.

14. The system of claim 9, wherein when the type of damage is a deep scratch, to generate the set of final images, a deep scratch detection engine of the server is configured to process one or more image frames of the video by superimposing each image frame of the one or more image frames with its neighboring image frames to create the set of final images having enhanced resolution.

15. The system of claim 9, wherein when the type of damage is a deep scratch, a deep scratch detection engine of the server is configured to transform the modified set of final images to a colorspace stretched image that highlights the deep scratch.

16. The system of claim 15, wherein to transform the modified set of images to a colorspace stretched image that highlights the deep scratch, the deep scratch detection engine is configured to:

stretching a colorspace of each modified final image to generate a set of colorspace stretched images, each colorspace stretched image defining an edge of the deep scratch in more detail;

processing each colorspace stretched image to determine a color gradient in the respective colorspace stretched image;

determining that pixels of each colorspace stretched image corresponding to the color gradient are within a threshold narrow band of pixels and that the pixels are aligned in a line;

identifying the pixels corresponding to the color gradient as a deep scratch; and

selecting the colorspace stretched image from the set of colorspace stretched images.

17. A portable computing device that is configured to detect and size a damage on a surface of interest, the portable computing device comprising:

a flash that is configured to direct artificial light towards a portion of the surface of interest that has the damage and a ruler that is placed adjacent the damage;

a camera that is configured to capture one or more images of the portion of the surface of interest that has the damage along with the ruler that is placed adjacent to the damage;

a dent detection engine that is configured to;

create, using the one or more captured images, a final image having an optimal reflection pattern; and

responsive to identifying a ruler in the final image based on a geometry and/or a color of the ruler and removing pixels corresponding to the ruler from the final image by a damage sizing engine of the portable computing device, convert the final image to a false color image that highlights the dent;

a deep scratch detection engine that is configured to;

create another final image having an enhanced resolution by superimposing the one or more captured images;

responsive to identifying a ruler in the other final image based on a geometry and/or a color of the ruler and removing pixels corresponding to the ruler from the other final image by a damage sizing engine of the portable computing device, convert the other final image to a colorspace stretched image that highlights the deep scratch,

wherein the damage sizing engine is configured to add the pixels corresponding to the ruler to at least one of the false color image and the colorspace stretched image, and wherein the damage sizing engine is configured to calculate a size of at least one of the dent and the deep scratch using the ruler as a reference;

a display that is configured to visually present at least one of the false color image, the colorspace stretched image, the size of the dent, and the size of the deep scratch based on a type of the damage.

18. The portable computing device of claim 17, wherein the surface of interest is a reflective surface.

19. The portable computing device of claim 17, wherein to convert the final image to the false color image, the dent detection engine is configured to:

blur the final image by removing high frequency edge components of the final image;

generate a saliency map comprising saliency values of each pixel of the blurred image by applying a saliency algorithm on the blurred image;

normalize the saliency values of each pixel in the saliency map; and

multiply the normalized saliency values of each pixel that exceed a certain threshold with corresponding pixels of the blurred image to generate the false color image in which less salient features of the blurred image are damped and more salient features of the blurred image are enhanced, wherein the less salient features include reflections from the surface of interest and the more salient features include the dent on the surface of interest.

20. The portable computing device of claim 17, wherein to convert the other final image to the colorspace stretched image, the deep scratch detection engine is configured to:

stretch a colorspace of the final image to define an edge of the deep scratch in more detail within the final image;

process the colorspace stretched image to determine a color gradient in the colorspace stretched image;

determine that pixels of the colorspace stretched image corresponding to color gradient are within a threshold narrow band of pixels and that the pixels are aligned in a line; and

identify the pixels corresponding to the color gradient as a deep scratch.

21. A system comprising:

a handheld portable computing device that comprises at least one image capture device, wherein the portable computing device is configured to;

capture, using the at least one image capture device, one or more images of a portion of a surface of interest that has a damage;

process the one or more captured images by a dent detection engine and a deep scratch detection engine of the portable computing device in parallel;

convert, by the dent detection engine using a saliency algorithm, the one or more captured images to a false color image in which less salient features are damped and more salient features are enhanced, wherein the less salient features include reflections from the surface of interest and the more salient features include the damage on the surface of interest;

convert, by the deep scratch detection engine, the one or more captured images to a colorspace stretched image;

determine a type of the damage based on whether the dent detection engine or the deep scratch detection engine provides a positive identification of the damage;

wherein when the dent detection engine provides the positive identification of the damage, the damage is a dent, and wherein when the deep scratch detection engine provides the positive identification of the damage, the damage is a deep scratch;

present at least one of the false color image and the colorspace stretched image via a display of the portable computing device.

22. The system of claim 21, wherein to convert the one or more captured images to the false color image, the dent detection engine is configured to:

generate, using the one or more captured images, a final image having an optimal reflection pattern;

blur the final image by removing high frequency edge components of the final image;

generate a saliency map comprising saliency values of each pixel of the blurred image by applying a saliency algorithm on the blurred image;

normalize the saliency values of each pixel in the saliency map; and

multiply the normalized saliency values of each pixel that exceed a certain threshold with the corresponding pixels of the blurred image to generate the false color image in which less salient features of the blurred image are damped and more salient features of the blurred image are enhanced, wherein the less salient features include reflections from the surface of interest and the more salient features include the dent on the surface of interest.

23. The portable computing device of claim 21, wherein to convert the one or more captured images to the colorspace stretched image, the deep scratch detection engine is configured to:

determine if a pixel density of the one or more captured images is less than a threshold pixel density value;

responsive to a determining that the pixel density of the one or more captured images is less than the threshold pixel density value, creating a final image having an enhanced resolution by superimposing the one or more captured images;

responsively, stretching a colorspace of the final image to define an edge of the deep scratch in more detail within the final image;

processing the colorspace stretched image to determine a color gradient in the colorspace stretched image;

determining that pixels of the colorspace stretched image corresponding to the color gradient are within a threshold narrow band of pixels and that the pixels are aligned in a line; and

identifying the pixels corresponding to the color gradient as a deep scratch.