Article damage detection

• US 10,929,717 B2
• Filed: 05/29/2020
• Issued: 02/23/2021
• Est. Priority Date: 04/03/2018
• Status: Active Grant

• Current Assignee: Advanced New Technologies Company Limited (Ant Group Co., Ltd.)
• Original Assignee: Advanced New Technologies Company Limited (Ant Group Co., Ltd.)
• Inventors: Liu, Yongchao, Zhang, Haitao, Guo, Yufeng
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US16/888,568
• Publication Number:
  US 20200293830A1
• Time in Patent Office: 270 Days
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: Y02W 90/00 Enabling technologies or technologies with a potential or indirect contribution to greenhouse gas [GHG] emissions mitigation

Abstract

The present specification provides an article damage detection method, including: obtaining at least two images that are time sequentially related and show a detected article at different angles; and inputting the images to a detection model in time order, to determine a damage detection result, where the detection model includes a first sub-model and a second sub-model, the first sub-model identifies respective features of each image, a feature processing result of each image is input to the second sub-model, the second sub-model performs time series analysis on the feature processing result to determine the damage detection result, and the first sub-model and the second sub-model are obtained by performing joint training by using training samples labeled with article damage.

Claims

1. A method for evaluating damages to an article, comprising:

obtaining at least two images that are time sequentially related and show the article at different angles;

providing the at least two images as input to a detection model in time order, wherein the detection model comprises a deep convolutional neural network and a long short-term memory (LSTM) network that have been jointly trained on a set of training samples, each training sample comprising multiple training images associated with labels indicating respective article damage degrees of an article shown in the multiple training images;

processing the at least two images using the deep convolutional neural network to output a feature processing result for each image based on respective features identified from the image;

processing the feature processing result using the long short-term memory (LSTM) network to receive the feature processing results and to output a damage detection result based on performing time series analysis on the feature processing results, wherein the damage detection result comprises a classification result of each of one or more types of damage; and

obtaining, as output from the detection model, the damage detection result.

2. The method of claim 1, wherein the long short-term memory (LSTM) network is an attention-based LSTM network.

3. The method of claim 1, wherein the at least two images that are time sequentially related and show the article at different angles comprise at least one of:

photos of the article in motion captured by a stationary camera, videos of the article in motion recorded by a stationary camera, photos of the article in a stationary state captured by a moving camera, or videos of the article in a stationary state recorded by a moving camera.

4. The method of claim 1, wherein the classification result of each of one or more types of damage comprises a binary variable indicating whether the article bears the type of damage.

5. The method of claim 1, wherein processing the at least two images using the deep convolutional neural network to output the feature processing result comprises, for each image:

performing feature extraction, damage discovery, and feature fusion to determine the feature processing result.

6. The method of claim 1, wherein obtaining the at least two images comprises:

moving the article in accordance with a first set of movement instructions;

or

moving a camera in accordance with a second set of movement instructions.

7. The method of claim 1, wherein obtaining the at least two images further comprises:

illuminating the article in accordance with a set of illumination instructions.

8. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations for evaluating damages to an article, wherein the operations comprise:

obtaining at least two images that are time sequentially related and show the article at different angles;

providing the at least two images as input to a detection model in time order, wherein the detection model comprises a deep convolutional neural network and a long short-term memory (LSTM) network that have been jointly trained on a set of training samples, each training sample comprising multiple training images associated with labels indicating respective article damage degrees of an article shown in the multiple training images;

processing the at least two images using the deep convolutional neural network to output a feature processing result for each image based on respective features identified from the image;

processing the feature processing result using the long short-term memory (LSTM) network to receive the feature processing results and to output a damage detection result based on performing time series analysis on the feature processing results, wherein the damage detection result comprises a classification result of each of one or more types of damage; and

obtaining, as output from the detection model, the damage detection result.

9. The non-transitory, computer-readable medium of claim 8, wherein the long short-term memory (LSTM) network is an attention-based LSTM network.

10. The non-transitory, computer-readable medium of claim 8, wherein the at least two images that are time sequentially related and show the article at different angles comprise at least one of:

photos of the article in motion captured by a stationary camera, videos of the article in motion recorded by a stationary camera, photos of the article in a stationary state captured by a moving camera, or videos of the article in a stationary state recorded by a moving camera.

11. The non-transitory, computer-readable medium of claim 8, wherein the classification result of each of one or more types of damage comprises a binary variable indicating whether the article bears the type of damage.

12. The non-transitory, computer-readable medium of claim 8, wherein processing the at least two images using deep convolutional neural network to output the feature processing result comprises, for each image:

performing feature extraction, damage discovery, and feature fusion to determine the feature processing result.

13. A computer-implemented system, comprising:

one or more computers; and

one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations for evaluating damages to an article, wherein the operations comprise;

obtaining at least two images that are time sequentially related and show the article at different angles;

providing the at least two images as input to a detection model in time order, wherein the detection model comprises a deep convolutional neural network and a long short-term memory (LSTM) network that have been jointly trained on a set of training samples, each training sample comprising multiple training images associated with labels indicating respective article damage degrees of an article shown in the multiple training images;

processing the at least two images using the deep convolutional neural network to output a feature processing result for each image based on respective features identified from the image;

processing the feature processing result using the long short-term memory (LSTM) network to receive the feature processing results and to output a damage detection result based on performing time series analysis on the feature processing results, wherein the damage detection result comprises a classification result of each of one or more types of damage; and

obtaining, as output from the detection model, the damage detection result.

14. The computer-implemented system of claim 13, wherein the long short-term memory (LSTM) network is an attention-based LSTM network.

15. The computer-implemented system of claim 13, wherein the at least two images that are time sequentially related and show the article at different angles comprise at least one of:

photos of the article in motion captured by a stationary camera, videos of the article in motion recorded by a stationary camera, photos of the article in a stationary state captured by a moving camera, or videos of the article in a stationary state recorded by a moving camera.

16. The computer-implemented system of claim 13, wherein the classification result of each of one or more types of damage comprises a binary variable indicating whether the article bears the type of damage.

17. The computer-implemented system of claim 13, wherein processing the at least two images using the deep convolutional neural network to output the feature processing result comprises, for each image:

performing feature extraction, damage discovery, and feature fusion to determine the feature processing result.