Automatic annotation for vehicle damage

• US 10,846,322 B1
• Filed: 02/10/2020
• Issued: 11/24/2020
• Est. Priority Date: 02/10/2020
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Ghamsari, Avid, Dagley, Geoffrey, Price, Micah, Tang, Qiaochu
• Primary Examiner(s): Ly, Anh
• Application Number:
  US16/786,695
• Time in Patent Office: 288 Days
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/45 Clustering; Classification
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

Aspects described herein may allow an automated generation of an interactive multimedia content with annotations showing vehicle damage. In one method, a server may receive vehicle-specific identifying information of a vehicle. Image sensors may capture multimedia content showing aspects associated with exterior regions of the vehicle, and may send the multimedia content to the server. For each of the exterior regions of the vehicle, the server may determine, using a trained classification model, instances of damage. Furthermore, the server may generate an interactive multimedia content that shows images with annotations indicating instances of damage. The interactive multimedia content may be displayed via a user interface.

Claims

1. A system comprising:

a first device comprising;

one or more first processors; and

a first memory storing first instructions that, when executed by the one or more first processors, cause the first device to;

receive vehicle-specific identifying information of a vehicle;

receive, from a second device, multimedia content showing one or more aspects of the vehicle, wherein the one or more aspects are associated with one or more exterior regions of the vehicle;

determine from the received multimedia content, image data corresponding to each of the one or more aspects of the vehicle;

for each of the one or more aspects of the vehicle,

determine, using a trained classification model, one or more instances of damage to the aspect;

annotate, based on the one or more instances of damage, the image data corresponding to the aspect to indicate the one or more instances of damage; and

associate, to the one or more instances of damage, any accident-specific data determined from an accident report of the vehicle; and

generate an interactive multimedia content associated with the vehicle, wherein the interactive multimedia content comprises the annotated image data indicating the one or more instances of damage and the accident-specific data associated with the one or more instances of damage; and

the second device comprising;

one or more image sensors;

one or more second processors; and

a second memory storing second instructions that, when executed by the one or more second processors, cause the second device to;

detect, using the one or more image sensors, that the vehicle is entering a predetermined area associated with the second device;

generate, using the one or more image sensors, the multimedia content showing the one or more aspects of the vehicle; and

send the multimedia content to the first device.

2. The system of claim 1, wherein the first instructions, when executed by the one or more first processors, cause the first device to:

prior to the determining the one or more instances of damage,

receive, for each of a plurality of aspects of reference vehicles, a first set of images and a second set of images, wherein the first set of images shows a first level of damage to the aspect of the reference vehicle, and wherein the second set of images shows a second level of damage to the aspect of the reference vehicle;

train, using the first set of images and the second set of images, and based on detecting the first level of damage and the second level of damage, a classification model for the vehicle; and

store the trained classification model in the first memory.

3. The system of claim 2, wherein the first instructions, when executed by the one or more first processors, cause the first device to:

prior to the determining the one or more instances of damage,

identify, based on the vehicle-specific identifying information of the vehicle, a trained classification model from a plurality of trained classification models stored in the first memory.

4. The system of claim 1, wherein the first instructions, when executed by the one or more first processors, cause the first device to determine the image data by:

inputting, into a trained machine learning model, the received multimedia content to detect the one or more exterior regions of the vehicle.

5. The system of claim 4, wherein the first instructions, when executed by the one or more first processors, cause the first device to:

prior to the inputting the received multimedia content into the trained machine learning model,

receive, for each of the one or more aspects of a plurality of reference vehicles, training data comprising;

multimedia content showing the aspect of the plurality of reference vehicles, and

an identification of image data of the multimedia content corresponding to the aspect;

train, by learned recognition of the image data corresponding to each of the one or more aspects from the multimedia content, a machine learning model; and

store the trained machine learning model in the first memory.

6. The system of claim 5, wherein each of the plurality of reference vehicles shares one or more vehicle-specific identifying information with the vehicle.

7. The system of claim 1, wherein the first instructions, when executed by the one or more first processors, cause the first device to:

receive, from the second device, a time of generation of the multimedia content showing the one or more aspects of the vehicle, and

annotate, based on the time of generation of the multimedia content, the image data to further indicate a time of detecting the one or more instances of damage.

8. The system of claim 7, wherein the time of generation of the multimedia content is received as metadata of the received multimedia content showing the one or more aspects of the vehicle.

9. The system of claim 1, wherein the first instructions, when executed by the one or more first processors, cause the first device to:

prompt an input comprising further information associated with the one or more instances of damage;

receive, in response to the prompting, the input comprising the further information associated with the one or more instances of damage; and

generate a revised interactive multimedia content of the vehicle, wherein the revised interactive multimedia content comprises the annotated image data indicating the one or more instances of damage and the further information associated with the one or more instances of damage.

10. The system of claim 1, wherein the first instructions, when executed by the one or more first processors, cause the first device to:

send, to a third device, the interactive multimedia content of the vehicle; and

wherein the system further comprises;

the third device comprising;

a user interface;

a display screen one or more third processors; and

a third memory storing second instructions that, when executed by the one or more third processors, cause the third device to;

receive, from the first device, the interactive multimedia content of the vehicle

receive a request to view the interactive multimedia content of the vehicle;

display the interactive multimedia content of the vehicle on the display screen; and

enable, through the user interface, an interaction with the interactive multimedia content to display the one or more aspects of the vehicle.

11. A method comprising:

receiving vehicle-specific identifying information of a vehicle;

generating, using one or more image sensors, multimedia content showing one or more aspects of the vehicle, wherein the one or more aspects are associated with one or more exterior regions of the vehicle;

determining from the generated multimedia content, image data corresponding to each of the one or more aspects of the vehicle;

for each of the one or more exterior regions of the vehicle,

determining, using a trained classification model, any one or more instances of damage to the aspect;

annotating, based on the one or more instances of damage, the image data corresponding to the aspect to reflect an indication of the one or more instances of damage; and

associating, to the one or more instances of damage, any accident-specific data determined from an accident report of the vehicle;

generating an interactive multimedia content associated with the vehicle, wherein the interactive multimedia content comprises the annotated image data indicating the one or more instances of damage and the accident-specific data associated with the one or more instances of damage; and

displaying, through a user interface and responsive to a request, the interactive multimedia content of the vehicle.

12. The method of claim 11, further comprising:

prior to the determining the any one or more instances of damage of the aspect,

receiving, for each of a plurality of exterior regions of a plurality of reference vehicles, a first set of images of the aspect showing no instances of damage, and a second set of images of the aspect showing one or more instances of damage;

training a classification model using the first set of images and the second set of images for each of the one or more aspects of the reference vehicles, wherein the classification model is trained to detect one or more instances of damage on an aspect of the vehicle; and

storing the trained classification model.

13. The method of claim 11, wherein the determining, from the generated multimedia content, the image data corresponding to each of the one or more aspects of the vehicle further comprises:

inputting the generated multimedia content into a trained machine learning model to detect the one or more aspects of the vehicle.

14. The method of claim 13, further comprising:

prior to the inputting the generated multimedia content into the trained machine learning model,

receiving, for each of the one or more aspects of a plurality of reference vehicles, training data comprising;

multimedia content showing the aspect of the plurality of reference vehicles, and

an identification of image data of the multimedia content corresponding to the aspect;

training a machine learning model by learning recognition of the image data corresponding to each of the one or more aspects from the received multimedia content; and

storing the trained machine learning model.

15. The method of claim 11, further comprising:

receiving, as metadata of the generated multimedia content showing the one or more aspects of the vehicle, a time of generation of the multimedia content; and

annotating, based on the time of generation of the multimedia content, the image data to further indicate a time of detecting the one or more instances of damage.

16. The method of claim 11, further comprising:

prompting an input comprising further information associated with the one or more instances of damage;

receiving, in response to the prompting, the input comprising the further information associated with the one or more instances of damage; and

generating a revised interactive multimedia content of the vehicle, wherein the revised interactive multimedia content comprises the annotated image data reflecting the indication of the one or more instances of damage and the further information associated with the one or more instances of damage.

17. One or more non-transitory storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:

a training phase comprising;

receiving, for each of one or more aspects of a plurality of reference vehicles, a first training data comprising

multimedia content comprising image data showing the aspect of the plurality of reference vehicles, and

an identification of each aspect of the one or more aspects shown by the image data, wherein the one or more aspects of the plurality of reference vehicles are associated with one or more exterior regions of the plurality of reference vehicles; and

training, using learned recognition of each aspect, and based on the image data, a machine learning model from the multimedia content;

storing the trained machine learning model for an application phase;

receiving, for each of a plurality of exterior regions of the plurality of reference vehicles, a second training data comprising;

a first set of images showing a first level of damage to the aspect of the reference vehicles, and

a second set of images showing a second level of damage to the aspect of the reference vehicles;

training a classification model using the first set of images and the second set of images for each of the one or more aspects of the reference vehicles, wherein the classification model is trained to detect one or more instances of damage on an aspect of a vehicle different from the reference vehicles; and

storing the trained classification model for the application phase; and

the application phase comprising;

receiving vehicle-specific identifying information of the vehicle;

generating, using one or more image sensors, multimedia content showing one or more aspects of the vehicle, wherein the one or more aspects are associated with one or more exterior regions of the vehicle;

inputting the generated multimedia content into the trained machine learning model to detect the one or more aspects of the vehicle;

determining image data corresponding to each of the one or more aspects of the vehicle;

for each of the one or more aspects of the vehicle,

determining, using the trained classification model, one or more instances of damage to the aspect;

annotating, based on the determined one or more instances of damage, the image data corresponding to the aspect to indicate the one or more instances of damage; and

associating, to the one or more instances of damage, any accident-specific data determined from an accident report of the vehicle;

generating an interactive multimedia content associated with the vehicle, wherein the interactive multimedia content comprises the annotated image data indicating the one or more instances of damage and accident-specific data associated with the one or more instances of damage; and

sending, to a device for display and responsive to a request, the interactive multimedia content of the vehicle.

18. The one or more non-transitory storage media of claim 17, storing instructions that, when executed by the one or more processors, cause the one or more processors to perform steps further comprising:

receiving, as metadata of the generated multimedia content showing the one or more aspects of the vehicle, a time of generation of the multimedia content; and

annotating, based on the time of generation of the multimedia content, the image data to further indicate a time of detecting the one or more instances of damage.