Systems, methods, and apparatus for object classification based on localized information

• US 10,290,060 B2
• Filed: 12/23/2014
• Issued: 05/14/2019
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: The Travelers Indemnity Company (The Travelers Cos., Inc.)
• Original Assignee: The Travelers Indemnity Company (The Travelers Cos., Inc.)
• Inventors: Buss, Eamonn R., Durlin, Matthew C., Wilson, Brandon M., Gandhi, Nehal B.
• Primary Examiner(s): Niquette, Robert R
• Assistant Examiner(s): Nguyen, Liz P
• Application Number:
  US14/582,064
• Publication Number:
  US 20160180468A1
• Time in Patent Office: 1,603 Days
• Field of Search: 705 4
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/36 Indoor scenes

Abstract

Systems, apparatus, methods, and articles of manufacture provide for image analysis using localized information. In one embodiment, an object (e.g., content of a building or other type of property) may be classified by determining the type of object and determining classification information associated with previously classified objects of the same type and associated with the same geographical area.

Claims

1. A system for enhancing displayed real-time images of objects with augmented content on a mobile device, the system comprising:

a computerized mobile device comprising;

a processor,

a display device in communication with the processor,

a geolocation system in communication with the processor,

an image capture device in communication with the processor,

a wireless communications interface in communication with the processor, and

a computer-readable memory in communication with the processor, the computer-readable memory storing a property content classification mobile application that when executed by the processor directs the processor to;

determine, using the geolocation system, a location of a first property;

determine property information associated with the first property;

transmit, using the wireless communications interface, the location of the first property and the property information to a property content classification server;

receive, from the property content classification server, content information for at least one similar property;

receive, via the image capture device, a real-time image captured at the first property, the real-time image including at least one object;

display, using the display device, the real-time image;

augment the displayed real-time image using the display device to generate a user interface, the user interface including object image highlighting of at least one object represented in the real-time image;

transmit an electronic image file including a representation of the at least one object to the property content classification server;

receive, from the property content classification server, content classification information; and

determine property content information for the first property based on the content information for the at least one similar property and the content classification information;

and

the property content classification server, comprising;

a second processor;

a second computer-readable memory in communication with the second processor, the second computer-readable memory storing instructions that when executed by the second processor direct the second processor to;

receive, from the computerized mobile device, the location of the first property and the property information;

identify, based on the location of the first property and the property information, the at least one similar property;

determine the content information for the at least one similar property;

transmit the content information for the at least one similar property to the mobile device;

receive, from the mobile device, the electronic image file;

recognize, using an image recognition application, the at least one object represented in the electronic image file;

for each at least one recognized object, determine respective content classification information; and

transmit the content classification information to the mobile device.

2. The system of claim 1, wherein the displayed real-time image is augmented with data supplemental to the at least one object represented in the real-time image.

3. The system of claim 1, wherein the object image highlighting alters a portion of the user interface corresponding to the at least one object represented in the displayed real-time image.

4. The system of claim 1, wherein the object image highlighting presents an indication that a highlighted object displayed using the user interface has been identified as an object for which a specific electronic image file has previously been created.

5. The system of claim 1, wherein the object image highlighting presents an indication that a highlighted object displayed using the user interface has been identified as a potential object for which an image may be captured.

6. The system of claim 1, wherein the object image highlighting presents an indication of a first type if a corresponding highlighted object has been captured previously, and presents an indication of a second type if the corresponding highlighted object has not been captured previously.

7. The system of claim 1, wherein the object image highlighting presents an indication of a first type if a corresponding highlighted object has been classified previously, and presents an indication of a second type if the corresponding highlighted object has not been classified previously.

8. The system of claim 1, wherein the instructions of the property content classification server when executed by the second processor further direct the second processor to:

identify a respective object type of each of the at least one object represented in the electronic image file, by comparing the electronic image file to stored object image data using the image recognition application.

9. The system of claim 1, wherein the at least one object comprises one of:

a shelving unit,

an electronic appliance,

a cash register,

a finished product,

product inventory,

a vehicle,

an office appliance, and

a building fixture.

10. The system of claim 1, wherein the instructions of the property content classification server when executed by the second processor further direct the second processor to:

generate a map of the first property based on the respective content classification information for each at least one recognized object.

11. The system of claim 1, wherein the instructions of the property content classification server when executed by the second processor further direct the second processor to:

transmit the content classification information to a mapping application system for generating a map of the property.

12. The system of claim 1, wherein the instructions of the property content classification server when executed by the second processor further direct the second processor to:

analyze a security status of the at least one recognized object represented in the electronic image file based on the respective content classification information.

13. The system of claim 1, wherein the instructions of the property content classification server when executed by the second processor further direct the second processor to:

transmit the content classification information to a security application system for analyzing a security status of the at least one recognized object represented in the electronic image file.

20. The system of claim 1, wherein the object image highlighting comprises a representation of a line around an object represented in the displayed real-time image.

21. The system of claim 1, wherein the object image highlighting comprises highlighting around an object represented in the displayed real-time image.

22. The system of claim 1, wherein augmenting the displayed real-time image using the display device to generate a user interface, the user interface including object image highlighting of at least one object represented in the real-time image, comprises:

using the display device to generate the user interface to display the object image highlighting of the at least one object in combination with the displayed real-time image.

14. A mobile device providing enhanced displayed real-time images of objects with augmented content, the mobile device comprising:

a processor,

a display device in communication with the processor,

a geolocation system in communication with the processor,

an image capture device in communication with the processor,

a wireless communications interface in communication with the processor, and

a computer-readable memory in communication with the processor, the computer-readable memory storing a property content classification mobile application that when executed by the processor directs the processor to;

determine, using the geolocation system, a location of a first property;

determine property information associated with the first property;

transmit, using the wireless communications interface, the location of the first property and the property information to a property content classification server;

receive, from the property content classification server, content information for at least one similar property;

receive, via the image capture device, a real-time image captured at the first property, the real-time image including at least one object;

display, using the display device, the real-time image;

augment the displayed real-time image using the display device to generate a user interface, the user interface including object image highlighting of at least one object represented in the real-time image;

transmit an electronic image file including a representation of the at least one object to the property content classification server;

receive, from the property content classification server, content classification information; and

determine property content information for the first property based on the content information for the at least one similar property and the content classification information.

15. The mobile device of claim 14, wherein the displayed real-time image is augmented with data supplemental to the at least one object represented in the real-time image.

16. The mobile device of claim 14, wherein the object image highlighting presents an indication that a highlighted object displayed using the user interface has been identified as an object for which a specific electronic image file has previously been created.

17. The mobile device of claim 14, wherein the object image highlighting presents an indication that a highlighted object displayed using the user interface has been identified as a potential object for which an image may be captured.

18. The mobile device of claim 14, wherein the object image highlighting presents an indication of a first type if a corresponding highlighted object has been captured previously, and presents an indication of a second type if the corresponding highlighted object has not been captured previously.

19. The mobile device of claim 14, wherein the object image highlighting presents an indication of a first type if a corresponding highlighted object has been classified previously, and presents an indication of a second type if the corresponding highlighted object has not been classified previously.

23. The mobile device of claim 14, wherein the object image highlighting alters a portion of the user interface corresponding to the at least one object represented in the displayed real-time image.

25. The mobile device of claim 14, wherein the object image highlighting comprises highlighting around an object represented in the displayed real-time image.

26. The mobile device of claim 14, wherein augmenting the displayed real-time image using the display device to generate a user interface, the user interface including object image highlighting of at least one object represented in the real-time image, comprises:

using the display device to generate the user interface to display the object image highlighting of the at least one object in combination with the displayed real-time image.

24. A mobile device providing enhanced displayed real-time images of objects with augmented content, the mobile device comprising:

a processor,

a display device in communication with the processor,

a geolocation system in communication with the processor,

an image capture device in communication with the processor,

a wireless communications interface in communication with the processor, and

a computer-readable memory in communication with the processor, the computer-readable memory storing a property content classification mobile application that when executed by the processor directs the processor to;

determine, using the geolocation system, a location of a first property;

determine property information associated with the first property;

transmit, using the wireless communications interface, the location of the first property and the property information to a property content classification server;

receive, from the property content classification server, content information for at least one similar property;

receive, via the image capture device, a real-time image captured at the first property, the real-time image including at least one object;

display, using the display device, the real-time image;

augment the displayed real-time image using the display device to generate a user interface, the user interface including object image highlighting of at least one object represented in the real-time image;

transmit an electronic image file including a representation of the at least one object to the property content classification server;

receive, from the property content classification server, content classification information; and

determine property content information for the first property based on the content information for the at least one similar property and the content classification information,

wherein the object image highlighting comprises a representation of a line around an object represented in the displayed real-time image.