Categorizing captured images for subsequent search

• US 9,811,536 B2
• Filed: 01/27/2016
• Issued: 11/07/2017
• Est. Priority Date: 01/27/2016
• Status: Active Grant

• Current Assignee: Dell Products LP (Dell Technologies Inc.)
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Morris, Cheryl, Silberman, Gabriel M.
• Primary Examiner(s): PARK, SOO JIN
• Application Number:
  US15/007,919
• Publication Number:
  US 20170212910A1
• Time in Patent Office: 650 Days
• Field of Search: None
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Systems and methods are described to identify in the images text, people, landmarks, objects, or any combination thereof and to store and search for images based on extracted data. A portable computing device, such as a wireless phone or tablet computer, may capture an image, determine context data associated with the image, and send the image and context data to an analyzer. The analyzer may extract data from the image and send the extracted data to the portable computing device. For example, the analyzer may determine if the image includes alphanumeric characters and perform character recognition (e.g., using optical character recognition (OCR) or similar technology). A format of the characters that are recognized in the image may be analyzed to determine additional information, such as whether the characters are a phone number, a uniform resource locator (URL), a name of a person, a name of a location, or the like. The analyzer may determine if the image includes a person, a landmark, an object, etc. by comparing at least a portion of the image with other images. The analyzer may send the extracted data to the portable computing device for storage. The portable computing device may associate the context data and the extracted data with a downsampled (e.g., thumbnail) version of the image to enable a user to query the context data and extracted data.

Claims

1. A computer-implemented method, comprising:

receiving an image;

determining context data associated with the image;

storing the image and the context data in a local database;

sending the image and the context data to an image analyzer;

receiving, from the image analyzer, extracted data that is determined based on the image and based on a user profile that includes user preferences specifying types of data that are not to be extracted from the image, the extracted data comprising a type of a recognized subject matter included in the image, a tag associated with a recognized subject matter, and an action associated with the recognized subject matter in the image;

reducing a size of the image to create a thumbnail of the image by performing at least one of downsampling the image, cropping the image, or reducing a number of colors in the image;

associating and storing the extracted data with the thumbnail in the local database;

receiving a query;

determining results of the query from the local database; and

displaying the results.

2. The computer-implemented method of claim 1, further comprising:

capturing the image using a computing device, and wherein the context data is determined using at least one of the following components on the computing device;

a clock, a GPS device, a Bluetooth positioning device.

3. The computer-implemented method of claim 1, wherein the image analyzer determines extracted data based on information from other data sources, including one or more of a social media database, a company directory, a map database, a website, an image library, or a news feed.

4. The computer-implemented method of claim 1, further comprising:

receiving user verification of the extracted data associated with the image; and

sending the user verification to the image analyzer.

5. The computer-implemented method of claim 4, wherein the action associated with the recognized subject matter is one of:

initiating a call to a phone number;

creating an electronic communication to a person;

navigating a web browser to a website;

or displaying a location on a map.

6. The computer-implemented method of claim 4, wherein the extracted data comprises a position of the recognized subject matter, the position comprising an area on the image that the recognized subject matter approximately occupies.

7. The computer-implemented method of claim 6, further comprising:

displaying the image with at least a part of the extracted data, wherein the tag of the recognized subject matter is displayed when a user pointer is moved in proximity to the position of the recognized subject matter, the tag including textual data associated with a recognized subject matter.

8. The computer-implemented method of claim 6, further comprising:

displaying the image with at least a part of the extracted data, wherein the action associated with the recognized subject matter is performed in response to receiving a user selection within the position associated with the recognized subject matter.

9. The computer-implemented method of claim 1, further comprising:

receiving user input to modify at least a part of the extracted data; and

sending the extracted data to the image analyzer.

10. One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to perform operations comprising:

receiving an image;

determining context data associated with the image;

storing the image and the context data in a local database;

sending the image and the context data to an image analyzer service, wherein the image analyzer service determines extracted data based on;

the image,

the context data, and

a user profile that includes user preferences specifying types of data that are to be extracted from the image;

receiving, from the image analyzer service, extracted data associated with the image, the extracted data comprising a type of a recognized subject matter included in the image, a tag associated with the recognized subject matter, and an action associated with the recognized subject matter;

reducing a size of the image to create a thumbnail of the image by performing at least one of downsampling the image, cropping the image, or reducing a number of colors in the image;

associating and storing the extracted data with the thumbnail in the local database;

receiving a query;

displaying results of the query, the results extracted from the local database.

11. The one or more non-transitory computer-readable media of claim 10, wherein the image analyzer service determines extracted data based on information from other data sources, including one or more of a social media database, a company directory, a map database, a website, an image library, or a news feed.

12. The one or more non-transitory computer-readable media of claim 10, wherein the image analyzer service determines that the image includes a URL by matching text recognized in the image against a regular expression.

13. The one or more non-transitory computer-readable media of claim 10, wherein the image analyzer service stores images from users in user-specific databases.

14. The one or more non-transitory computer-readable media of claim 10, wherein the image analyzer service maintains accounts associated with users including user preference data and user history data, and determines extracted data based on at least one of the user preference data or the user history data.

15. The one or more non-transitory computer-readable media of claim 14, wherein the image analyzer service determines extracted data for a first image associated with a first user based on a second image associated with a second user or extracted data associated with the second image.

16. The one or more non-transitory computer-readable media of claim 12, wherein the image analyzer service validates the URL and obtains a risk rating for the URL.

17. A computer device, comprising:

one or more processors; and

one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to perform operations comprising;

capturing an image;

determining context data associated with the image;

storing the image and the context data in a local database;

sending the image and the context data to an image analyzer, wherein the image analyzer determines extracted data based on;

the image;

the context data; and

a user profile specifying types of data that are to be extracted from the image;

reducing a size of the image to create a thumbnail of the image by performing at least one of downsampling the image, cropping the image, or reducing a number of colors in the image;

receiving from the image analyzer extracted data associated with the image, the extracted data comprising a type of recognized subject matter included in the image, a tag associated with the recognized subject matter, and an action associated with the recognized subject matter;

associating and storing the extracted data with the thumbnail in the local database;

receiving a query directed to the local database; and

displaying results of the query, the results extracted from the local database.

18. The computer device of claim 17, the one or more non-transitory computer-readable media storing further instructions that are executable by the one or more processors to:

determining parameters from the query, the parameters comprising one or more of a search context data or a search tag;

determining the results of the query based on the parameters;

ranking the results according to a rank criteria.

19. The computer device of claim 18, wherein the rank criteria is based on a relevance score based on the query parameters and context data or tags associated with images in the results.

20. The computer device of claim 18, wherein the rank criteria is based on user preference data.