Augmented project lists with price information

• US 10,650,422 B1
• Filed: 04/17/2019
• Issued: 05/12/2020
• Est. Priority Date: 04/17/2019
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Benkreira, Abdelkader M'Hamed, Mossoba, Michael, Shah, Salik
• Primary Examiner(s): Fadok, Mark A
• Application Number:
  US16/386,424
• Time in Patent Office: 391 Days
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06Q 30/0283 Price estimation or determination
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

In some embodiments, the present disclosure relates to a method for augmenting project lists with price information. The method can include: capturing an image using a camera of a user device; identifying a project list within the captured image; extracting, from the captured image, a list of items associated with the project list, the list of items represented as text; retrieving price information for one or more of the list items; and displaying, on the user device, an augmented image comprising the captured image and the price information.

Claims

1. A method for augmenting project lists with price information, the method comprising:

capturing an image using a camera of a user device;

identifying a project list within the captured image using computer vision (CV), by;

selecting a first machine learning (ML) model based on a project type associated with the project list; and

applying the CV to the project list, wherein the CV comprises the selected first ML model;

extracting from the captured image, using natural language processing (NLP), a list of items associated with the project list, the list of items represented as text, by;

selecting a second machine learning (ML) model based on the project type associated with the project list; and

applying the NLP to the project list, wherein the NLP comprises the selected second ML model;

retrieving price information for one or more of the list items from a database or from one or more vendors using application programming interfaces (APIs);

generating an augmented image comprising the captured image and the price information; and

displaying the augmented image on the user device.

2. The method of claim 1 comprising:

for the one or more list items, positioning the corresponding price information proximate to the corresponding list item within the augmented image.

3. The method of claim 1 comprising:

calculating a total price for the project list using the retrieved price information,

wherein the augmented image comprises the total price.

4. The method of claim 3 comprising:

positioning the total price proximate to the project list within the augmented image.

5. The method of claim 3 comprising:

identifying negative space within the captured image; and

positioning the total price over the negative space within the augmented image.

6. The method of claim 3 comprising:

displaying, on the user device, user interface controls to remove items from the project list; and

in response to the user removing an item from the project list, updating the total price for the project list within the augmented image.

7. The method of claim 1 wherein identifying the project list within the captured image comprises identifying a recipe within the captured image using machine learning models (ML) trained to recognize text within different categories of project lists, wherein retrieving the price information comprises retrieving the prices for one or more ingredients.

8. The method of claim 1 wherein identifying the project list within the captured image comprises identifying a grocery list within the captured image using machine learning models (ML) trained to recognize text within different categories of project lists, wherein retrieving the price information comprises retrieving the prices for one or more grocery items.

9. The method of claim 1 wherein retrieving the price information for the one or more list items comprises retrieving item data from a database.

10. The method of claim 1 comprising:

determining a preferred vendor associated with a user of the user device,

wherein retrieving the price information for the one or more list items comprises retrieving price information associated with the preferred vendor.

11. The method of claim 10 comprising:

displaying, on the user device, one or more user interface controls for modifying user preferences, wherein the user preferences comprise at least the user'"'"'s preferred vendor; and

in response to the user modifying the user preferences, sending the user preferences to a server device, the server device configured to store the user preferences in a database.

12. The method of claim 1 comprising:

determining a disposable income budget for a user of the user device; and

calculating a relative cost of the project list using the disposable income budget,

wherein the augmented image comprises the relative cost of the project list.

13. A server device comprising:

one or more processors;

an application programming interface (API) module configured for execution on the processors to receive requests from and send responses to a user device;

an items module configured for execution the processors to;

receive, from the user device via the API module, a list of items associated with a project list;

retrieve, from a database, price information for the list of items from a database or from one or more vendors using application programming interfaces (APIs); and

send the price information to the user device via the API module;

wherein the user device is configured to;

capture an image using a camera of the user device;

identify the project list within the captured image using computer vision (CV), by;

selecting a first machine learning (ML) model based on a project type associated with the project list; and

applying the CV to the project list, wherein the CV comprises the selected first ML model;

extract from the captured image, using natural language processing (NLP), the list of items associated with the project list, the list of items represented as text by;

selecting a second machine learning (ML) model based on the project type associated with the project list; and

applying the NLP to the project list, wherein the NLP comprises the selected second ML model;

generate an augmented image comprising the captured image and the price information; and

display the augmented image on the user device.