Digital image manipulation

• US 9,875,402 B2
• Filed: 05/09/2017
• Issued: 01/23/2018
• Est. Priority Date: 04/14/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Goyal, Ankur
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US15/591,070
• Publication Number:
  US 20170243059A1
• Time in Patent Office: 259 Days
• Field of Search: 382175, 382224, 382309, 382310
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 50/12 Hotels or restaurants
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

Techniques for assigning context to a digitally captured image, and for manipulating recognized data fields within such image. In an exemplary embodiment, a context of an image may be assigned based on, e.g., user input or pattern recognition. Based on the assigned context, recognized data fields within the image may be manipulated according to context-specific processing. In an aspect, processing specific to a sales receipt context may automatically manipulate certain data, e.g., calculate updated sales tax and subtotals based on user-designated fields, and display the automatically calculated data in an output receipt. Fields not designated by the user may be selectively concealed in the output receipt for privacy. Further aspects disclose processing techniques specific to other contexts such as restaurant menu, store shelf, and fillable form contexts.

Claims

1. An apparatus for generating an output image from at least one digital image, the apparatus comprising:

a field recognition block configured to digitally recognize one or more data fields in the at least one digital image;

a context assignment block configured to assign a context to the at least one digital image by correlating the digitally recognized one or more data fields with a plurality of predetermined context-specific templates;

a user designation block configured to receive input from the user designating at least one of the one or more recognized data fields;

a data manipulation block configured to automatically compute additional data fields based on the assigned context and the designated at least one of the one or more recognized data fields; and

a post-processing block configured to generate the output image by superimposing the computed additional data fields onto the at least one digital image.

2. The apparatus of claim 1, the plurality of predetermined context-specific templates comprising at least one template corresponding to a restaurant menu, the digitally recognized one or more data fields corresponding to menu items and prices as specified by the at least one template corresponding to a restaurant menu, and the user designation block configured to receive input from the user designating at least one of the menu items and prices, the computed additional data fields comprising a sub-total field, a tax field, a gratuity field, or a total price field.

3. The apparatus of claim 1, the plurality of predetermined context-specific templates comprising at least one template corresponding to a store shelf, the digitally recognized one or more data fields corresponding to a plurality of items for purchase and corresponding prices as specified by the at least one template corresponding to a store shelf, and the user designation block configured to receive input from the user designating at least one of the plurality of items for purchase, the computed additional data fields comprising a sub-total field, a tax field, or a total price field.

4. The apparatus of claim 3, the at least one digital image comprising a plurality of images, the output image comprising the plurality of images displayed in a single image, the sub-total field comprising a sum of recognized data fields corresponding to price extracted from each of the plurality of images.

5. The apparatus of claim 4, each of the plurality of images being assigned the same context.

6. The apparatus of claim 3, the received input from the user designating at least one of the plurality of items for purchase comprising designation of the digitally recognized one or more data fields corresponding to prices.

7. The apparatus of claim 1, the apparatus comprising a mobile communications device comprising a camera, the at least one digital image being generated by the camera.

8. The apparatus of claim 1, the computed additional fields comprising a calculated sales tax field.

9. The apparatus of claim 8, the at least one digital image comprising geographical information identifying location of where each image was captured, the sales tax field being calculated based on a sales tax rate corresponding to the identified location.

10. The apparatus of claim 8, the user designation block further configured to receive input from the user designating a sales tax rate to be used when calculating the sales tax field.

11. A method comprising:

digitally recognizing one or more data fields in at least one digital image;

assigning a context to the at least one digital image by correlating the digitally recognized one or more data fields with a plurality of predetermined context-specific templates;

receiving input from a user designating at least one of the one or more recognized data fields;

automatically computing additional data fields based on the assigned context and the designated at least one of the one or more recognized data fields; and

generating an output image by superimposing the computed additional data fields onto the at least one digital image.

12. The method of claim 11, the plurality of predetermined context-specific templates comprising at least one template corresponding to a restaurant menu, the digitally recognized one or more data fields corresponding to menu items and prices as specified by the at least one template corresponding to a restaurant menu, and the receiving the input from the user comprising receiving user designation of at least one of the menu items and prices, the computed additional data fields comprising a sub-total field, a tax field, a gratuity field, or a total price field.

13. The method of claim 11, the plurality of predetermined context-specific templates comprising at least one template corresponding to a store shelf, the digitally recognized one or more data fields corresponding to a plurality of items for purchase and corresponding prices as specified by the at least one template corresponding to a store shelf, and the receiving the input from the user comprising receiving user designation at least one of the plurality of items for purchase, the computed additional data fields comprising a sub-total field, a tax field, or a total price field.

14. The method of claim 11, further comprising generating the at least one digital image using a digital camera coupled to a mobile communications device.

15. The method of claim 11, the computed additional fields comprising a calculated sales tax field.

16. The method of claim 15, the at least one digital image comprising geographical information identifying location of where each image was captured, the sales tax field being calculated based on a sales tax rate corresponding to the identified location.

17. The method of claim 15, the receiving the input from the user further comprising receiving user designation of a sales tax rate to be used when calculating the sales tax field.

18. A computing device including a processor and a memory holding instructions executable by the processor to:

digitally recognize one or more data fields in at least one digital image;

assign a context to the at least one digital image by correlating the digitally recognized one or more data fields with a plurality of predetermined context-specific templates;

receive input from a user designating at least one of the one or more recognized data fields;

automatically compute additional data fields based on the assigned context and the designated at least one of the one or more recognized data fields; and

generate an output image by superimposing the computed additional data fields onto the at least one digital image.

19. The device of claim 18, the plurality of predetermined context-specific templates comprising at least one template corresponding to a restaurant menu, the digitally recognized one or more data fields corresponding to menu items and prices as specified by the at least one template corresponding to a restaurant menu, and the instructions executable by the processor to receive the input from the user comprising instructions executable by the processor to receive user designation of at least one of the menu items and prices, the computed additional data fields comprising a sub-total field, a tax field, a gratuity field, or a total price field.

20. The device of claim 18, the plurality of predetermined context-specific templates comprising at least one template corresponding to a store shelf, the digitally recognized one or more data fields corresponding to a plurality of items for purchase and corresponding prices as specified by the at least one template corresponding to a store shelf, and the instructions executable by the processor to receive the input from the user comprising instructions executable by the processor to receive user designation at least one of the plurality of items for purchase, the computed additional data fields comprising a sub-total field, a tax field, or a total price field.