Note capture and recognition with manual assist

• US 9,292,186 B2
• Filed: 01/29/2015
• Issued: 03/22/2016
• Est. Priority Date: 01/31/2014
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Moore, Richard J., Mahli, David M., Axelsson, Pontus, Moran, Cristin E., Persson, Roland, Rotstein, Michael, Sivalingam, Ravishankar, Somasundaram, Guruprasad, Wolk, Diane R., kerlund, Linus
• Primary Examiner(s): RIES, LAURIE ANNE
• Application Number:
  US14/608,955
• Publication Number:
  US 20150220257A1
• Time in Patent Office: 418 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

At least some aspects of the present disclosure feature a computing device configured to receive an input image of an environment having a plurality of physical notes. The computing device automatically processes the input image to identify at least some of the plurality of the physical notes in the input image and displays the input image and indications indicative of the identified physical notes on a user interface. The computing device receives a user input indicating a position within the input image via a user interface and, responsive to the user input, recognizes proximate to the position a missed one of the physical notes that was not identified by the computing device when initially processing the input image.

Claims

1. A computing device having a processor, the computing device comprising:

an image collection module configured to receive an input image of a plurality of physical notes;

a note identification module, implemented by the processor, configured to process the input image to identify at least some of the plurality of physical notes in the input image; and

a user interface configured to display the input image and indications indicative of the identified physical notes,

wherein the user interface is further configured to receive a user input indicating a position in the input image, and

wherein the note identification module is further configured to identify, based on the position, a missed one of the physical notes in the input image that was not identified by the note identification module when processing the input image.

2. The computing device of claim 1, wherein the note identification module is further configured to apply, responsive to the user input, a color segmentation algorithm to determine a boundary of the missed physical note based on the position in the input image.

3. The computing device of claim 1, wherein the note identification module is further configured to apply, responsive to the user input, a marching algorithm to determine a boundary of the missed physical note based on the position in the input image.

4. The computing device of claim 1, wherein the note identification module is further configured to determine an area of interest based on areas of the identified notes in the input image, and wherein the note identification module is further configured to determine, responsive to the user input, a boundary of the missed physical note based on the position in the input image and the area of interest.

5. The computing device of claim 1, wherein the note identification module is further configured to identify a plurality of candidate missed physical notes in the input image based on the position, and generate a score for each candidate missed physical note based on at least one factor selected from a group consisting of area, aspect ratio, position, and color values, and

wherein the note identification module is further configured to choose a candidate missed physical note having a highest score among the plurality of candidate missed physical notes as the missed physical note.

6. The computing device of claim 1, wherein the note identification module is further configured to determine a color of the missed physical note and identify one or more other missed physical notes based on the color of the missed physical note.

7. The computing device claim 1, wherein the note identification module is further configured to generate digital notes, each digital note corresponding to a note of the plurality of notes identified in the input image.

8. The computing device of claim 1, wherein the user interface is further configured to display the input image overlaid with an indication indicative of the missed physical note.

9. The computing device of claim 1, wherein the input image comprises at least one of a static image, an animated image, a sequence of images, and a video.

10. The computing device of claim 1, wherein the input image depicts a visual representation of a scene having the plurality of physical notes.

11. A method of capturing and recognizing notes implemented on a computing device having a processor, a memory, and a user interface, the method comprising:

receiving an input image of a plurality of physical notes;

identifying, by the processor, at least some of the plurality of physical notes in the input image;

displaying the input image and indications indicative of the identified physical notes on the user interface;

receiving a user input indicating a position in the input image; and

identifying, by the processor and responsive to the position indicated by the user input, a missed physical note in the input image that was not identified by the processor when initially processing the input image.

12. The method of claim 11, further comprising:

applying, by the processor, a color segmentation algorithm to determine a boundary of the missed physical note based on the position in the input image.

13. The method of claim 11, further comprising:

applying, by the processor, a marching algorithm to determine a boundary of the missed physical note based on the position in the input image.

14. The method of claim 11, further comprising:

determining an area of interest based on areas of the identified notes in the input image; and

determining a boundary of the missed physical note based on the position in the input image and the area of interest.

15. The method of claim 11, further comprising:

identifying a plurality of candidate missed physical notes in the image based on the position;

generating a score for each candidate missed physical note based on at least one factor selected from a group consisting of area, aspect ratio, position, and color values; and

choosing a candidate missed physical note having a highest score among the plurality of candidate missed physical notes as the missed physical note.

16. The method of claim 11, further comprising:

determining a color of the missed physical note; and

identifying one or more other missed physical notes based on the color of the missed physical note.

17. The method of claim 11, further comprising:

generating digital notes, each digital note representing a note identified in the input image.

18. The method of claim 11, further comprising:

displaying the input image overlaid with an indication indicative of the missed physical note on the user interface.

19. The method claim 11, wherein the input image is overlaid with indications indicative of the identified notes.

20. The method of claim 11, wherein the input image depicts a visual representation of a scene having the plurality of physical notes.