Systems and methods for note content extraction and management using segmented notes

• US 9,779,295 B2
• Filed: 07/19/2016
• Issued: 10/03/2017
• Est. Priority Date: 07/09/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Wolk, Diane R., Moran, Cristin E., Mahli, David M., Somasundaram, Guruprasad, Moore, Richard J.
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/213,585
• Publication Number:
  US 20160328609A1
• Time in Patent Office: 441 Days
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A note management system comprises a note recognition module configured to receive image data capturing a note having a plurality of color segments, wherein the note recognition module is further configured to generate a plurality of indicators, each indicator indicative of a color class of a pixel or group of pixels within the image data and based on color values of the pixel or group of pixels; and a note extraction module configured to determine general boundaries of the color segments of the note based on the plurality of indicators and extract content using the general boundaries, the content comprising a plurality of content pieces, each of the content pieces corresponding to one of the color segments of the note.

Claims

1. A method of extracting content from a note with color segments using a computer system having one or more processors and memories, comprising:

receiving, by the one or more processors, image data comprising a visual representation of a physical note having a plurality of segments, each of the plurality of segments having at least one mark;

identifying, by the one or more processors, marks in the image data;

based on the identified marks, determining, by the one or more processors, general boundaries within the image data of the segments; and

extracting, by the one or more processors and from the image data, content comprising a set of content pieces using the general boundaries, each of the content pieces corresponding to a different one of the plurality of segments of the physical note.

2. The method of claim 1, wherein the marks comprise at least one of lines, arrows, star-shaped marks, elbow-shaped marks, rectangular marks, circular marks, ellipse-shaped marks, polygon-shaped marks, and geometric-shaped marks.

3. The method of claim 1, further comprising:

identifying a color for each of the one or more marks; and

associating each of the content pieces with a category based on the identified color of the at least one mark for the corresponding segment.

4. The method of claim 1, further comprising:

creating a digital note representative of the physical note; and

associating the content extracted from the image data with the digital note.

5. The method of claim 4,

wherein creating a digital note comprises creating a respective record within a database to store attributes associated with the physical note, the record having a plurality of fields for storing respective content;

and

wherein associating the content with the digital note comprises updating each of the fields for the respective record with the database to store respective ones of the content pieces for the different segments.

6. The method of claim 1, further comprising:

identifying a characteristic for each of the marks;

comparing the identified characteristic with a set of known characteristics for marks to be affixed to notes; and

associating each of the pieces of content extracted from a respective one of the plurality of segments with a category based on the comparison of the identified characteristic to the set of known characteristics.

7. The method of claim 6, wherein the characteristic comprises at least one of a shape, a color, a size, a color property, and a geometry property.

8. The method of claim 6, further comprising:

presenting the set of the content pieces on a graphical interface of a computing device with an indicator indicative its category.

9. The method of claim 6, further comprising:

presenting the set of content pieces on a graphical interface of a computing device in one or more groups, wherein each of the content pieces in a same one of the groups is associated with a same category.

10. The system of claim 1, further comprising:

a digital note generation module configured to create a digital note representative of the physical note and associate the content extracted from the image data with the digital note.

11. The system of claim 10,

wherein the digital note generation module is further configured to create a respective record within a database to store attributes associated with the physical note, the record having a plurality of fields for storing respective content;

and

wherein the digital note generation module is further configured to update each of the fields for the respective record within the database to store respective ones of the content pieces for the different segments.

12. A note management system having one or more processors and memories, comprising:

a note recognition module executing on the one or more processors and configured to receive image data capturing a physical note having a plurality of segments, each of the plurality of segments having at least one mark,

wherein the note recognition module is further configured to identify the marks and determine general boundaries within the image data of the segments based on the identified marks; and

a note extraction module executing on the one or more processors and configured to determine general boundaries within the image data of the segments based on the identified marks and extract from the image data, content comprising a set of content pieces using the general boundaries,

wherein each of the content pieces is corresponding to a different one of the plurality of segments of the physical note.

13. The system of claim 12, wherein the marks comprise at least one of lines, arrows, star-shaped marks, elbow-shaped marks, rectangular marks, circular marks, ellipse-shaped marks, polygon-shaped marks, and geometric-shaped marks.

14. The system of claim 12, wherein the note recognition module is further configured to identify a color for each of the one or more marks, and

wherein the note extraction module is further configured to associate each of the content pieces with a category based on the identified color of the at least one mark for the corresponding segment.

15. The system of claim 12, wherein the note recognition module is further configured to identifying a characteristic for each of the marks;

wherein the note extraction module is further configured to compare the identified characteristic with a set of known characteristics for marks to be affixed to notes and associate each of the pieces of content extracted from a respective one of the plurality of segments with a category based on the comparison of the identified characteristic to the set of known characteristics.

16. The system of claim 15, wherein the characteristic comprises at least one of a shape, a color, a size, a color property, and a geometry property.

17. The system of claim 15, further comprising:

a note presentation module configured to present the set of the content pieces on a graphical interface of a computing device with an indicator indicative its category.

18. The system of claim 15, further comprising:

a note presentation module configured to present the set of content pieces on a graphical interface of a computing device in one or more groups, wherein each of the content pieces in a same one of the groups is associated with a same category.

19. A non-transitory computer-readable storage device comprising instructions that cause a programmable processor to:

receive image data comprising a visual representation of a physical note having a plurality of segments, each of the plurality of segments having at least one mark;

identify marks in the image data;

determine general boundaries within the image data of the segments based on the identified marks;

extract from the image data, content comprising a set of content pieces using the general boundaries, each of the content pieces corresponding to a different one of the plurality of segments of the physical note,

create a digital note representative of the note, the digital note having data indicating each of the segments for the note; and

associate the respective content pieces extracted from the image data with the corresponding segment data of the digital note.

20. The computer-readable storage device of claim 19, wherein the instructions cause the programmable processor to:

create the digital note corresponding to the physical note as a record within a database to store attributes associated with the physical note, and

update the record with the database to associate the respective record with the content.