Systems and methods for note content extraction and management by segmenting notes

• US 9,390,322 B2
• Filed: 07/08/2014
• Issued: 07/12/2016
• Est. Priority Date: 07/09/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Wolk, Diane R., Moran, Cristin E., Mahli, David M., Somasundaram, Guruprasad, Moore, Richard J.
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US14/326,026
• Publication Number:
  US 20150016716A1
• Time in Patent Office: 735 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A note management system comprises a sensor configured to capture an image data of a physical note, wherein the note is separated into one or more segments using marks, wherein each of the segments is marked by at least one of the marks. The note management system further comprises a note recognition module coupled to the sensor, the note recognition module configured to receive the captured image data and identify the marks on the note, and a note extraction module configured to determine general boundaries of the one or more segments within the captured image data based on the identified marks and extract content using the general boundaries, the content comprises content pieces, each of the content pieces corresponding to one of the one or more segments of the note.

Claims

1. A method of extracting content from a note using a computer system having one or more processors and memories, comprising:

capturing, by a sensor, image data comprising a visual representation of a note, wherein the note comprises a physical note having one or more segments, each of the segments having at least one mark having a known characteristic;

identifying, by the one or more processors, marks within the image data;

based on the known characteristics of identified marks, determining, by the one or more processors, a general boundary of the physical note;

based on the known characteristics of identified marks and the determined boundary of the physical note, determining, by the one or more processors, general boundaries within the image data for the one or more segments of the physical note;

extracting, by the one or more processors and from the image data, content comprising a set of content pieces using the general boundaries, each of the content pieces corresponding to a different one of the one or more segments of the note;

creating, by the one or more processors, a digital note representative of the physical note, the digital note having data indicating each of the segments for the physical note; and

associating, by the one or more processors, the respective content pieces extracted from the image data with the corresponding segments of the digital note.

2. The method of claim 1, wherein the marks comprise at least one of lines, arrows, star-shaped marks, elbow-shaped marks, rectangular marks, circular marks, ellipse-shaped marks, polygon-shaped marks, and geometric-shaped marks.

3. The method of claim 1, further comprising:

identifying a shape for each of the one or more marks;

comparing the identified shape with a set of known shapes for marks to be affixed to physical notes; and

associating each of the pieces of content extracted from a respective one of the one or more segments with a category based on the comparison of the identified shape of the at least one mark for the corresponding segment to the set of known shapes.

4. The method of claim 3, further comprising:

presenting the one or more of the content pieces on a graphical interface of a computing device with an indicator indicative of its category.

5. The method of claim 3, further comprising:

presenting the content pieces on a graphical interface of a computing device in one or more groups, wherein each of the content pieces in a same one of the groups is associated with a same category.

6. The method of claim 1, further comprising:

identifying a color for each of the one or more marks; and

associating each of the pieces of content extracted from a respective one of the one or more segments with a category based on the identified color of the at least one mark for the corresponding segment.

7. The method of claim 6, further comprising:

presenting the one or more of the content pieces on a graphical interface of a computing device with an indicator indicative of its category.

8. The method of claim 6, further comprising:

presenting the content pieces on a graphical interface of a computing device in one or more groups, wherein each of the content pieces in a same one of the groups is associated with a same category.

9. The method of claim 1, wherein the determining step comprises determining the general boundaries based upon the identified marks and known relative positions of the marks for the one or more segments.

10. A note management system having one or more processors and memories, comprising:

a sensor configured to capture an image data of a physical note, wherein the note is separated into one or more segments using marks, wherein each of the segments is marked by at least one of the marks having a known characteristic,

a note recognition module coupled to the sensor, the note recognition module executing on the one or more processor and configured to receive the captured image data and identify the marks within the image data; and

a note extraction module executing on the one or more processors and configured to determine a general boundary of the physical note based on the known characteristics of identified marks and determine general boundaries of the one or more segments within the captured image data based on the identified marks and the determined boundary of the physical note and extract content using the general boundaries of the one or more segments, the content comprises content pieces, each of the content pieces corresponding to one of the one or more segments of the note,

wherein the note extraction module is further configured to create a digital note representative of the physical note, the digital note having data indicating each of the segments for the physical note, and associate the respective content pieces extracted from the image data with the corresponding segments of the digital note.

11. The note management system of claim 10, wherein the note extraction module is further configured to determine the general boundaries of the one or more note segments based upon the identified marks and predefined relative positions of the marks for the one or more note segments.

12. The note management system of claim 10, further comprising:

a note presentation module executing on the one or more processors and configured to present the set of content on a user interface.

13. The note management system of claim 12, wherein the note presentation module presents at least one of the content pieces with an indicator indicative its category.

14. The note management system of claim 12, wherein the note presentation module is further configured to present the content pieces in one or more groups, wherein each of the content pieces a group is associated with a same category.

15. The note management system of claim 10, wherein the marks comprise at least one of lines, arrows, star-shaped marks, elbow-shaped marks, rectangular marks, circular marks, ellipse-shaped marks, polygon-shaped marks, and geometric-shaped marks.

16. The note management system of claim 10, wherein the note extraction module identifies a shape for each of the one or more marks and associates each of the content pieces with a category by the shape of the at least one mark for the corresponding segment.

17. The note management system of claim 10, wherein the note extraction module identifies a color for each of the one or more marks and associates each of the content pieces with a category identified by the color of the at least one mark for the corresponding segment.

18. The note management system of claim 12, further comprising:

a database having a plurality of records storing data indicative of a plurality of digital notes,

wherein the note extraction module creates a digital note within the database representative of the physical note captured by the image data, the digital note stored within the database and having data indicating each of the segments identified for the physical note, and

wherein the note extraction module updates the records to associate the respective content pieces extracted from the image data with the corresponding color segments of the digital note.

19. A non-transitory computer-readable storage device comprising instructions that cause a programmable processor to:

receive image data containing a visual representation of a scene having a plurality of physical notes, each of the plurality of physical notes having a plurality of segments, each of the segments having at least one mark having a known characteristic;

process the image data to identify the marks within the image data and determine, based on the known characteristics of the marks, a boundary within the image data of a first physical note of the plurality of physical notes;

process the image data to determine, based on the known characteristic of the marks and the boundary determined for the physical note, boundaries for each of the segments of the first physical note;

extract, from the image data, respective content for each of the segments of the first physical note using and the boundaries determined for each of the segments of the physical note;

create a digital note representative of the first physical note, the digital note having data indicating each of the segments for the physical note; and

associate the respective content pieces extracted from the image data with the corresponding segments of the digital note.