Systems and methods for note content extraction and management using segmented notes

• US 9,412,018 B2
• Filed: 02/02/2015
• Issued: 08/09/2016
• Est. Priority Date: 07/09/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Wolk, Diane R., Moran, Cristin E., Mahli, David M., Somasundaram, Guruprasad, Moore, Richard J.
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US14/612,009
• Publication Number:
  US 20150186719A1
• Time in Patent Office: 554 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A note management system comprises a note recognition module configured to receive image data capturing a note having a plurality of color segments, wherein the note recognition module is further configured to generate a plurality of indicators, each indicator indicative of a color class of a pixel or group of pixels within the image data and based on color values of the pixel or group of pixels; and a note extraction module configured to determine general boundaries of the color segments of the note based on the plurality of indicators and extract content using the general boundaries, the content comprising a plurality of content pieces, each of the content pieces corresponding to one of the color segments of the note.

Claims

1. A method of extracting content from a note with color segments using a computer system having one or more processors and memories, comprising:

receiving, by the one or more processors, image data comprising a visual representation of a note having a plurality of predefined color segments, wherein at least two of the plurality of color segments are different in size;

generating, by the one or more processors, a plurality of indicators, each indicator indicative of a color class of a respective pixel or group of pixels in the image data and based on color values of the pixel or group of pixels;

based on the generated plurality of indicators, determining, by the one or more processors, general boundaries within the image data of the color segments; and

extracting, by the one or more processors and from the image data, content comprising a set of multiple content pieces using the general boundaries, each of the content pieces corresponding to a different one of the color segments of the note.

2. The method of claim 1, wherein at least one of the color segments has a single color.

3. The method of claim 2, wherein at least two adjacent color segments have different colors.

4. The method of claim 1, further comprising:

creating a digital note representative of the note for which the boundary is determined; and

associating the content extracted from the image data with the digital note.

5. The method of claim 4,

wherein creating a digital note comprises creating a respective record within a database to store attributes associated with the note for which the boundary is determined, the record having a plurality of fields for storing respective content;

and

wherein associating the content with the digital note comprises updating each of the fields for the respective record with the database to store respective ones of the content pieces for the different color segments.

6. The method of claim 1, wherein at least one of the color segments has a first color at the boundary of the color segment and a second color within the boundary of the color segment, the second color being different from the first color.

7. The method of claim 1, further comprising:

categorizing, with the processor, each of the content pieces associated with a color of the corresponding color segment.

8. The method of claim 1, further comprising:

categorizing, with the processor, each of the content pieces associated with a size of the corresponding color segment.

9. The method of claim 1, further comprising:

labelling, with the processor, each of the content pieces with a status associated with a color of the corresponding color segment.

10. A note management system having one or more processors and memories, comprising:

a note recognition module executing on the one or more processors and configured to receive image data capturing a note having a plurality of color segments, wherein at least two of the plurality of color segments are different in size,

wherein the note recognition module is further configured to generate a plurality of indicators, each indicator indicative of a color class of a pixel or group of pixels within the image data and based on color values of the pixel or group of pixels; and

a note extraction module executing on the one or more processors and configured to determine general boundaries of the color segments of the note based on the plurality of indicators and extract content using the general boundaries, the content comprising a plurality of content pieces, each of the content pieces corresponding to one of the color segments of the note.

11. The note management system of claim 10, wherein at least one of the color segments has a first color at the boundary of the color segment and a second color within the boundary of the color segment, the second color being different from the first color.

12. The note management system of claim 11, wherein the note extraction module categorizes each of the content pieces by the color at the boundary of the corresponding color segment.

13. The note management system of claim 10, wherein the note extraction module associates each of the content pieces with a category, wherein, when presenting the content, the note presentation module is further configured to group at least some of the content pieces in accordance with the associated categories.

14. The note management system of claim 10, wherein the note extraction module is further configured to group adjacent pixels having a same indicator and form a region.

15. The note management system of claim 10, wherein the note recognition module is further configured generate each of the indicators by applying a function to the color values of the respective pixel or group of pixels.

16. The note management system of claim 10, wherein the note extraction module is further configured to categorize each of the content pieces associated with a color of the corresponding color segment.

17. The note management system of claim 10, wherein the note extraction module is further configured to categorize each of the content pieces associated with a size of the corresponding color segment.

18. The note management system of claim 10, wherein the note extraction module is further configured to label each of the content pieces with a status associated with a color of the corresponding color segment.

19. A non-transitory computer-readable storage device comprising instructions that cause a programmable processor to:

receive image data containing a visual representation of a scene having a plurality of physical notes, each of the plurality of physical notes having a plurality of color segments, wherein at least two of the plurality of color segments are different in size;

generate, by the one or more processors, a plurality of indicators, each indicator indicative of a color class of color values of one or more pixel within the image data;

process the image data to determine, based on the plurality of indicators, a boundary within the image data of a first physical note of the plurality of physical notes;

process the image data to determine, based on the indicators and the boundary for the physical note, boundaries for each of the color segments of the first physical note;

extract, from the image data, respective content for each of the color segments of the first physical note using and the boundaries for each of the color segments of the physical note;

create a digital note representative of the first physical note, the digital note having data indicating each of the color segments for the physical note; and

associate the respective content pieces extracted from the image data with the corresponding color segments of the digital note.

20. The computer-readable storage device of claim 19, wherein the instructions cause the programmable processor to:

create the digital note corresponding to the first physical note as a record within a database to store attributes associated with the physical note, and

update the record with the database to associate the respective record with the content.