Note recognition and management using color classification

• US 8,891,862 B1
• Filed: 07/08/2014
• Issued: 11/18/2014
• Est. Priority Date: 07/09/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Wolk, Diane R., Moran, Cristin E., Mahli, David M., Somasundaram, Guruprasad, Moore, Richard J., Sivalingam, Ravishankar, Schumacher, Jennifer F.
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US14/326,074
• Time in Patent Office: 133 Days
• Field of Search: 382/162, 382/164, 382/165, 382/166, 382/167, 382/173, 382/190, 382/199, 382/276, 382/277, 358/3.06, 358/448, 358/515, 375/240.25
• US Class Current: 382/165
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

This disclosure describes techniques for creating and manipulating software notes representative of physical notes. For example, techniques are described for recognizing physical notes present within a physical environment, capturing information therefrom and creating corresponding digital representations of the physical notes, referred to herein as digital notes or software-based notes. At least some aspects of the present disclosure feature system and methods for note recognition using color classification. The system receives a visual representation of a scene having one or more notes, where each note has a color. The system generates indicators indicative of color classes of pixels in the visual representation. The system further determines a general boundary of one of the notes based on the indicators.

Claims

1. A method of extracting note content using a computer system having one or more processors and memories, comprising:

receiving image data containing a visual representation of a scene having a plurality of notes, each of the plurality of notes having a color;

generating, by the one or more processors, a plurality of indicators, each indicator indicative of a color class of a respective pixel or group of pixels in the image data and based on color values of the pixel or group of pixels;

based on the plurality of indicators, determining, by the one or more processors, a general boundary within the image data of one of the plurality of notes; and

extracting, by the one or more processors and from the image data, content of the one of the plurality of notes using the general boundary.

2. The method of claim 1, wherein the generating step comprises applying a function to the color values of the pixel or group of pixels and generating a function output as an indicator.

3. The method of claim 2, wherein the function is a linear discriminant function.

4. The method of claim 2, wherein the function is a quadratic discriminant function.

5. The method of claim 1, further comprising:

converting, by the one or more processors, the image data to LAB color space, wherein each of the pixel or group of pixels is associated with L, a, and b values.

6. The method of claim 5, wherein the generating step comprises comparing the b value of the pixel or group of pixels to a predetermined threshold.

7. The method of claim 6, wherein the predetermined threshold relates to a color.

8. The method of claim 1, wherein the visual representation is the image data.

9. The method of claim 1, wherein the determining step comprises grouping adjacent pixels having a same indicator and forming a region.

10. The method of claim 1, wherein the generated indicators comprise at least one of a number, a shape, a code, a grayscale value, a color value, and a symbol.

11. The method of claim 1, further comprising:

creating a digital note representative of the one of the plurality of notes for which the boundary is determined; and

associating the content extracted from the image data with the digital note.

12. The method of claim 11,

wherein creating a digital note comprises creating a respective record within a database to store attributes associated with the one of the plurality of notes for which the boundary is determined, and

wherein associating the content with the digital note comprises updating the respective record with the database to associate the respective record with the content.

13. The method of claim 12, further comprising:

determining a group for the one of the plurality of notes for which the boundary is determined; and

updating the respective record with the database to associate the digital note with the group.

14. The method of claim 1, wherein the image data comprises one of an image, a plurality of images or video data.

15. The method of claim 1, wherein two or more of the notes are of different colors.

16. The method of claim 1, wherein at least one of the notes comprises multiple color segments of different colors.

17. A note recognition system having one or more processors and memories, comprising:

a note recognition module executing on the one or more processors and configured to receive image data capturing a visual representation of a scene having a plurality of notes, each of the plurality of notes having a color,

wherein the note recognition module is further configured to generate a plurality of indicators, each indicator indicative of a color class of a pixel or group of pixels within the image data and based on color values of the pixel or group of pixels; and

a note extraction module executing on the one or more processors and configured to determine a general boundary within the image data of one of the plurality of notes based on the plurality of indicators and to extract content of the one of the plurality of notes using the determined general boundary of the one of the plurality of notes.

18. The note recognition system of claim 17, wherein the note recognition module is further configured to apply a function to the color values of the pixel or group of pixels and generate a function output as an indicator.

19. The note recognition system of claim 18, wherein the function is a linear discriminant function.

20. The note recognition system of claim 18, wherein the function is a quadratic linear discriminant function.

21. The note recognition system of claim 18, wherein the note recognition module is further configured to receive a set of training images and obtain coefficients of the function using the training images.

22. The note recognition system of claim 17, wherein the note recognition module is further configured to convert the image data to LAB color space, wherein each pixel or group of pixels is associated with L, a, and b values.

23. The note recognition system of claim 22, wherein the note recognition module is further configured to compare the b value of the pixel or group of pixels to a predetermined threshold.

24. The note recognition system of claim 23, wherein the predetermined threshold relates to a color.

25. The note recognition system of claim 17, wherein the note extraction module is further configured to group adjacent pixels having a same indicator and form a region.

26. The note recognition system of claim 17, wherein the generated indicators comprise at least one of a number, a shape, a code, a grayscale value, a color value, and a symbol.

27. The note recognition system of claim 17, wherein the visual representation is the image.

28. A non-transitory computer-readable storage device comprising instructions that cause a programmable processor to:

receive image data containing a visual representation of a scene having a plurality of physical notes, each of the plurality of physical notes having a color;

generate, by the one or more processors, a plurality of indicators, each indicator indicative of a color class of color values of one or more pixel within the image data;

based on the plurality of indicators, determine a boundary within the image data of a first physical note of the plurality of physical notes;

extract, from the image data, content of the first physical note using the general boundary;

creating a digital note representative of the first physical note; and

associating the content extracted from the image data with the digital note.

29. The computer-readable storage device of claim 28, wherein the instructions cause the programmable processor to:

create the digital note corresponding to the first physical note as a record within a database to store attributes associated with the physical note, and

update the record with the database to associate the respective record with the content.

30. The computer-readable storage device of claim 29, wherein the instructions cause the programmable processor to:

determine, based on the image data, a group for the first physical note based on at least one attribute of the first physical note; and

update the respective record with the database to associate the digital note with the group.