Note recognition and management using color classification

• US 9,508,001 B2
• Filed: 01/04/2016
• Issued: 11/29/2016
• Est. Priority Date: 07/09/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Wolk, Diane R., Moran, Cristin E., Mahli, David M., Somasundaram, Guruprasad, Moore, Richard J., Sivalingam, Ravishankar, Schumacher, Jennifer F.
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US14/987,221
• Publication Number:
  US 20160117570A1
• Time in Patent Office: 330 Days
• Field of Search: 382/195
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

This disclosure describes techniques for creating and manipulating software notes representative of physical notes. For example, techniques are described for recognizing physical notes present within a physical environment, capturing information therefrom and creating corresponding digital representations of the physical notes, referred to herein as digital notes or software-based notes. At least some aspects of the present disclosure feature system and methods for note recognition using color classification. The system receives a visual representation of a scene having one or more notes, where each note has a color. The system generates indicators indicative of color classes of pixels in the visual representation. The system further determines a general boundary of one of the notes based on the indicators.

Claims

1. A method of extracting note content using a computer system having one or more processors and memories, comprising:

receiving image data containing a visual representation of a scene having a plurality of notes, each of the plurality of notes having a color;

generating, by the one or more processors, a plurality of indicators, each indicator indicative of a color class of a respective pixel or group of pixels in the image data and based on color values of the pixel or group of pixels;

based on the plurality of indicators, determining, by the one or more processors, a content location within the image data of one of the plurality of notes; and

extracting, by the one or more processors and from the image data, content of the one of the plurality of notes using the content location.

2. The method of claim 1, wherein the generating step comprises applying a function to the color values of the pixel or group of pixels and generating a function output as an indicator.

3. The method of claim 2, wherein the function is a linear discriminant function.

4. The method of claim 1, further comprising:

creating a digital note representative of the one of the plurality of notes for which the content location is determined; and

associating the content extracted from the image data with the digital note.

5. The method of claim 1, further comprising:

converting, by the one or more processors, the image data to LAB color space, wherein each of the pixel or group of pixels is associated with L, a, and b values.

6. The method of claim 5, wherein the generating step comprises comparing the b value of the pixel or group of pixels to a predetermined threshold.

7. The method of claim 6, wherein the predetermined threshold relates to a color.

8. The method of claim 1, wherein the visual representation is the image data.

9. The method of claim 1, wherein the determining step comprises grouping adjacent pixels having a same indicator and forming a region.

10. The method of claim 1, wherein the generated indicators comprise at least one of a number, a shape, a code, a grayscale value, a color value, and a symbol.

11. A note recognition system having one or more processors and memories, comprising:

a note recognition module executing on the one or more processors and configured to receive image data capturing a visual representation of a scene having a plurality of notes, each of the plurality of notes having a color,

wherein the note recognition module is further configured to generate a plurality of indicators, each indicator indicative of a color class of a pixel or group of pixels within the image data and based on color values of the pixel or group of pixels; and

a note extraction module executing on the one or more processors and configured to determine a content location within the image data of one of the plurality of notes based on the plurality of indicators and to extract content of the one of the plurality of notes using the determined content location of the one of the plurality of notes.

12. The note recognition system of claim 11, wherein the note recognition module is further configured to apply a function to the color values of the pixel or group of pixels and generate a function output as an indicator.

13. The note recognition system of claim 11, wherein the note recognition module is further configured to receive a set of training images and obtain coefficients of the function using the training images.

14. The note recognition system of claim 11, wherein the note extraction module is further configured to group adjacent pixels having a same indicator and form a region.

15. The note recognition system of claim 11, wherein the generated indicators comprise at least one of a number, a shape, a code, a grayscale value, a color value, and a symbol.

16. The note recognition system of claim 11, wherein the visual representation is the image.

17. The note recognition system of claim 11, wherein the image data comprises at least one of an image, a plurality of images, or video data.

18. A non-transitory computer-readable storage device comprising instructions that cause a programmable processor to:

receive image data containing a visual representation of a scene having a plurality of physical notes, each of the plurality of physical notes having a color;

generate, by the one or more processors, a plurality of indicators, each indicator indicative of a color class of color values of one or more pixel within the image data;

based on the plurality of indicators, determine a content location within the image data of a first physical note of the plurality of physical notes;

extract, from the image data, content of the first physical note using the content location;

creating a digital note representative of the first physical note; and

associating the content extracted from the image data with the digital note.

19. The computer-readable storage device of claim 18, wherein the instructions cause the programmable processor to:

create the digital note corresponding to the first physical note as a record within a database to store attributes associated with the physical note, and

update the record with the database to associate the respective record with the content.

20. The computer-readable storage device of claim 19, wherein the instructions cause the programmable processor to:

determine, based on the image data, a group for the first physical note based on at least one attribute of the first physical note; and

update the respective record with the database to associate the digital note with the group.