Note recognition for overlapping physical notes

• US 10,296,789 B2
• Filed: 07/19/2016
• Issued: 05/21/2019
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Somasundaram, Guruprasad, Sivalingam, Ravishankar, Mahli, David M., Moran, Cristin E., Moore, Richard J., Axelsson, Pontus, Rotstein, Michael, Persson, Roland
• Primary Examiner(s): Baderman, Scott T
• Assistant Examiner(s): Jung, Seung Woon
• Application Number:
  US15/213,598
• Publication Number:
  US 20160328857A1
• Time in Patent Office: 1,036 Days
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 51/066 Format adaptation, e.g. format conversion or compression
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04N 1/04 Scanning arrangements , i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versaH04N1/387 takes precedence

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A computing device includes a processor, an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes, and an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image. The image processing engine determines a boundary of each note in the plurality of overlapping physical notes in the input image, and generates a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

Claims

1. A method comprising:

receiving, by a computing device, an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors;

processing the input image with the computing device to identify each of the overlapping physical notes in the input image, wherein identifying the overlapping physical notes comprises;

classifying pixels of the input image into one of a plurality of color classes, wherein each of the different background colors corresponds to a respective color class of the plurality of color classes, wherein classifying the pixels comprises;

grouping adjacent pixels having the same color class into regions that each correspond to a note of the plurality of overlapping physical notes in the input image; and

determining, based on boundaries of the regions and according to a machine-learning pattern recognition algorithm, boundaries of each of the overlapping physical notes in the input image;

extracting, by the computing device, content for each of the respective overlapping physical notes based on the respective determined boundary;

creating, by the computing device, a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and

associating, by the computing device, the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

2. The method of claim 1, further comprising:

applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and

generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

3. The method of claim 1, further comprising:

determining, by the computing device, whether each of the plurality of overlapping physical notes in the input image comprises a border color different from the background color;

upon determining that the plurality of overlapping physical notes includes one or more physical notes with a border color, applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and

generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

4. The method of claim 2, further comprising:

displaying, by the computing device, the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and

displaying, by the computing device, the plurality of segmented digital notes on a second portion of the graphical user interface.

5. The method of claim 1, further comprising:

determining, by the computing device, whether at least one of the overlapping physical notes in the input image includes one or more fiducial markers using a marker detection module;

based on the determination that at least one of the overlapping physical notes in the input image includes one or more fiducial markers, detecting one or more fiducial markers which indicate the size and shape of the note;

determining, by the computing device and based on the fiducial markers, the boundary of each of the physical notes in the plurality of overlapping physical notes;

segmenting, by the computing device and based on the determined boundaries, each of the physical notes in the plurality of overlapping physical notes;

generating, by the computing device, a plurality of segmented digital notes corresponding to the plurality of segmented physical notes.

6. The method of claim 1, wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the size and shape of each note in the plurality of overlapping physical notes.

7. The method of claim 1, wherein the machine-learning pattern recognition algorithm includes a support vector machine.

8. A non-transitory computer-readable medium comprising instructions for causing a programmable processor to:

receive an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors;

process the input image to identify each of the overlapping physical notes in the input image, wherein the instructions that cause the processor to identify the overlapping physical notes comprise instructions that cause the processor to;

classify pixels of the input image into one of a plurality of color classes, wherein each of the different background colors corresponds to a respective color class of the plurality of color classes, wherein the instructions that cause the processor to classify the pixels comprise instructions that cause the processor to;

group adjacent pixels having the same color class into regions that each correspond to a note of the plurality of overlapping physical notes in the input image; and

determine, based on boundaries of the regions and according to a machine-learning pattern recognition algorithm, boundaries of each of the overlapping physical notes in the input image;

extract content for each of the respective overlapping physical notes based on the respective determined boundary;

generate a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and

associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

9. The non-transitory computer-readable medium of claim 8, wherein the machine-learning pattern recognition algorithm includes a support vector machine.

10. A computing device comprising:

a processor; and

a memory storing instructions that, when executed, cause the processor to;

receive an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors;

process the input image with the computing device to identify the plurality of overlapping physical notes in the input image, wherein, to identify the plurality of overlapping physical notes, the processor is configured to;

classify pixels of the input image into one of a plurality of color classes, wherein each of the different background colors corresponds to a respective color class of the plurality of color classes, wherein, to classify the pixels, the processor is configured to;

group adjacent pixels having the same color class into regions that each correspond to a note of the plurality of overlapping physical notes in the input image; and

determine, based on boundaries of the regions and according to a machine-learning pattern recognition algorithm, boundaries of each of the overlapping physical notes in the input image;

extract content for each of the respective overlapping physical notes based on the respective determined boundary;

generate a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and

associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

11. The device of claim 10, wherein the image collection module includes an image capture device configured to capture the input image.

12. The device of claim 10, wherein the image collection module includes a communication interface configured to receive the input image.

13. The device of claim 10, wherein the image processing engine is further configured to:

apply a color segmenting process to segment each physical note in the plurality of overlapping physical notes; and

generate a plurality of segmented digital notes corresponding to the plurality of segmented notes.

14. The device of claim 13, where the image processing engine is further configured to:

display the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and

display the plurality of segmented digital notes on a second portion of the graphical user interface.

15. The device of claim 10, wherein the image processing engine is further configured to:

determine whether the plurality of overlapping physical notes in the input image comprise a different border color;

upon determining that the plurality of overlapping physical notes includes two or more physical notes with different border colors, applying a color segmenting process to segment each physical note in the plurality of overlapping physical notes; and

generate a plurality of segmented digital notes corresponding to the plurality of segmented notes.

16. The device of claim 10, wherein the image processing engine is further configured to:

determine whether at least one of the plurality of physical notes in the input image includes one or more fiducial markers using a marker detection module,

based on the determination that at least one of the plurality of notes in the input image includes one or more fiducial markers, detecting one or more fiducial markers which indicate the size and shape of the note;

determine based on the fiducial markers, the boundary of each physical note in the plurality of overlapping physical notes;

segment based on the determined boundaries, each of the physical notes in the plurality of overlapping physical notes;

generate a plurality of segmented digital notes corresponding to the plurality of segmented physical notes.

17. The device of claim 10, wherein identifying the plurality of overlapping physical notes includes identifying the size and shape of each note in the plurality of overlapping physical notes.

18. The computing device of claim 10, wherein the machine-learning pattern recognition algorithm includes a support vector machine.