Note recognition for overlapping physical notes

• US 9,412,174 B2
• Filed: 10/15/2014
• Issued: 08/09/2016
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Somasundaram, Guruprasad, Sivalingam, Ravishankar, Mahli, David M., Moran, Cristin E., Moore, Richard J., Axelsson, Pontus, Rotstein, Michael, Persson, Roland
• Primary Examiner(s): Baderman, Scott
• Assistant Examiner(s): JUNG, SEUNG WOON
• Application Number:
  US14/514,473
• Publication Number:
  US 20150106699A1
• Time in Patent Office: 664 Days
• Field of Search: 715/244
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 51/066 Format adaptation, e.g. format conversion or compression
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04N 1/04 Scanning arrangements , i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versaH04N1/387 takes precedence

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A computing device includes a processor, an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes, and an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image. The image processing engine determines a boundary of each note in the plurality of overlapping physical notes in the input image, and generates a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

Claims

1. A method comprising:

receiving, by a computing device, an input image of an environment having a plurality of overlapping physical notes, each the plurality of physical notes having a background color and a border having a border color different from the background color;

processing the input image with the computing device to apply a color segmenting process to identify, based on the background colors and the border colors of the overlapping physical notes, borders of each of the overlapping physical notes in the input image;

processing the identified borders of each of the overlapping physical notes as fiducial markers for the respective physical note to extract, based on the fiducial markers, content for each of the respective overlapping physical notes, wherein the extracted content for a respective physical note includes portions of the border of the respective overlapping physical note not covered by another physical note of the plurality of overlapping physical notes;

creating, by the computing device, a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and

associating, by the computing device, the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

2. The method of claim 1,

wherein the color segmenting process is applied in response to determining that the background colors of the plurality of overlapping physical notes are different.

3. The method of claim 1, further comprising:

displaying, by the computing device, the input image on a first portion of a graphical user interface; and

displaying, by the computing device, the digital notes on a second portion of the graphical user interface.

4. The method of claim 1, further comprising:

detecting one or more fiducial markers in the input image that are different than the identified borders of each of the overlapping physical notes,

wherein extracting the content from at least one of the overlapping physical notes comprises extracting, by the computing device and based on the identified fiducial markers that are different than the identified borders, the content from the at least one overlapping physical note.

5. The method of claim 1, wherein extracting the content from each of the plurality of overlapping physical notes includes identifying, by the computing device and based on the fiducial markers for the respective physical note, a size and a shape for each of the respective overlapping physical notes.

6. The method of claim 1, wherein extracting the content from each of the plurality of overlapping physical notes includes identifying the plurality of notes according to a pattern recognition algorithm.

7. The method of claim 6, wherein the pattern recognition algorithm includes at least one of a group consisting of:

a support vector machine;

a cross-correlation; and

a template matching.

8. A non-transitory computer-readable medium comprising instructions for causing a programmable processor to:

receive, by a computing device, an input image of an environment having a plurality of overlapping physical notes, each the plurality of physical notes having a background color and a border having a border color different from the background color;

process the input image with the computing device to apply a color segmenting process to identify, based on the background colors and the border colors of the overlapping physical notes, borders of each of the overlapping physical notes in the input image;

process the identified borders of each of the overlapping physical notes as fiducial markers for the respective physical note to extract, based on the fiducial markers, content for each of the respective overlapping physical notes, wherein the extracted content for a particular physical note includes portions of the border of the particular overlapping physical note not covered by another physical note of the plurality of overlapping physical notes;

create a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and

associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

9. A computing device comprising:

one or more processors;

an image collection module executable by the one or more processors to receive an input image of an environment having a plurality of overlapping physical notes, each the plurality of physical notes having a background color and a border having a border color different from the background color; and

an image processing engine executable by the one or more processors to;

process the input image to apply a color segmenting process to identify, based on the background colors and the border colors of the overlapping physical notes, borders of each of the overlapping physical notes in the input image having the border color;

process the identified borders of each of the overlapping physical notes as fiducial markers for the respective physical note to extract, based on the fiducial markers, content for each of the respective overlapping physical notes, wherein the extracted content for a respective physical note includes portions of the border of the respective overlapping physical note not covered by another physical note of the plurality of overlapping physical notes;

create a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and

associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

10. The device of claim 9, wherein the image collection module includes an image capture device configured to capture the input image.

11. The device of claim 9, wherein the image collection module includes a communication interface configured to receive the input image.

12. The device of claim 11, wherein the communication interface includes a wireless communication device configured to receive the input image.

13. The device of claim 11, wherein the communication interface includes a wired communication device configured to receive the input image.

14. The device of claim 9, wherein the image processing engine is executable by the one or more processors to:

apply the color segmenting process in response to determining that the background colors of the plurality of overlapping physical notes are different.

15. The device of claim 9, where the image processing engine further configured to:

display the input image on a first portion of a graphical user interface; and

display the digital notes on a second portion of the graphical user interface.

16. The device of claim 9, wherein the image processing engine is further executable by the one or more processors to:

detect one or more fiducial markers in the input image that are different than the identified borders of each of the overlapping physical notes,

wherein, to extract the content from at least one of the overlapping physical notes, the image processing engine is executable by the one or more processors to determine, based on the identified fiducial markers that are different than the identified borders, the content from the at least one overlapping physical note.

17. The device of claim 9, wherein, to extract the content from each of the plurality of overlapping physical notes, the image processing engine is executable by the one or more processors to:

identify a size and a shape for each of the respective overlapping physical notes.

18. The device of claim 9, wherein, to extract the content from each of the plurality of overlapping physical notes, the image processing engine is executable by the one or more processors to:

identify the plurality of notes according to a pattern recognition algorithm.

19. The device of claim 18, wherein the pattern recognition algorithm includes at least one of a group consisting of:

a support vector machine;

a cross-correlation; and

a template matching.