Note recognition and management using multi-color channel non-marker detection

• US 9,542,756 B2
• Filed: 06/03/2015
• Issued: 01/10/2017
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Axelsson, Pontus, Persson, Roland, Ergenekon, Emre Berge, kerlund, Linus, Somasundaram, Guruprasad
• Primary Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US14/729,965
• Publication Number:
  US 20150269751A1
• Time in Patent Office: 587 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Techniques are described for creating and manipulating software notes representative of physical notes. A computing device comprises a processor and a note identification module executable on the processor and configured to separate an input image into a plurality of channelized input images. Each of the channelized input images are associated with a different color. The note identification module is configured to apply edge detection and feature extraction to identify polygons within each of the channelized input images and select, from the polygons from the channelized input images, a representative polygon for each of the physical notes in the input image.

Claims

1. A method, comprising:

receiving, by a computing device, an input image of an environment having a plurality of physical notes;

separating, by the computing device, the input image into a plurality of channelized input images, each of the channelized input images associated with a different color;

generating, by the computing device, a plurality of transformed channelized input images by applying edge detection to each of the plurality of channelized input images;

processing, by the computing device, the plurality of transformed channelized input images to identify, based on edges detected within the transformed channelized input images, polygons for respective physical notes in the input image, wherein each identified polygon corresponds to one of the plurality of physical notes;

creating, by the computing device, a plurality of digital notes, each of the digital notes representing a particular one of the physical notes;

extracting, by the computing device and from the input image, a respective sub-image for each respective physical note based on the identified polygons for the respective physical note; and

associating, by the computing device and in a database, the extracted sub-image for each respective physical note with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

2. The method of claim 1, further comprising:

displaying, by the computing device, the plurality of digital notes on a graphical user interface, wherein the plurality of digital notes are overlaid the plurality of physical notes within the input image.

3. The method of claim 1, further comprising:

generating, by the computing device, an overlay image including the plurality of digital notes overlaid the plurality of physical notes within the input image; and

storing, by the computing device, the overlay image to a digital data storage device for retrieval by a user.

4. The method of claim 1, further comprising:

prior to processing the input image to separate the input image into a plurality of different color channels, normalizing the input image to create a normalized input image to remove noise and shadows from the input image.

5. The method of claim 1, wherein normalizing, further includes thresholding the input image to remove noise and shadows from the input image.

6. The method of claim 1, wherein the plurality of different color channels comprises at least one of a group consisting of:

a Y luminance channel;

a U chrominance channel; and

a V chrominance channel.

7. The method of claim 1, wherein the input image comprises a gray scale image and each of the different colors comprise a different range of grayscale values.

8. The method of claim 1, wherein at least two of the identified polygons have different shapes from each other.

9. The method of claim 1, wherein at least two of the identified polygons have different sizes from each other.

10. The method of claim 1, further comprising removing, by the computing device, one or more of the polygons that do not match criteria,

wherein the criteria includes pre-determined size of the polygon or a requirement for a substantially rectangular shape.

11. A method, comprising:

receiving, by a computing device, an input image of an environment having a plurality of physical notes;

generating, by the computing device, a transformed image by applying edge detection to the input image;

processing, by the computing device, the transformed image to identify, based on edges detected within the transformed input image, a plurality of polygons corresponding to the physical notes within the input image;

grouping, by the computing device, the plurality of polygons from the transformed image into a plurality of groups, each of the groups comprising one or more of the polygons and corresponding to different one of the physical notes;

selecting, by the computing device, a representative one of the polygons from each of the plurality of groups for each respective physical note in the input image;

creating, by the computing device, a plurality of digital notes, each of the digital notes representing a particular one of the physical notes;

extracting, by the computing device and from the input image, a respective sub-image for each respective physical note based on the selected representative polygon for the respective physical note; and

associating, by the computing device and in a database, the extracted sub-image for each respective physical note with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

12. The method of claim 11, further comprising:

displaying, by the computing device, the plurality of digital notes on a graphical user interface, wherein the plurality of digital notes are overlaid the plurality of physical notes within the input image.

13. The method of claim 11, wherein selecting, by the computing device, a representative one of the polygons further comprises analyzing the plurality of polygon sets to select one polygon in each polygon set as a representative polygon.

14. The method of claim 13, wherein analyzing the plurality of polygon sets to select one polygon in each polygon set as a representative polygon includes selecting the polygon associated with the respective physical note in the input image based on an evaluation of the relative similarity of each polygon in the polygon set to an expected shape of the corresponding note in the input image.

15. The method of claim 13, wherein analyzing the plurality of polygon sets to select one polygon in each polygon set as a representative polygon includes selecting the polygon associated with the respective physical note in the input image based on a comparison of the symmetry of each polygon in the polygon set,

wherein the comparison of the symmetry is based on a score assigned by the computing device to each polygon in the polygon set, and wherein the polygon with the highest score is selected as the representative polygon.

16. A computing device comprising:

a processor;

a note identification module executable on the processor and configured to;

obtain an input image of an environment having a plurality of physical notes,

separate the input image into a plurality of channelized input images, each of the channelized input images associated with a different color,

apply edge detection and use detected edges to identify polygons within each of the channelized input images,

select, based on the polygons from the channelized input images, a representative polygon for each of the physical notes in the input image; and

a digital note generation module executable on the processor and configured to;

create a plurality of digital notes, each of the digital notes representing a particular one of the physical notes,

extract, from the input image, a respective sub-image for each respective physical note based on the selected representative polygon for the respective physical note, and

associate, in a database, the extracted sub-image for each respective physical note with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

17. The device of claim 16, wherein the selected representative polygon defines a shape and boundary of a sub-image extracted from the input image for the respective digital note.

18. The device of claim 16, wherein the computing device is configured to display the plurality of digital notes on a graphical user interface, and wherein the plurality of digital notes overlay the plurality of physical notes within the input image.

19. The device of claim 16, wherein the note identification module is further configured to normalize the input image to remove noise and shadows from the input image.

20. The device of claim 16, wherein the plurality of different color channels comprises at least one of a group consisting of:

a Y luminance channel;

a U chrominance channel; and

a V chrominance channel.

21. The device of claim 16, wherein the note identification module is further configured to determine the polygon associated with the respective physical note in the input image based on an evaluation of the relative similarity of the polygons to an expected shape of the corresponding note in the input image.