Note recognition and management using multi-color channel non-marker detection

• US 9,082,184 B2
• Filed: 10/15/2014
• Issued: 07/14/2015
• Est. Priority Date: 10/16/2013
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Axelsson, Pontus, Persson, Roland, Ergenekon, Emre Berge, kerlund, Linus
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US14/514,607
• Publication Number:
  US 20150104098A1
• Time in Patent Office: 272 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Techniques are described for creating and manipulating software notes representative of physical notes. A computing device comprises a processor and a note identification module executable on the processor and configured to separate an input image into a plurality of channelized input images. Each of the channelized input images are associated with a different color. The note identification module is configured to apply edge detection and feature extraction to identify polygons within each of the channelized input images and select, from the polygons from the channelized input images, a representative polygon for each of the physical notes in the input image.

Claims

1. A method comprising:

receiving, by a computing device, an input image of an environment having a plurality of physical notes;

separating, by the computing device, the input image into a plurality of channelized input images, each of the channelized input images associated with a different color;

generating, by the computing device, a plurality of transformed channelized input images by applying edge detection to each of the plurality of channelized input images;

processing, by the computing device, the plurality of transformed channelized input images to identify, based on edges detected within the transformed channelized input images, a plurality of polygons corresponding to the physical notes within the input image;

grouping, by the computing device, the plurality of polygons from the plurality of transformed channelized input images into a plurality of groups, each of the groups comprising one or more of the polygons and corresponding to different one of the physical notes; and

selecting, by the computing device, a representative one of the polygons from each of the plurality of groups for each respective physical note in the input image.

2. The method of claim 1 further comprising generating, by the computing device, a plurality of digital notes based on the identified data of the plurality of representative polygons, wherein the identified data includes the shape and boundary of sub-images extracted for each digital note in the plurality of digital notes from the input image.

3. The method of claim 1 further comprising displaying, by the computing device, the plurality of digital notes on a graphical user interface, wherein the plurality of digital notes are overlaid on the plurality of physical notes within the input image.

4. The method of claim 1, further comprising:

generating, by the computing device, an overlay image including the plurality of digital notes overlaid on the plurality of physical notes within the input image; and

storing, by the computing device, the overlay image to a digital data storage device for retrieval by a user.

5. The method of claim 1, further comprising, prior to processing the input image to separate the input image into a plurality of different color channels, normalizing the input image to create a normalized input image to remove noise and shadows from the input image.

6. The method of claim 5, wherein normalizing further includes thresholding the input image to remove noise and shadows from the input image.

7. The method of claim 1, wherein grouping, by the computing device, the plurality of polygons comprises grouping the polygons from the different color channels based on size, location, and orientation.

8. The method of claim 1, wherein the plurality of different color channels comprises at least one of a group consisting of:

a blue color channel;

a green color channel; and

a red color channel.

9. The method of claim 1, wherein selecting, by the computing device, a representative one of the polygons further comprises analyzing the plurality of polygon sets to select one polygon in each polygon set as a representative polygon.

10. The method of claim 9, wherein analyzing the plurality of polygon sets to select one polygon in each polygon set as a representative polygon includes selecting the polygon associated with the respective physical note in the input image based on an evaluation of the relative similarity of each polygon in the polygon set to an expected shape of the corresponding note in the input image.

11. The method of claim 9, wherein analyzing the plurality of polygon sets to select one polygon in each polygon set as a representative polygon includes selecting the polygon associated with the respective physical note in the input image based on a comparison of the symmetry of each polygon in the polygon set,

wherein the comparison of the symmetry is based on a score assigned by the computing device to each polygon in the polygon set, and wherein the polygon with the highest score is selected as the representative polygon.

12. The method of claim 1, further comprising removing, by the computing device, one or more of the polygons that do not match criteria,

wherein the criteria includes pre-determined size of the polygon or a requirement for a substantially rectangular shape.

13. The method of claim 1, wherein the input image comprises at least one of a group consisting of:

a digital image;

a digital video; and

a continuous digital scan.

14. The method of claim 1, wherein the input image comprises a gray scale image and each of the different colors comprise a different range of grayscale values.

15. The method of claim 1, wherein the plurality of physical notes in the input image comprises at least one of a group consisting of:

a 7.62 by 7.62 centimeter (cm) note;

a 66.04 by 99.06 cm poster;

a triangular metal sign;

a sticker;

a legal size sheet of paper;

an A3 size sheet of paper;

an A4 size sheet of paper; and

a 8.5 by 11 inch sheet of paper.

16. A computing device comprising:

a processor; and

a note identification module executable on the processor and configured to separate an input image into a plurality of channelized input images, each of the channelized input images associated with a different color,

wherein the note identification module is configured to apply edge detection and feature extraction to identify polygons within each of the channelized input images, and

wherein the note identification module is configured to select, from the polygons from the channelized input images, a representative polygon for each of the physical notes in the input image.

17. The device of claim 16 further comprising a digital note generation module configured to generate a respective digital note for each of the physical notes based on the corresponding representative polygon, wherein the representative polygon defines a shape and boundary of a sub-image extracted from the input image for the respective digital note.

18. The device of claim 16 wherein the computing device is configured to display the plurality of digital notes on a graphical user interface, and wherein the plurality of digital notes overlay the plurality of physical notes within the input image.

19. The device of claim 16 further comprising, a digital data storage device configured to store an overlay image for retrieval by a user.

20. The device of claim 19, wherein the overlay image includes the plurality of digital notes overlaid on the plurality of physical notes in the input image.

21. The device of claim 16, wherein the note identification module is further configured to normalize the input image to remove noise and shadows from the input image.

22. The device of claim 16, wherein the plurality of different color channels comprises at least one of a group consisting of:

a blue color channel;

a green color channel; and

a red color channel.

23. The device of claim 16, wherein the note identification module is further configured to select the polygon associated with the respective physical note in the input image based on an evaluation of the relative similarity of the polygons to an expected shape of the corresponding note in the input image.

24. A computer system having at least one processor configured to:

receive an input image of an environment having a plurality of physical notes;

separate the input image into a plurality of channelized input images, each of the channelized input images associated with a different color;

generate a plurality of transformed channelized input images by applying edge detection to each of the plurality of channelized input images;

process the plurality of transformed channelized input images to identify, based on edges detected within the transformed channelized input images, a plurality of polygons corresponding to the physical notes within the input image;

group the plurality of polygons from the plurality of transformed channelized input images into a plurality of groups, each of the groups comprising one or more of the polygons and corresponding to different one of the physical notes; and

select a representative one of the polygons from each of the plurality of groups for each respective physical note in the input image.

25. A non-transitory computer readable medium comprising instructions that cause a programmable processor to:

receive an input image of an environment having a plurality of physical notes;

separate the input image into a plurality of channelized input images, each of the channelized input images associated with a different color;

generate a plurality of transformed channelized input images by applying edge detection to each of the plurality of channelized input images;

process the plurality of transformed channelized input images to identify, based on edges detected within the transformed channelized input images, a plurality of polygons corresponding to the physical notes within the input image;

group the plurality of polygons from the plurality of transformed channelized input images into a plurality of groups, each of the groups comprising one or more of the polygons and corresponding to different one of the physical notes; and

select a representative one of the polygons from each of the plurality of groups for each respective physical note in the input image.