Invisible junction feature recognition for document security or annotation

• US 9,373,029 B2
• Filed: 03/31/2008
• Issued: 06/21/2016
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Uda, Shigeharu, Ke, Qifa
• Primary Examiner(s): Huang, Tsan-Yu J
• Application Number:
  US12/059,583
• Publication Number:
  US 20090016615A1
• Time in Patent Office: 3,004 Days
• Field of Search: 382/305, 382181-231, 385181-231
• US Class Current: 1/1
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

The present invention uses invisible junctions which are a set of local features unique to every page of the electronic document to match the captured image to a part of an electronic document. The present invention includes: an image capture device, a feature extraction and recognition system and database. When an electronic document is printed, the feature extraction and recognition system captures an image of the document page. The features in the captured image are then extracted, indexed and stored in the database. Given a query image, the features in the query image are extracted and compared against those stored in the database to identify the query image. The feature extraction and recognition system of the present invention is integrated into a multifunction peripheral. This allows the feature extraction and recognition system to be used in conjunction with other modules to provide security and annotation applications.

Claims

1. A computer-implemented method for utilizing invisible junction features for performing an action, the method comprising:

receiving, with one or more processors, an input image;

detecting, with the one or more processors, a location for each of the invisible junction features in a skeleton by applying a distance transformation to a binary image of the input image;

determining, with the one or more processors, a region surrounding each of the invisible junction features based on the skeleton, the region including pixels from one or more characters of the input image and the location for each of the invisible junction features being a center of the region;

recognizing, with the one or more processors, an electronic document corresponding to the input image using the location for each of the invisible junction features and the region surrounding each of the invisible junction features; and

performing, with the one or more processors, the action related to the electronic document in response to recognizing the electronic document.

2. The method of claim 1, wherein the steps of receiving, recognizing and performing are performed by a multifunction peripheral.

3. The method of claim 1, wherein the action comprises enforcing a security procedure related to the electronic document.

4. The method of claim 1, wherein performing the action comprises:

determining a security procedure associated with the electronic document; and

processing the input image according to the security procedure.

5. The method of claim 1, wherein performing the action comprises:

requesting a password;

receiving input from a user;

comparing the input from the user to a password for the electronic document; and

processing the input image if the input from the user matches the password for the electronic document input.

6. The method of claim 5, wherein processing the input image comprises one from the group of outputting a copy of the input image, printing the electronic document, sending a copy of the input image as a facsimile message, sending the electronic document as part of a facsimile message, emailing a copy of the input image, emailing the electronic document, sending a message including the input image and sending a message including the electronic document.

7. The method of claim 1, wherein responsive to performing the action, further comprising storing a record indicating the action taken.

8. The method of claim 7, wherein the record includes one from the group of a time, a number of copies output, a type of output, a recipient and whether image or document was acted upon.

9. The method of claim 1, wherein performing the action comprises inhibiting output of the input image or electronic document.

10. The method of claim 1, wherein responsive to performing the action, further comprising generating and sending a notification indicating the action taken on the electronic document.

11. The method of claim 1, wherein performing the action comprises printing the electronic document.

12. The method of claim 1, wherein:

the input image is one from the group of a document that has been updated since the input image was produced, a page or portion of a much longer document and a previously copied or printed version of the document; and

the electronic document comprises one from the group of an updated version of the electronic document, an original electronic document and an entire electronic document.

13. The method of claim 1, wherein the action comprises updating a database with an entry related to the electronic document.

14. The method of claim 13, wherein updating the database comprises modifying a mixed media reality (MMR) record in the database to add one from the group of text, voice, video and metadata.

15. The method of claim 13, wherein updating the database comprises:

prompting a user for updating instructions;

receiving updating instructions from the user; and

updating the database according to the updating instructions.

16. The method of claim 13, wherein updating the database comprises:

prompting a user for processing instructions;

receiving processing instructions from the user; and

processing the electronic document or the input image according to the processing instructions.

17. The method of claim 1, wherein the action comprises adding an annotation for the electronic document.

18. The method of claim 17, wherein adding the annotation comprises modifying a mixed media reality (MMR) record in the database corresponding to the input image to add one from the group of text, voice, video and metadata.

19. The method of claim 17, wherein adding the annotation comprises:

prompting a user for processing instructions;

receiving processing instructions from the user;

receiving an annotation from the user; and

processing the input image and the annotation according to the processing instructions.

20. The method of claim 1, wherein recognizing the electronic document corresponding to the input image using the invisible junction features comprises creating a patch map.

21. The method of claim 20, wherein creating the patch map comprises:

defining a grid having a plurality of cells;

generating a query using invisible junction features of a cell of the grid;

applying the query to a database to generate a list of candidate documents; and

defining a cell in the patch map corresponding to the list of candidate documents.

22. The method of claim 1, wherein recognizing the electronic document corresponding to the input image using the invisible junction features comprises preprocessing invisible junction features prior to generation of a patch map.

23. The method of claim 1, wherein recognizing the electronic document corresponding to the input image using the invisible junction features comprises post-processing using a patch map to determine at least one secure document, a significant portion of which is included in the input image.

24. A computer program product comprising a non-transitory computer readable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive an input image;

detect a location for each of the invisible junction features in a skeleton by applying a distance transformation to a binary image of the input image;

determine a region surrounding each of the invisible junction features based on the skeleton, the region including pixels from one or more characters of the input image and the location for each of the invisible junction features being a center of the region;

recognize an electronic document corresponding to the input image using the location for each of the invisible junction features and the region surrounding each of the invisible junction features; and

perform the action related to the electronic document in response to recognizing the electronic document.

25. The computer program product of claim 24, wherein the steps of receiving, recognizing and performing are performed by a multifunction peripheral.

26. The computer program product of claim 24, wherein the action comprises enforcing a security procedure related to the electronic document.

27. The computer program product of claim 24, wherein performing the action comprises:

determining a security procedure associated with the electronic document; and

processing the input image according to the security procedure.

28. The computer program product of claim 24, wherein performing the action comprises:

requesting a password;

receiving input from a user;

comparing the input from the user to a password for the electronic document; and

processing the input image if the input from the user matches the password for the electronic document input.

29. The computer program product of claim 28, wherein processing the input image comprises one from the group of outputting a copy of the input image, printing the electronic document, sending a copy of the input image as a facsimile message, sending the electronic document as part of a facsimile message, emailing a copy of the input image, emailing the electronic document, sending a message including the input image and sending a message including the electronic document.

30. The computer program product of claim 24, wherein responsive to performing the action, further comprising storing a record indicating the action taken.

31. The computer program product of claim 30, wherein the record includes one from the group of a time, a number of copies output, a type of output, a recipient and whether image or document was acted upon.

32. The computer program product of claim 24, wherein performing the action comprises inhibiting output of the input image or electronic document.

33. The computer program product of claim 24, wherein responsive to performing the action, further comprising generating and sending a notification indicating the action taken on the electronic document.

34. The computer program product of claim 24, wherein performing the action comprises printing the electronic document.

35. The computer program product of claim 24, wherein:

the input image is one from the group of a document that has been updated since the input image was produced, a page or portion of a much longer document and a previously copied or printed version of the document; and

the electronic document comprises one from the group of an updated version of the electronic document, an original electronic document and an entire electronic document.

36. The computer program product of claim 24, wherein the action comprises updating a database with an entry related to the electronic document.

37. The computer program product of claim 36, wherein updating the database comprises modifying a mixed media reality (MMR) record in the database to add one from the group of text, voice, video and metadata.

38. The computer program product of claim 36, wherein updating the database comprises:

prompting a user for updating instructions;

receiving updating instructions from the user; and

updating the database according to the updating instructions.

39. The computer program product of claim 24, wherein updating the database comprises:

prompting a user for processing instructions;

receiving processing instructions from the user; and

processing the electronic document or the input image according to the processing instructions.

40. The computer program product of claim 24, wherein the action comprises adding an annotation for the electronic document.

41. The computer program product of claim 40, wherein adding the annotation comprises modifying a mixed media reality (MMR) record in the database corresponding to the input image to add one from the group of text, voice, video and metadata.

42. The computer program product of claim 24, wherein adding the annotation comprises:

prompting a user for processing instructions;

receiving processing instructions from the user;

receiving an annotation from the user; and

processing the input image and the annotation according to the processing instructions.

43. The computer program product of claim 24, wherein recognizing the electronic document corresponding to the input image using the invisible junction features comprises creating a patch map.

44. The computer program product of claim 43, wherein creating the patch map comprises:

defining a grid having a plurality of cells;

generating a query using invisible junction features of a cell of the grid;

applying the query to a database to generate a list of candidate documents; and

defining a cell in the patch map corresponding to the list of candidate documents.

45. The computer program product of claim 24, wherein recognizing the electronic document corresponding to the input image using the invisible junction features comprises preprocessing the invisible junction features prior to generation of a patch map.