Recognition and Tracking Using Invisible Junctions

• US 20090015676A1
• Filed: 07/11/2007
• Published: 01/15/2009
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Ke, Qifa
• Granted Patent: US 8,184,155 B2
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

The present invention uses invisible junctions which are a set of local features unique to every page of the electronic document to match the captured image to a part of an electronic document. The present invention includes: an image capture device, a feature extraction and recognition system and database. When an electronic document is printed, the feature extraction and recognition system captures an image of the document page. The features in the captured image are then extracted, indexed and stored in the database. Given a query image, usually a small patch of some document page captured by a low resolution image capture device, the features in the query image are extracted and compared against those stored in the database to identify the query image. The present invention also includes methods for recognizing and tracking the viewing region and look at point corresponding to the input query image. This information is combined with a rendering of the original input document to generate a new graphical user interface to the user. This user interface can be displayed on a conventional browser or even on the display of an image capture device.

Claims

1. A method for recognizing and tracking a point and viewing region of an electronic document using invisible junction features, the method comprising:

receiving an input image;

first attempting to recognize a electronic document page corresponding to the input image using invisible junctions, and generating previous recognition results; and

processing the input image using previous recognition results to determine the point, viewing region and the electronic document page.

2. The method of claim 1, wherein:

receiving the input image includes receiving a plurality of input images;

first attempting to recognize the electronic document page performs recognition on the plurality of input images; and

the previous recognition results include one or more results of recognition on the plurality of input images.

3. The method of claim 1, wherein the previous recognition results includes an estimated mode of an input device.

4. The method of claim 3, wherein the previous recognition results includes a user intention generated from input device motion state.

5. The method of claim 3, wherein the mode of the input device is one from the group of hovering, scanning, zooming and abrupt change.

6. The method of claim 1, wherein processing the input image includes estimating a mode of an input device.

7. The method of claim 1, wherein the previous recognition results includes a tracking state of an input device.

8. The method of claim 7, wherein the tracking state is one from the group of a video tracking state, a recognizing state, a paper tracking state or a drift correcting state.

9. The method of claim 1, wherein processing the input image includes determining a current state of tracking.

10. The method of claim 1, wherein processing the input image includes:

estimating a mode of an input device;

determining a current state of tracking;

performing video tracking if an estimated mode of the input device is other than hovering; and

performing paper tracking if the electronic document page is recognized.

11. The method of claim 10, wherein performing video tracking includes tracking input device motion as a project of transformation across a plurality of video frames with a first video frame is a reference frame.

12. The method of claim 10, wherein performing paper tracking includes tracking input device motion with respect to paper with a paper plane acting as a reference frame.

13. The method of claim 1, wherein processing the input image includes correcting for drift.

14. The method of claim 1, wherein processing the input image includes recognizing the point and the viewing region in the electronic document page.

15. The method of claim 1, further comprising displaying the point and the viewing region.

16. The method of claim 14, wherein the point and the viewing region are displayed overlaid upon a representation of the electronic document.

17. The method of claim 1, further comprising a second attempting to recognize the electronic document page corresponding to the input image using invisible junctions and the recognition results from the first attempting to recognize.

18. A system for recognizing and tracking a point and viewing region of an electronic document using invisible junction features, the system comprising:

a recognizer unit having an input and an output for retrieving an electronic document page using an invisible junction feature descriptor, the input of the recognizer coupled to receive an input image, the output of the recognizer providing information about the electronic document page; and

a tracker having an input and an output for receiving information about the electronic document page and identifying the point and viewing region for the electronic document page, the input of the tracker coupled to the output of the recognizer.

19. The system of claim 18, wherein the tracker creates a link between the between the input image and the point and viewing region on paper.

20. The system of claim 18, wherein recognizer unit generates previous recognition results and provides them to the tracker, and the tracker processes the input image using previous recognition results to determine the point, viewing region and the electronic document page.

21. The system of claim 18, wherein the previous recognition results includes an estimated mode of an input device.

22. The system of claim 18, wherein the previous recognition results includes a user intention generated from input device motion state.

23. The system of claim 18, wherein the previous recognition results includes a tracking state of an input device.

24. The system of claim 18, wherein the tracker is a video tracker having an input and an output for tracking input device motion as a project of transformation across a plurality of video frames with a first video frame is a reference frame, the input of the video tracker coupled to the output of the recognizer.

25. The system of claim 18, wherein the tracker is a paper tracker for tracking input device motion with respect to paper with a paper plane acting as a reference frame, the input of the paper tracker coupled to the output of the recognizer.

26. The system of claim 18, further comprising a drift corrector adapted for communication with the tracker, the drift corrector for correcting a link between paper and a video frame.

27. The system of claim 18, wherein the tracker outputs information usable to show the point and viewing region.

28. The system of claim 18, wherein the tracker outputs information for generating a representation of the electronic document overlaid with the point and the viewing region.