Recognition and tracking using invisible junctions

• US 8,184,155 B2
• Filed: 07/11/2007
• Issued: 05/22/2012
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Ke, Qifa, Hull, Jonathan J.
• Primary Examiner(s): BATES, KEVIN T
• Application Number:
  US11/776,530
• Publication Number:
  US 20090015676A1
• Time in Patent Office: 1,777 Days
• Field of Search: 348/169, 348/142, 382/180, 382/174
• US Class Current: 348/142
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

The present invention uses invisible junctions which are a set of local features unique to every page of the electronic document to match the captured image to a part of an electronic document. The present invention includes: an image capture device, a feature extraction and recognition system and database. When an electronic document is printed, the feature extraction and recognition system captures an image of the document page. The features in the captured image are then extracted, indexed and stored in the database. Given a query image, usually a small patch of some document page captured by a low resolution image capture device, the features in the query image are extracted and compared against those stored in the database to identify the query image. The present invention also includes methods for recognizing and tracking the viewing region and look at point corresponding to the input query image. This information is combined with a rendering of the original input document to generate a new graphical user interface to the user. This user interface can be displayed on a conventional browser or even on the display of an image capture device.

Claims

1. A method for recognizing and tracking a point and a viewing region of an electronic document using invisible junction features, the method comprising:

receiving an input image;

applying a distance transformation to generate invisible junctions within the input image;

determining the point at one of the invisible junctions;

retrieving a previous recognition result;

processing the input image using the previous recognition result to identify the point and the viewing region within the electronic document page; and

identifying an electronic document page corresponding to the input image based on the invisible junctions, the previous recognition result and the point.

2. The method of claim 1, wherein:

receiving the input image includes receiving a plurality of input images;

identifying the electronic document page comprises performing recognition on the plurality of input images; and

wherein the previous recognition result is associated with the plurality of input images.

3. The method of claim 1, wherein the previous recognition result includes an estimated mode of an input device.

4. The method of claim 3, wherein the previous recognition result includes a user intention generated from an input device motion state.

5. The method of claim 3, wherein the mode of the input device is one from the group of hovering, scanning, zooming and abrupt change.

6. The method of claim 1, wherein processing the input image includes estimating a mode of an input device.

7. The method of claim 1, wherein the previous recognition result includes a tracking state of an input device.

8. The method of claim 7, wherein the tracking state is one from the group of a video tracking state, a recognizing state, a paper tracking state or a drift correcting state.

9. The method of claim 1, wherein processing the input image includes determining a current state of tracking.

10. The method of claim 1, wherein processing the input image includes:

estimating a mode of an input device;

determining a current state of tracking;

performing video tracking if an estimated mode of the input device is other than hovering; and

performing paper tracking if the electronic document page is recognized.

11. The method of claim 10, wherein performing video tracking includes tracking input device motion as a projective transformation across a plurality of video frames with a first video frame as a reference frame.

12. The method of claim 10, wherein performing paper tracking includes tracking input device motion with respect to paper with a paper plane acting as a reference frame.

13. The method of claim 1, wherein processing the input image includes correcting for drift.

14. The method of claim 1, wherein processing the input image includes estimating a feature orientation for determining the input image orientation.

15. The method of claim 1, further comprising displaying the point and the viewing region.

16. The method of claim 1, wherein the point and the viewing region are displayed overlaid upon a representation of the electronic document.

17. The method of claim 1, further comprising identifying the electronic document page a second time.

18. A system for recognizing and tracking a point and a viewing region of an electronic document using invisible junction features, the system comprising:

a processor;

a feature extraction unit stored on a memory and executed by the processor, the feature extraction unit for receiving an input image, applying a distance transformation to generate invisible junctions within the input image and determining the point at one of the invisible junctions;

a recognizer unit coupled to the feature extraction unit, the recognizer unit for retrieving a previous recognition result;

a tracker coupled to the recognizer unit, the tracker for processing the input image using the previous recognition result to identify the point and the viewing region within the electronic document page; and

a feature retrieval unit coupled to the tracker, the feature retrieval unit for identifying an electronic document page corresponding to the input image based on the invisible junctions, the previous recognition result and the point.

19. The system of claim 18, wherein the tracker creates a link between the input image and the point and viewing region on paper.

20. The system of claim 18, wherein the tracker estimates a feature orientation for determining the input image orientation.

21. The system of claim 18, wherein the previous recognition result includes an estimated mode of an input device.

22. The system of claim 18, wherein the previous recognition result includes a user intention generated from input device motion state.

23. The system of claim 18, wherein the previous recognition result includes a tracking state of an input device.

24. The system of claim 18, wherein the tracker is a video tracker having an input and an output for tracking input device motion as a projective transformation across a plurality of video frames with a first video frame as a reference frame, the input of the video tracker coupled to the output of the recognizer.

25. The system of claim 18, wherein the tracker is a paper tracker for tracking input device motion with respect to paper with a paper plane acting as a reference frame, the input of the paper tracker coupled to the output of the recognizer.

26. The system of claim 18, further comprising a drift corrector adapted for communication with the tracker, the drift corrector for correcting a link between paper and a video frame.

27. The system of claim 18, wherein the tracker outputs information usable to show the point and viewing region.

28. The system of claim 18, wherein the tracker outputs information for generating a representation of the electronic document overlaid with the point and the viewing region.