User interface for three-dimensional navigation

• US 20090019402A1
• Filed: 07/11/2007
• Published: 01/15/2009
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hart, Peter E., Hull, Jonathan J., Ke, Qifa, Kishi, Hidenobu
• Granted Patent: US 8,276,088 B2
• Time in Patent Office: Days
• US Class Current: 715/849
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1456 based on user interactions

Abstract

The present invention uses invisible junctions which are a set of local features unique to every page of the electronic document to match the captured image to a part of an electronic document. The present invention includes: an image capture device, a feature extraction and recognition system and database. When an electronic document is printed, the feature extraction and recognition system captures an image of the document page. The features in the captured image are then extracted, indexed and stored in the database. Given a query image, usually a small patch of some document page captured by a low resolution image capture device, the features in the query image are extracted and compared against those stored in the database to identify the query image. The present invention also includes methods for recognizing and tracking the viewing region and look at point corresponding to the input query image. This information is combined with a rendering of the original input document to generate a new graphical user interface to the user. This user interface can be displayed on a conventional browser or even on the display of an image capture device.

Claims

1. A method for utilizing invisible junction features for generating a graphical user interface, the method comprising:

receiving a first input image;

recognizing a electronic document corresponding to the first input image using invisible junctions; and

retrieving information associated with the electronic document; and

presenting the retrieved information in a graphical user interface.

2. The method of claim 1, wherein the recognizing determines a first camera pose and further comprising determining a second camera pose, determining a change in pose and updating the presenting of the retrieved information based on the change in pose.

3. The method of claim 1, further comprising:

receiving a second input image;

determining a change in camera pose from the first input image and the second input image;

modifying the graphical user interface based upon the change in camera pose.

4. The method of claim 3, wherein the change in camera pose is one from the group of left, right, up and down.

5. The method of claim 4, wherein the graphical user interface display a view of a three-dimensional model of an object.

6. The method of claim 5, wherein the graphical user interface is modified to adjust the view of the three-dimensional model to the left if the change in camera pose is left and to adjust the view of the three-dimensional model to the right if the change in camera pose is right.

7. The method of claim 6, wherein the graphical user interface is modified to adjust the view of the three-dimensional model upward if the change in camera pose is up and to adjust the view of the three-dimensional model downward if the change in camera pose is down.

8. The method of claim 1, wherein the graphical user interface is modified to provide data related to an object described by the electronic document in response to a change in the input image.

9. The method of claim 3, wherein the change in camera pose is one from the group of left, right, up, down, east, west, north, south, clockwise and counterclockwise.

10. The method of claim 9, wherein the graphical user interface is modified based upon the change in camera pose and the retrieved information displayed in the graphical user interface.

11. The method of claim 10, wherein the graphical user interface is modified to adjust the view of the three-dimensional model if the change in camera pose is left, right, up, down and the retrieved information is a three-dimensional model.

12. The method of claim 10, wherein the graphical user interface is modified to adjust the position of an aerial view if the change in camera pose is east, west, north, south, clockwise and counterclockwise, and the retrieved information is an aerial view.

13. The method of claim 1, wherein the information is a three-dimensional model.

14. The method of claim 1, wherein the information is an aerial view of terrain.

15. The method of claim 1, wherein the information is a representation of the electronic document with a viewing region having a size and shape corresponding to the input image overlaid on the representation of the electronic document.

16. The method of claim 1, wherein the method further comprises:

receiving an instruction from a user;

retrieving additional information related to the electronic document; and

displaying the additional information related to the electronic document in the graphical user interface.

17. The method of claim 16, wherein the retrieved information is the input image and the additional information is an aerial view of terrain.

18. The method of claim 16, the wherein method further comprises:

receiving a second input image;

determining a change in camera pose from the first input image and the second input image;

modifying the graphical user interface based upon the change in camera pose.

19. The method of claim 17, wherein modifying the graphical user interface includes displaying a three-dimensional model.

20. The method of claim 17, wherein determining the change in camera pose includes determining a position of the camera imaging plane with respect to a document plane.

21. The method of claim 3, wherein the change in camera pose is a function of a position of a printed document corresponding to the input image and a camera position relative to that printed document.

30. The system of claim 1, wherein the information is a three-dimensional model.

31. The system of claim 1, wherein the information is an aerial view of terrain.

32. The system of claim 1, wherein the information is a representation of the electronic document with a viewing region having a size and shape corresponding to the input image overlaid on the representation of the electronic document.

33. The system of claim 1, wherein the user interface module receive an instruction from a user and in response retrieves and displays additional information in the graphical user interface related to the electronic document.

34. The system of claim 1, wherein the user interface module determines a camera pose by determining a position of the camera imaging plane with respect to a document plane.

22. A system for utilizing invisible junction features for generating a graphical user interface, the system comprising:

a recognizer having an input and an output for identifying an electronic document using an invisible junction feature descriptor from an input image, the input of the recognizer coupled to receive the input image, the output of the recognizer providing information about the electronic document; and

a user interface module having an input and an output for retrieving information associated with the electronic document and presenting the retrieved information in a graphical user interface in response to changes in the input image, the input of the user interface module coupled to the output of the recognizer.

23. The system of claim 22, wherein the recognizer determines a first camera pose, a second camera pose and a change in camera pose and the recognizer send the change in camera pose to the user interface module.

24. The system of claim 22, wherein the recognizer determines a change in camera pose from a first input image and a second input image, and wherein the user interface module modifies the graphical user interface based upon the change in camera pose.

25. The system of claim 23, wherein the change in camera pose is one from the group of left, right, up and down.

26. The system of claim 25, wherein the user interface module causes the graphical user interface to display a view of a three-dimensional model of an object.

27. The system of claim 26, wherein the user interface module modifies the graphical user interface to a view of the three-dimensional model to the left if the change in camera pose is left;

to a view of the three-dimensional model to the right if the change in camera pose is right;

to a view of the three-dimensional model upward if the change in camera pose is up; and

to a view of the three-dimensional model downward if the change in camera pose is down.

28. The system of claim 22 wherein the user interface module modifies the graphical user interface to provide data related to an object described by the electronic document in response to a change in the input image.

29. The system of claim 24 wherein the user interface module modifies the graphical user interface based upon the change in camera pose and the retrieved information displayed in the graphical user interface.