User interface for three-dimensional navigation

• US 8,276,088 B2
• Filed: 07/11/2007
• Issued: 09/25/2012
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Ke, Qifa, Hart, Peter E., Hull, Jonathan J., Kishi, Hidenobu
• Primary Examiner(s): Stevens, Robert
• Assistant Examiner(s): BECKER, SHASHI KAMALA
• Application Number:
  US11/827,530
• Publication Number:
  US 20090019402A1
• Time in Patent Office: 1,903 Days
• Field of Search: 715/768, 715/764, 715/788
• US Class Current: 715/768
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1456 based on user interactions

Abstract

The present invention uses invisible junctions which are a set of local features unique to every page of the electronic document to match the captured image to a part of an electronic document. The present invention includes: an image capture device, a feature extraction and recognition system and database. When an electronic document is printed, the feature extraction and recognition system captures an image of the document page. The features in the captured image are then extracted, indexed and stored in the database. Given a query image, usually a small patch of some document page captured by a low resolution image capture device, the features in the query image are extracted and compared against those stored in the database to identify the query image. The present invention also includes methods for recognizing and tracking the viewing region and look at point corresponding to the input query image. This information is combined with a rendering of the original input document to generate a new graphical user interface to the user. This user interface can be displayed on a conventional browser or even on the display of an image capture device.

Claims

1. A method for utilizing invisible junction features for generating a graphical user interface, the method comprising:

receiving a first input image from an image capture device;

detecting a skeleton within the first input image;

detecting locations of invisible junctions within the first input image based on the skeleton;

identifying an electronic document corresponding to the first input image and a first pose of the image capture device using the invisible junctions, the first pose including a position and an orientation of the image capture device;

retrieving information associated with the electronic document; and

providing the retrieved information at a viewing angle in the graphical user interface, wherein the viewing angle of the retrieved information is based on the first pose.

2. The method of claim 1, further comprising:

identifying a second pose of the image capture device;

determining a change between the first pose and the second pose of the image capture device; and

updating the providing of the retrieved information based on the change.

3. The method of claim 1, further comprising:

receiving a second input image;

identifying a second pose of the image capture device;

determining a change between the first pose and the second pose of the image capture device; and

modifying the graphical user interface based on the change.

4. The method of claim 3, wherein the change is one from a group of left, right, up and down.

5. The method of claim 4, wherein the graphical user interface includes a view of a three-dimensional model of an object.

6. The method of claim 5, wherein the graphical user interface is modified to adjust the view of the three-dimensional model to the left if the change is left and to adjust the view of the three-dimensional model to the right if the change is right.

7. The method of claim 6, wherein the graphical user interface is modified to adjust the view of the three-dimensional model upward if the change is up and to adjust the view of the three-dimensional model downward if the change is down.

8. The method of claim 1, wherein the graphical user interface is modified to provide data related to an object described by the electronic document in response to a change in the first input image.

9. The method of claim 3, wherein the change is one from a group of left, right, up, down, east, west, north, south, clockwise and counterclockwise.

10. The method of claim 9, wherein the graphical user interface is modified based on the change and the retrieved information included in the graphical user interface.

11. The method of claim 10, wherein the graphical user interface is modified to adjust a view of a three-dimensional model if the change is one from a group of left, right, up and down and the retrieved information is the three-dimensional model.

12. The method of claim 10, wherein the graphical user interface is modified to adjust a position of an aerial view if the change is one from a group of east, west, north, south, clockwise and counterclockwise, and the retrieved information is the aerial view.

13. The method of claim 1, wherein the retrieved information is a three-dimensional model.

14. The method of claim 1, wherein the retrieved information is an aerial view of terrain.

15. The method of claim 1, wherein the retrieved information is a representation of the electronic document with a viewing region having a size and shape corresponding to the first input image overlaid on the representation of the electronic document.

16. The method of claim 1, wherein the method further comprises:

receiving an instruction from a user;

retrieving additional information related to the electronic document; and

providing the additional information related to the electronic document in the graphical user interface.

17. The method of claim 16, wherein the retrieved information is the first input image and the additional information is an aerial view of terrain.

18. The method of claim 16, wherein the method further comprises:

receiving a second input image;

identifying a second pose of the image capture device;

determining a change between the first pose and the second pose of the image capture device; and

modifying the graphical user interface based on the change.

19. The method of claim 18, wherein modifying the graphical user interface includes providing a three-dimensional model.

20. The method of claim 18, wherein determining the change includes determining a position of an image capture device imaging plane with respect to a document plane.

21. The method of claim 3, wherein the change is a function of a position of a printed document corresponding to the first input image and the position of the image capture device relative to the printed document.

22. A system for utilizing invisible junction features for generating a graphical user interface, the system comprising:

a processor;

a feature extraction unit stored on a memory and executable by the processor, the feature extraction unit for receiving a first input image from an image capture device, for detecting a skeleton within the first input image and for detecting locations of invisible junctions within the first input image based on the skeleton;

a recognizer stored on the memory and executable by the processor, the recognizer for identifying an electronic document corresponding to the first input image and a first pose of the image capture device using the invisible junctions, the recognizer providing retrieved information associated with the electronic document, the first pose including a position and an orientation of the image capture device; and

a user interface module stored on the memory and executable by the processor, the user interface module for receiving the retrieved information associated with the electronic document from the recognizer and providing the retrieved information at a viewing angle in the graphical user interface, wherein the viewing angle of the retrieved information is based on the first pose.

23. The system of claim 22, wherein the recognizer determines a second pose of the image capture device and a change between the first pose and the second pose of the image capture device and the recognizer sends the change to the user interface module.

24. The system of claim 22, wherein the recognizer identifies a second pose of the image capture device, determines a change between the first pose and the second pose of the image capture device, and wherein the user interface module modifies the graphical user interface based on the change.

25. The system of claim 23, wherein the change is one from a group of left, right, up and down.

26. The system of claim 25, wherein the user interface module causes the graphical user interface to provide a view of a three-dimensional model of an object.

27. The system of claim 26, wherein the user interface module modifies the graphical user interface to the view of the three-dimensional model to the left if the change is left, to the view of the three-dimensional model to the right if the change is right, to the view of the three-dimensional model upward if the change is up, and to the view of the three-dimensional model downward if the change is down.

28. The system of claim 22 wherein the user interface module modifies the graphical user interface to provide data related to an object described by the electronic document in response to a change in the first input image.

29. The system of claim 24 wherein the user interface module modifies the graphical user interface based on the change between the first pose and the second pose and the retrieved information provided in the graphical user interface.

30. The system of claim 22, wherein the received information is a three-dimensional model.

31. The system of claim 22, wherein the received information is an aerial view of terrain.

32. The system of claim 22, wherein the received information is a representation of the electronic document with a viewing region having a size and shape corresponding to the first input image overlaid on the representation of the electronic document.

33. The system of claim 22, wherein the user interface module receives an instruction from a user and in response retrieves and provides additional information in the graphical user interface related to the electronic document.

34. The system of claim 22, wherein the recognizer identifies the first pose by determining a position of an image capture device imaging plane with respect to a document plane.

35. A computer program product comprising a non-transitory computer readable storage medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a first input image from an image capture device;

detect a skeleton within the first input image;

detect locations of invisible junctions within the first input image based on the skeleton;

identify an electronic document corresponding to the first input image and a first pose of the image capture device using the invisible junctions, the first pose including a position and an orientation of the image capture device;

retrieve information associated with the electronic document; and

provide the retrieved information at a viewing angle in a graphical user interface, wherein the viewing angle of the retrieved information is based on the first pose.