Image matching in support of mobile navigation

• US 9,683,853 B2
• Filed: 01/23/2009
• Issued: 06/20/2017
• Est. Priority Date: 01/23/2009
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Perttula, Arttu, Carter, Scott, Denoue, Laurent
• Primary Examiner(s): Reyes, Reginald R
• Application Number:
  US12/359,222
• Publication Number:
  US 20100191459A1
• Time in Patent Office: 3,070 Days
• Field of Search: 701200, 701211, 701206, 345427
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0086 Image transceiver H04N2201/0093 and H04N2201/0094 take precedence

Abstract

Method, apparatus and system for improving navigation using mobile devices by adding previously captured media to navigation maps provided by mobile device. Added media include still images and video clips. Media captured at the same location using various camera orientations are available in an image database. Images in database include metadata indicating location, orientation and other parameters associated with each image. User location and pose are matched against metadata indicating location and camera orientation of captured images. User motion may also be matched to metadata indicating camera motion of stored video clips. Other parameters like lighting, seasonal factors, and tasks of walking, biking or driving may also be matched. User pose may be derived from an accelerometer embedded in the mobile device. Some or all of user parameters desired by user may be directly input to mobile device by user instead of being automatically detected by mobile device.

Claims

1. A method of accessing an image database, the method comprising:

receiving image data photographed by a camera;

receiving camera location data provided by a global position system (GPS) locator and camera orientation data corresponding to the photographed image data, wherein the camera orientation data is an angle of a vector extending along a pointing direction of a camera with respect to a coordinate axis;

associating, by using a processor, the camera location data and the camera orientation data as associated metadata with corresponding photographed image data;

forming a database of the photographed image data including the associated metadata;

receiving an instruction to generate a two-dimensional navigation map of a location of the user, the instruction comprising the location of a user and an orientation of a mobile device of a user at the location;

determining a viewing direction of the user based on the orientation of the mobile device of the user;

matching at least one photograph image in the database corresponding to the viewing direction of the user and the location of the user based on the camera location data and the camera orientation data associated with the database of photographed image data;

generating the two-dimensional navigation map of the location of the user, the two-dimensional navigation map comprising the at least one photograph image; and

displaying the generated two-dimensional navigation map.

2. The method of claim 1, wherein the photographed image data comprise video images, and

wherein the associating the camera location data and the camera orientation data as associated metadata comprises deriving a list of time stamped camera location change events and camera orientation change events for each video image and tagging transitions for the video images as the associated metadata.

3. The method of claim 1, wherein the photographed image data comprise picture images, and

wherein the associating the camera location data and the camera orientation data as associated metadata comprises embedding corresponding ones of the camera location and the camera orientation as header with the picture images as the associated metadata.

4. The method of claim 1, further comprising:

forming an image diary by associating the camera orientation data, as associated diary metadata, with the photographed image data.

5. The method of claim 4, further comprising:

receiving a request for an image corresponding to a selected user pose;

searching the image diary by searching the associated diary metadata; and

retrieving the photographed image data having the associated diary metadata matching the selected user pose.

6. A method for navigation using a mobile device of a user, the method comprising:

continuously detecting in real-time mobile device information of the mobile device from a global position system (GPS) locator, the mobile device information comprising a current mobile device location and a current direction of travel of the mobile device;

receiving an instruction to generate a two-dimensional navigation map from the location of the mobile device to a specified destination;

generating the two-dimensional map from the location of the mobile device to the specified location, based on the current direction of travel of the mobile device;

matching at least one photograph image of a database of photograph images associated with a location at which the photograph images are photographed and a camera orientation of a camera capturing the photograph images corresponding to the current direction of travel of the mobile device and the two-dimensional navigation map from the location of the mobile device to the specified location; and

displaying the at least one image together with the two-dimensional navigation map.

7. The method of claim 6, wherein the photograph images comprise pictures and video clips.

8. The method of claim 6, wherein the at least one photograph image is selected to be a confirmatory matched image, confirming a current location of the mobile device, or an anticipatory matched image, anticipating an upcoming location of the mobile device.

9. The method of claim 6, wherein the map is an interpolated map of a path from the location of the mobile device to the specified location.

10. The method of claim 6, wherein the map is an extrapolated map of a path from the location of the mobile device to the specified location.

11. The method of claim 6, wherein the mobile device information further comprises time data, light data, and user task data, the time data including season data, the light data including overcast condition and night condition, and

wherein the matching comprises matching the at least one photograph image by further matching the time data, the light data, and the user task data respectively to time of capture, light condition of capture, and task data associated as further metadata with each of the photograph images.

12. The method of claim 11, wherein the user task data is riding a bicycle,

wherein the at least one photograph image was procured by a camera riding on a bicycle along a same path, during a same season, and under similar lighting conditions as the user, and

wherein the at least one photograph image corresponds to an image of interest to a bicyclist.

13. The method of claim 6, wherein the obtaining comprises receiving an input by the user indicating the current mobile device location and the current direction of travel of the mobile device.

14. A database system comprising:

an interface configured to receive image data photographed by a camera and receiving camera location data provided by a global position system (GPS) locator and camera orientation data corresponding to the photographed image;

a processor configured to associate the camera location data and the camera orientation data, as associated metadata with corresponding photographed image data; and

a database configured to store the photographed image data together with the associated metadata,

wherein the interface is further configured to receive user requests comprising a location of a user and an orientation of a mobile device of a user at the location,

wherein the processor is further configured to determine a viewing direction of the user based on the orientation of the mobile device of the user, match at least one photograph image in the database corresponding to the viewing direction of the user and the location of the user based on the camera location data and the camera orientation data associated with the database of photographed image data, generate the two-dimensional navigation map of the location of the user, the two-dimensional navigation map comprising the at least one photograph image, and control display of the generated two-dimensional navigation map.

15. A mobile device for navigation, the mobile device comprising:

a global positioning system module configured to continuously detect in real-time a location of the mobile device;

an accelerometer configured to continuously detect in real-time a direction of travel of the mobile device;

a processor configured to process an instruction to generate a two-dimensional navigation map from the location of the mobile device to a specified destination, generate the two-dimensional navigation map from the location of the mobile device to the specified location, based on the current direction of travel of the mobile device, match at least one photograph image of a database of photograph images associated with a location at which the photograph images are photographed and a camera orientation of a camera capturing the photograph images corresponding to the current direction of travel of the mobile device and the map from the location of the mobile device to the specified location; and

a display configured to display the at least one photograph image together with the two-dimensional navigation map.