IMAGE MATCHING IN SUPPORT OF MOBILE NAVIGATION

• US 20100191459A1
• Filed: 01/23/2009
• Published: 07/29/2010
• Est. Priority Date: 01/23/2009
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Perttula, Arttu, Denoue, Laurent, Carter, Scott
• Granted Patent: US 9,683,853 B2
• Time in Patent Office: Days
• US Class Current: 701/208
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0086 Image transceiver H04N2201/0093 and H04N2201/0094 take precedence

Abstract

Method, apparatus and system for improving navigation using mobile devices by adding previously captured media to navigation maps provided by mobile device. Added media include still images and video clips. Media captured at the same location using various camera orientations are available in an image database. Images in database include metadata indicating location, orientation and other parameters associated with each image. User location and pose are matched against metadata indicating location and camera orientation of captured images. User motion may also be matched to metadata indicating camera motion of stored video clips. Other parameters like lighting, seasonal factors, and tasks of walking, biking or driving may also be matched. User pose may be derived from an accelerometer embedded in the mobile device. Some or all of user parameters desired by user may be directly input to mobile device by user instead of being automatically detected by mobile device.

Claims

7. A method for navigation using a mobile device of a user, the method comprising:

obtaining mobile device information, the mobile device information comprising mobile device location and mobile device pose;

receiving a map corresponding to a user request;

receiving a matched image corresponding to the map; and

displaying the matched image together with the map,

wherein the matched image is selected from a database including database images by matching the mobile device pose and the mobile device location respectively to camera orientation and camera location associated as metadata with each of the database images.

1. A method for forming an image database, the method comprising:

receiving captured image data;

receiving camera location data and camera orientation data corresponding to the captured image data;

associating the camera location data and the camera orientation data as associated metadata with corresponding captured image data; and

forming a database of the captured image data including the associated metadata.

2. The method of claim 1, wherein the camera orientation data is derived from camera acceleration data.

3. The method of claim 1,

wherein the captured image data comprise video images, and

wherein the associating the camera location data and the camera orientation data as associated metadata comprises deriving a list of time stamped camera location change events and camera orientation change events for each video image and tagging transitions for the video images as the associated metadata.

4. The method of claim 1,

wherein the captured image data comprise picture images, and

wherein the associating the camera location data and the camera orientation data as associated metadata comprises embedding corresponding ones of the camera location and the camera orientation as header with the picture images as the associated metadata.

5. The method of claim 1, further comprising:

forming an image diary by associating the camera orientation data, as associated diary metadata, with the captured image data.

6. The method of claim 5, further comprising:

receiving a request for an image corresponding a selected user pose;

searching the image diary by searching the associated diary metadata; and

retrieving the captured image data having the associated diary metadata matching the selected user pose.

8. The method of claim 7,

wherein the database images comprise pictures and video clips.

9. The method of claim 7, wherein the matched image is selected to be a confirmatory matched image, confirming a current location of the mobile device, or an anticipatory matched image, anticipating an upcoming location of the mobile device.

10. The method of claim 7,

wherein the mobile device information further comprises mobile device motion, and

wherein the matched image is selected from the database by further matching the mobile device motion to camera motion associated as further metadata with each of the database images.

11. The method of claim 7,

wherein the map is a map of the mobile device location.

12. The method of claim 7,

wherein the map is an interpolated map of a path from a first location to a second location, the first location corresponding to an initial mobile device location.

13. The method of claim 10,

wherein the map is an extrapolated map of a path from the mobile device location and being generated according to the mobile device motion, the mobile device motion including mobile device velocity.

14. The method of claim 7,

wherein the mobile device information further comprises time data, light data, and user task data, the time data including season data, the light data including overcast condition and night condition, and

wherein the matched image is selected from the database including the database images by further matching the time data, the light data, an the user task data respectively to time of capture, light condition of capture, and task data associated as further metadata with each of the database images.

15. The method of claim 14,

wherein the user task data is riding a bicycle,

wherein the matched image was procured by a camera riding on a bicycle along a same path, during a same season, and under similar lighting conditions as the user, and

wherein the matched image corresponds to an image of interest to a bicyclist.

16. The method of claim 7, wherein the obtaining of the mobile device information comprises receiving an input by the user indicating the mobile device location and the mobile device pose.

15-1. The method of claim 7,

wherein the obtaining of the mobile device information comprises detecting the mobile device location and mobile device acceleration, and

wherein the mobile device pose is derived from the mobile device acceleration.

17. A database system comprising:

an input interface for receiving captured image data and receiving camera location data and camera orientation data corresponding to the captured image data;

a first processor for associating the camera location data and the camera orientation data, as associated metadata, with a corresponding captured image data;

a database for storing the captured image data together with the associated metadata as stored images; and

a receiver for receiving user requests including user pose and user location;

a second processor for determining matched images, the matched images matching the user pose and the user location to the camera location data and camera orientation data of the stored images; and

a transmitter for transmitting the matched images to a user sending the user request.

18. A mobile device for navigation, the mobile device comprising:

a global positioning system module for detecting mobile device location;

an accelerometer for detecting mobile device acceleration;

a processor for deriving user pose from the mobile device acceleration;

a transceiver for transmitting a user request and receiving a map corresponding to the user request and receiving a matched image corresponding to the map; and

a display for displaying the matched image together with the map,

wherein the matched image is selected from a database including database images by matching the user pose and the user device location respectively to camera orientation and camera location associated as metadata with each of the database images.

19. A navigation system comprising:

a mobile device having navigation capabilities; and

an image database in communication with the mobile device,

wherein the mobile device collects and transmits mobile device location data and mobile device pose data to a machine having access to the image database,

wherein the mobile device location data and the mobile device pose data are matched to metadata indicating camera location and camera orientation associated with an image in the images database to obtain matched images,

wherein the matched image is transmitted to the mobile device, and

wherein the matched image is displayed with a map on the mobile device.

20. The navigation system of claim 19,

wherein the map is selected from a group consisting of a map of a current user location, a map of a path between a first user location and a destination location, and a map being extrapolated from the first user location responsive to motion of the mobile device, and

wherein the images comprise pictures and video clips.