System and method for producing multi-angle views of an object-of-interest from images in an image dataset

• US 8,531,449 B2
• Filed: 12/18/2007
• Issued: 09/10/2013
• Est. Priority Date: 12/18/2007
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: Navteq B.V.
• Inventors: Lynch, James, Gale, William
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US12/002,746
• Publication Number:
  US 20090153549A1
• Time in Patent Office: 2,093 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

Disclosed are a system and method for creating multi-angle views of an object-of-interest from images stored in a dataset. A user specifies the location of an object-of-interest. As the user virtually navigates through the locality represented by the image dataset, his current virtual position is determined. Using the user'"'"'s virtual position and the location of the object-of-interest, images in the image dataset are selected and interpolated or stitched together, if necessary, to present to the user a view from his current virtual position looking toward the object-of-interest. The object-of-interest remains in the view no matter where the user virtually travels. From the same image dataset, another user can select a different object-of-interest and virtually navigate in a similar manner, with his own object-of-interest always in view. The object-of-interest also can be “virtual,” added by computer-animation techniques to the image dataset. For some image datasets, the user can virtually navigate through time as well as through space.

Claims

1. A method for using images in a photo image dataset to produce multi-angle views of an object-of-interest in a navigation system, the method comprising:

specifying a first viewpoint location;

based, at least in part, on a photo image read from the photo image dataset, producing a first photo image view of the object-of-interest as seen from the first viewpoint location, the photo image dataset including multiple photo images taken via a collection vehicle at different respective points along a road and accessible for one or more users remote from the collection vehicle, wherein the photo image dataset is stored on a non-transitory computer readable medium;

providing the first photo image view of the object-of-interest to display the first photo image view locally to at least one of the one or more users;

specifying a second viewpoint location;

associating at least portions of the photo image with range distance data;

transforming photo images read from the photo image dataset based on the range distance data to create another image with a second photo image view of the object-of-interest as seen from the second viewpoint location, wherein the second photo image view is different than the first photo image view; and

providing the second photo image view of the object-of-interest to display the second photo image view locally to the at least one or another of the one or more users.

2. The method of claim 1 wherein the photo image dataset comprises multiple images taken from multiple points of view, at least some of which images overlap.

3. The method of claim 1 wherein the photo image dataset represents a real-world locality.

4. The method of claim 1 wherein the object-of-interest is selected from the group consisting of:

a structure, a landmark, and a location.

5. The method of claim 1 further comprising:

specifying a location of the object-of-interest,

wherein specifying the location of the object-of-interest comprises specifying a location in 3D space.

6. The method of claim 1 further comprising:

stitching together the photo images taken on different image-capture passes along a driving path.

7. The method of claim 1 wherein the photo image dataset comprises a time coordinate for at least some photo images in the photo image dataset, the method further comprising:

specifying a first time associated with the first viewpoint location;

wherein producing the first photo image view of the object-of-interest as seen from the first viewpoint location is based, at least in part, on photo images read from the photo image dataset whose time coordinates are correlated to the first time associated with the first viewpoint location.

8. The method of claim 7 further comprising:

specifying a second time associated with the second viewpoint location;

wherein producing the second photo image view of the object-of-interest as seen from the second viewpoint location is based, at least in part, on photo images read from the photo image dataset whose time coordinates are correlated to the second time associated with the second viewpoint location.

9. The method of claim 1 wherein the photo image dataset comprises data representing a first object and a second object, the first object at least partially occluding the second object from at least one point of view, the method further comprising:

determining relative positions of the first and second objects; and

based, at least in part, on the relative positions of the first and second objects, transforming one or more photo images of the photo image dataset to reveal the second object.

12. The method of claim 1 further comprising:

stitching together the photo images taken on different image-capture passes of the collection vehicle, or another collection vehicle.

13. The method of claim 1, wherein transforming photo images read from the photo image dataset comprises interpolating data from the photo images to produce the second photo image view.

14. The method of claim 1, wherein the range distance data is collected by a range finding device in the collection vehicle.

15. The method of claim 14, wherein the range finding device is a light detection and ranging (LIDAR) device.

16. The method of claim 1, wherein pixels of the photo image are associated with range distance data.

10. A non-transitory computer-readable medium containing computer-executable instructions for performing a method for using images in a photo image dataset to produce multi-angle views of an object-of-interest in a navigation system, the method comprising:

specifying a first viewpoint location;

based, at least in part, on a photo image read from the photo image dataset, producing a first photo image view of the object-of-interest as seen from the first viewpoint location, the photo image dataset including multiple photo images taken via a collection vehicle at different respective points along a road and accessible for one or more users remote from the collection vehicle;

providing the first photo image view of the object-of-interest to display the first photo image view;

specifying a second viewpoint location;

associating at least portions of the photo image with range distance data

interpolating the photo images read from the photo image dataset according to the range distance data to create another image with a second photo image view of the object-of-interest as seen from the second viewpoint location wherein the second photo image view is different than the first photo image view; and

providing the second photo image view of the object-of-interest to display the second photo image view.

17. The non-transitory computer-readable medium of claim 10, wherein the range distance data is collected by a range finding device in the collection vehicle.

11. A navigation device for using images in a video image dataset to produce multi-angle views of an object-of-interest, the navigation device method comprising:

a user interface configured to specify a first viewpoint location;

a display configured to display a first video image view of the object-of-interest as seen from the first viewpoint location, wherein based, at least in part, on a video image read from the video image dataset, the first video image view of the object-of-interest as seen from the first viewpoint location is generated to display, the video image dataset including multiple video images taken via a collection vehicle at different respective points along a road and accessible for one or more users remote from the collection vehicle;

wherein the user interface is further configured to specify a second viewpoint location; and

wherein the display is further configured to display a second video image view of the object-of-interest as seen from the second viewpoint location, wherein based, at least in part, on video images read from the video image dataset, the second video image view of the object-of-interest as seen from the second viewpoint location is generated to display, and wherein the second video image view of the object-of-interest is generated by stitching together video images in the video image dataset according to range distance data associated with the video images, the second video image view different than the first video image.