System and method for creating, storing, and utilizing composite images of a geographic location

• US 6,895,126 B2
• Filed: 01/11/2001
• Issued: 05/17/2005
• Est. Priority Date: 10/06/2000
• Status: Expired due to Term

• Current Assignee: Vederi LLC
• Original Assignee: Enrico Di Bernardo, Luis F. Goncalves
• Inventors: Di Bernardo, Enrico, Goncalves, Luis F.
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/758,717
• Publication Number:
  US 20020047895A1
• Time in Patent Office: 1,587 Days
• Field of Search: 701/25, 701/213, 382/284, 382/294, 382/103, 382/295, 382/100, 382/305, 382/104, 382/312, 707/102, 348/36, 348/37, 348/169, 348/175
• US Class Current: 382/284
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/387 Organisation of map data, e.g. version management or database structures
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

A system and method synthesizing images of a locale to generate a composite image that provide a panoramic view of the locale. A video camera moves along a street recording images of objects along the street. A GPS receiver and inertial navigation system provide the position of the camera as the images are being recorded. The images are indexed with the position data provided by the GPS receiver and inertial navigation system. The composite image is created on a column-by-column basis by determining which of the acquired images contains the desired pixel column, extracting the pixels associated with the column, and stacking the columns side by side. The composite images are stored in an image database and associated with a street name and number range of the street being depicted in the image. The image database covers a substantial amount of a geographic area allowing a user to visually navigate the area from a user terminal.

Claims

1. A method for creating a composite image of at least one object, the method comprising:

recording a plurality of images of the objects using an image recording device moving along a path;

obtaining position information of the image recording device as the image recording device moves along the path;

associating the position information with the plurality of images; and

processing image data acquired from the plurality of images to create a composite image representing the object, wherein the composite image simulates a view of the object from a particular location that is situated off of the path of the image recording device.

2. The method of claim 1, wherein the obtaining of position information comprises obtaining Global Positioning System (GPS) data.

3. The method of claim 1, wherein the obtaining of position information comprises:

obtaining acceleration information of the image recording device as the image recording device moves along the path; and

deriving the position information from the acceleration information.

4. The method of claim 1, wherein the obtaining of position information comprises deriving the position information of the image recording device at a particular time by computing a motion of objects in a plurality of images closest to the particular time.

5. The method of claim 1, wherein the associating of the position information comprises correlating times associated with the position information to times of acquisition of the plurality of images.

6. The method of claim 1, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

7. The method of claim 6, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

8. The method of claim 6, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting an image recorded closest to the calculated time.

9. The method of claim 6, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

computing a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

10. The method of claim 1, wherein the image data comprises a set of pixel values.

11. The method of claim 1, wherein the recording of the images comprises recording the images from multiple viewing directions using multiple image recording devices.

12. A method for creating a composite image database of a particular geographic area, the method comprising:

recording a plurality of images of a series of objects using an image recording device moving along a path;

obtaining position information of the image recording device as the image recording device moves along the path;

associating the position information with the plurality of images;

processing image data acquired from the plurality of images to create a composite image depicting a view of the series of objects from a particular location;

partitioning the path into a plurality of discrete segments;

associating the composite image to one of the discrete segments; and

storing the composite image and association information in the composite image database.

13. The method of claim 12, wherein the recording of the images comprises recording the images from multiple viewing directions using multiple image recording devices.

14. The method of claim 12, wherein the obtaining of position information comprises obtaining Global Positioning System (GPS) data.

15. The method of claim 12, wherein the obtaining of position information comprises:

obtaining acceleration information of the image recording device as the image recording device moves along the path; and

deriving the position information from the acceleration information.

16. The method of claim 12, wherein the obtaining of position information comprises deriving the position information of the image recording device at a particular time by computing a motion of objects in a plurality of images closest to the particular time.

17. The method of claim 12, wherein the associating of the position information comprises correlating times associated with the position information to times of acquisition of the plurality of images.

18. The method of claim 12, wherein the object is located on a first side of the path and the composite image simulates a view of the object from the particular location on a second side of the path opposite from the first side.

19. The method of claim 12, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

20. The method of claim 19, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

21. The method of claim 19, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

selecting an image recorded closest to the calculated time.

22. The method of claim 19, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

calculating a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

23. The method of claim 12, wherein the image data comprises a set of pixel values.

24. The method of claim 12, wherein the partitioning of the path into a plurality of discrete segments comprises:

detecting an intersection on the path; and

identifying the position of the intersection.

25. The method of claim 24 wherein the detecting of the intersection comprises detecting a point of maximum curvature on the path.

26. The method of claim 12 wherein each discrete segment is associated with a plurality of composite images, each composite image depicting a portion of the associated segment.

27. The method of claim 12, wherein each discrete segment is a portion of a street and the method further comprises associating each discrete segment with a street name and number range.

28. A system for creating a composite image of a series of objects, the system comprising:

an image recording device moving along a path and recording a plurality of images of the objects;

a means for receiving position information of the image recording device as the image recording device moves along the path; and

a processor receiving the plurality of images and position information, the processor being operable to execute program instructions including;

associating the position information with the plurality of images; and

processing image data acquired from the plurality of images to create a composite image representing the object, wherein the composite image simulates a view of the object from a particular location that is situated off of the path of the image recording device.

29. The system of claim 28 wherein multiple image recording devices record the images from multiple viewing directions.

30. The system of claim 28, wherein the means for receiving position information comprises means for obtaining acceleration information of the image recording device as the image recording device moves along the path.

31. The system of claim 28, wherein the means for receiving position information comprises means for receiving GPS data.

32. The system of claim 28, wherein the associating of the position information comprises correlating times associated with the position information to times of acquisition of the plurality of images.

33. The system of claim 28, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

34. The system of claim 33, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

35. The system of claim 33, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting an image recorded closest to the calculated time.

36. The system of claim 33, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

calculating a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

37. The system of claim 28, wherein the image data comprises a set of pixel values.

38. A system for creating a composite image database of a particular geographic area, the system comprising:

an image recording device moving along a path and recording a plurality of images of a series of objects;

means for obtaining position information of the image recording device as the image recording device moves along the path;

means for associating the position information with the plurality of images;

means for processing image data acquired from the plurality of images to create a composite image depicting a view of the series of objects from a particular location;

means for partitioning the path into a plurality of discrete segments;

means for associating the composite image to one of the discrete segments; and

means for storing the composite image and association information in the composite image database.

39. The system of claim 38 wherein multiple image recording devices record the images from multiple viewing directions.

40. The system of claim 38, wherein the means for associating comprises means for correlating times associated with the position information to times of acquisition of the plurality of images.

41. The system of claim 38, wherein the object is located on a first side of the path and the composite image simulates a view of the object from the particular location on a second side of the path opposite from the first side.

42. The system of claim 38, wherein the means for processing the image data comprises:

means for identifying a plurality of optical rays originating from the particular location;

means for selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

means for extracting image data for the corresponding optical ray from the selected images; and

means for combining the extracted image data to form the composite image.

43. The system of claim 42, wherein the means for selecting an image for each optical ray comprises:

means for determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

means for calculating a time associated with the determined position; and

means for identifying an image recorded at the calculated time.

44. The system of claim 42, wherein the means for selecting an image for each optical ray comprises:

means for determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

means for calculating a time associated with the determined position; and

means for selecting an image recorded closest to the calculated time.

45. The system of claim 42, wherein the means for selecting of an image for each optical ray comprises:

means for determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

means for calculating a time associated with the determined position;

means for selecting a plurality of images recorded closest to the calculated time;

means for calculating a motion of objects in the selected images; and

means for creating a new image frame for the calculated time based on the computed motion of objects.

46. The system of claim 38, wherein the image data comprises a set of pixel values.

47. The system of claim 38, wherein the means for partitioning the path comprises:

means for detecting an intersection on the path; and

means for identifying the position of the intersection.

48. The system of claim 38 wherein each discrete segment is associated with a plurality of composite images, each composite image depicting a portion of the associated segment.

49. The system of claim 38, wherein each discrete segment is a portion of a street and the system further comprises means for associating each discrete segment with a street name and number range.

50. A computer-readable medium comprising:

a program code embodied in the computer readable medium for creating a composite image of a series of objects from a plurality of images acquired by an image recording device moving along a path, the computer-readable program segment comprising instructions for performing the steps of;

identifying a plurality of optical rays originating from a particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

51. The computer-readable medium of claim 50, wherein the step of selecting an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

52. The computer-readable medium of claim 50, wherein the step of selecting an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting an image recorded closest to the calculated time.

53. The computer-readable medium of claim 50, wherein the step of selecting an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

calculating a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

54. The computer-readable medium of claim 50, wherein the image data comprises a set of pixel values.