System and method for creating, storing, and utilizing composite images of a geographic location

• US 20020047895A1
• Filed: 01/11/2001
• Published: 04/25/2002
• Est. Priority Date: 10/06/2000
• Status: Active Grant

• Current Assignee: Vederi LLC
• Original Assignee: Enrico Di Bernardo, Luis F. Goncalves
• Inventors: Goncalves, Luis F., Bernardo, Enrico Di
• Granted Patent: US 6,895,126 B2
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/387 Organisation of map data, e.g. version management or database structures
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

A system and method synthesizing images of a locale to generate a composite image that provide a panoramic view of the locale. A video camera moves along a street recording images of objects along the street. A GPS receiver and inertial navigation system provide the position of the camera as the images are being recorded. The images are indexed with the position data provided by the GPS receiver and inertial navigation system. The composite image is created on a column-by-column basis by determining which of the acquired images contains the desired pixel column, extracting the pixels associated with the column, and stacking the columns side by side. The composite images are stored in an image database and associated with a street name and number range of the street being depicted in the image. The image database covers a substantial amount of a geographic area allowing a user to visually navigate the area from a user terminal.

Claims

1. A method for creating a composite image of at least one object, the method comprising:

recording a plurality of images of the object using an image recording device moving along a path;

obtaining position information of the image recording device as the image recording device moves along the path;

associating the position information with the plurality of images; and

processing image data acquired from the plurality of images to create a composite image representing the object.

2. The method of claim 1, wherein the object is located on a first side of the path and the composite image simulates a view of the object from a particular location on a second side of the path opposite from the first side.

3. The method of claim 2, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

4. A method for creating a composite image of at least one object, the method comprising:

recording a plurality of images of the objects using an image recording device moving along a path;

obtaining position information of the image recording device as the image recording device moves along the path;

associating the position information with the plurality of images; and

processing image data acquired from the plurality of images to create a composite image depicting a view of the object from a particular location.

5. The method of claim 4, wherein the object is located on a first side of the path and the composite image simulates a view of the object from the particular location on a second side of the path opposite from the first side.

6. The method of claim 4, wherein the obtaining of position information comprises obtaining Global Positioning System (GPS) data.

7. The method of claim 4, wherein the obtaining of position information comprises:

obtaining acceleration information of the image recording device as the image recording device moves along the path; and

deriving the position information from the acceleration information.

8. The method of claim 4, wherein the obtaining of position information comprises deriving the position information of the image recording device at a particular time by computing a motion of objects in a plurality of images closest to the particular time.

9. The method of claim 4, wherein the associating of the position information comprises correlating times associated with the position information to times of acquisition of the plurality of images.

10. The method of claim 4, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

11. The method of claim 10, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

12. The method of claim 10, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting an image recorded closest to the calculated time.

13. The method of claim 10, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

computing a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

14. The method of claim 4, wherein the image data comprises a set of pixel values.

15. The method of claim 4, wherein the recording of the images comprises recording the images from multiple viewing directions using multiple image recording devices.

17. The method of claim 16, wherein the recording of the images comprises recording the images from multiple viewing directions using multiple image recording devices.

18. The method of claim 16, wherein the obtaining of position information comprises obtaining Global Positioning System (GPS) data.

19. The method of claim 16, wherein the obtaining of position information comprises:

obtaining acceleration information of the image recording device as the image recording device moves along the path; and

deriving the position information from the acceleration information.

20. The method of claim 16, wherein the obtaining of position information comprises deriving the position information of the image recording device at a particular time by computing a motion of objects in a plurality of images closest to the particular time.

21. The method of claim 16, wherein the associating of the position information comprises correlating times associated with the position information to times of acquisition of the plurality of images.

22. The method of claim 16, wherein the object is located on a first side of the path and the composite image simulates a view of the object from the particular location on a second side of the path opposite from the first side.

23. The method of claim 16, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

24. The method of claim 23, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

25. The method of claim 23, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

selecting an image recorded closest to the calculated time.

26. The method of claim 23, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

calculating a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

27. The method of claim 16, wherein the image data comprises a set of pixel values.

28. The method of claim 16, wherein the partitioning of the path into a plurality of discrete segments comprises:

detecting an intersection on the path; and

identifying the position of the intersection.

29. The method of claim 28 wherein the detecting of the intersection comprises detecting a point of maximum curvature on the path.

30. The method of claim 16 wherein each discrete segment is associated with a plurality of composite images, each composite image depicting a portion of the associated segment.

31. The method of claim 16, wherein each discrete segment is a portion of a street and the method further comprises associating each discrete segment with a street name and number range.

16. A method for creating a composite image database of a particular geographic area, the method comprising:

recording a plurality of images of a series of objects using an image recording device moving along a path;

obtaining position information of the image recording device as the image recording device moves along the path;

associating the position information with the plurality of images;

processing image data acquired from the plurality of images to create a composite image depicting a view of the series of objects from a particular location;

partitioning the path into a plurality of discrete segments;

associating the composite image to one of the discrete segments; and

storing the composite image and association information in the composite image database.

32. In a system including an image database and a user terminal having a screen and an input device, a method for visually navigating a geographic area from a user terminal, the method comprising:

storing a plurality of images in the image database representing substantially all static objects within the geographic area;

identifying a current location in the geographic area;

retrieving an image corresponding to the current location from the image database;

monitoring a change of the current location in the geographic area; and

retrieving an image corresponding to the changed location.

33. The method of claim 32, wherein image database resides at a remote site and a host computer at the remote site receives a request via a communications network for an image corresponding to the current or changed location, retrieves the requested image from the image database, and transmits the retrieved image to the user terminal via the communications network.

34. The method of claim 32, wherein the image database resides at the user terminal and the method further comprises:

displaying the image corresponding to the current location on the screen of the user terminal; and

updating the image on the screen with the image corresponding to the changed location.

35. The method of claim 32 further comprising retrieving a map of a portion of the geographic area for displaying the map on the screen of the user terminal.

36. The method of claim 35, wherein the current or changed location is identified by a user selection of the location on the map using the input device.

37. The method of claim 32, wherein the current or changed location is identified by a specific address entered by the user using the input device.

38. The method of claim 32, wherein the retrieving of the image corresponding to the current or changed location comprises:

identifying a street segment including the current or changed location;

identifying a position on the street segment corresponding to the current or changed location; and

identifying an image associated with said position.

39. The method of claim 38, wherein the image simulates a view of objects on one side of the street segment and the method further comprises retrieving a second image depicting a view of objects on an opposite side of the street segment in response to a user request.

40. The method of claim 32, wherein the image is a composite image created by processing a plurality of image frames acquired from an image recording device moving through the geographic area.

41. The method of claim 32 further comprising:

displaying an indicia that information is available about an object in the image; and

retrieving information about the object in response to a user request to display the information on the screen of the user terminal.

42. The method of claim 32 further comprising:

displaying a navigation button on the screen of the user terminal; and

retrieving the image associated with the changed location from the image database upon actuation of the navigation button using the user input device.

43. The method of claim 32 further comprising:

displaying a map of a portion of the geographic area;

identifying the current location with an identifier on the map; and

advancing the identifier on the map from the current location to the changed location.

44. The method of claim 32 further comprising identifying a current viewing direction associated with the retrieved image.

46. The system of claim 45 wherein multiple image recording devices record the images from multiple viewing directions.

47. The system of claim 45, wherein the object is located on a first side of the path and the composite image simulates a view of the object from the particular location on a second side of the path opposite from the first side.

48. The system of claim 45, wherein the means for receiving position information comprises means for obtaining acceleration information of the image recording device as the image recording device moves along the path.

49. The system of claim 45, wherein the means for receiving position information comprises means for receiving GPS data.

50. The system of claim 45, wherein the associating of the position information comprises correlating times associated with the position information to times of acquisition of the plurality of images.

51. The system of claim 45, wherein the processing of the image data comprises:

identifying a plurality of optical rays originating from the particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.

52. The system of claim 51, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

53. The system of claim 51, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting an image recorded closest to the calculated time.

54. The system of claim 51, wherein the selecting of an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

calculating a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

55. The system of claim 45, wherein the image data comprises a set of pixel values.

57. The system of claim 56 wherein multiple image recording devices record the images from multiple viewing directions.

58. The system of claim 56, wherein the means for associating comprises means for correlating times associated with the position information to times of acquisition of the plurality of images.

59. The system of claim 56, wherein the object is located on a first side of the path and the composite image simulates a view of the object from the particular location on a second side of the path opposite from the first side.

60. The system of claim 56, wherein the means for processing the image data comprises:

means for identifying a plurality of optical rays originating from the particular location;

means for selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

means for extracting image data for the corresponding optical ray from the selected images; and

means for combining the extracted image data to form the composite image.

61. The system of claim 60, wherein the means for selecting an image for each optical ray comprises:

means for determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

means for calculating a time associated with the determined position; and

means for identifying an image recorded at the calculated time.

62. The system of claim 60, wherein the means for selecting an image for each optical ray comprises:

means for determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

means for calculating a time associated with the determined position; and

means for selecting an image recorded closest to the calculated time.

63. The system of claim 60, wherein the means for selecting of an image for each optical ray comprises:

means for determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

means for calculating a time associated with the determined position;

means for selecting a plurality of images recorded closest to the calculated time;

means for calculating a motion of objects in the selected images; and

means for creating a new image frame for the calculated time based on the computed motion of objects.

64. The system of claim 56, wherein the image data comprises a set of pixel values.

65. The system of claim 56, wherein the means for partitioning the path comprises:

means for detecting an intersection on the path; and

means for identifying the position of the intersection.

66. The system of claim 56 wherein each discrete segment is associated with a plurality of composite images, each composite image depicting a portion of the associated segment.

67. The system of claim 56, wherein each discrete segment is a portion of a street and the system further comprises means for associating each discrete segment with a street name and number range.

69. The system of claim 68, wherein the image database resides at a remote site and the system further comprises a host computer including means for receiving a request for an image corresponding to the current or changed location, means for retrieving the requested image from the image database, and means for transmitting the retrieved image to the user terminal.

70. The system of claim 68, wherein the image database resides at the user terminal and the system further comprises:

a display screen for displaying the image of the current location; and

means for updating the image on the screen with the image corresponding to the changed location.

71. The system of claim 68 further comprising means for retrieving a map of a portion of the geographic area for displaying the map on the screen of the user terminal.

72. The system of claim 71, wherein the current or changed location is identified by a user selection of the location on the map using in the input device.

73. The system of claim 68, wherein the current or changed location is identified by a specific address entered by the user using an input device.

74. The system of claim 68, wherein the means for retrieving the image corresponding to the current or changed location comprises:

means for identifying a street segment including the current or changed location;

means for identifying a position on the street segment corresponding to the current or changed location; and

means for identifying an image associated with said position.

75. The system of claim 74, wherein the image simulates a view of objects on one side of the street segment and the system further comprises means for retrieving a second image depicting a view of objects on an opposite side of the street segment in response to a user request.

76. The system of claim 68 further comprising means for processing a plurality of image frames acquired from an image recording device moving through the geographic area.

77. The system of claim 68 further comprising:

means for indicating that information is available about an object in the image;

means for retrieving information about the object in response to a user request to display the information on the screen of the user terminal.

78. The system of claim 68 further comprising:

means for navigating the image database; and

means for retrieving the image associated with the next location from the image database as the user navigates the image database.

79. The system of claim 68 further comprising:

means for displaying a map of a portion of the geographic area;

means for identifying the current location on the map; and

means for identifying the next location on the map as the user navigates the image database.

80. The system of claim 68 further comprising means for identifying a current viewing direction associated with the retrieved image.

82. The computer-readable medium of claim 79, wherein the step of selecting an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position; and

identifying an image recorded at the calculated time.

83. The computer-readable medium of claim 79, wherein the step of selecting an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting an image recorded closest to the calculated time.

84. The computer-readable medium of claim 79, wherein the step of selecting an image for each optical ray comprises:

determining the particular position on the path where the image recording device would have been located for recording an image including the corresponding optical ray;

calculating a time associated with the determined position;

selecting a plurality of images recorded closest to the calculated time;

calculating a motion of objects in the selected images; and

creating a new image frame for the calculated time based on the computed motion of objects.

85. The computer-readable medium of claim 79, wherein the image data comprises a set of pixel values.

45. A system for creating a composite image of a series of objects, the system comprising:

an image recording device moving along a path and recording a plurality of images of the objects;

a means for receiving position information of the image recording device as the image recording device moves along the path; and

a processor receiving the plurality of images and position information, the processor being operable to execute program instructions including;

associating the position information with the plurality of images; and

processing image data acquired from the plurality of images to create a composite image depicting a view of the object from a particular location.

56. A system for creating a composite image database of a particular geographic area, the system comprising:

an image recording device moving along a path and recording a plurality of images of a series of objects;

means for obtaining position information of the image recording device as the image recording device moves along the path;

means for associating the position information with the plurality of images;

means for processing image data acquired from the plurality of images to create a composite image depicting a view of the series of objects from a particular location;

means for partitioning the path into a plurality of discrete segments;

means for associating the composite image to one of the discrete segments; and

means for storing the composite image and association information in the composite image database.

68. A system for visually navigating a geographic area from a user terminal, the system comprising:

means for storing a plurality of images in the image database representing substantially all static objects within the geographic area;

means for identifying a current location in the geographic area;

means for retrieving an image corresponding to the current location from the image database;

means for monitoring a change of the current location in the geographic area; and

means for retrieving an image corresponding to the changed location.

81. A computer-readable medium comprising:

a program code embodied in the computer readable medium for creating a composite image of a series of objects from a plurality of images acquired by an image recording device moving along a path, the computer-readable program segment comprising instructions for performing the steps of;

identifying a plurality of optical rays originating from a particular location;

selecting for each optical ray an image including a corresponding optical ray originating from a position on the path;

extracting image data for the corresponding optical ray from the selected images; and

combining the extracted image data to form the composite image.