Synchronization between image data and location information for panoramic image synthesis

• US 6,335,754 B1
• Filed: 06/17/1998
• Issued: 01/01/2002
• Est. Priority Date: 12/03/1997
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Mixed Reality Systems Laboratory Inc.
• Inventors: Katayama, Akihiro, Endo, Takaaki
• Primary Examiner(s): Rao, Andy
• Application Number:
  US09/098,546
• Time in Patent Office: 1,294 Days
• Field of Search: 348/36, 348/37, 348/39, 348/42, 348/43, 348/44, 348/45, 348/46, 348/47, 348/49, 348/50, 348/52, 348/118, 348/113, 348/120, 348/159, 348/148, 348/149
• US Class Current: 348/37
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/42646 for reading from or writing on a non-volatile solid state storage medium, e.g. DVD, CD-ROM
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal
• CPC Class Codes: H04N 9/8233 the additional signal being a character code signal

Abstract

An image recording apparatus uses a plurality of cameras point in a plurality of discrete azimuth directions so as to realize a walkthrough space. The apparatus stores, in video tapes, data frames of images generated by sequentially sensing an environment and time codes TC that respectively specify the data frames; acquires location information LOC associated with the image sensing location of each data frame, and stores the location information together with acquisition time information indicating an acquisition time TIM of the location information in a disk HD; and stores a pair of time code of the data frame and acquisition time information corresponding to the location information acquired at the time of generation of the data frame in the HD.

Claims

1. A method of recording images obtained by sensing an object using a plurality of cameras which point in a plurality of directions so as to obtain a basis for forming a sequence of three dimensional image spaces, comprising the steps of:

storing, in a first memory, data frames of images generated by sequentially sensing an object using a camera of said plurality of cameras with specifying information, including a time code, that specifies each data frame;

using the time code of a specific one of said plurality of cameras as a reference time code, and calculating differences between the time code of said specific one camera and time codes of the other cameras;

acquiring location in formation indicating an image sensing location of said camera, and storing, in a second memory, the location information together with acquisition time information indicating cm acquisition time when the location information is acquired; and

storing, in a third memory, a pair of specifying information that specifies each data frame, and acquisition time information corresponding to the location information acquired at the time when the data frame is generated.

2. The method according to claim 1, wherein said first memory comprises a sequential memory, and the specifying information is a frame number indicating a recording order of the data frames in said first memory.

3. The method according to claim 1, wherein the specifying information is information indicating a relative time upon recording the data frame in said first memory.

4. The method according to claim 1, wherein said first memory comprises video tape memories arranged in units of cameras, and said second memory comprises a magnetic disk.

5. The method according to claim 1, wherein said plurality of cameras are mounted on a vehicle.

6. The method according to claim 1, wherein posture information indicating a posture of the camera is recorded together in said second memory.

7. The method according to claim 1, wherein the location information is obtained from a GPS sensor.

8. The method according to claim 1, further comprising the steps of:

reading out, using a value of the acquisition time information as a key, the specifying information corresponding to the acquisition time information from said third memory, and the location information corresponding to the acquisition time information from said second memory; and

recording the image data frame in said first memory specified by the specifying information read out from said third memory, and the location information read out from said second memory, in a fourth memory in association with each other.

9. An image database generated by a recording method of claim 1.

10. A method of recording images obtained by sensing an object using a plurality of cameras which point in a plurality of directions so as to obtain a basis for forming a sequence of three dimensional image spaces, comprising the steps of:

storing, in a first memory, data frames of images generated by sequentially sensing an object using a camera of said plurality of cameras with generation time information that indicates a generation time of each data frame and differences between the generation time information of one specific camera and the generation time information of the other cameras; and

storing, in a second memory, location information associated with an image sensing location of each data frame, and acquisition time information indicating an acquisition time of the location information.

11. The method according to claim 10, wherein when a value of the generation time information of the image data frame in said first memory equals a value of the acquisition time information of the location information in said second memory, the image data frame and the location information are recorded in a fourth memory in association with each other.

12. An image database system obtained by sensing an object using a plurality of cameras which point in a plurality of directions, comprising:

a location-time database including location information associated with an image sensing location with first time information, including a time code, representing an acquisition time of the location information, and time differences between the time code of a specific one of said plurality of cameras and the time codes of the other cameras; and

an image-time database including image data together with second time information indicating a generation time of the image data.

13. A recording apparatus for recording images obtained by sensing an object using a plurality of cameras which point in a plurality of directions so as to obtain a basis for forming a sequence of three dimensional image spaces, comprising:

a first memory storing data frames of images generated by sequentially sensing an object using a camera of said plurality of cameras with specifying information, including a time code, that specifies data frame;

using the time code of a specific one of said plurality of cameras as a reference time code, calculating differences between the time code of said specific one camera, and time codes of the other cameras;

a second memory storing location information indicating an image sensing location of said camera with acquisition time information indicating an acquisition time of the location information; and

a third memory storing a pair of specifying information that specifies each data frame, and acquisition time information corresponding to the location information acquired at the time of generation of the data frame.

14. The apparatus according to claim 13, wherein said first memory comprises a sequential memory, and the specifying information is a frame number indicating a recording order of the data frames in said first memory.

15. The apparatus according to claim 13, wherein the specifying information is information indicating a relative time upon recording the data frame in said first memory.

16. The apparatus according to claim 13, wherein said first memory comprises video tape memories arranged in units of cameras, and said second memory comprises a magnetic disk.

17. The apparatus according to claim 13, wherein said plurality of cameras are mounted on a vehicle.

18. The apparatus according to claim 13, wherein posture information indicating a posture of the camera is recorded together in said second memory.

19. The apparatus according to claim 13, wherein the location information is obtained from a GPS sensor.

20. The apparatus according to claim 13, further comprising:

a reader unit reading out, using a value of the acquisition time information as a key, the specifying information corresponding to the acquisition time information from said third memory, and the location information corresponding to the acquisition time information from said second memory; and

a recorder unit recording the image data frame in said first memory specified by the specifying information read out from said third memory, and the location information read out from said second memory, in a fourth memory in association with each other.

21. A recording apparatus for recording images obtained by sensing an object using a plurality of cameras which point in a plurality of directions so as to obtain a basis for forming a sequence of three dimensional image spaces, comprising:

a first memory storing data frames of images generated by sequentially sensing an object using a camera of said plurality of cameras with generation time information that indicates a generation time of each data frame and differences between the generation time information of one specific camera and the generation time information of the other cameras; and

a second memory storing location information associated with an image sensing location of each data frame together with acquisition time information indicating an acquisition time of the location information.

22. The apparatus according to claim 21, wherein when a value of the generation time information of the image data frame in said first memory equals a value of the acquisition time information of the location information in said second memory, the image data frame and the location information are recorded in a fourth memory in association with each other.

23. A recording medium of a computer program that makes a computer execute control for recording images obtained by sensing an object using a plurality of cameras which point in a plurality of directions so as to obtain a basis for forming a sequence of three dimensional image spaces, said medium storing:

first program code means for storing, in a first memory, data frames of images generated by sequentially sensing an object using a camera of said plurality of cameras together with specifying information, including a time code, that specifies each data frame;

second program code means for using the time code of a specific one of said plurality of cameras as a reference time code, and calculating differences between the time code of said specific one camera, and time codes of the other cameras;

third program code means for acquiring location information indicating an image sensing location of said camera, and storing, in a second memory, the location information together with acquisition time information indicating an acquisition time of the location information; and

fourth program code means for storing a pair of specifying information that specifies each data frame, and acquisition time information corresponding to the location information acquired at the time of generation of the data frame, in a third memory.

24. A recording medium of a computer program that makes a computer execute control for recording images obtained by sensing an object using a plurality of cameras which point in a plurality of directions so as to obtain a basis for forming a sequence of three dimensional image spaces, said medium storing:

first program code means for storing, in a first memory, data frames of images generated by sequentially sensing an object using a camera of said plurality of cameras with generation time information that indicates a generation time of each data frame and differences between the generation time information of one specific camera and the generation time information of the other cameras; and

second program code means for acquiring location information associated with an image sensing location of each data frame, and storing the location information together with acquisition time information indicating an acquisition time of the location information in a second memory.