Video collecting device, video searching device, and video collecting/searching system

• US 20020154213A1
• Filed: 11/28/2001
• Published: 10/24/2002
• Est. Priority Date: 01/31/2000
• Status: Active Grant

• Current Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Inventors: Sibyama, Zyunn?apos;iti, Tanaka, Satoshi, Hisanaga, Satoshi, Nagahisa, Hiroto
• Granted Patent: US 6,950,535 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G01S 3/784 using a mosaic of detectors
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G09B 29/106 using electronic means
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/775 between a recording apparatus and a television receiver
• CPC Class Codes: H04N 5/9201 involving the multiplexing of an additional signal and the video signal
• CPC Class Codes: H04N 5/9206 the additional signal being a character code signal
• CPC Class Codes: H04N 5/926 by pulse code modulation H04N5/919 takes precedence

Abstract

An image recording medium (101) and a position-time recording medium (102) are provided in an image collecting device (10). In an image retrieving device (20), a matching section (24) allows image data read from the image recording medium (101) and position-time data read by a data reading section (21) to be matched with each other based upon time so as to generate an image database. An image pickup locus display processing section (32) retrieves for image data having its image-pickup position on a map within a map display section (28), and displays the image-pickup position as a locus. When a position on the map is specified by a map input section (29) by reference to the locus, image data in the vicinity of this position is reproduced by an image display processing section (33).

Claims

1. An image retrieving device comprising:

an image reading unit which reads a sequence of image data recorded with image pickup times;

an image data holding, unit which holds the sequence of image data that has been read by the image reading unit;

an attribute information reading unit which reads attribute information containing at least image pickup positions where the sequence of image pickup data has been obtained and the image pickup times thereof;

a matching unit which matches the sequence of image data held in the image data holding unit with the attribute information read by the attribute information reading unit based upon the image pickup times;

an image database which holds the matching relationship that has been determined by the matching unit;

a map data holding unit which holds map data;

a map display processing unit which displays the map data on a map display unit based upon the map data;

an image retrieving unit which retrieves the image database;

a locus display processing unit which allows the image retrieving unit to retrieve for image data having image pickup positions within a map displayed by the map display unit, and displays the retrieved pickup positions on the map as a locus;

an image display unit which displays the sequence of image data;

a position specifying unit which specifies a position of the map displayed on the map display unit; and

an image processing unit which acquires image data corresponding to the image pickup position in the vicinity of the position specified by the position specifying unit from the image data holding unit, and reproduces and displays the resulting image data on the image display unit.

2. The image retrieving device according to claim 1, wherein the attribute information further includes information related to the- image pickup orientation, image pickup direction, image pickup angle or combinations of these.

3. The image retrieving device according to claim 1, wherein the locus display processing unit further comprises a locus-type button display processing unit which allows the image retrieving unit to retrieve for a sequence of image data having image pickup positions within the map displayed by the map display unit, and displays a route formed by connecting the image pickup positions of the sequence of image data thus retrieved and a slide bar that slides on the route, and is constituted by an inputting button for indicating a reproduction start point of the image data on the map.

4. The image retrieving device according to claim 1, further comprising a route searching unit which allows the image retrieving unit to retrieve for a sequence of image data located between two positions indicating the image pickup start and the image pickup end specified by the position specifying unit, generates a route between the two positions that passes through the image pickup positions indicated by the sequence of image data, displays the locus of the image pickup positions along the route on the map display unit, and, when an image pickup position is specified by the position specifying unit, displays image data on the route succeeding to the image pickup position.

5. The image retrieving device according to claim 1, further comprising:

a junction image holding unit which holds a crossing point image picked up on the periphery of a crossing point at which sequences of image data intersect each other;

a crossing-point database which holds the matching relationship in which the crossing-point image and the attribute information of the crossing-point image are matched with each other; and

a connection interpolating unit which, when image data passing through the crossing point exists, retrieves the crossing-point database, and interpolates images on the periphery of the crossing point by using the crossing-point image held in the junction image holding unit.

6. The image retrieving device according to claim 1, further comprising an image editing unit which carries out an editing process including cutting and composing processes of the sequence of image data.

7. The image retrieving device according to claim 1, further comprising an image adjusting unit which carries out a thinning process or an interpolating process on the image data so that the image pickup position gaps between the respective pieces of image data constituting the sequence of image data are made virtually the same.

8. The image retrieving device according to claim 1, wherein the map data holding unit holds three-dimensional map data, and the map display processing unit displays the three-dimensional map on the map display unit stereoscopically based upon the three-dimensional map data.

9. The image retrieving device according to claim 1, further comprising an image pickup position display processing unit which, based upon the attribute information, displays the image pickup range displayed on the image display unit on the map display unit.

10. The image retrieving device according to claim 8, further comprising a synchronization processing unit which provides a three-dimensional display having the same three-dimensional display position, direction and angle as the image pickup position, image pickup direction and image pickup angle of the image displayed on the image display unit, on the map display unit in synchronism with the image.

11. The image retrieving device according to claim 8, further comprising:

an image position specifying unit which specifies a position on the display screen of the image display unit; and

a three-dimensional position display processing unit which calculates the three-dimensional position corresponding to the position specified by the image position specifying unit based upon the image-pickup position, the image-pickup direction and the image-pickup angle of the image data displayed on the image display unit, and displays the resulting three-dimensional position on the map display unit.

12. The image retrieving device according to claim 8, further comprising:

an image position specifying unit which specifies a position on the display screen of the image display unit;

a three-dimensional model holding unit which holds a three-dimensional model; and

a three-dimensional model image composing unit which composes the three-dimensional model into the image and for displaying the resulting image at the position specified by the image position specifying unit in a manner so as to match the image displayed on the image display unit.

13. The image retrieving device according to claim 12, further comprising a three-dimensional model and map composing unit which calculates a three-dimensional position corresponding to the position specified by the image position specifying unit based upon the image-pickup position, image-pickup direction and image-pickup angle of the image data displayed on the image display unit, and composes the three-dimensional model and the map and displays the resulting map at the three-dimensional position on the map displayed by the map display unit.

14. The image retrieving device according to claim 1, further comprising:

a map attribute retrieving unit which retrieves the map data holding unit for map attribute information corresponding to the image pickup position at which the image data is obtained; and

a map attribute information display unit which displays the map attribute information.

15. The image retrieving device according to claim 14, further comprising a map retrieving unit which retrieves a position on the two-dimensional map based upon the specified map attribute.

16. The image retrieving device according to claim 1, further comprising a subject-position matching unit which matches the subject position of an image and the pickup position thereof with each other.

17. The image retrieving device according to claim 16, further comprising:

a subject angle detection unit which detects an angle between the subject face of an image and the lens face of the image collecting device for collecting the sequence of image data; and

an image angle correction unit which corrects the distortion of the image due to the angle with respect to the image data.

18. The image retrieving device according to claim 1, which collects the sequence of image data with the lens angle having a known lens angle difference with respect to the reference direction, further comprising:

an image angle correction unit which corrects the distortion of an image resulting from the difference in the lens angle.

19. The image retrieving device according to claim 1, which has all-around image data obtained by a fish-eye lens as the sequence of image data, further comprising:

an image upright correction unit which extracts an image in a specified direction from the all-around image data and for correcting it into an upright image.

20. The image retrieving device according to claim 1, which has stereoscopic image data obtained by using two stereoscopic lenses spaced with a predetermined gap as the sequence of image data, further comprising:

a polarization processing unit which carries out a polarizing process on each piece of the stereoscopic image data.

21. The image retrieving device according to claim 16, further comprising:

a subject-distance acquiring unit which detects the distance between the subject face of an image and the lens face of the image collecting device for collecting the sequence of image data; and

an image size correction unit which corrects a difference in the image size caused by the distance with respect to the image data.

22. The image retrieving device according to claim 6, further comprising:

a junction detection unit which detects a crossing point from the map data and;

a junction data holding unit which holds the data of the crossing point detected by the junction detection unit,

wherein the image editing unit carries out a cutting process of the sequence of image databased upon the crossing-point data held by the junction data holding unit.

25. The image collecting and retrieving system according to claim 24, wherein the at least one image collecting device further comprises a transfer adjusting unit which thins the image data to be transmitted so as to adjust the amount of data to be transmitted.

26. The image collecting and retrieving system according to claim 24, wherein the image retrieving device further comprises a communication destination selection unit which switches the receipt of the sequence of image data and attribute information transmitted from the at least one image collecting device in a time-divided manner.

27. The image collecting and retrieving system according to claim 24, further comprising a collection instructing unit which gives instructions for collecting operations including the start and finish of the image collection to the image collecting device,

wherein the image collecting device further includes an image collection control unit which controls the image collecting device based upon the collection instruction by the collection instructing unit.

23. An image collecting device comprising:

an image recording unit which records a sequence of picked-up image data together with the image pickup times;

a position acquiring unit which acquires attribute information containing at least an image pickup position and image pickup time;

a position-time recording unit which records the attribute information acquired by the position acquiring unit; and

a recording control unit which controls the image recording unit and the position-time recording unit to carry out the recording operations with the respective recording times being synchronous to each other.

24. An image collecting and retrieving system comprising:

at least one image collecting device which includes an image recording unit which records a sequence of picked-up image data together with the image pickup times;

an image reading unit which reads the sequence of image data;

a position acquiring unit which acquires attribute information containing at least an image pickup position and image pickup time;

a position-time recording unit which records the attribute information acquired by the position acquiring unit;

a recording control unit which controls the image recording unit and the position-time recording unit to carry out the recording operations with the respective recording times being synchronous to each other; and

a transmission processing unit which successively transmits the sequence of image data read by the image reading unit and the attribute information, and

an image retrieving device connected to the at least one image collecting device, the image retrieving device includes

a receiving processing unit which receives the sequence of image data and the attribute information transmitted from the at least one image collecting device;

an image data holding unit which holds the sequence of image data received by the receiving processing unit;

an attribute information holding unit which holds the attribute information received by the receiving processing unit;

a matching unit which matches the sequence of image data held in the image data holding unit with the attribute information read by the attribute information reading unit based upon the image pickup times;

an image database which holds the matching relationship that has been determined by the matching unit;

a map data holding unit which holds map data;

a map display processing unit which displays the map data on a map display unit based upon the map data;

an image retrieving unit which retrieves the image database;

a locus display processing unit which allows the image retrieving unit to retrieve for image data having image pickup positions within a map displayed by the map display unit, and displays the retrieved pickup positions on the map as a locus;

an image display unit which displays the sequence of image data;

a position specifying unit which specifies a position of the map displayed on the map display unit; and

an image processing unit which acquires image data corresponding to the image pickup position in the vicinity of the position specified by the position specifying unit from the image data holding unit, and reproduces and displays the resulting image data on the image display unit.