Video image search method and system using the same

• US 5,576,950 A
• Filed: 07/26/1994
• Issued: 11/19/1996
• Est. Priority Date: 07/28/1993
• Status: Expired due to Term

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Kishimoto, Tomio, Suzuki, Gen, Akutsu, Akihito, Tonomura, Yoshinobu, Taniguchi, Yukinobu
• Primary Examiner(s): Voeltz, Emanuel T.
• Assistant Examiner(s): PEESO, THOMAS R
• Application Number:
  US08/280,507
• Time in Patent Office: 847 Days
• Field of Search: 364/514 A, 358/311, 358/335, 358/300, 358/537, 360/11.1, 360/14.3, 348/222, 348/430
• US Class Current: 386/200
• CPC Class Codes: G11B 2220/90 Tape-like record carriers
• CPC Class Codes: G11B 27/028 with computer assistance
• CPC Class Codes: G11B 27/102 Programmed access in sequence to addressed parts of tracks of operating record carriers access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816
• CPC Class Codes: G11B 27/107 of operating tapes
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 1/00283 with a television apparatus
• CPC Class Codes: H04N 2201/3226 of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
• CPC Class Codes: H04N 2201/3269 of machine readable codes or marks, e.g. bar codes or glyphs
• CPC Class Codes: H04N 2201/3271 Printing or stamping
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus

Abstract

A video signal, replay by a video player, is subjected to image processing by a calculation control part of a video image searching device to calculate the image quality value and another feature value. An image whose image quality satisfies a predetermined condition and whose feature value matches a predetermined condition is detected as an image in which an event has occurred. Such images are printed on an output paper together with associated addition information. A main control part of a video image access device reads out the additional information on the output paper by a scanner and effects control via a video control part to search the image corresponding to that information by a video player.

Claims

1. A video image search system comprising:

video signal source means for outputting a video signal in a sequence of frame images;

layout storage means for storing images and additional information in a layout;

calculation control means which repeats processing in which each of said frame images of said video signal is subjected to image processing to obtain, as event items, a quality value of each of said frame images and at least one feature value characterizing contents of said video signal, and when both said quality value and said feature value satisfy predetermined conditions, it is decided that an event is present, said frame image decided to have an event is stored in a predetermined desired layout in a predetermined area in said layout storage means, and at least one part of additional information corresponding to said frame image, obtained at the time of said image processing is arranged in said area of said layout storage means, together with said frame image, in accordance with a predetermined array condition, and which reads out said arranged frame image and additional information; and

event output means for outputting, as an event, at least one of said stored frame images and said additional information both of which are read out from said layout storage means.

2. The system of claim 1, wherein said video signal source includes video replay means which replays and outputs recorded video signals.

3. The system of claim 1, wherein said video signal source is a signal stream through which said video signal supplied from the outside is fed to said system.

4. The system of claim 1, wherein said event output means includes printer means which stored frame image and at least one part of said additional information on a sheet.

5. The system of claim 1, further comprising a video access device which includes:

video replay monitor means controllable from the outside;

readout means for reading out said additional information being output; and

a main control part which converts said additional information read out by said readout means into video replay control information and controls said video replay monitor means by said converted information to access a video image corresponding to said additional information.

6. The system of claim 5, wherein said event output means includes printer means for stored frame image and at least one part of said additional information on a sheet and said readout means includes scanner means for optically reading out said at least one part of said additional information.

7. The system of claim 5, wherein said event output means includes magnetic recording means for magnetically writing at least one part of said additional information on a magnetic medium formed at a predetermined position on said sheet and said readout means includes means for reading out said at least one part of said additional information from said magnetic medium on said sheet.

8. The system of claim 5, wherein said event output means includes package media recording means for recording said additional information on package media, readout means includes package media readout means for reading out said additional information recorded on said package media and said main control part controls aid package media readout means to read out said additional information.

9. The system of claim 1 or 5, wherein said video signal contains a video signal and an audio signal, which further comprises sampling means for sampling said video signal from said video signal source means and for outputting sample data and data storage means for storing said sample data, and wherein when said video signal is decided to have an event, said calculation control means controls said data storage means to store therein at least said sample data of said audio signal in said video signal as part of said additional information for a fixed period of time upon said decision.

10. The system of claim 5, wherein said readout means is facsimile means and said main control part controls said video replay monitor provided at a remote place via communication.

11. The system of claim 1 or 5, further comprising storage means for storing image and additional information and wherein when either one or both of the result of image processing or sound processing of said video signal satisfy, as said feature values, a particular condition, said calculation control means decides that an event condition for said feature value is satisfied, stores in said storage means said image decided to have an event, or said image and sound and said additional information corresponding thereto, and arranges said stored image and additional information in said storage area of said layout storage means in accordance with predetermined size and position conditions upon each occurrence of an event or after termination of said video signal.

12. The system of claim 11, which further comprises means whereby, when said calculation control means decides the occurrence of an event and stores an image or said image, sound and additional information in said storage area, data of at least said image is stored in compressed form.

13. A video image search method comprising the steps of:

(a) subjecting an input video signal to image processing for each of a plurality of frame images to obtain, as event items, a quality value of each frame image and at least one feature value characterizing contents of said video signal;

(b) deciding that an event is present when said quality value and said feature value both satisfy particular conditions;

(c) storing a frame image decided to have an event, in a predetermined desired layout in a predetermined storage area in layout storage means and arranging additional information corresponding to said image, obtained in said image processing, in said storage area of said layout storage means in accordance with a predetermined position condition and in correspondence to said image;

(d) repeating steps (a), (b) and (c) and, upon completion of arrangement into said storage area, reading out said arranged image and additional information from said storage area; and

(e) outputting, as an event, at least one of said stored images and said additional information from said storage area.

14. The method of claim 13, further comprising a step of reading out, by readout means said at least one part of additional information read out from said storage area and output as said event, converting said additional information to a control signal and controlling video replay monitor means by said control signal to access an image corresponding to said additional information.

15. The method of claim 14, wherein said event outputting step includes a step of printing said image and said at least one part of additional information read out of said storage means on a sheet and said reading-out step includes a step of optically reading out said additional information printed on said sheet.

16. The method of claim 15, further comprising a step of reading out said printed additional information by facsimile and controlling said video replay monitor means provided at a remote place by communication.

17. The method of claim 15, further comprising a step of computing and presenting the size and the number of optimum arranged images on the basis of predetermined size of a print and the number of images to be arranged, when said images and additional information in said storage area.

18. The method of claim 17, further comprising at step of selecting images stored in decreasing order of the matching value and the quality value in accordance with said computed optimum number of images when arranging said frame images and additional information and arranging said selected images after transforming their sizes by geometric transforming means in accordance with said computed optimum image layout.

19. The method of claim 14, further comprising a step of writing an index showing the position of occurrence of an event in accordance with said read-out information, in a video medium, by controlling said video replay monitor means.

20. The method of claim 13 or 14, wherein said step (c) comprises a step of storing said image in storage means in a data compressed form, and a step of reading out said image from said storage means and arranging it in said storage area of said layout storage means upon occurrence of an event or after termination of said video signal.

21. The method of claim 13, further comprising a step of printing said image read out of said storage area on a sheet, magnetically recording said at least one part of additional information on a magnetic medium formed on said sheet, reading out said recorded additional information, converting said read-out information to information for controlling video replay monitor means and controlling said video replay monitor means by said converted information to access an image corresponding to said additional information.

22. The method of claim 13, further comprising a step of printing said image read out of said storage area on a sheet, recording at least one part of said additional information on package media, reading out said additional information recorded on said package media, converting said read-out information to information for controlling video replay monitor means and controlling said video replay monitor means by said converted information to access an image corresponding to said additional information.

23. The method of claim 13, further comprising a step of subjecting said image to geometrical transformation to satisfy a predetermined particular arrangement condition when said image is arranged in said storage area.

24. The method of claim 13, further comprising a step of sampling and recording only an audio signal in said video signal or both of an image signal and said audio signal for a fixed time upon deciding that said video signal has an event, said recorded information being used as one part of said additional information.

25. The method of claim 13, wherein said feature value as said event item is a predetermined particular camera work and which comprises a step of detecting said camera work and changing the size and position of an image to be arranged in said storage area, in accordance with said detected camera work.

26. The method of claim 13, wherein said feature value as said event item is sound information, said video signal is subjected to image processing and sound processing and when the results of said image processing and sound processing satisfy a particular condition, it is decided that an event condition is satisfied.

27. The method of claim 13, wherein a template description corresponding to a fixed condition is prepared as a predetermined condition for arranging a frame image and additional information when detecting an event and a changing part is added to said description upon each occurrence of an event, thereby arranging said frame image and said additional information.

28. The method of claim 13, wherein the result of image processing or sound processing of said video signal is used as an event item, and when one or both of them satisfy a particular condition, it is judged that the event condition of that item is satisfied, an image decided to have an event or said image and sound and additional information obtained during said processing are stored, and upon each occurrence of an event, or after termination of said video signal, said stored image and additional information are arranged in said storage area in accordance with predetermined size and positional condition.

29. The method of claim 13, further comprising a step of computing, as said feature value for the decision of an event, a matching value between a reference image and the image of said video signal and deciding that the event condition of said feature value is satisfied when said matching value matches a predetermined condition.

30. The method of claim 29, wherein when it is decided that said event is present, said reference image is updated with the image decided to have said event.

31. The method of claim 29, wherein said event condition deciding step if a step of computing, as said matching value, the distance between said reference image and the image of said video signal in a feature space and deciding that the event condition is satisfied when said distance is larger than a predetermined threshold value.

32. The method of claim 29, wherein said event deciding step is a step of computing the number of pixels larger or smaller than a threshold value provided by the quantity of blur of an image or its intensity signal, or the presence or absence of a synchronizing signal of a replayed video signal as said quality value, and a camera work parameter as said feature value and deciding that an event is present when these computed results match a predetermined condition.

33. The method of claim 29, further comprising a step of selecting images decided to have an event in decreasing order of said matching value and the quality value of the video signal in accordance with predetermined number of images and arranging them in said storage area.

34. The method of claim 29, further comprising a step of correcting an image of a small quality value to improve its quality when arranging image decided to have an event and additional information in said storage area.

35. The method of claim 34, wherein said image correcting means includes means for performing intensity conversion, color conversion, emphasized field interpolation and intraframe pixel interpolation of images in the step of correcting images of small quality value when arranging said images and additional information in said storage means.