Embedded metadata engines in digital capture devices

• US 7,403,224 B2
• Filed: 09/10/2004
• Issued: 07/22/2008
• Est. Priority Date: 09/01/1998
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Virage, Inc. (HP Inc.)
• Inventors: Fuller, Charles, Humphrey, Richard D., Gorkani, Mojgan Monika
• Primary Examiner(s): Villecco; John M
• Application Number:
  US10/939,216
• Publication Number:
  US 20050033760A1
• Time in Patent Office: 1,411 Days
• Field of Search: 348/239, 348/552, 348/231.99, 348/231.2, 348/231.3, 348/231.4, 348/231.5, 348/231.6, 348/211.4, 348/211.9, 382/227, 382/218, 382/194, 382/170
• US Class Current: 348/231.3
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/787 using geographical or spatial information, e.g. location
• CPC Class Codes: H04N 1/00326 with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00
• CPC Class Codes: H04N 1/00331 with an apparatus performing optical character recognition character recognition G06V30/10
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 1/32106 separate from the image data, e.g. in a different computer file
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 21/4184 providing storage capabilities, e.g. memory stick
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3214 of a date
• CPC Class Codes: H04N 2201/3215 of a time or duration
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3226 of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
• CPC Class Codes: H04N 2201/3232 : of a page, copy or picture number
• CPC Class Codes: H04N 2201/3252 : Image capture parameters, e.g. resolution, illumination conditions, orientation of the image capture device H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3253 : Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 5/772 : the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8233 : the additional signal being a character code signal

Abstract

A digital capture system, such as a digital still camera or digital video recorder, has an embedded real-time content-based analysis function in the capture device to extract metadata from the digital signals. In one embodiment, metadata (descriptive information about the digital content) is formatted and stored separately from the content. In another embodiment, the metadata may be formatted and combined with the digital content in a container format such as MPEG-7, QuickTime, or FlashPix.

Claims

1. An integrated data and real-time metadata capture system, comprising:

a digital capture device producing a digital representation of one or more forms of media content;

a feature extraction engine integrated with the digital capture device, the feature extraction engine having a plurality of feature extractors to automatically extract metadata in real-time from the digital content simultaneously with encoding of the content; and

a storage device capable of storing the encoded media content and the metadata, wherein selected portions of the metadata are associated with selected portions of the encoded media content.

2. The system defined in claim 1, additionally comprising at least one collateral metadata source, wherein the collateral metadata is simultaneously captured with the media content.

3. The system defined in claim 2, wherein the collateral metadata source includes at least one of the following:

user input, device properties, and external sensors.

4. The system defined in claim 2, wherein the digital capture device comprises a digital video camera capable of marking a video clip, wherein the marking is automatically triggered by a recording process of the camera.

5. The system defined in claim 4, wherein the collateral metadata is associated with the video clip and is stored on the storage device with the metadata.

6. The system defined in claim 5, wherein the collateral metadata includes at least one of the following:

time/date, location, or scene/take.

7. The system defined in claim 1, additionally comprising an output interface in data communication with the storage device and generating output data, wherein the output interface is capable of preserving a temporal relationship between a sequence of the metadata and the digital content.

8. The system defined in claim 7, wherein the digital capture device comprises a digital still camera, and the output data comprises a FlashPix image containing image feature vectors as auxiliary metadata within an image file.

9. The system defined in claim 7, wherein the digital capture device comprises a digital video camera including an audio capture capability.

10. The system defined in claim 7, wherein the output data comprises an MPEG-7 digital video file containing a plurality of metadata tracks as embedded metadata within an MPEG-7 metadata framework.

11. The system defined in claim 7, wherein the output data is transmitted to a computer.

12. The system defined in claim 11, wherein the computer executes a media processing function.

13. The system defined in claim 1, wherein the digital capture device comprises a digital still camera.

14. The system defined in claim 13, wherein the metadata comprises a plurality of feature vectors.

15. The system defined in claim 1, additionally comprising a format module configured to associate a time code with the encoded media content and the metadata.

16. The system defined in claim 1, additionally comprising a time code module configured to provide a time code for association with the encoded media content and the metadata.

17. An integrated data and real-time metadata capture method, comprising:

sensing analog signals;

converting the analog signals to a digital representation of one or more forms of media content;

compressing the digital media content;

automatically extracting metadata in real-time from the digital media content simultaneously with the compressing of the digital media content; and

storing the digital media content and the metadata, wherein selected portions of the metadata are associated with selected portions of the digital media content.

18. The method defined in claim 17, wherein automatically extracting metadata in real-time includes extracting a plurality of feature vectors from the digital media content.

19. The method defined in claim 17, additionally comprising associating a time code with the digital media content and the metadata.

20. The method defined in claim 17, additionally comprising generating a time code for association with the digital media content and the metadata.

21. The method defined in claim 17, wherein the storing includes storing to a single storage device for the digital media content and the metadata.

22. An integrated data and real-time metadata capture system, comprising:

a digital capture device producing a digital representation of one or more forms of media content;

a frame buffer storing a frame of the digital content;

a feature extraction engine integrated with the digital capture device, the feature extraction engine having a plurality of feature extractors to automatically extract metadata in real-time from the digital content stored in the frame buffer simultaneously with encoding of the digital content; and

a storage device capable of storing the digital content and the metadata, wherein selected portions of the metadata are associated with selected portions of the digital content.

23. The system defined in claim 22, additionally comprising at least one collateral metadata source, wherein the collateral metadata is simultaneously captured with the media content.

24. The system defined in claim 22, additionally comprising a format module configured to associate a time code with the digital content and the metadata.

25. The system defined in claim 22, additionally comprising a time code module configured to provide a time code for association with the digital content and the metadata.

26. The system defined in claim 22, wherein the feature extraction engine generates a plurality of time-based tracks of metadata that index the digital content.

27. An integrated data and real-time metadata capture system, comprising:

a digital capture device producing a digital representation of one or more forms of media content;

a compressor integrated with the digital capture device, the compressor configured to encode the media content;

a feature extraction engine integrated with the digital capture device, the feature extraction engine having a plurality of feature extractors to automatically extract metadata in real-time from the digital content simultaneously with encoding of the content; and

a storage device capable of storing the encoded media content and the metadata, wherein selected portions of the metadata are associated with selected portions of the encoded media content.

28. The system defined in claim 27, additionally comprising at least one collateral metadata source, wherein the collateral metadata is simultaneously captured with the media content.

29. The system defined in claim 28, wherein the collateral metadata source includes at least one of the following:

user input, device properties, and external sensors.

30. The system defined in claim 28, wherein the digital capture device comprises a digital video camera capable of marking a video clip, wherein the marking is automatically triggered by a recording process of the camera.

31. The system defined in claim 30, wherein the collateral metadata is associated with the video clip and is stored on the storage device with the metadata.

32. The system defined in claim 31, wherein the collateral metadata includes at least one of the following:

time/date, location, or scene/take.

33. The system defined in claim 27, additionally comprising an output interface in data communication with the storage device and generating output data, wherein the output interface is capable of preserving a temporal relationship between a sequence of the metadata and the digital content.

34. The system defined in claim 27, wherein the digital capture device comprises a digital still camera.

35. The system defined in claim 31, wherein the metadata comprises a plurality of feature vectors.

36. The system defined in claim 27, additionally comprising a format module configured to associate a time code with the encoded media content and the metadata.

37. The system defined in claim 27, additionally comprising a time code module configured to provide a time code for association with the encoded media content and the metadata.

38. An integrated data and real-time metadata capture system, comprising:

a digital capture device producing a digital representation of one or more forms of media content including at least video content;

a feature extraction engine integrated with the digital capture device, the feature extraction engine having a plurality of feature extractors to automatically extract metadata via feature identification in real-time from the video content simultaneously with encoding of the video content; and

a storage device capable of storing the encoded video content and the metadata, wherein selected portions of the metadata are associated with selected portions of the encoded video content.

39. The system of claim 38, wherein the feature identification includes speaker identification.

40. The system of claim 38, wherein the feature identification includes face identification.

41. The system of claim 38, wherein the feature identification includes optical character recognition (OCR).

42. The system of claim 38, wherein the video content includes audio content.