Method and apparatus for enhancing digital images with textual explanations

• US 6,970,185 B2
• Filed: 01/31/2001
• Issued: 11/29/2005
• Est. Priority Date: 01/31/2001
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Halverson, Paul Steven
• Primary Examiner(s): Ometz, David L.
• Assistant Examiner(s): Ye, Lin
• Application Number:
  US09/774,731
• Publication Number:
  US 20020101513A1
• Time in Patent Office: 1,763 Days
• Field of Search: 348/207.1, 348/211.1, 348/211.4, 348/211.6, 348/231.3, 348/231.4, 348/375, 348/552, 348/515, 704/275, 704/251, 704/270, 704/254, 704/255, 704/257, 704/231
• US Class Current: 348/207.1
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8233 the additional signal being a character code signal

Abstract

A photographer adds explanatory text to a captured image by speaking the text at approximately the time the image is captured by the digital camera. The spoken information is reduced to text by recognizing the user'"'"'s speech, and is associated with the digital image. Preferably, the digital camera contains an on-board speech reduction capability to produce an intermediate symbolic form expressing the user'"'"'s speech as a series of basic sounds, or phonemes, which can be later reduced to natural language text by a computer system having access to sophisticated vocabulary lists and syntactical analysis.

Claims

1. A digital camera, comprising:

a housing;

a digital optical sensing apparatus mounted within said housing, said digital optical sensing apparatus sensing optical images;

a storage medium for storing digital optical images captured by said digital optical sensing apparatus;

an acoustic sensor capable of sensing human speech;

a speech reduction apparatus coupled to said acoustic sensor, said speech reduction apparatus converting human speech sensed by said acoustic sensor to a symbolic text form; and

a controller which stores said symbolic text form in said storage medium in a relationship associated with a captured digital image, wherein said controller;

(a) receives a user indication of a plurality of discrete time intervals;

(b) records a plurality of discrete human speech segments sensed by said acoustic sensor in respective said discrete time intervals;

(c) causes said speech reduction apparatus to convert each said human speech segment to a corresponding symbolic text segment; and

(d) automatically associates a respective digital optical image captured by said digital optical sensing apparatus with each said symbolic text segment based on a temporal relationship between the time interval in which the discrete human speech segment corresponding to the symbolic text segment was recorded and the capturing of said digital optical image;

wherein said controller associates a respective digital image with each symbolic text segment according to all of the following association priorities;

(1) if a first digital image is captured during the recording of a human speech segment corresponding to the symbolic text segment, the symbolic text segment is associated with the first digital image;

(2) if no digital image is captured from a time the digital camera is powered on until the end of the recoding of the human speech segment corresponding to the symbolic text segment, and a second digital image is captured after recording the human speech segment but before the digital camera is powered off, then the symbolic text segment is associated with the second digital image; and

(3) in all other cases, the symbolic text is associated with the digital image most recently captured before the recording of the human speech segment corresponding to the symbolic text segment.

2. The digital camera of claim 1, wherein said controller comprises a programmable processor executing a control program for controlling the operation of said digital camera.

3. The digital camera of claim 2, wherein said speech reduction apparatus comprises a speech reduction algorithm embodied as a plurality of instructions executable on said programmable processor.

4. The digital camera of claim 1, wherein said speech reduction apparatus converts said human speech sensed by said acoustic sensor to an intermediate symbolic form comprising a symbolic representation of phonemes, said intermediate symbolic form being subsequently reduced to natural language text by a separate apparatus.

5. A method of operating a digital camera, comprising the steps of:

capturing a plurality of digital images of respective objects of interest with optical sensing apparatus of said digital camera;

recording a plurality of discrete segments of human speech of a user in said digital camera during a plurality of respective discrete time intervals, each respective discrete time interval occurring substantially contemporaneously with capturing of each respective digital image of said plurality of digital images;

rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text using speech reduction apparatus within said digital camera; and

automatically associating each respective digital image of said plurality of digital images with a respective corresponding segment of symbolic text rendered from a respective corresponding segment of human speech based on a temporal relationship between the respective discrete time interval during which the corresponding segment of human speed was recorded and the capturing of the respective digital image, and storing each said symbolic text segment in a relationship associated with each respective said captured digital image;

wherein said step of automatically associating each respective digital image of said plurality of digital images with a respective corresponding segment of symbolic text comprises automatically associating according to all of the following association priorities;

(1) if a first digital image is captured during the recording of a human speech segment corresponding to the symbolic text segment, the symbolic text segment is associated with the first digital image;

(2) if no digital image is captured from a time the digital camera is powered on until the end of the recoding of the human speech segment corresponding to the symbolic text segment, and a second digital image is captured after recording the human speech segment but before the digital camera is powered off, then the symbolic text segment is associated with the second digital image; and

(3) in all other cases, the symbolic text is associated with the digital image most recently captured before the recording of the human speech segment corresponding to the symbolic text segment.

6. The method of operating a digital camera of claim 5, wherein said step of rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text converts said human speech to an intermediate symbolic form comprising a symbolic representation of phonemes, said intermediate symbolic form being subsequently reduced to natural language text by an apparatus separate from said digital camera.

7. The method of operating a digital camera of claim 5, wherein said step of rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text is performed by a programmable processor executing a speech reduction program.

8. The method of operating a digital camera of claim 7, wherein said programmable processor further executes a control program for controlling the operation of said digital camera, and said step of rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text is performed by said programmable processor in the background when said control program is otherwise unoccupied.

9. A program product for controlling the operation of a digital camera, said program product comprising a plurality of processor executable instructions recorded on signal-bearing media, wherein said instructions, when executed by at least one programmable processor within said digital camera, cause the camera to perform the steps of:

capturing a plurality of digital images of respective objects of interest with optical sensing apparatus of said digital camera;

recording a plurality of discrete segments of human speech of a user in said digital camera during a plurality of respective discrete time intervals, each respective discrete time interval occurring substantially contemporaneously with capturing of each respective digital image of said plurality of digital images;

rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text using speech reduction apparatus within said digital camera; and

associating each respective digital image of said plurality of digital images with a respective corresponding segment of symbolic text rendered from a respective corresponding segment of human speech based on a temporal relationship between the respective discrete time interval during which the corresponding segment of human speech was recorded and the capturing of the respective digital image, and storing each said symbolic text segment in a relationship associated with each respective said captured digital image;

wherein said step of associating each respective digital image of said plurality of digital images with a respective corresponding segment of symbolic text comprises associating according to all of the following priorities;

(1) if a first digital image is captured during the recording of a human speech segment corresponding to the symbolic text segment, the symbolic text segment is associated with the first digital image;

(2) if no digital image is captured from a time the digital camera is powered on until the end of the recoding of the human speech segment corresponding to the symbolic text segment, and a second digital image is captured after recording the human speech segment but before the digital camera is powered off, then the symbolic text segment is associated with the second digital image; and

(3) in all other cases, the symbolic text is associated with the digital image most recently captured before the recording of the human speech segment corresponding to the symbolic text segment.

10. The program product for controlling the operation of a digital camera of claim 9, wherein said step of rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text converts said human speech to an intermediate symbolic form comprising a symbolic representation of phonemes, said intermediate symbolic form being subsequently reduced to natural language text by an apparatus separate from said digital camera.

11. A method of recording information with digital images, comprising the steps of:

capturing a plurality of digital images of respective objects of interest with optical sensing apparatus of a digital camera;

recording a plurality of discrete segments of human speech of a user in said digital camera during a plurality of respective discrete time intervals occurring substantially contemporaneously with capturing of each respective digital image of said plurality of digital images;

rendering each said segment of said plurality of discrete segments of human speech into a respective corresponding segment of symbolic text using speech reduction apparatus within said digital camera;

automatically associating each respective digital image of said plurality of digital images with a respective corresponding segment of symbolic text rendered from a respective corresponding segment of human speech based on a temporal relationship between the respective discrete time interval during which the corresponding segment of human speech was recorded and the capturing of the respective digital image, and recording said association in a memory of said digital camera;

uploading said at least one digital image and said at least one segment of symbolic text to a digital image formatting apparatus; and

formatting said plurality of digital images and said plurality of segments of symbolic text for viewing by a user using said digital image formatting apparatus, wherein each said segment of symbolic text is formatted for viewing in a human readable form associated with its corresponding digital image;

wherein said step of automatically associating each respective digital image of said plurality of digital images with a respective corresponding segment of symbolic text comprises automatically associating according to all of the following association priorities;

(1) if a first digital image is captured during the recording of a human speech segment corresponding to the symbolic text segment, the symbolic text segment is associated with the first digital image;

(2) if no digital image is captured from a time the digital camera is powered on until the end of the recoding of the human speech segment corresponding to the symbolic text segment, and a second digital image is captured after recording the human speech segment but before the digital camera is powered off, then the symbolic text segment is associated with the second digital image; and

(3) in all other cases, the symbolic text is associated with the digital image most recently captured before the recording of the human speech segment corresponding to the symbolic text segment.

12. The method of recording information with digital images of claim 11, wherein said step of rendering each said segment of said plurality of discrete segments of human speech in a respective corresponding segment of symbolic text converts said human speech to an intermediate symbolic form comprising a symbolic representation of phonemes, and wherein said step of formatting said plurality of digital images and said plurality of segments of symbolic text for viewing comprises reducing said intermediate symbolic form to natural language text.

13. The method of recording information with digital images of claim 11, wherein said digital image formatting apparatus is a general-purpose digital computer executing a digital image formatting program.

14. The method of recording information with digital images of claim 11, wherein said step of formatting said plurality of digital images and said plurality of segments of symbolic text comprises formatting for output on paper, wherein formatted text is printed on paper with a corresponding digital image.

15. The method of recording information with digital images of claim 11, wherein said step of formatting said plurality of digital images and said plurality of segments of symbolic text comprises formatting for viewing from an output screen of a digital device, wherein formatted text is displayed on said output screen with a corresponding digital image.