Dynamic range video recording and playback system and method

• US 20020118293A1
• Filed: 02/28/2002
• Published: 08/29/2002
• Est. Priority Date: 08/17/1999
• Status: Active Grant

• Current Assignee: Applied Vision Systems, Inc.
• Original Assignee: Applied Vision Systems, Inc.
• Inventors: Hui, Michael, Hori, Koichiro
• Granted Patent: US 7,193,652 B2
• Time in Patent Office: Days
• US Class Current: 348/362
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/45 Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen

Abstract

An improved method and system for capturing and displaying a series of successive video images using at least first and second different exposure times, so that first video images captured using said first exposure time occur alternately in said series with second video images captured using said second exposure time. The method employs a digital computer and a software program for generating and storing digital data representative of said first and second video images, and utilizing that digital data to selectively display (a) side-by-side reproductions of said first and second images or (b) reproductions of only said first or said second video images or (c) composite images derived from the first and second video images digital data.

Claims

1. Method of producing a video recording with improved dynamic range comprising:

(a) providing a video camera capable of converting a viewed optical image into a stream of analog video image signals defining a series of video fields or frames representing said optical image;

(b) operating said video camera to capture an optical image and simultaneously varying the amount of light received by said video camera during the time frame of each video field or frame representing the captured optical image so as to produce a stream of analog video image signals defining at least a plurality of first fields or frames representing a first exposure value of said captured optical image and a plurality of second fields or frames representing a second exposure value of said captured optical image, and said first fields or frames being interspersed among said second fields or frames in a discernible sequence;

(c) providing a computer comprising video capturing means adapted to convert analog video signals that define a series of video fields or frames into digital video signals comprising digital data representative of said video fields or frames, an application memory for storing said digital video signals, a display memory for storing said digital video signals, signal processing means coupled to said display memory for converting digital video signals from said display memory into output analog video signals, and display means for generating a visual display in response and according to said output analog video signals,

(d) applying said stream of analog video image signals to said video capturing means of said computer so that said analog video signals are converted to digital video signals representative of the fields or frames of the captured optical image; and

(e) operating said computer so that (1) digital video signals are stored in said application memory in the order that they were generated from said analog video image signals but segregated according to the exposure values of the fields or frames which they represent, (2) said digital video signals are transferred from said application memory to said display memory according to the exposure values of the fields or frames represented by said digital video signals and in the order that said digital format data signals were generated from said analog signals, (3) said digital video signals are transferred from said display memory to said video signal processing means and converted thereby to output analog video signals, and (4) said output analog video signals are applied to said display means so as to cause said display means to generate a visual side by side display of captured images with different exposure values.

2. A method according to claim 1 wherein said camera is operated so that said stream of analog video image signals comprises first signals each defining a first field or frame representing a first exposure value of said captured optical image and second signals each defining a second field or frame representing a second exposure value of said captured optical image, with said first and second signals occurring alternately in said stream of signals, and further wherein said computer is operated so that said display means displays two video images in side by side relation with one another in response to said first and second signals, with one of said side by side video images being representative of said first exposure value and the other of said side by side video images being representative of said second exposure value.

3. A method according to claim 1 wherein said analog video signals comprise first signals each representative of a first field of an interlaced scanned image captured at a first exposure level and second signals each representative of the second field of an interlaced scanned image captured at a second different exposure level, and further wherein the number of lines of pixels in each of said fields is doubled when stored in said application memory.

4. Method of producing a video recording with improved dynamic range comprising:

(a) providing a video camera capable of converting a viewed optical image into a stream of video image signals defining a series of video fields or frames representing said optical image;

(b) operating said video camera to capture an optical image and simultaneously varying the amount of light received by said video camera during the time frame of each video field or frame representing the captured optical image so as to produce a stream of video image signals defining at least a plurality of first fields or frames representing a first exposure value of said captured optical image and a plurality of second fields or frames representing a second different exposure value of said captured optical image, and said first fields or frames being interspersed among said second fields or frames in a discernible sequence;

(c) providing a computer comprising video capturing means adapted to convert video image signals that define a series of video fields or frames into digital video signals comprising digital data representative of said video fields or frames, an application memory for storing said digital video signals, a display memory for storing said digital video signals, signal processing means coupled to said display memory for converting digital video signals from said display memory into output video signals, and display means for generating a visual display in response and according to said output video signals, (d) applying said stream of video image signals to said video capturing means of said computer so that said video signals are converted to digital video signals comprising data representative of the fields or frames of the captured optical image; and

(e) operating said computer so that (1) digital video signals are stored in said application memory in the order that they were generated from said video image signals but segregated according to the exposure values of the fields or frames which they represent, (2) said digital video signals are transferred from said application memory to said display memory according to the exposure values of the fields or frames represented by said digital video signals and in the order that said digital format data signals were generated from said video image signals, (3) said digital video signals are transferred from said display memory to said video signal processing means and converted thereby to output video signals in a format suitable for driving said display means, and (4) said output video signals are applied to said display means so as to cause said display means to generate a display reproducing the images captured according to either or both of said first and second exposure values.

5. Method according to claim 4 wherein said a video image signals comprise first signals each representative of a first field of an interlaced scanned image characterized by said first exposure value and second signals each representative of the second field of an interlaced scanned image characterized by said second different exposure value, and further wherein the number of lines of pixels in each of said fields is doubled when stored in said application memory.

6. Method of producing a video recording comprising:

(a) generating a stream of analog image signals defining a sequence of interlaced video fields representing a captured optical image, with said stream of analog image signals comprising first analog image signals defining first video fields of n lines of pixels representing the captured optical image at a first exposure level and second analog image signals defining second video fields of n lines of pixels representing the captured optical image at a second exposure level, with said first image signals occurring alternately with said second image signals in said stream;

(b) providing a computer comprising a central processor, image capturing means adapted to receive analog image signals that define a sequence of video fields representing an optical image and transform said analog image signals into digital field signals comprising digital data representative of said video fields, an application memory for storing said digital field signals, a display memory for storing said digital field signals, a video signal processing means coupled to said display memory for recovering digital field signals from said display memory and converting said digital field signals into output video signals having an analog format, display means for generating a visual display of captured images in response and according to said output video signals, and a stored application software program operable through said central processor for (1) directing said digital field signals to said application memory for storage according to the exposure values of the captured images represented by said digital field signals, (2) transferring said digital field signals from said application memory to said display memory in accordance with the exposure values of captured images represented by said digital field signals and in the order that said digital field signals were directed to said application memory for storage, with each of said digital field signals being transformed to define video fields of 2n lines of pixels, and (3) transferring stored digital field signals from said display memory to said video signal processing means so as to generate analog video signals in response and corresponding to said digital field signals; and

(c) applying said stream of analog image signals to said image capturing means so that (1) said first and second analog image signals are converted to first and second digital field signals representative of said first and second fields and said first and second exposure levels respectively, (2) said stored application software program causes said first and second digital field signals to be stored in said application memory in the order that they were generated from said analog image signals and according to the exposure values of said captured images, with each of said digital field signals stored in said application memory defining a video field of 2n lines of pixels, (3) said stored application software program transfers stored first and second digital field signals from said application memory to said display memory in the order that they were generated and according to the exposure values of said captured images;

(4) said stored application software program causes said stored first and second digital field signals to be transferred from said display memory to said video signal processing means, whereby said video signal processing means converts said first and second digital field or frame signals to first and second analog output video signals respectively, and (5) said stored application software program causes said first and second analog output video signals to be applied to said display means so as to cause said display means to generate a simultaneous display of a first video reproduction of the captured optical image at a first exposure level and a second video reproduction of the captured optical image at a second exposure level, with each video reproduction comprising 2 a lines of pixels.

7. A method of recording and displaying video images comprising:

capturing successive first optical images according to a first exposure time and capturing successive second optical images according to a second exposure time, with said first optical images occurring alternately with said second optical images;

producing first and second analog image signals representative of said first and second optical images, with said first and second image signals occurring alternately in the order of capture of said first and second optical images;

applying said first and second analog image signals to a digital computer having an image capture card that is adapted to convert analog image signals to corresponding digital data image signals, a video card adapted to convert said digital data image signals into corresponding analog output video signals, and display means for generating visual video images in response to said analog output video signals; and

operating said digital computer so that in sequence (a) said first and second analog image signals are converted by said image capture card to first and second digital data image signals representative of said first and second optical images, (b) said first and second digital data image signals are stored in first and second separate buffers in said application memory, (c) said first digital data image signals are transferred in sequence from said first application memory buffer to a first image buffer in said display memory, (d) said second digital data image signals are transferred in sequence from said second buffer application memory to a second image buffer on said display memory, with each of said second digital data image signals being transferred in step with said first digital data image signals, (e) said first and second digital data image signals are transferred from said first and second image buffers to said video card and converted by said video card to corresponding first and second analog output video signals; and

(f) said first and second analog output video signals are applied to said display means so as to cause said display means to generate a display comprising first and second video images in side by side relation, with said first video images corresponding in exposure to said first optical images and said second video images corresponding in exposure to said second optical images.

8. A method according to claim 7 wherein said digital computer is coupled to a write/read storage device, and further wherein said digital computer includes means for causing said first and second digital data image signals to be written into said storage device and means for causing said first and second digital data image signals to be read out of said storage device into said application memory, whereby analog output video signals suitable for producing a visual video display of recorded images represented by digital data image signals stored in said storage device may be generated by transferring said digital data image signals from said application memory to said video card via said display memory.

9. A method of recording and displaying a scene captured as a series of first and second alternately occurring analog video signals with each first analog video signal and the next occurring second analog video signal representing first and second interlaced fields of a video frame comprising n scanning lines of image information, with each of said first and second fields comprising ½

n scanning lines and said first fields containing image information representing the scene captured according to a first length exposure time and said second fields containing image information representing the said scene captured according to a second different length exposure time, said method comprising the following steps;

(a) converting said first and second analog video signals to corresponding first and second digital format video signals each representative of a video field comprising ½

n lines of pixels;

(b) converting said first and second digital format video signals to first and second composite digital video signals each representative of a frame of n lines of pixels, with ½

of said n lines of pixels being derived from the remainder of said n lines of pixels and said first and second composite digital video signals being representative of said scene captured according to said first and second length exposure times respectively;

(c) storing said first and second composite digital video signals in first and second buffers respectively; and

(d) utilizing said stored first and second composite digital video signals to generate first and second side-by-side video displays of the scene captured according to said first and second exposure times respectively.

10. Method of capturing and recording video images comprising:

(a) capturing a sequence of video images wherein said sequence comprises at least first video images captured at a first exposure level and second video images captured at a second different exposure level, with said second video images being interspersed among said first video images in said sequence;

(b) generating a series of video image signals comprising first video image signals representative of said first video images and second video image signals representative of said second video images;

(c) converting said first and second video image signals to first and second digital format data respectively;

(d) storing said first and second digital format data;

(e) retrieving said first and second digital format data;

(f) converting said first and second digital format data to first and second output video signals respectively; and

(g) utilizing said first and second output video signals to generate sideby-side displays of said first and second video images respectively.

11. Method according to claim 10 wherein said first and second video image signals are analog video signals derived from an analog video camera.

12. Method according to claim 10 wherein said first and second video image signals are digital video signals.

13. Method according to claim 10 wherein said digital video signals are derived from a digital video camera.

14. Method according to claim 10 further including the step of assigning codes to said first and second digital format data whereby to identify each of said first and second video images by date, time and frame number.

15. Method according to claim 10 wherein said first and second digital format data is stored temporarily in first and second buffers.

16. Method according to claim 10 wherein said first and second digital format data is stored in a non-volatile memory device.

17. Method according to claim 10 wherein said sequence of captured video images includes third video images captured at a third exposure level, with said third video images being interspersed among said first and second video images in said sequence, and further comprising the steps of generating third video image signals image signals representative of said third images, converting said third video image signals to third digital format data, storing said third digital data format, retrieving said third digital format data and converting said third digital format data to third output video signals, and utilizing said third output video signals to generate a display of said third video images concurrently with and in side-by-side relation with said displays of said first and second video images.

19. Method according to claim 18 further including the step of storing said digital data in a non-volatile storage for later retrieval.

20. Method according to claim 19 wherein said step of storing said digital data in a non-volatile storage precedes step (b), and further including the step of retrieving said digital data from said non-volatile storage, after which said retrieved digital data is processed according to steps (b) and (c).

21. Method according to claims 19 wherein the process of storing said digital data in said non-volatile storage includes the step of assigning date, time and frame codes to separately identify each of said relatively bright and relatively dark images.

18. Method of capturing and recording video images comprising:

(a) capturing in memory in the form of digital data a sequence of video images of a scene wherein said sequence comprises first relatively bright video images and second relatively dark video images, with said bright and dark video images occurring alternately in said sequence; and

(b) converting said digital data to video signals representative of said first relatively bright video images and said second relatively dark images; and

(c) using said video signals to generate a side-by-side display of said first relatively bright video images and said second relatively dark images.

22. Method of capturing and recording video images comprising:

(a) capturing a sequence of video images wherein said sequence comprises at least first video images captured at a first exposure level and second video images captured at a second different exposure level, with said second video images being interspersed among said first video images in said sequence;

(b) generating a series of video image signals comprising first video image signals representative of said first video images and second video image signals representative of said second video images;

(c) converting said first and second video image signals to first and second digital format data respectively;

(d) storing said first and second digital format data;

(e) retrieving said first and second digital format data;

(f) processing said first and second digital format data to generate therefrom composite digital video format data; and

(g) converting said composite digital format data to a video output signal that comprises the image information represented by said first and second digital data.

27. Method according to claim 26 wherein said sequence comprises additional video images with an exposure value intermediate the exposure value of said relatively bright and relatively dark video images, with said additional video images occurring alternately in said sequence with said relatively bright and relatively dark video images, and further wherein the digital data representing said additional video images is utilized to generate output video signals for selectively producing a display of said additional video images independently or concurrently with a side by side display of said relatively bright and said relatively dark video images.

29. A computer system according to claim 28 further including a storage device, and further wherein said processing means comprises means for storing said digital field or frame signals in said storage device.

30. A computer system according to claim 29 wherein said processing means includes means for retrieving digital field or frame signals from said storage device and transmitting said retrieved digital field or frame signals to said application memory for playback purposes.

31. A computer system according to claim 28 wherein each of said first and, second video image signals constitutes one field of a video frame that comprises n lines of pixels representing two interlaced fields, with each field comprising ½

n lines of pixels, and further wherein said processing means modifies said digital field signals so that each of said digital field signals is stored in said application memory as representing n lines of pixels.

32. A computer system according to claim 28 wherein said series of video image signals comprises third video image signals defining third fields or frames constituting data representing images captured at a third exposure level different than said first and second exposure levels, said image capturing means of said image capture card comprises image is adapted to receive said series of third video image signals and convert said third video image signals into third digital field or frame signals constituting data representative of the images captured at said third exposure level, said video card is adapted to converting said third digital field or frame signals into third output video signals in a format suitable for driving a display device to provide a visual video display of the images represented by said third fields or frames, and said processing means is adapted to (a) store said third digital frame or field signals in said application memory in sequence with said first and second field or frame signals with said stored first, second and third digital field or frame signals being categorized according to their exposure values, (b) sequentially move said stored first, second and third digital field or frame signals from said application memory and sequentially storing said first, second and third digital field or frame signals in said display memory with said stored digital field or frame signals being categorized in said display memory according to their exposure values, and (c) transferring from said display memory to said video card in sequence each of said first digital field or frame signals and the second and third digital field or frame signals generated next in time relative to said each first digital field or frame signal, whereby to convert said first, second and third digital field or frame signals into first, second and third output video signals respectively in a format suitable for driving a display means so as to generate a display constituting side by side images with three different exposure levels.

23. Method of capturing and recording video images comprising:

(a) capturing in memory in the form of digital data a sequence of video images of a scene wherein said sequence comprises first relatively bright video images and second relatively dark video images, with said bright and dark video images occurring alternately in said sequence. said digital data comprising first digital data representative of said first relatively bright video images and second digital data representative of said second relatively dark video images; and

(b) processing said first and second digital data to generate therefrom composite digital data;

(c) converting said composite digital data to video output signals that comprise image information represented by said first and second digital data; and

(d) using said video output signals to display images comprising said image information.

24. Method of capturing and recording video images comprising:

(a) capturing a sequence of video images wherein said sequence comprises at least first video images captured at a first exposure level and second video images captured at a second different exposure level, with said second video images being interspersed among said first video images in said sequence;

(b) generating a series of video image signals comprising first video image signals representative of said first video images and second video image signals representative of said second video images;

(c) converting said first and second video image signals to first and second digital format data respectively;

(d) storing said first and second digital format data;

(e) retrieving said first and second digital format data;

(f) converting said first and second digital format data to first and second output video signals respectively; and

(g) generating a video display from said first or second output video signals.

25. Method of capturing and recording video images comprising:

(a) capturing in memory in the form of digital data a sequence of video images of a scene wherein said sequence comprises relatively bright video images and relatively dark video images, with said bright and dark images occurring alternately in said sequence; and

(b) converting said digital data to video signals representative of said first relatively bright video images and said second relatively dark video images; and

(c) using said video signals to displaying the image information of one of said relatively bright video images or one of said second relatively dark video images.

26. Method of capturing and recording video images comprising:

(a) capturing in memory in the form of digital data a sequence of video images of a scene wherein said sequence comprises relatively bright video images and relatively dark video images, with said bright and dark images occurring alternately in said sequence; and

(b) utilizing said digital data to generate output video signals for selectively producing any of the following;

a side by side display of said relatively bright and relatively dark video images, a singular display of only said relatively bright or said relatively dark images, a composite image display that comprises information derived from said relatively bright and said relatively dark video images, and said composite image display plus a display of only said relatively bright or relatively dark images.

28. A computer system for processing, storing and displaying video image information generated as a series of video image signals comprising first video image signals defining first fields or frames constituting data representing images captured at a first exposure level and second video image signals defining second fields or frames constituting data representing images captured at a second exposure level different than said first exposure level, with said first and second analog video image signals occurring alternately in sequence with one another, said system comprising:

an image capture card comprising image capturing means for receiving said series of video image signals and converting said first and second video image signals into first and second digital field or frame signals constituting data representative of the images captured at said first and second exposure levels;

an application memory;

a display memory;

a video card for converting said first and second digital field or frame signals into first and second output video signals respectively in a format suitable for driving a display device to provide a visual video display of the images represented by said fields or frames; and

processing means including a central processor for (a) sequentially storing said first and second digital field or frame signals in said application memory with said stored digital field or frame signals being categorized according to their exposure values, (b) sequentially moving said stored first and second digital field or frame signals from said application memory and sequentially storing said first and second digital field or frame signals in said display memory with said stored digital field or frame signals being categorized in said display memory according to their exposure values, and (c) transferring from said display memory to said video card in sequence each of said first digital field or frame signals and the second digital field or frame signals generated next in time relative to said each first digital field or frame signal, whereby to convert said first and second digital field or frame signals into first and second output video signals respectively in a format suitable for driving a display means so as to generate a display constituting side by side images with two different exposure values.