METHOD, DEVICE AND SYSTEM FOR PRODUCING A MERGED DIGITAL VIDEO SEQUENCE

• US 20150036736A1
• Filed: 06/26/2014
• Published: 02/05/2015
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Axis AB (Canon Inc.)
• Inventors: Lundberg, Stefan
• Granted Patent: US 9,756,348 B2
• Time in Patent Office: Days
• US Class Current: 375/240.02
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

A video processing device for producing a frame of a merged digital video sequence. A memory storing a first and a second digital video sequence depicting the same scene. The first digital video sequence has a higher pixel density than the second digital video sequence. A scaler generating an up-scaled version having the same pixel density as the first video sequence. A decoder decoding a frame of the first digital video sequence and a skip block identifying a position for a skip block and a non-skip block in the frame of the first digital video sequence. A block extractor extracting a block of pixels from the frame of the second digital video sequence based on the skip block and a block of pixels from the frame of the first digital video sequence based on the non-skip block. A merging unit merging both extracted blocks to produce the merged video sequence.

Claims

1. A method for producing a frame of a merged digital video sequence, comprising:

obtaining, using an image processor a first digital video sequence;

analyzing, using an analysis unit frames of the first digital video sequence to identify pixel blocks considered to be of relevance;

producing, using a mask generator a mask indicating pixel blocks of the frames of the first digital video sequence being of relevance;

encoding, using an encoding unit using a video compression algorithm, a frame of the first digital video sequence using the mask into a frame in an encoded version of the first video sequence, wherein the pixel blocks being masked out as not being of relevance are encoded using skip blocks;

decoding, using a decoder, the encoded frame of the first digital video sequence;

obtaining, using the image processor, a second digital video sequence, wherein the first and second digital video sequences being spatially and temporally similar depicting the same scene, the frames of the first digital video sequence having a higher number of pixels than the frames of the second digital video sequence;

scaling, using a scaler, a frame of the second digital video sequence such that an up-scaled frame of the second digital video sequence having the same number of pixels as the frame of the first video sequence is obtained;

identifying, while decoding the encoded frame of the first digital video sequence and using a skip block identifier, a position for at least one skip block and a position for at least one non-skip block in the frame of the first digital video sequence;

extracting, using a block extractor and based on the identified position for the at least one skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the up-scaled frame of the second digital video sequence;

extracting, using the block extractor and based on the identified position for the at least one non-skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the frame of the first digital video sequence; and

merging, using a merging unit, the extracted block of pixels from the frame of the second digital video sequence and the extracted block of pixels from the frame of the first digital video sequence such that the frame of the merged video sequence is produced.

2. The method according to claim 1, further comprising filtering, using a filter, the scaled frame of the second digital video sequence.

3. The method according to claim 1, wherein the frames of the second digital video sequence is encoded using a video compression algorithm, wherein the method further comprises decoding, using the decoder, a frame of the second digital video sequence.

4. The method according to claim 1, wherein the encoded frame of the first digital video sequence is an INTER-frame.

5. The method according to claim 1, wherein the video compression algorithm is ITU-H.26X, ISO-MPEG, WMV, On2 or WebM compliant.

6. The method according to claim 1, further comprising storing the frame of the merged video sequence in a memory.

7. The method according to claim 1, further comprising play back of the frame of the merged video sequence.

8. A computer-readable recording medium having recorded thereon a program for implementing the method according to claim 1 when executed on a device having processing capabilities.

9. A video processing device arranged to execute the program recorded on the computer-readable recording medium according to claim 8.

10. A system for producing a frame of a merged digital video sequence, the system comprising:

an image processor arranged to obtain a first digital video sequence;

an analysis unit arranged to analyze frames of the first digital video sequence to identify pixel blocks considered to be of relevance;

a mask generator arranged to output a mask indicating pixel blocks of relevance in the frames of the first digital video sequence;

an encoding unit arranged to encode a frame of the first digital video sequence using the mask indicating pixel blocks of relevance into a frame in an encoded version of the first video sequence, wherein the pixel blocks being masked out as not being of relevance are encoded using skip blocks;

the image processor is further arranged to obtain a second digital video sequence, wherein the first and second digital video sequences are spatially and temporally similar depicting the same scene, the frames of the first digital video sequence having a higher number of pixels than the second digital video sequence,

the system further comprises a scaler arranged to scale a frame of the second digital video sequence such that an up-scaled frame of the second digital video sequence is obtained, the up-scaled frame of the second digital video sequence having the same number of pixels as the frame of the first video sequence;

the system further comprises a decoder arranged to decode the encoded frame of the first digital video sequence;

the system further comprises a skip block identifier arranged to, while decoding, using the decoder, the encoded frame of the first digital video sequence, identify a position for at least one skip block and a position for at least one non-skip block in the frame of the first digital video sequence;

the system further comprises a block extractor arranged to extract based on the identified position for the at least one skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the up-scaled frame of the second digital video sequence and based on the identified position for the at least one non-skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the frame of the first digital video sequence; and

a merging unit arranged to merge the extracted block of pixels from the frame of the second digital video sequence and the extracted block of pixels from the frame of the first digital video sequence such that the frame of the merged video sequence is produced.

11. The system according to claim 10, further comprising an image sensor arranged to capture a video sequence of a scene;

and wherein the image processor is arranged to process the video sequence of the scene captured by the image sensor into the first and the second digital video sequence depicting the same scene.

12. The system according to claim 10, wherein the system is comprised in a digital video camera.

13. A method for producing a frame of a merged digital video sequence, comprising:

obtaining a first digital video sequence using an image processor;

obtaining a second digital video sequence using the image processor, wherein the first digital video sequence and the second digital video sequence are spatially and temporally similar depicting a same scene, and wherein frames of the first digital video sequence has a higher number of pixels than the frames of the second digital video sequence;

up-scaling a frame of the second digital video sequence using a scaler, wherein an up-scaled frame has a same number of pixels as the frame of the first digital video sequence;

identifying relevant pixel blocks in the frames of the first digital video sequence using an analysis unit;

generating a mask based on the identified relevant pixel blocks of the frames of the first digital video sequence;

encoding a frame of the first digital video sequence using the mask into a frame in an encoded version of the first video sequence using an encoder, wherein non-relevant pixel blocks being masked out are encoded using skip blocks;

decoding an encoded frame of the first digital video sequence using a decoder;

identifying a position for at least one skip block and a position for at least one non-skip block in the frame of the first digital video sequence while decoding the encoded frame of the first digital video sequence and using a skip block identifier;

extracting at least one corresponding block of pixels from the up-scaled frame of the second digital video sequence using a block extractor and based on the identified position for the at least one skip block in the frame of the first digital video sequence;

extracting at least one corresponding block of pixels from the frame of the first digital video sequence using the block extractor and based on the identified position for the at least one non-skip block in the frame of the first digital video sequence; and

merging the extracted block of pixels from the frame of the second digital video sequence and the extracted block of pixels from the frame of the first digital video sequence using a merging unit to produce the frame of the merged video sequence.

14. The method according to claim 13, further comprising:

filtering the up-scaled frame of the second digital video sequence using a filter.

15. The method according to claim 13, wherein the frames of the second digital video sequence are encoded, further comprising:

decoding a frame of the second digital video sequence using the decoder.

16. The method according to claim 13, wherein the encoded frame of the first digital video sequence is an INTER-frame.

17. The method according to claim 13, wherein a video compression algorithm used in the encoder is at least one of ITU-H.26X, ISO-MPEG, WMV, On2 or WebM compliant.

18. The method according to claim 13, further comprising:

storing the frame of the merged video sequence in a memory.

19. The method according to claim 13, further comprising:

playing back of the frame of the merged video sequence.