Method, device and system for producing a merged digital video sequence

• US 9,756,348 B2
• Filed: 06/26/2014
• Issued: 09/05/2017
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Axis AB (Canon Inc.)
• Inventors: Lundberg, Stefan
• Primary Examiner(s): Rahman, Mohammad J
• Application Number:
  US14/316,227
• Publication Number:
  US 20150036736A1
• Time in Patent Office: 1,167 Days
• Field of Search: None
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

A video processing device for producing a frame of a merged digital video sequence. A memory storing a first and a second digital video sequence depicting the same scene. The first digital video sequence has a higher pixel density than the second digital video sequence. A scaler generating an up-scaled version having the same pixel density as the first video sequence. A decoder decoding a frame of the first digital video sequence and a skip block identifying a position for a skip block and a non-skip block in the frame of the first digital video sequence. A block extractor extracting a block of pixels from the frame of the second digital video sequence based on the skip block and a block of pixels from the frame of the first digital video sequence based on the non-skip block. A merging unit merging both extracted blocks to produce the merged video sequence.

Claims

1. A method for producing a frame of a merged digital video sequence, comprising:

obtaining, using an image processor, a first digital video sequence;

analyzing, using an analysis unit, frames of the first digital video sequence to identify pixel blocks considered to be of relevance;

producing, using a mask generator, a mask indicating pixel blocks of the frames of the first digital video sequence being of relevance;

encoding, using an encoding unit using a video compression algorithm, a frame of the first digital video sequence using the mask into a frame in an encoded version of the first video sequence, wherein the pixel blocks being masked out as not being of relevance are encoded using skip blocks;

transmitting the encoded version of the first video sequence over a network to a client;

obtaining, using the image processor, a second digital video sequence, wherein the first and second digital video sequences both include at least one block of pixels that depicts approximately the same area of relevance and are not encoded as skip blocks, wherein the frames of the first digital video sequence have a higher number of pixels than the frames of the second digital video sequence;

transmitting the second digital video sequence over the network to the client;

scaling, using a scaler of the client, a frame of the second digital video sequence such that an up-scaled frame of the second digital video sequence having the same number of pixels as the frame of the first video sequence is obtained;

identifying, at the client, while decoding the encoded frame of the first digital video sequence and using a skip block identifier, a position for at least one skip block and a position for at least one non-skip block in the frame of the first digital video sequence;

extracting, using a block extractor of the client and based on the identified position for the at least one skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the up-scaled frame of the second digital video sequence;

extracting, using the block extractor and based on the identified position for the at least one non-skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the frame of the first digital video sequence; and

merging, using a merging unit of the client, the extracted block of pixels from the frame of the second digital video sequence and the extracted block of pixels from the frame of the first digital video sequence such that the frame of the merged video sequence is produced, wherein the pixels of the extracted block of pixels from the frame of the first digital video sequence are generated independently of the pixels of the extracted block of pixels from the frame of the second digital video sequence.

2. The method according to claim 1, further comprising filtering, using a filter, the scaled frame of the second digital video sequence.

3. The method according to claim 1, wherein the frames of the second digital video sequence is encoded using a video compression algorithm, wherein the method further comprises decoding, using the decoder, a frame of the second digital video sequence.

4. The method according to claim 1, wherein the encoded frame of the first digital video sequence is an INTER-frame.

5. The method according to claim 1, wherein the video compression algorithm is ITU-H.26X, ISO-MPEG, WMV, On2 or WebM compliant.

6. The method according to claim 1, further comprising storing the frame of the merged video sequence in a memory.

7. The method according to claim 1, further comprising play back of the frame of the merged video sequence.

8. A non-transitory computer-readable recording medium having recorded thereon a program for implementing the method according to claim 1 when executed on a device having processing capabilities.

9. A video processing device arranged to execute the program recorded on the non-transitory computer-readable recording medium according to claim 8.

10. A system for producing a frame of a merged digital video sequence, the system comprising:

a digital video camera arranged to obtain a first digital video sequence;

an analysis unit, of the digital video camera, the analysis unit arranged to analyze frames of the first digital video sequence to identify pixel blocks considered to be of relevance;

a mask generator, of the analysis unit, the mask generator arranged to output a mask indicating pixel blocks of relevance in the frames of the first digital video sequence;

an encoding unit, of the digital video camera, the encoding unit arranged to encode a frame of the first digital video sequence using the mask indicating pixel blocks of relevance into a frame in an encoded version of the first video sequence, wherein the pixel blocks being masked out as not being of relevance are encoded using skip blocks;

the digital video camera is further arranged to transmit the encoded version of the first video sequence over a network to a client,

the digital video camera is further arranged to obtain a second digital video sequence, wherein the first and second digital video sequences both include at least one block of pixels that depicts approximately the same area of relevance and are not encoded as skip blocks wherein the frames of the first digital video sequence have a higher number of pixels than the second digital video sequence,

the digital video camera being further arranged to transmit the second digital video sequence over the network to the client,

the system further comprises a scaler at the client arranged to scale a frame of the second digital video sequence such that an up-scaled frame of the second digital video sequence is obtained, the up-scaled frame of the second digital video sequence having the same number of pixels as the frame of the first video sequence;

the system further comprises a decoder at the client arranged to decode the encoded frame of the first digital video sequence;

the system further comprises a skip block identifier arranged to, at the client, while decoding, using the decoder, the encoded frame of the first digital video sequence, identify a position for at least one skip block and a position for at least one non-skip block in the frame of the first digital video sequence;

the system further comprises a block extractor at the client arranged to extract based on the identified position for the at least one skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the up-scaled frame of the second digital video sequence and based on the identified position for the at least one non-skip block in the frame of the first digital video sequence, at least one corresponding block of pixels from the frame of the first digital video sequence; and

a merging unit at the client arranged to merge the extracted block of pixels from the frame of the second digital video sequence and the extracted block of pixels from the frame of the first digital video sequence such that the frame of the merged video sequence is produced, wherein the pixels of the extracted block of pixels from the frame of the first digital video sequence are generated independently of the pixels of the extracted block of pixels from the frame of the second digital video sequence.

11. The system according to claim 10, further comprising an image sensor arranged to capture a video sequence of a scene;

and wherein the digital video camera is arranged to process the video sequence of the scene captured by the image sensor into the first and the second digital video sequence depicting the same scene.

12. A method for producing a frame of a merged digital video sequence, comprising:

obtaining a first digital video sequence using an image processor;

obtaining a second digital video sequence using the image processor, wherein frames of the first digital video sequence have a higher number of pixels than the frames of the second digital video sequence;

identifying relevant pixel blocks in the frames of the first digital video sequence using an analysis unit, wherein the second digital video sequence includes at least one pixel block that is not encoded as a skip block and that depicts approximately the same area of relevance as that depicted in at least one relevant pixel block identified in the frames of the first digital video sequence;

generating a mask based on the identified relevant pixel blocks of the frames of the first digital video sequence;

encoding a frame of the first digital video sequence using the mask into a frame in an encoded version of the first video sequence using an encoder, wherein non-relevant pixel blocks being masked out are encoded using skip blocks;

transmitting the encoded version of the first video sequence and the second digital video sequence to a client;

up-scaling a frame of the second digital video sequence using a scaler at the client, wherein an up-scaled frame has a same number of pixels as the frame of the first digital video sequence;

identifying, at the client, a position for at least one skip block and a position for at least one non-skip block in the frame of the first digital video sequence while decoding the encoded frame of the first digital video sequence and using a skip block identifier;

extracting at least one corresponding block of pixels from the up-scaled frame of the second digital video sequence using a block extractor at the client and based on the identified position for the at least one skip block in the frame of the first digital video sequence;

extracting at least one corresponding block of pixels from the frame of the first digital video sequence using the block extractor and based on the identified position for the at least one non-skip block in the frame of the first digital video sequence; and

merging the extracted block of pixels from the frame of the second digital video sequence and the extracted block of pixels from the frame of the first digital video sequence using a merging unit at the client to produce the frame of the merged video sequence, wherein the pixels of the extracted block of pixels from the frame of the first digital video sequence are generated independently of the pixels of the extracted block of pixels from the frame of the second digital video sequence.

13. The method according to claim 12, further comprising:

filtering the up-scaled frame of the second digital video sequence using a filter.

14. The method according to claim 12, wherein the frames of the second digital video sequence are encoded, further comprising:

decoding a frame of the second digital video sequence using the decoder.

15. The method according to claim 12, wherein the encoded frame of the first digital video sequence is an INTER-frame.

16. The method according to claim 12, wherein a video compression algorithm used in the encoder is at least one of ITU-H.26X, ISO-MPEG, WMV, On2 or WebM compliant.

17. The method according to claim 12, further comprising:

storing the frame of the merged video sequence in a memory.

18. The method according to claim 12, further comprising:

playing back of the frame of the merged video sequence.