Encoding video frames using generated region of interest maps

• US 10,349,060 B2
• Filed: 06/30/2017
• Issued: 07/09/2019
• Est. Priority Date: 06/30/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Varadarajan, Srenivas, Liao, Yiting, Somayazulu, Vallabhajosyula S., Tickoo, Omesh, Ndiour, Ibrahima J., Perez-Ramirez, Javier
• Primary Examiner(s): Aghevli, Reza
• Application Number:
  US15/640,202
• Publication Number:
  US 20190007690A1
• Time in Patent Office: 739 Days
• Field of Search: None
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock

Abstract

An example apparatus for encoding video frames includes a receiver to receive video frames and a heat map from a camera and expected object regions from a video database. The apparatus also includes a region of interest (ROI) map generator to detect a region of interest in a video frame based on the expected object regions. The ROI map generator can also detect a region of interest in the video frame based on the heat map. The ROI map generator can then generate an ROI map based on the detected regions of interest. The apparatus further includes a parameter adjuster to adjust an encoding parameter based on the ROI map. The apparatus also further includes a video encoder to encode the video frame using the adjusted encoding parameter.

Claims

1. An apparatus for encoding video frames, comprising:

a receiver of an edge device to receive video frames and a heat map representing temporal activity of objects in the video frames from a camera comprising a non-visual modality and expected object regions from a video database;

a processor of the edge device to;

detect a first region of interest in a video frame based on the expected object regions;

detect a second region of interest in the video frame based on the heat map;

generate a ROI map based on the detected first and second regions of interest; and

adjust an encoding parameter based on the ROI map; and

a video encoder of the edge device to encode the video frame using the adjusted encoding parameter.

2. The apparatus of claim 1, wherein the processor is to further detect a third region of interest based on a received object detection in a previous frame received from a remote server.

3. The apparatus of claim 1, wherein the processor is to further detect a third region of interest based on a detected new object, wherein the new object is detected in response to not detecting that the new object corresponds to an object that is not present in or tracked from a previous frame.

4. The apparatus of claim 1, comprising a video transmitter of the edge device to transmit the encoded video frame to an object detector of a remote server.

5. The apparatus of claim 1, wherein the heat map comprises hot areas corresponding to new temporal activity.

6. The apparatus of claim 1, wherein the encoded video frame comprises an area corresponding to the first region of interest and the second region of interest encoded in higher fidelity and an area not corresponding to any region of interest encoded in lower fidelity.

7. The apparatus of claim 1, wherein the ROI map comprises a union of the detected first and second regions of interest.

8. The apparatus of claim 1, wherein the first and second regions of interest are defined to preserve semantic inference.

9. The apparatus of claim 1, wherein the expected object regions comprise a priori expected regions obtained from a large amount of video data.

10. The apparatus of claim 1, wherein the camera comprises a dynamic vision sensor to detect changes in an intensity of light.

11. A method for encoding video frames, comprising:

receiving, at a processor of an edge device, video frames and a heat map representing temporal activity of objects in the video frames from a camera comprising a non-visual modality and expected object regions from a video database;

detecting, via the processor, a first region of interest (ROI) in a video frame based on the expected object regions;

detecting, via the processor, a second region of interest in the video frame based on the heat map;

generating, via the processor, a ROI map based on the detected first and second regions of interest;

adjusting, via the processor, an encoding parameter based on the ROI map; and

encoding, via the processor, the video frame using the adjusted parameter.

12. The method of claim 11, comprising sending, via the processor, the encoded video frame to an object detector of a remote server.

13. The method of claim 11, comprising receiving, from a remote server via a feedback channel, an object detection in a previous frame detecting a third region of interest based on the received object detection in the previous frame.

14. The method of claim 11, comprising detecting a third region of interest based on a detected new object, wherein detecting the new object comprises detecting that the new object corresponds to an object that is not present in or tracked from a previous frame.

15. The method of claim 11, wherein encoding the video frame comprises encoding an area of the video frame corresponding to the first region of interest and the second region of interest in higher fidelity and encoding an area of the video frame not corresponding to any region of interest in lower fidelity.

16. The method of claim 11, wherein detecting the ROI based on the expected object regions comprises empirically obtaining the expected object regions from a large amount of video data.

17. The method of claim 11, wherein generating the ROI map comprises detecting overlapping regions of interest and including the overlapping regions of interest in the ROI map.

18. The method of claim 11, wherein adjusting the encoding parameter based on the ROI map comprises applying a positive quantization parameter (QP) offset to a macroblock of the video frame in response to detecting the macroblock is not within a union of ROIs of the ROI map.

19. The method of claim 11, wherein adjusting the encoding parameter based on the ROI map comprises applying a negative quantization parameter (QP) offset to a macroblock of the video frame in response to detecting the macroblock is within a union of ROIs of the ROI map.

20. The method of claim 11, comprising defining the regions of interest to preserve semantic inference.

21. At least one non-transitory computer readable medium for encoding video frames having instructions stored therein that, in response to being executed on edge computing device, cause the edge computing device to:

receive video frames and a heat map representing temporal activity of objects in the video frames from a camera comprising a non-visual modality and expected object regions from a video database;

detect a first region of interest in a video frame based on the expected object regions;

detect a second region of interest (ROI) in the video frame based on the heat map;

generate a ROI map based on the detected first and second regions of interest;

adjust an encoding parameter based on the ROI map; and

encode the video frame using the adjusted encoding parameter.

22. The at least one non-transitory computer readable medium of claim 21, comprising instructions to transmit the encoded video frame to an object detector of a remote server.

23. The at least one non-transitory computer readable medium of claim 21, comprising instructions to detect a third region of interest based on a received object detection in a previous frame received from a remote server.

24. The at least one non-transitory computer readable medium of claim 21, comprising instructions to detect a third region of interest based on a detected new object.

25. The at least one non-transitory computer readable medium of claim 21, comprising instructions to encode an area of the video frame corresponding to the first region of interest and the second region of interest in higher fidelity and an area of the video frame not corresponding to any region of interest in lower fidelity.