Generating heat maps using dynamic vision sensor events

• US 10,582,196 B2
• Filed: 06/30/2017
• Issued: 03/03/2020
• Est. Priority Date: 06/30/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Perez-Ramirez, Javier, Varadarajan, Srenivas, Liao, Yiting, Somayazulu, Vallabhajosyula S., Tickoo, Omesh, Ndiour, Ibrahima J.
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US15/640,198
• Publication Number:
  US 20190007678A1
• Time in Patent Office: 977 Days
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/126 Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field

Abstract

An example apparatus for encoding video frames includes a receiver to receive events from a dynamic vision sensor and a video frame from an image sensor. The apparatus also includes a heat map generator to generate a heat map based on the received events. The apparatus further includes a region of interest (ROI) map generator generate a ROI map based on the heat map. The apparatus includes a parameter adjuster to adjust an encoding parameter based on the ROI map. The apparatus also further includes a video encoder to encode the video frame using the adjusted parameter.

Claims

1. An apparatus for encoding video frames, comprising a processor to:

receive events from a dynamic vision sensor and a video frame from an image sensor;

generate a heat map based on the received events;

generate a ROI map based on the heat map;

adjust an encoding parameter based on the ROI map; and

encode the video frame using the adjusted parameter.

2. The apparatus of claim 1, wherein the processor is to transmit the encoded video frame to an object detector.

3. The apparatus of claim 1, wherein the heat map comprises hot areas corresponding to new temporal activity.

4. The apparatus of claim 1, wherein the encoded video frame comprises an area of new temporal activity encoded in higher fidelity and an area of no new temporal activity encoded in lower fidelity.

5. The apparatus of claim 1, wherein the events comprise a positive event or a negative event.

6. The apparatus of claim 1, wherein the processor is to generate the heat map based on a combined event matrix generated from a positive event matrix and a negative event matrix.

7. The apparatus of claim 1, wherein the processor is to receive expected object regions from a video database, detect regions of interest based on the expected object regions and the heat map, and generate the ROI map based on the detected regions of interest.

8. The apparatus of claim 1, wherein the processor is to receive an object detection in a previous frame, detect regions of interest based on the object detection and the heat map, and generate the ROI map based on the detected regions of interest.

9. The apparatus of claim 1, wherein the processor is to detect regions of interest based on a detected new object in the video frame and the heat map and generate the ROI map based on the detected regions of interest.

10. The apparatus of claim 1, wherein the dynamic vision sensor comprises a plurality of smart pixels to capture video frames at a high frame rate.

11. A method for encoding video frames, comprising:

receiving, at a processor, events from a dynamic vision sensor and a video frame from an image sensor;

generating, via the processor, a heat map based on the received events;

generating, via the processor, a region of interest (ROI) map based on the heat map;

adjusting, via the processor, an encoding parameter based on the ROI map; and

encoding, via the processor, the video frame using the adjusted encoding parameter.

12. The method of claim 11, comprising sending, via the processor, the encoded video frame to an object detector.

13. The method of claim 11, wherein generating the heat map comprises generating a positive event matrix and a negative event matrix, generating a combined event matrix based on the positive event matrix and the negative event matrix, and generating the heat map based on the combined event matrix.

14. The method of claim 11, wherein generating the ROI map comprises detecting regions of interest based on received expected object regions and the heat map and generating the ROI map based on the detected regions of interest.

15. The method of claim 11, wherein generating the ROI map comprises detecting regions of interest based on an object detection from a previous frame and the heat map and generating the ROI map based on the detected regions of interest.

16. The method of claim 11, wherein generating the ROI map comprises detecting regions of interest based on a detected new object in the video frame and the heat map and generating the ROI map based on the detected regions of interest.

17. The method of claim 11, wherein adjusting the encoding parameter comprises increasing a quantization parameter of a macroblock of the video frame in response to detecting that the macroblock does not correspond to any region of interest in the ROI map.

18. The method of claim 11, wherein adjusting the encoding parameter comprises decreasing a quantization parameter of a macroblock of the video frame in response to detecting that the macroblock corresponds to a region of interest in the ROI map.

19. The method of claim 11, wherein encoding the video frame using the adjusted encoding parameter comprises compressing macroblocks with higher quantization parameter (QP) values more than macroblocks with lower QP values.

20. The method of claim 11, wherein the events comprise a negative event or a positive event corresponding to a change in light intensity detected at a smart pixel of the dynamic vision sensor.

21. At least one non-transitory computer readable medium for encoding video frames having instructions stored therein that, in response to being executed on a computing device, cause the computing device to:

receive events from a dynamic vision sensor and a video frame from an image sensor;

generate a heat map based on the received events;

generate a region of interest (ROI) map based on the heat map;

adjust an encoding parameter based on the ROI map; and

encode the video frame using the adjusted parameter.

22. The at least one non-transitory computer readable medium of claim 21, comprising instructions to transmit the encoded video frame to an object detector.

23. The at least one non-transitory computer readable medium of claim 21, comprising instructions to generate a positive event matrix and a negative event matrix, and to generate a combined event matrix based on the positive event matrix and the negative event matrix.

24. The at least one non-transitory computer readable medium of claim 21, comprising instructions to detect regions of interest based on expected object regions from a video database and the heat map, and to generate the ROI map based on the detected regions of interest.

25. The at least one non-transitory computer readable medium of claim 21, comprising instructions to detect regions of interest based on object detection in a previous frame and the heat map, and to generate the ROI map based on the detected regions of interest.