Adaptive object detection and recognition

• US 10,699,126 B2
• Filed: 04/27/2018
• Issued: 06/30/2020
• Est. Priority Date: 01/09/2018
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Karyodisa, Ronald, Huang, Pengjun
• Primary Examiner(s): Woldemariam, Aklilu K
• Application Number:
  US15/965,665
• Publication Number:
  US 20190213420A1
• Time in Patent Office: 795 Days
• Field of Search: 348 25, 348143, 348159, 382103, 382104, 382195, 382199, 382298
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 10/7788 the supervisor being a human, e.g. interactive learning with a human teacher
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Techniques and systems are provided for performing object recognition for video frames. For example, one or more video frames of a first video are obtained. The one or more video frames have a first resolution and include one or more objects. The one or more objects are detected in the one or more video frames of the first video based on object detection being performed on the one or more video frames. One or more video frames of a second video are also obtained. The one or more video frames of the second video also include the one or more objects and have a second resolution that is greater than the first resolution. An object recognition process is performed on the one or more video frames of the second video to recognize or identify the one or more objects detected in the one or more video frames of the first video. In some cases, feedback associated with the object detection and/or the object recognition process can be obtained, and a resolution of the first video and/or a resolution of the second video can be adjusted based on the feedback.

Claims

1. A method of performing object recognition for video frames, the method comprising:

obtaining one or more video frames of a first video, the one or more video frames including one or more objects, wherein the one or more video frames of the first video have a first resolution;

performing object detection to detect the one or more objects in the one or more video frames of the first video;

obtaining one or more video frames of a second video, the one or more video frames including the one or more objects, wherein the one or more video frames of the second video have a second resolution that is greater than the first resolution;

performing an object recognition process on the one or more video frames of the second video to recognize the one or more objects detected in the one or more video frames of the first video;

obtaining one or more of feedback associated with the object detection performed on the one or more video frames of the first video or feedback associated with the object recognition process performed on the one or more video frames of the second video; and

adjusting one or more of a resolution of the first video to a higher resolution or a lower resolution for performing object detection on one or more subsequent frames of the first video based on the feedback associated with the object detection or a resolution of the second video to a higher resolution or a lower resolution for performing the object recognition process on one or more subsequent frames of the second video based on the feedback associated with the object recognition process.

2. The method of claim 1, further comprising:

obtaining the feedback associated with the object detection performed on the one or more video frames of the first video;

adjusting the resolution of frames of the first video based on the feedback associated with the object detection; and

detecting at least one object in a subsequent video frame of the first video after the resolution of the frames of the first video is adjusted, the at least one object being detected based on object detection being performed on the subsequent video frame having the adjusted resolution.

3. The method of claim 1, further comprising:

obtaining the feedback associated with the object recognition process performed on the one or more video frames of the second video;

adjusting the resolution of frames of the second video based on the feedback associated with the object recognition process; and

performing the object recognition process on at least one object in a subsequent video frame of the second video after the resolution of the frames of the second video is adjusted, the object recognition process being performed on the subsequent video frame having the adjusted resolution.

4. The method of claim 1, further comprising:

adjusting the resolution of frames of the first video to a first adjusted resolution based on the feedback associated with the object detection;

adjusting the resolution of frames of the second video to a second adjusted resolution based on the feedback associated with the object recognition process;

detecting at least one object in a subsequent video frame of the first video after the resolution of the frames of the first video is adjusted, the at least one object being detected based on object detection being performed on the subsequent video frame having the first adjusted resolution; and

performing the object recognition process on the at least one object detected in the subsequent video frame of the first video, the object recognition process being performed on a subsequent video frame of the second video after the resolution of the frames of the second video is adjusted, the subsequent video frame of the second video having the second adjusted resolution.

5. The method of claim 1, wherein the feedback associated with the object detection is based on one or more of a latency of the object detection or a quality of results from the object detection performed on one or more objects in the one or more video frames of the first video, and wherein the feedback associated with the object recognition process is based on one or more of a latency of the object recognition process or a quality of results from the object recognition process performed on one or more objects in the one or more video frames of the second video.

6. The method of claim 1, wherein adjusting one or more of the resolution of the first video or the resolution of the second video includes:

determining, based on the feedback associated with the object detection, that a latency of the object detection performed on the one or more video frames of the first video is greater than a latency threshold; and

reducing a resolution of at least one video frame of the first video based on the latency of the object detection being greater than the latency threshold.

7. The method of claim 1, wherein adjusting one or more of the resolution of the first video or the resolution of the second video based on the feedback includes:

determining, based on the feedback associated with the object recognition process, that a confidence score generated for the one or more objects by the object recognition process is less than a quality threshold; and

increasing a resolution of at least one video frame of the second video based on the confidence score being less than the quality threshold.

8. The method of claim 1, further comprising:

determining object information associated with the one or more detected objects based on the object detection performed on the one or more video frames of the first video;

modifying the object information from the first resolution to the second resolution; and

performing the object recognition process on the one or more video frames of the second video using the modified object information.

9. The method of claim 8, wherein modifying the object information from the first resolution to the second resolution includes upscaling the object information from the first resolution to the second resolution.

10. The method of claim 8, wherein object information associated with a detected object includes information defining a bounding region generated for the detected object.

11. The method of claim 10, wherein modifying the object information from the first resolution to the second resolution includes upscaling the bounding region from a first size to a second size, wherein the second size is based on a ratio between the first resolution and the second resolution.

12. The method of claim 11, wherein performing the object recognition process on the one or more video frames of the second video using the modified object information includes performing the object recognition process on a region of the one or more video frames of the second video corresponding to the upscaled bounding region.

13. The method of claim 8, wherein object information associated with a detected object includes information defining one or more landmarks on the detected object.

14. The method of claim 13, wherein modifying the object information from the first resolution to the second resolution includes changing a location of a landmark from a first location to a second location, wherein the second location is based on a ratio between the first resolution and the second resolution.

15. The method of claim 1, wherein the first video and the second video are obtained from two different video streams.

16. The method of claim 1, wherein the first video is a downscaled version of the second video.

17. An apparatus for performing object recognition for video frames, comprising:

a memory configured to store one or more video frames; and

a processor configured to;

obtain one or more video frames of a first video, the one or more video frames including one or more objects, wherein the one or more video frames of the first video have a first resolution;

perform object detection to detect the one or more objects in the one or more video frames of the first video;

obtain one or more video frames of a second video, the one or more video frames including the one or more objects, wherein the one or more video frames of the second video have a second resolution that is greater than the first resolution;

perform an object recognition process on the one or more video frames of the second video to recognize the one or more objects detected in the one or more video frames of the first video;

obtain one or more of feedback associated with the object detection performed on the one or more video frames of the first video or feedback associated with the object recognition process performed on the one or more video frames of the second video; and

adjust one or more of a resolution of the first video to a higher resolution or a lower resolution for performing object detection on one or more subsequent frames of the first video based on the feedback associated with the object detection or a resolution of the second video to a higher resolution or a lower resolution for performing the object recognition process on one or more subsequent frames of the second video based on the feedback associated with the object recognition process.

18. The apparatus of claim 17, wherein the processor is further configured to:

obtain the feedback associated with the object detection performed on the one or more video frames of the first video;

adjust the resolution of frames of the first video based on the feedback associated with the object detection; and

detect at least one object in a subsequent video frame of the first video after the resolution of the frames of the first video is adjusted, the at least one object being detected based on object detection being performed on the subsequent video frame having the adjusted resolution.

19. The apparatus of claim 17, wherein the processor is further configured to:

obtain the feedback associated with the object recognition process performed on the one or more video frames of the second video;

adjust the resolution of frames of the second video based on the feedback associated with the object recognition process; and

perform the object recognition process on at least one object in a subsequent video frame of the second video after the resolution of the frames of the second video is adjusted, the object recognition process being performed on the subsequent video frame having the adjusted resolution.

20. The apparatus of claim 17, wherein the processor is further configured to:

adjust the resolution of frames of the first video to a first adjusted resolution based on the feedback associated with the object detection;

adjust the resolution of frames of the second video to a second adjusted resolution based on the feedback associated with the object recognition process;

detect at least one object in a subsequent video frame of the first video after the resolution of the frames of the first video is adjusted, the at least one object being detected based on object detection being performed on the subsequent video frame having the first adjusted resolution; and

perform the object recognition process on the at least one object detected in the subsequent video frame of the first video, the object recognition process being performed on a subsequent video frame of the second video after the resolution of the frames of the second video is adjusted, the subsequent video frame of the second video having the second adjusted resolution.

21. The apparatus of claim 17, wherein the feedback associated with the object detection is based on one or more of a latency of the object detection or a quality of results from the object detection performed on one or more objects in the one or more video frames of the first video, and wherein the feedback associated with the object recognition process is based on one or more of a latency of the object recognition process or a quality of results from the object recognition process performed on one or more objects in the one or more video frames of the second video.

22. The apparatus of claim 17, wherein adjusting one or more of the resolution of the first video or the resolution of the second video includes:

determining, based on the feedback associated with the object detection, that a latency of the object detection performed on the one or more video frames of the first video is greater than a latency threshold; and

reducing a resolution of at least one video frame of the first video based on the latency of the object detection being greater than the latency threshold.

23. The apparatus of claim 17, wherein adjusting one or more of the resolution of the first video or the resolution of the second video based on the feedback includes:

determining, based on the feedback associated with the object recognition process, that a confidence score generated for the one or more objects by the object recognition process is less than a quality threshold; and

increasing a resolution of at least one video frame of the second video based on the confidence score being less than the quality threshold.

24. The apparatus of claim 17, further comprising:

determining object information associated with the one or more detected objects based on the object detection performed on the one or more video frames of the first video;

modifying the object information from the first resolution to the second resolution; and

performing the object recognition process on the one or more video frames of the second video using the modified object information.

25. The apparatus of claim 24, wherein object information associated with a detected object includes information defining a bounding region generated for the detected object.

26. The apparatus of claim 25, further comprising:

wherein modifying the object information from the first resolution to the second resolution includes upscaling the bounding region from a first size to a second size, wherein the second size is based on a ratio between the first resolution and the second resolution; and

wherein performing the object recognition process on the one or more video frames of the second video using the modified object information includes performing the object recognition process on a region of the one or more video frames of the second video corresponding to the upscaled bounding region.

27. The apparatus of claim 24, wherein object information associated with a detected object includes information defining one or more landmarks on the detected object, and wherein modifying the object information from the first resolution to the second resolution includes changing a location of a landmark from a first location to a second location, wherein the second location is based on a ratio between the first resolution and the second resolution.

28. The apparatus of claim 17, wherein the first video and the second video are obtained from two different video streams.

29. The apparatus of claim 17, wherein the first video is a downscaled version of the second video.

30. The apparatus of claim 17, wherein the apparatus comprises a mobile device with a camera for capturing the video frames and a display for displaying the video frames.