Object-Of-Interest Detection And Recognition With Split, Full-Resolution Image Processing Pipeline

• US 20150347861A1
• Filed: 05/30/2014
• Published: 12/03/2015
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank, Hunter, Kevin
• Granted Patent: US 9,251,431 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Abstract

Differing embodiments of this disclosure may employ one or all of the several techniques described herein to utilize a “split” image processing pipeline, wherein one part of the “split” image processing pipeline runs an object-of-interest recognition algorithm on scaled down (also referred to herein as “low-resolution”) frames received from a camera of a computing device, while the second part of the “split” image processing pipeline concurrently runs an object-of-interest detector in the background on full resolution (also referred to herein as “high-resolution”) image frames received from the camera. If the object-of-interest detector detects an object-of-interest that can be read, it then crops the object-of-interest out of the “high-resolution” camera buffer, optionally performs a perspective correction, and/or scaling on the object-of-interest to make it the desired size needed by the object-of-interest recognition algorithm, and then sends the scaled, high-resolution representation of the object-of-interest to the object-of-interest recognition algorithm for further processing.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to:

obtain a first representation of a first image, wherein the first representation comprises a first plurality of pixels;

downscale the first representation of the first image to create a first low-resolution representation of the first image;

in a first thread, identify a first low-resolution representation of an object-of-interest within the first low-resolution representation of the first image;

in a second thread, identify and crop out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest, wherein the first and second threads are executed concurrently by the one or more processing units;

evaluate a quality metric for each of the first low-resolution representation of the object-of-interest and the first full-resolution representation of the object-of-interest; and

select the first representation of the object-of-interest that is evaluated as having a quality metric exceeding a first quality threshold value.

2. The non-transitory program storage device of claim 1, wherein the instructions to identify and crop out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest further comprise instructions to:

perform an edge detection process on the first representation of the first image.

3. The non-transitory program storage device of claim 2, wherein the instructions to perform edge detection further comprise instructions to perform a Canny edge detection process.

4. The non-transitory program storage device of claim 1, further comprising instructions to guide placement of the object-of-interest within the first image if neither the first low-resolution representation of the object-of-interest nor the first full-resolution representation of the object-of-interest have been identified within a first predetermined amount of time.

5. The non-transitory program storage device of claim 1, wherein the instructions to identify and crop out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest further comprise instructions to perform at least one of the following operations on the first representation of the first image:

perspective correction, scaling, and corner detection.

6. The non-transitory program storage device of claim 1, wherein the object-of-interest comprises one or more sub-regions-of-interest.

7. The non-transitory program storage device of claim 6, wherein the instructions to evaluate a quality metric for each of the first low-resolution representation of the object-of-interest and the first full-resolution representation of the object-of-interest further comprise instructions to:

evaluate a quality metric for each of the one or more sub-regions-of-interest.

8. A system, comprising:

a memory having, stored therein, computer program code;

a digital camera; and

one or more processing units operatively coupled to the digital camera and memory and configured to execute instructions in the computer program code that cause the one or more processing units to;

obtain a first representation of a first image from the digital camera, wherein the first representation comprises a first plurality of pixels;

downscale the first representation of the first image to create a first low-resolution representation of the first image;

in a first thread, identify a first low-resolution representation of an object-of-interest within the first low-resolution representation of the first image;

in a second thread, identify and crop out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest, wherein the first and second threads are executed concurrently by the one or more processing units;

evaluate a quality metric for each of the first low-resolution representation of the object-of-interest and the first full-resolution representation of the object-of-interest; and

select the first representation of the object-of-interest that is evaluated as having a quality metric exceeding a first quality threshold value.

9. The system of claim 8, wherein the instructions to identify and crop out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest further comprise instructions to:

perform an edge detection process on the first representation of the first image.

10. The system of claim 9, wherein the instructions to perform edge detection further comprise instructions to perform a Canny edge detection process.

11. The system of claim 8, wherein the computer program code further comprises instructions to guide placement of the object-of-interest within the first image if neither the first low-resolution representation of the object-of-interest nor the first full-resolution representation of the object-of-interest have been identified within a first predetermined amount of time.

12. The system of claim 8, wherein the instructions to identify and crop out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest further comprise instructions to perform at least one of the following operations on the first representation of the first image:

perspective correction, scaling, and corner detection.

13. The system of claim 8, wherein the object-of-interest comprises one or more sub-regions-of-interest.

14. The system of claim 13, wherein the instructions to evaluate a quality metric for each of the first low-resolution representation of the object-of-interest and the first full-resolution representation of the object-of-interest further comprise instructions to:

evaluate a quality metric for each of the one or more sub-regions-of-interest.

15. A computer-implemented method, comprising:

obtaining a first representation of a first image from a first digital camera, wherein the first representation comprises a first plurality of pixels;

downscaling, using a computer, the first representation of the first image to create a first low-resolution representation of the first image;

in a first computer thread, identifying a first low-resolution representation of an object-of-interest within the first low-resolution representation of the first image;

in a second computer thread, identifying and cropping out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest, wherein the first and second threads are executed concurrently by the one or more processing units;

evaluating, using a computer, a quality metric for each of the first low-resolution representation of the object-of-interest and the first full-resolution representation of the object-of-interest; and

selecting, using a computer, the first representation of the object-of-interest that is evaluated as having a quality metric exceeding a first quality threshold value.

16. The computer-implemented method of claim 15, wherein the acts of identifying and cropping out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest further comprise the act of:

performing an edge detection process on the first representation of the first image.

17. The computer-implemented method of claim 16, wherein the act of performing an edge detection process further comprises performing a Canny edge detection process.

18. The computer-implemented method of claim 15, wherein the acts of identifying and cropping out a first region within the first representation of the first image to create a first full-resolution representation of the object-of-interest further comprise performing at least one of the following acts on the first representation of the first image:

perspective correction, scaling, and corner detection.

19. The computer-implemented method of claim 15, wherein the object-of-interest comprises one or more sub-regions-of-interest.

20. The computer-implemented method of claim 19, wherein the act of evaluating a quality metric for each of the first low-resolution representation of the object-of-interest and the first full-resolution representation of the object-of-interest further comprises the act of evaluating a quality metric for each of the one or more sub-regions-of-interest.