Method and system for imaging documents, such as passports, border crossing cards, visas, and other travel documents, in mobile applications

• US 9,916,500 B2
• Filed: 10/15/2015
• Issued: 03/13/2018
• Est. Priority Date: 10/17/2014
• Status: Expired due to Fees

• Current Assignee: Blackshark Tech, LLC
• Original Assignee: SimonComputing, Inc.
• Inventors: Solano, Arturo M., Dao, Truc C.
• Primary Examiner(s): BELLA, MATTHEW C
• Assistant Examiner(s): Torres, Jose M
• Application Number:
  US14/883,731
• Publication Number:
  US 20160110597A1
• Time in Patent Office: 880 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables

Abstract

A mobile computing device-implemented method of imaging an object to read information, includes capturing, by running an image capturing thread, a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in an image queue in a first memory location; processing, by running an image processing thread, one or more raw images to extract one or more potential machine readable zone (MRZ) candidates and placing in a MRZ candidate queue in a second memory location; analyzing, by running an image analysis thread, an MRZ candidate to detect an MRZ and placing in an MRZ queue in a third memory location; and creating a composite MRZ if a timer has expired, or the MRZ queue has reached a predetermined threshold.

Claims

1. A mobile computing device-implemented method of imaging an object to read information, comprising the steps of:

a) capturing a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in a first queue in a first memory location;

b) processing, in one or more processors of the mobile device, a plurality of raw images substantially simultaneously to extract potential machine readable zone (MRZ) candidates and placing in a second queue in a second memory location;

c) analyzing one of the MRZ candidates from the second queue to detect the presence of a predetermined size MRZ rectangle;

d) creating an MRZ blob if an MRZ rectangle is detected in step c);

e) processing the MRZ blob to form an MRZ;

f) placing the MRZ of step e) in a third queue in a third memory location; and

g) creating a composite MRZ if a timer has expired, or the third queue has reached a predetermined threshold.

2. The method of claim 1, wherein:

the step b) comprises processing a plurality of raw images separately by individual processors of the mobile device.

3. The method of claim 2, wherein:

the processing step b) is carried out by two to six processors of the mobile computing device.

4. The method of claim 2, wherein the processing step b) comprises:

i) cropping a raw image around a predetermined MRZ window area;

ii) converting the cropped image to a gray scale image;

iii) scrubbing the gray scale image; and

iv) extracting text characters from the scrubbed gray scale image.

5. The method of claim 4, wherein:

the extracted text characters comprise one or more character strings.

6. The method of claim 5, wherein:

the step c) comprises finding in the character strings two or three lines of characters of a predetermined length to detect an MRZ rectangle.

7. The method of claim 6, wherein:

a two-line MRZ rectangle comprises 44 characters in each line.

8. The method of claim 6, wherein:

a three-line MRZ rectangle comprises 30 characters in each line.

9. The method of claim 6, wherein:

the timer in step g) is set from 3 to 8 seconds.

10. The method of claim 6, wherein:

the predetermined threshold for creating a composite MRZ in step g) is three MRZs.

11. The method of claim 1, wherein:

the step g) comprises creating a composite MRZ from a plurality of MRZs by selecting each character appearing at a highest frequency among all of the MRZs in the third queue.

12. The method of claim 1, further comprising:

discarding, in step a), the oldest raw image if the first queue is full.

13. The method of claim 1, further comprising:

discarding, in step b), the oldest MRZ candidate if the second queue is full.

14. The method of claim 1, wherein:

the processing in step b) is carried out at a faster speed than the steps c), d), and e).

15. The method of claim 1, wherein:

the processing in step b) is carried out 20-30 times faster than the steps c), d), and e).

16. The method of claim 1, further comprising:

h) creating a final MRZ by parsing and correcting the composite MRZ.

17. The method of claim 6, wherein the MRZ blob processing step e) comprises:

i) parsing the characters in the MRZ blob to determine object type; and

ii) validating object type by comparing with a predetermined object template.

18. A mobile computing device-implemented method of imaging an object to read information, comprising the steps of:

a) initializing a plurality of image processing threads;

b) initializing one or more of image capturing and image analysis threads;

c) capturing, by running the image capturing thread, a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in an image queue in a first memory location;

d) processing, by running the image processing threads, a plurality of raw images to extract potential machine readable zone (MRZ) candidates and placing in a MRZ candidate queue in a second memory location;

e) analyzing, by running the image analysis thread, an MRZ candidate to detect an MRZ and placing in an MRZ queue in a third memory location; and

f) creating a composite MRZ if a timer has expired, or the MRZ queue has reached a predetermined threshold.

19. The method of claim 18, further comprising:

starting the timer in step e) if the MRZ is first in the MRZ queue.

20. The method of claim 19, comprising:

creating the composite MRZ in step f) when three MRZs are placed in the MRZ queue.

21. The method of claim 19, comprising:

creating the composite MRZ in step f) when the timer has run a preselected time period from 3 to 8 seconds.

22. The method of claim 18, wherein:

the step a) comprises initializing two to six image processing threads.

23. The method of claim 18, wherein:

the number of image processing threads initialized in step a) corresponds to the number of slots in the image queue.

24. The method of claim 23, comprising:

the oldest raw image is discarded in step c) if the image queue is full.

25. The method of claim 18, comprising:

the oldest MRZ candidate is discarded in step d) if the MRZ candidate queue is full.

26. The method of claim 18, wherein each image processing thread in step d) comprises:

i) cropping a raw image around a predetermined MRZ window area;

ii) converting the cropped image to a gray scale image;

iii) scrubbing the gray scale image; and

iii) extracting text characters from the scrubbed gray scale image.

27. The method of claim 26, wherein:

the extracted text characters comprise one or more character strings.

28. The method of claim 27, wherein:

the step e) comprises finding in the character strings two or three lines of characters of a predetermined length to detect an MRZ.

29. The method of claim 28, wherein:

a two-line MRZ comprises 44 characters in each line.

30. The method of claim 28, wherein:

a three-line MRZ comprises 30 characters in each line.

31. The method of claim 18, wherein:

at least one of the image processing threads in step e) runs faster than the image analysis thread in step e).

32. The method of claim 31, wherein:

the at least one image processing thread runs 20-30 times faster than the image analysis thread.

33. The method of claim 18, further comprising:

detecting number of processors on the mobile device prior to step a).

34. The method of claim 33, wherein:

the step a) comprises initializing a plurality of image processing threads corresponding in number to the number of processors detected on the mobile device.

35. A non-transitory computer-readable medium with instructions stored thereon, that when executed by a processing device, perform the steps comprising:

a) initializing a plurality of image processing threads;

b) initializing one or more of image capturing and image analysis threads;

c) capturing, by running the image capturing thread, a plurality of raw images of an object by an image capturing component of the processing device and placing in an image queue in a first memory location;

d) processing, by running the image processing threads, a plurality of raw images to extract potential machine readable zone (MRZ) candidates and placing in an MRZ candidate queue in a second memory location;

e) analyzing, by running the image analysis thread, an MRZ candidate to detect an MRZ and placing in an MRZ queue in a third memory location; and

f) creating a composite MRZ if a timer has expired, or the MRZ queue has reached a predetermined threshold.

36. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to:

start the timer in step e) if the MRZ is first in the MRZ queue.

37. The non-transitory computer-readable medium of claim 36, comprising instructions for causing the processing device to:

create the composite MRZ in step f) when three MRZs are placed in the MRZ queue.

38. The non-transitory computer-readable medium of claim 36, comprising instructions for causing the processing device to:

create the composite MRZ in step f) when the timer has run a preselected time period from 3 to 8 seconds.

39. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to:

initialize two to six image processing threads in step a).

40. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to:

initialize in step a) the number of image processing threads corresponding to the number of slots in the image queue.

41. The non-transitory computer-readable medium of claim 40, comprising instructions for causing the processing device to:

discard in step c) the oldest raw image if the image queue is full.

42. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to:

discard in step d) the oldest MRZ candidate if the MRZ candidate queue is full.

43. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to perform, in each image processing thread in step d), the steps comprising:

i) cropping a raw image around a predetermined MRZ window area;

ii) converting the cropped image to a gray scale image;

iii) scrubbing the gray scale image; and

iv) extracting characters from the scrubbed gray scale image.

44. The non-transitory computer-readable medium of claim 43, wherein:

the extracted text characters comprise one or more character strings.

45. The non-transitory computer-readable medium of claim 44, comprising instructions for causing the processing device in step e) to:

find in the character strings two or three lines of characters of a predetermined length to detect an MRZ.

46. The non-transitory computer-readable medium of claim 45, wherein:

a two-line MRZ comprises 44 characters in each line.

47. The non-transitory computer-readable medium of claim 45, wherein:

a three-line MRZ comprises 30 characters in each line.

48. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to:

run at least one of the image processing threads in step d) faster than the image analysis thread in step e).

49. The non-transitory computer-readable medium of claim 48, comprising instructions for causing the processing device to:

run the at least one image processing thread 20-30 times faster than the image analysis thread.

50. The non-transitory computer-readable medium of claim 35, comprising instructions for causing the processing device to:

detect number of processors on the mobile device prior to step a).

51. The non-transitory computer-readable medium of claim 50, comprising instructions for causing the processing device to:

initialize, in step a), a plurality of image processing threads corresponding in number to the number of processors detected on the mobile device.

52. The non-transitory computer-readable medium of claim 35, wherein:

the processing device comprises a mobile device.

53. A system for imaging an object to read information, comprising:

a) an image capturing component;

b) a control module that runs;

i) an image capturing thread to capture a plurality of raw images of the object by the image capturing component and place in an image queue;

ii) a plurality of image processing threads, each of which processes a raw image to extract a potential machine readable zone (MRZ) candidate and places in an MRZ candidate queue;

iii) an image analysis thread that analyzes an MRZ candidate to detect an MRZ and places in an MRZ queue;

c) a first memory location for the image queue;

d) a second memory location for the MRZ candidate queue;

e) a third memory location for the MRZ queue; and

f) wherein the control module creates a composite MRZ if a timer expires, or the MRZ queue has reached a predetermined threshold.

54. The system of claim 53, wherein:

the control module runs image processing threads corresponding in number to the number of slots in the image queue.

55. The system of claim 53, wherein:

the control module runs image processing threads corresponding in number to the number of processors in the system.

56. The system of claim 53, wherein:

the control module runs two to six image processing threads substantially simultaneously.

57. The system of claim 53, wherein:

the control module creates a composite MRZ when three MRZs are placed in the MRZ queue.

58. The system of claim 53, wherein:

the control module creates a composite MRZ when the timer has run a preselected time period from 3 to 8 seconds.

59. The system of claim 53, wherein:

the control module, when running an image processing thread, crops a raw image around a predetermined MRZ window area, gray scales the cropped image, and scrubs the gray scale image to extract an MRZ candidate.

60. The system of claim 53, wherein:

the control module runs at least one of the image processing threads faster than the image analysis thread.

61. The system of claim 53, wherein:

the control module runs at least one of the image processing threads 20-30 times faster than the image analysis thread.

62. The system of claim 53, wherein the system comprises a mobile device.

63. A mobile computing device-implemented method of imaging an object to read information, comprising the steps of:

a) capturing a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in a first queue in a first memory location;

b) processing, in one or more processors of the mobile device, a plurality of raw images substantially simultaneously to extract potential machine readable zone (MRZ) candidates and placing in a second queue in a second memory location;

c) analyzing one of the MRZ candidates from the second queue to detect the presence of an MRZ;

d) placing the MRZ of step c) in a third queue in a third memory location; and

e) creating a composite MRZ if a timer has expired, or the third queue has reached a predetermined threshold.

64. A mobile computing device-implemented method of imaging an object to read information, comprising the steps of:

a) capturing, by running an image capturing thread, a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in an image queue in a first memory location;

b) processing, by running an image processing thread, one or more raw images to extract one or more potential machine readable zone (MRZ) candidates and placing in a MRZ candidate queue in a second memory location;

c) analyzing, by running an image analysis thread, an MRZ candidate to detect an MRZ and placing in an MRZ queue in a third memory location; and

d) creating a composite MRZ if a timer has expired, or the MRZ queue has reached a predetermined threshold.

65. A mobile computing device-implemented method of imaging an object to read information, comprising the steps of:

a) capturing a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in a first queue in a first memory location;

b) processing, in one or more processors of the mobile device, a plurality of raw images substantially simultaneously to extract potential machine readable zone (MRZ) candidates and placing in a second queue in a second memory location;

c) analyzing one of the MRZ candidates from the second queue to detect the presence of a predetermined size MRZ rectangle;

d) creating an MRZ blob if an MRZ rectangle is detected in step c);

e) processing the MRZ blob to form an MRZ;

f) placing the MRZ of step e) in a third queue in a third memory location;

g) creating a composite MRZ if a timer has expired, or the third queue has reached a predetermined threshold; and

h) the step g) comprising creating a composite MRZ from a plurality of MRZs by selecting each non-handwritten character appearing at a highest frequency among all of the MRZs in the third queue.

66. A mobile computing device-implemented method of imaging an object to read information, comprising the steps of:

a) capturing a plurality of raw images of the object by an image capturing component of the mobile computing device and placing in a first queue in a first memory location;

b) processing, in one or more processors of the mobile device, a plurality of raw images substantially simultaneously to extract potential machine readable zone (MRZ) candidates and placing in a second queue in a second memory location;

c) analyzing one of the MRZ candidates from the second queue to detect the presence of a predetermined size MRZ rectangle;

d) creating an MRZ blob if an MRZ rectangle is detected in step c);

e) processing the MRZ blob to form an MRZ;

f) placing the MRZ of step e) in a third queue in a third memory location;

g) creating a composite MRZ if a timer has expired, or the third queue has reached a predetermined threshold; and

h) the processing step b) comprising;

i) cropping a raw image around a predetermined MRZ window area forming a rectangle of pixels;

ii) converting the cropped image to a gray scale image;

iii) scrubbing the gray scale image to remove background image information; and

iv) extracting text characters from the scrubbed gray scale image.