Text Detection on Mobile Communications Devices

• US 20080118162A1
• Filed: 11/20/2006
• Published: 05/22/2008
• Est. Priority Date: 11/20/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Siegemund, Frank
• Granted Patent: US 7,787,693 B2
• Time in Patent Office: Days
• US Class Current: 382/229
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/162 Quantising the image signal

Abstract

A mobile communications device with an integrated camera is directed towards text. A video stream is analyzed in real time to detect one or more words in a specified region of the video frames and to indicate the detected words on a display. Users can select a word in a video stream and subsequently move or extend the initial selection. It is thus possible to select multiple words. A subregion of the video frame comprising the detected word(s) is pre-processed and compressed before being sent to a remote optical character recognition (OCR) function which may be integrated in an online service such as an online search service.

Claims

1. A method comprising:

(i) receiving at least one image at a camera associated with a mobile communications device, the image comprising text;

(ii) defining a designated display area in the image;

(iii) detecting a subregion of the image comprising a word in the designated display area, the process of detecting the subregion comprising using a first thresholding process;

(iv) displaying the received image on the mobile communications device and indicating the detected subregion on the display of the image;

(v) applying a second thresholding process to the detected subregion, that second thresholding process being different from the first thresholding process, to produce a processed subregion;

(vi) sending information about the processed subregion to a remote entity comprising optical character recognition functionality.

2. A method as claimed in claim 1 which further comprises receiving a video stream at the camera, the video stream comprising the at least one image.

3. A method as claimed in claim 2 which further comprises repeating step (iii) of detecting a subregion, and step (iv) of displaying the received image and indicating the detected subregion on the display of that image, for subsequent frames of the video stream in real time.

4. A method as claimed in claim 1 wherein the second thresholding process is carried out on a subregion of a still image rather than a video frame of a real time video stream.

5. A method as claimed in claim 1 wherein step (iv) of displaying the received image on the mobile communications device and indicating the detected subregion on the display of the image further comprises receiving a user input indicating acceptance of the detected subregion and additionally monitoring for user input indicating requirements to modify the detected subregion.

6. A method as claimed in claim 3 wherein step (iii) of detecting a subregion of the image comprising a word in the designated display area, comprises computing information about a bounding box for that subregion in a current video frame, and wherein the method further comprises, for a subsequent video frame, shifting the designated display area on the basis of the computed information about the bounding box.

7. A method as claimed in claim 6 wherein the computed information about the bounding box comprises the position of the centre of that bounding box.

8. A method as claimed in claim 3 wherein the first thresholding process comprises using a specified threshold which is adjusted for each video frame on the basis of information about the detected subregion for that video frame.

9. A method as claimed in claim 1 wherein the first thresholding process comprises assessing an image element with respect to its neighbouring image elements.

10. A method as claimed in claim 1 wherein the second thresholding process comprises using a threshold which is dynamically selected on the basis of information about potential text elements detected after using various thresholds.

11. A mobile communications device comprising:

(i) a camera arranged to receive at least one image comprising text;

(ii) a processor arranged to define a designated display area in the image;

(iii) the processor being arranged to detect a subregion of the image comprising a word in the designated display area, the process of detecting the subregion comprising using a first thresholding process;

(iv) a display arranged to display the received image and indicate the detected subregion on the display of the image;

(v) the processor being arranged to apply a second thresholding process to the detected subregion, that second thresholding process being different from the first thresholding process, to produce a processed subregion;

(vi) an output arranged to send information about the processed subregion to a remote entity comprising optical character recognition functionality.

12. A mobile communications device as claimed in claim 11 wherein the camera is arranged to receive a video stream, the video stream comprising the at least one image.

13. A mobile communications device as claimed in claim 12 wherein the processor is arranged to repeat step (iii) of detecting a subregion, and step (iv) of displaying the received image and to indicate the detected subregion on the display of that image, for subsequent frames of the video stream in real time.

14. A mobile communications device as claimed in claim 11 wherein the processor is arranged to carry out the second thresholding process on a subregion of a still image rather than a video frame of a real time video stream.

15. A mobile communications device as claimed in claim 11 which further comprises a user interface arranged to receive a user input indicating acceptance of the detected subregion and wherein the processor is arranged to monitor for user input indicating requirements to modify the detected subregion.

16. A mobile communications device as claimed in claim 13 wherein the processor is arranged to compute information about a bounding box for the subregion in a current video frame, and wherein the processor is arranged, for a subsequent video frame, to shift the designated display area on the basis of the computed information about the bounding box.

17. A mobile communications device as claimed in claim 16 wherein the processor is arranged to adjust a specified threshold used in the first thresholding process for each video frame on the basis of information about the detected subregion for that video frame.

18. One or more device-readable media with device-executable instructions for performing steps comprising:

(i) receiving at least one image at a camera associated with a mobile communications device, the image comprising text;

(ii) defining a designated display area in the image;

(iii) detecting a subregion of the image comprising a word in the designated display area, the process of detecting the subregion comprising using a first thresholding process;

(iv) displaying the received image on the mobile communications device and indicating the detected subregion on the display of the image;

(v) applying a second thresholding process to the detected subregion, that second thresholding process being different from the first thresholding process, to produce a processed subregion;

(vi) sending information about the processed subregion to a remote entity comprising optical character recognition functionality.

19. One or more device-readable media as claimed in claim 18 with device-executable instructions for performing steps comprising receiving a video stream at the camera, the video stream comprising the at least one image.

20. One or more device-readable media as claimed in claim 19 with device-executable instructions for performing steps comprising repeating step (iii) of detecting a subregion, and step (iv) of displaying the received image and indicating the detected subregion on the display of that image, for subsequent frames of the video stream in real time.