Camera OCR with context information

• US 9,082,035 B2
• Filed: 04/18/2012
• Issued: 07/14/2015
• Est. Priority Date: 08/29/2011
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hwang, Kyuwoong, Lee, Te-Won, Kim, Duck Hoon, You, Kisun, Jin, Minho, Kim, Taesu, Cho, Hyun-Mook
• Primary Examiner(s): TABATABAI, ABOLFAZL
• Application Number:
  US13/450,016
• Publication Number:
  US 20130108115A1
• Time in Patent Office: 1,182 Days
• Field of Search: 382/106, 382/140, 382/159, 382/161, 382/229, 382/311, 382/317, 382/321, 348/169, 348/170, 348/171, 348/172, 704/2, 704/5, 704/9, 704/10, 704/235, 704/258
• US Class Current: 1/1
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/127 with the intervention of an operator
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 30/287 of Kanji, Hiragana or Katakana characters

Abstract

Embodiments of the invention describe methods and apparatus for performing context-sensitive OCR. A device obtains an image using a camera coupled to the device. The device identifies a portion of the image comprising a graphical object. The device infers a context associated with the image and selects a group of graphical objects based on the context associated with the image. Improved OCR results are generated using the group of graphical objects. Input from various sensors including microphone, GPS, and camera, along with user inputs including voice, touch, and user usage patterns may be used in inferring the user context and selecting dictionaries that are most relevant to the inferred contexts.

Claims

1. A method for performing Optical Character Recognition (OCR), the method comprising:

obtaining an image generated by a camera coupled to a device;

identifying a portion of the image comprising at least one graphical object;

inferring a non-textual context associated with the image;

selecting a group of one or more graphical objects based on the non-textual context associated with the image; and

determining an OCR result of the at least one graphical object using the group of graphical objects.

2. The method of claim 1, wherein determining the OCR result comprises performing an OCR using the group of graphical objects.

3. The method of claim 1, wherein determining the OCR result comprises:

generating a plurality of OCR candidates for the at least one graphical object by performing an OCR on the at least one graphical object; and

selecting an OCR candidate from the plurality of OCR candidates using the group of graphical objects.

4. The method of claim 1, further comprising modifying the inferred non-textual context using the OCR result.

5. The method of claim 1, wherein the at least one graphical object comprises one or more of symbols, characters, words, signs, and numbers.

6. The method of claim 1, wherein the group of one or more graphical objects comprises a plurality of graphical objects associated with each other through a common characteristic.

7. The method of claim 1, wherein the group of one or more graphical objects represents a dictionary.

8. The method of claim 1, wherein the group of one or more graphical objects represents a refinement of a dictionary.

9. The method of claim 1, wherein the non-textual context is inferred using a graphical illustration, wherein the graphical illustration comprises one or more gestures, one or more graphical representations of an object, or some combination thereof.

10. The method of claim 1, wherein the non-textual context of the image is inferred using sensor input.

11. The method of claim 10, wherein the sensor input is provided by the camera, a microphone, a light sensor, a clock, a GPS element, or some combination thereof.

12. The method of claim 1, wherein inferring the non-textual context comprises:

estimating a distance between the camera and the at least one graphical object; and

inferring the context based on the estimated distance between the camera and the at least one graphical object.

13. The method of claim 1, wherein inferring the non-textual context comprises:

detecting at least one characteristic associated with a plurality of light rays in an environment where the image is obtained; and

inferring the context based on the at least one characteristic associated with the plurality of light rays.

14. The method of claim 13, wherein the at least one characteristic associated with the plurality of light rays is brightness, wherein brightness under a threshold results in an inference of obtaining the image indoors and brightness over the threshold results in the inference of obtaining the image outdoors.

15. The method of claim 1, wherein inferring the non-textual context comprises:

receiving an audio input using a microphone;

performing pattern recognition on the audio input; and

inferring the context based on a recognized pattern in the audio input.

16. The method of claim 15, wherein the recognized pattern comprises a voice command from a user.

17. The method of claim 15, wherein the recognized pattern comprises an indication for a location of the camera at a time of receiving the audio input.

18. The method of claim 1, wherein inferring the non-textual context comprises:

receiving context-related information from a user; and

inferring the context based on the received context-related information from the user.

19. The method of claim 1, wherein inferring the non-textual context comprises:

estimating a location of the device; and

inferring the context based on the estimated location of the device.

20. The method of claim 19, wherein the location of the device is estimated using a GPS element of the device.

21. The method of claim 19, wherein the location of the device is estimated using a strength of a signal in relation to a cell tower.

22. The method of claim 19, wherein the location of the device is estimated using an audio input from a microphone.

23. The method of claim 1, further comprising inferring a textual context, comprising analyzing a plurality of content semantics associated with the image.

24. The method of claim 23, wherein analyzing the plurality of content semantics is based on analyzing a probability of a co-occurrence amongst a plurality of graphical objects.

25. The method of claim 23, wherein analyzing the plurality of content semantics is based on analyzing a relationship amongst at least one graphical object and at least one graphical illustration.

26. The method of claim 25, wherein the at least one graphical illustration comprises a gesture, a graphical representation of an object, or both.

27. The method of claim 23, wherein analyzing the plurality of content semantics is based on analyzing the plurality of content semantics based on multiple languages.

28. A method of claim 27, wherein analyzing the plurality of content semantics based on multiple languages comprises:

identifying one or more portions of the image comprising a first graphical object in a first language and a second graphical object in a second language, wherein the first graphical object and the second graphical object have a similar meaning;

identifying a first plurality of candidates for the first graphical object and a second plurality of candidates for the second graphical object; and

selecting a graphical object from the first plurality of candidates that has a similar meaning to at least one of the second plurality of candidates.

29. The method of claim 28, wherein selecting the graphical object from the first plurality of candidates comprises:

translating the second plurality of candidates for the second graphical object to the first language; and

finding at least one graphical object with the similar meaning from the first plurality of candidates to the translated second plurality of candidates.

30. The method of claim 23, wherein inferring the textual context comprises:

identifying at least one graphical object from a plurality of graphical objects; and

inferring the textual context based on the at least one identified graphical object from the plurality of graphical objects.

31. The method of claim 23, wherein inferring the textual context comprises inferring the textual context based on analyzing a structural layout of the image.

32. The method of claim 1, wherein inferring the non-textual context comprises:

detecting an environmental factors using input from at least one sensor;

retrieving a past behavioral response by a user during a presence of a similar environmental factors; and

inferring the non-textual context using the past behavioral response.

33. The method of claim 32, wherein the environmental factor corresponds to a time, a location, a sound, a distance of a subject of the image from the user, or some combination thereof.

34. A device, comprising:

memory; and

a processor coupled to the memory and configured to;

obtain an image from the memory;

identify a portion of the image comprising at least one graphical object;

infer a non-textual context associated with the image;

select a group of graphical objects based on the non-textual context associated with the image; and

determine an Optical Character Recognition (OCR) result of the at least one graphical object using the group of graphical objects.

35. The device of claim 34, wherein determining the OCR result comprises performing an OCR using the group of one or more graphical objects.

36. The device of claim 34, wherein determining the OCR result comprises the processor further configured to:

generate a plurality of OCR candidates for the at least one graphical object by performing an OCR on the at least one graphical object; and

select an OCR candidate from the plurality of OCR candidates using the group of one or more graphical objects.

37. The device of claim 34, further comprising modifying the inferred non-textual context using the OCR result.

38. The device of claim 34, wherein the at least one graphical object comprises one or more of symbols, characters, words, signs, and numbers.

39. The device of claim 34, wherein the group of one or more graphical objects comprises a plurality of graphical objects associated with each other through a common characteristic.

40. The device of claim 34, wherein the group of one or more graphical objects represents a dictionary.

41. The device of claim 34, wherein the group of one or more graphical objects represents a refinement of a dictionary.

42. The device of claim 34, wherein the non-textual context is inferred using a graphical illustration, wherein the graphical illustration comprises one or more gestures, one or more graphical representations of an object, or some combination thereof.

43. The device of claim 34, wherein the non-textual context of the image is inferred using a sensor input.

44. The device of claim 43, wherein the sensor input is provided by the camera, a microphone, a light sensor, a clock, a GPS element, or some combination thereof.

45. The device of claim 34, wherein inferring the non-textual context comprises the processor further configured to:

estimate a distance between the camera and the at least one graphical object; and

infer the context based on the estimated distance between the camera and the at least one graphical object.

46. The device of claim 34, wherein inferring the non-textual context comprises the processor further configured to:

detect at least one characteristic associated with a plurality of light rays in an environment where the image is obtained; and

infer the context based on the at least one characteristic associated with the plurality of light rays.

47. The device of claim 46, wherein the at least one characteristic associated with the plurality of light rays is brightness, wherein brightness under a threshold results in an inference of obtaining the image indoors and brightness over the threshold results in the inference of obtaining the image outdoors.

48. The device of claim 34, wherein inferring the non-textual context comprises the processor further configured to:

receive an audio input using a microphone;

perform pattern recognition on the audio input; and

infer the context based on a recognized pattern in the audio input.

49. The device of claim 48, wherein the recognized pattern comprises a voice command from a user.

50. The device of claim 48, wherein the recognized pattern comprises an indication for a location of the camera at a time of receiving the audio input.

51. The device of claim 50, wherein inferring the non-textual context comprises the processor further configured to:

receive context-related information from a user; and

infer the context based on the received context-related information from the user.

52. The device of claim 51, wherein the location of the device is estimated using a strength of a signal in relation to a cell tower.

53. The device of claim 34, wherein inferring the non-textual context comprises the processor further configured to:

estimate a location of the device; and

infer the context based on the estimated location of the device.

54. The device of claim 53, wherein the location of the device is estimated using a GPS element of the device.

55. The device of claim 53, wherein the location of the device is estimated using an audio input from a microphone.

56. The device of claim 34, further comprising inferring a textual context, comprising analyzing a plurality of content semantics associated with the image.

57. The device of claim 56, wherein analyzing the plurality of content semantics is based on analyzing a probability of a co-occurrence amongst a plurality of graphical objects.

58. The device of claim 56, wherein analyzing the plurality of content semantics is based on analyzing a relationship amongst at least one graphical object and at least one graphical illustration.

59. The device of claim 58, wherein the at least one graphical illustration comprises a gesture, a graphical representation of an object, or both.

60. A device of claim 58, wherein analyzing the plurality of content semantics based on multiple languages comprises:

identifying one or more portions of the image comprising a first graphical object in a first language and a second graphical object in a second language, wherein the first graphical object and the second graphical object have a similar meaning;

identifying a first plurality of candidates for the first graphical object and a second plurality of candidates for the second graphical object; and

selecting a graphical object from the first plurality of candidates that has a similar meaning to at least one of the second plurality of candidates.

61. The device of claim 60, wherein selecting the graphical object from the first plurality of candidates comprises:

translating the second plurality of candidates for the second graphical object to the first language; and

finding at least one graphical object with the similar meaning from the first plurality of candidates to the translated second plurality of candidates.

62. The device of claim 56, wherein analyzing the plurality of content semantics is based on analyzing the plurality of content semantics based on multiple languages.

63. The device of claim 56, wherein inferring the textual context comprises the processor further configured to:

identify at least one graphical object from a plurality of graphical objects; and

infer the textual context based on the at least one identified graphical object from the plurality of graphical objects.

64. The device of claim 56, wherein inferring the textual context comprises:

inferring the textual context based on analyzing a structural layout of the image.

65. The device of claim 34, wherein inferring the non-textual context comprises the processor further configured to:

detect an environmental factor using input from at least one sensor;

retrieve a past behavioral response by a user during a presence of a similar environmental factors; and

infer the context using the past behavioral response by the user.

66. The device of claim 65, wherein the environmental factors corresponds to a time, a location, a sound, a distance of a subject of the image from a user, or some combination thereof.

67. A non-transitory computer-readable storage medium comprising code stored therein and which is executable by a processor of a device to cause the process to:

obtain an image generated by a camera;

identify a portion of the image comprising at least one graphical object;

infer a non-textual context associated with the image;

select a group of graphical objects based on the non-textual context associated with the image; and

determine an Optical Character Recognition (OCR) result of the at least one graphical object using the group of graphical objects.

68. The non-transitory computer-readable storage medium of claim 67, wherein improving the OCR result comprises the code further executable to perform an OCR using the group of graphical objects.

69. The non-transitory computer-readable storage medium of claim 67, wherein improving the OCR result comprises the code further executable to:

generate a plurality of OCR candidates for the at least one graphical object by performing an OCR on the at least one graphical object; and

select an OCR candidate from the plurality of OCR candidates using the group of graphical objects.

70. An apparatus for performing Optical Character Recognition (OCR), comprising:

means for obtaining an image generated by a camera coupled to a device;

means for identifying a portion of the image comprising at least one graphical object;

means for inferring a non-textual context associated with the image;

means for selecting a group of graphical objects based on the non-textual context associated with the image; and

means for determining an OCR result of the at least one graphical object using the group of graphical objects.

71. The apparatus of claim 70, wherein determining the OCR result comprises a means for performing an OCR using the group of one or more graphical objects.

72. The apparatus of claim 70, wherein determining the OCR result comprises:

means for generating a plurality of OCR candidates for the at least one graphical object by performing an OCR on the at least one graphical object;

means for selecting an OCR candidate from the plurality of OCR candidates using the group of one or more graphical objects; and

means for using the selected OCR candidate in improving the OCR result.