Database assisted OCR for street scenes and other images

• US 7,917,286 B2
• Filed: 12/16/2005
• Issued: 03/29/2011
• Est. Priority Date: 12/16/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Vincent, Luc, Taylor, Bret
• Primary Examiner(s): MANCHO, RONNIE M
• Application Number:
  US11/305,694
• Publication Number:
  US 20070140595A1
• Time in Patent Office: 1,929 Days
• Field of Search: 701/201, 701/207, 701/208, 701/213, 725/91, 725/114, 725/211, 340/531, 340/105, 340/995.19, 340/933, 735/91, 735/114, 735/90, 382/229, 382/275, 382/272, 382/305, 382/104, 382/103, 382/299, 382/187, 382/310, 348/142, 342/357.01, 705/1, 705/11, 705/5, 705/28, 705/26, 707/3, 707/10, 707/102, 707/104.1
• US Class Current: 701/208
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/262 using context analysis, e.g. lexical, syntactic or semantic context

Abstract

Optical character recognition (OCR) for images such as a street scene image is generally a difficult problem because of the variety of fonts, styles, colors, sizes, orientations, occlusions and partial occlusions that can be observed in the textual content of such scenes. However, a database query can provide useful information that can assist the OCR process. For instance, a query to a digital mapping database can provide information such as one or more businesses in a vicinity, the street name, and a range of possible addresses. In accordance with an embodiment of the present invention, this mapping information is used as prior information or constraints for an OCR engine that is interpreting the corresponding street scene image, resulting in much greater accuracy of the digital map data provided to the user.

Claims

1. A method for constructing a database storing one or more street scene images, each street scene image associated with GPS location data identifying an actual location of the street scene image, the method executed by a computer system and comprising:

storing in the database of the computer system, a plurality business names, each business name associated with a street address and one or more keywords;

estimating a street address for a street scene image of the street scene images using the GPS location data associated with that street scene image;

identifying a target address range based on the estimated street address;

querying the database to identify a business name associated with a street address in the target address range and to identify the one or more keywords associated with the identified business name;

performing optical character recognition of the street scene image to recognize text contained in the street scene image, wherein the optical character recognition of the street scene is constrained by at least one of the identified keywords resulting from querying the database;

determining if the recognized text contained in the street scene image corresponds to at least one of the identified keywords;

updating the estimated street address in the database including the GPS location data in response to a determination that the recognized text corresponds to at least one of the identified keywords.

2. The method of claim 1, further comprising performing image analysis of the street scene image to determine if features associated with the identified business name are present in the street scene image.

3. The method of claim 1 wherein the street scene image includes one street address.

4. The method of claim 1 wherein the street scene image is a panoramic image that includes a plurality of street addresses.

5. The method of claim 1 wherein the database includes a business listings directory.

6. The method of claim 1, wherein estimating the street address for the street scene comprises determining a target GPS location for the street scene image using GPS data associated with that street scene image.

7. The method of claim 6 wherein the GPS data includes GPS locations for at least two locations captured in the street scene image.

8. A method for performing image analysis of a street scene image, the street scene image associated with GPS location data identifying an actual location of the street scene image, the method executed by a computer system, and comprising:

querying a database comprising a plurality of street addresses, each street address associated with image features, to identify an image feature that is present in a street scene image of a street address, the street scene image comprising a plurality of address locations and at least two of the plurality of address locations having known GPS data;

performing image analysis of the street scene image to recognize features in the street scene image, wherein the image analysis of the street scene is constrained by the identified image feature from querying the database, and the identified image feature is associated with a street address from the database;

determining if the recognized features in the street scene image correspond to the identified image feature associated with the street address from the database; and

in response to determining that the recognized features correspond to the identified image feature associated with the street address from querying the database, updating the street address in the database including the GPS location data.

9. The method of claim 8 wherein querying the database identifies a number of textual and non-textual features.

10. The method of claim 9 further comprising:

performing image analysis of the street scene image to determine if non-textual features are present.

11. The method of claim 8 wherein the street scene image includes one street address.

12. The method of claim 8 wherein the street scene image is a panoramic image that includes a plurality of street addresses.

13. The method of claim 8 wherein the known GPS data includes known GPS locations for at least two locations captured in the street scene image.

14. A method for performing optical character recognition (OCR) of an image, the image associated with GPS location data identifying an actual location of the image, the method executed by a computer and comprising:

storing in a database a plurality of images, each image associated with one or more keywords;

querying the database to select at least one image;

performing optical character recognition of the selected image to recognize text that may be contained in the image, wherein the optical character recognition of the image is constrained by at least one of the keywords associated with the image in the database;

determining if the recognized text contained in the image corresponds to the at least one of the keywords associated with the image; and

in response to determining that the recognized text contained in the image corresponds to at least one of the keywords associated with the image, updating a street address to be associated with the GPS location data, the street address being associated with the image.

15. The method of claim 14 wherein the keyword is further associated with a street address or a key event.

16. The method of claim 14 wherein querying the database identifies a number of textual and non-textual features that is in the image.

17. The method of claim 16 further comprising:

performing image analysis of the image to determine if non-textual features are present.

18. The method of claim 14 wherein the image is one of a photograph or video frame.

19. The method of claim 14 wherein the image is associated with GPS location data, and in response to determining that the keyword is present, determining an actual GPS location associated with that keyword.

20. A computer program product, comprising a computer-readable storage medium containing computer executable instructions for controlling a computer system to perform the operations of:

storing in a database of the computer system a plurality of business names, each business name associated with a street address and one or more keywords;

estimating a street address for a street scene image using data associated with that street scene image, the street scene image associated with GPS location data identifying an actual location of the street scene image;

identifying a target address range based on the estimated street address of the target GPS location;

querying the database with the target address range to identify a business name associated with a street address in the target address range and to identify the one or more keywords associated with the identified business name;

performing optical character recognition of the street scene image to recognize text contained in the street scene image, wherein the optical character recognition of the street scene is constrained by at least one of the identified keywords resulting from querying the database;

determining if the recognized text contained in the street scene image corresponds to at least one of the identified keywords;

updating the estimated street address including the GPS location data in response to a determination that the recognized text corresponds to at least one of the identified keywords.

21. The computer program product of claim 20, further comprising performing image analysis of the street scene image to determine if features associated with the identified business name are present in the street scene image.

22. The computer program product of claim 20, wherein the street scene image is a panoramic image that includes a plurality of street addresses.

23. The computer program product of claim 20, wherein the GPS data includes GPS locations for at least two locations captured in the street scene image.

24. The computer program product of claim 20, wherein the database includes a business listings directory.

25. The computer program product of claim 20, wherein the computer executable instructions for estimating the street address for the street scene further comprises computer executable instructions for determining a target GPS location for the street scene image using GPS data associated with that street scene image.

26. A computer program product, comprising a computer-readable storage medium containing computer executable instructions for controlling a computer system to perform the operations of:

querying a database comprising a plurality of street addresses, each street address associated with image features, to identify an image feature that is present in a street scene image of a street address of the street addresses, the street scene image comprising a plurality of address locations and at least two of the plurality of address locations having known GPS data;

performing image analysis of the street scene image to recognize features in the street scene image, wherein the image analysis of the street scene is constrained by the identified image feature from querying the database, and the identified image feature is associated with a street address from the database;

determining if the recognized features in the street scene image correspond to the identified image feature associated with the street address from the database; and

in response to determining that the recognized features correspond to the identified image feature associated with the street address from querying the database, updating the street address in the database including GPS location data associated with GPS location data, the GPS location data identifying an actual location of the street stress.

27. The computer program product of claim 26, wherein querying the database identifies a number of textual and non-textual features.

28. The computer program product of claim 27 further comprising:

performing image analysis of the street scene image to determine if non-textual features are present.

29. A computer program product, comprising a computer-readable storage medium containing computer executable instructions for controlling a computer system to perform the operations of:

storing in a database a plurality of images, each image associated with one or more keywords and GPS location data identifying an actual location of the image;

querying the database to select at least one image;

performing optical character recognition of the selected image to recognize text that may be contained in the image, wherein the optical character recognition of the image is constrained by at least one of the keywords associated with the image in the database;

determining if the recognized text contained in the image corresponds to the at least one of the keywords associated with the image; and

in response to determining that the recognized text contained in the image corresponds to at least one of the keywords associated with the image, updating a street address in the database to be associated with the GPS data, the street address being associated with image.

30. The computer program product of claim 29 wherein the keyword is further associated with a street address or a key event.

31. A computer system for constructing a database storing one or more street scenes, each street scene image associated with GPS location data identifying an actual location of the street scene image, comprising:

a database storing a plurality of business names and street addresses, each business name associated with a street address of the street addresses and one or more keywords;

a target address range estimator module coupled to the database and adapted to;

estimate a street address of the street addresses for the street scene image using data associated with that street scene image;

a target GPS calculator module coupled to the target address range estimator module and adapted to identify a target address range based on the estimated street address of the target GPS location;

an image registration module coupled to the database and the target address range estimator module and adapted to query the database to identify a business name associated with a street address in the target address range and to identify the one or more keywords associated with the identified business name;

an optical character recognition module coupled to the target address range estimator module and the image registration module and adapted to;

perform optical character recognition of the street scene image to recognize text contained in the street scene image, wherein the optical character recognition of the street scene is constrained by at least one of the identified keywords resulting from querying the database; and

determining if the recognized text contained in the street scene image corresponds to at least one of the identified keywords; and

the image registration module adapted to update the estimated street address in the database including the GPS location data in response to a determination that the recognized text corresponds to at least one of the identified keywords.

32. The computer system of claim 31, wherein the target address range estimator is further adapted to determine a target GPS location for the street scene image using GPS data associated with that street scene image.

33. A computer system for performing image analysis of a street scene image using a database, the street scene image associated with GPS location data identifying an actual location of the street scene image, comprising:

an image registration module adapted to query the database comprising a plurality of street addresses, each street address associated with image features, to identify an image feature that is present in the street scene image of the street address, the street scene image comprising a plurality of address locations and at least two of the plurality of address locations having known GPS data;

an optical character recognition module coupled to the image registration module and adapted to;

perform image analysis of the street scene image to recognize features in the street scene image, wherein the image analysis of the street scene is constrained by the identified image feature from querying the database, and the identified image feature is associated with the street address from the database; and

determine if the recognized features in the street scene image correspond to the identified image feature associated with the street address from the database; and

the image registration module, in response to determining that the recognized features correspond to the identified image feature associated with the street address from querying the database, adapted to

update the street address in the database including the GPS location data, the street address associated with the identified image feature.

34. A computer system for performing optical character recognition (OCR) of an image using a database, comprising:

the database storing a plurality of images, each image associated with one or more keywords, and GPS location data to identifying an actual location of the images;

an image registration module coupled to the database and adapted to query the database to select at least one image; and

an OCR module coupled to the image registration module and adapted to;

perform OCR of the selected image to recognize text that may be contained in the image, wherein the OCR of the image is constrained by at least one of the keywords associated with the image in the database;

determine if the recognized text contained in the image corresponds to the at least one of the keywords associated with the image; and

in response to determining that the recognized text contained in the image corresponds to at least one of the keywords associated with the image, assigning an actual GPS location to a street address associated with an image of the plurality of images; and

updating the street address in the database to be associated with the GPS location data.

35. A method for constructing a database storing one or more street scene images, each street scene image associated with GPS location data identifying an actual location of the one or more street scene images, the method executed by a computer system, comprising:

storing in the database of the computer system, a plurality business names, each business name associated with a street address and one or more keywords;

estimating a street address for a street scene image of the one or more street scene images using the GPS location data associated with the one or more street scene images;

identifying a target address range based on the estimated street address;

querying the database to identify a business name of the plurality of business names associated with a street address in the target address range, and to identify the one or more keywords associated with the identified business name;

performing optical character recognition of the one or more street scene images to recognize text contained in the one or more street scene images, wherein the optical character recognition of the one or more street scene images is constrained by at least one of the identified keywords resulting from querying the database;

determining if the recognized text contained in the one or more street scene images corresponds to at least one of the identified keywords; and

updating the business name associated with the street address in the database including the GPS location data of the street scene image.

36. A method for constructing a database storing one or more street scene images, each street scene image associated with GPS location data identifying an actual location of the one or more street scene images, the method executed by a computer system, comprising:

storing in the database of the computer system, a plurality business names, each business name associated with a street address and one or more keywords;

estimating a street address for a street scene image of the one or more street scene images using the GPS location data associated with the one or more street scene images;

identifying a target address range based on the estimated street address;

querying the database to identify a business name of the plurality of business names associated with a street address in the target address range, and to identify the one or more keywords associated with the identified business name;

performing optical character recognition of the one or more street scene images to recognize text contained in the one or more street scene images, wherein the optical character recognition of the one or more street scene images is constrained by at least one of the identified keywords resulting from querying the database;

determining if the recognized text contained in the one or more street scene images corresponds to at least one of the identified keywords; and

updating the one or more street scene images in database including the business name associated with the street address in response to determination that the recognized text corresponds to at least on of the identified keywords.