Business change detection from street level imagery

• US 10,339,193 B1
• Filed: 11/24/2015
• Issued: 07/02/2019
• Est. Priority Date: 11/24/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Stumpe, Martin Christian, Yatziv, Liron, Lee, Dar-Shyang, Yu, Qian, Fuhrmann, Oded, Shet, Vinay Damodar, Arnoud, Sacha Christophe
• Primary Examiner(s): Gart, Matthew S
• Assistant Examiner(s): Swartz, Stephen S
• Application Number:
  US14/950,795
• Time in Patent Office: 1,316 Days
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/19013 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching specially adapted for image segmentation G06T7/10; specially adapted for the analysis of motion G06T7/20; specially adapted for image alignment G06T7/30; specially adapted for the calculation of depth from stereo images G06T7/50; specially adapted for position determination G06T7/70

Abstract

Aspects of the disclosure provide for a method for updating business information. A business at a business location and a reference image for the business at the business location may be selected for an update. From a number of more recent images, a comparison image may be selected based on a likelihood that the comparison image captures the business location. Text and visual features reference image may be compared with text and visual features of the comparison image to determine a text similarity score and a features similarity score. A confidence level indicating whether the business in the reference image is in the comparison image may then be determined using the text similarity score and the feature similarity score. According to the confidence level, the business information of the business may be updated.

Claims

1. A method comprising:

selecting, using one or more processors, a business associated with stored business information;

selecting, from pre-stored images, using the one or more processors, a reference image depicting a business location of the business, the reference image having a timestamp;

identifying, using the one or more processors, one or more new images based on characteristics of at least one of the business and the reference image, each of the one or more new images being associated with a timestamp that is more recent than the timestamp of the reference image and location and orientation information;

for each new image, determining, using the one or more processors, a likelihood value indicative of the likelihood that the business location is shown in a given new image of the one or more new images by;

projecting one or more rays from the business location to the a location of the given new image, and

identifying an angular relationship between the one or more rays and the given new image'"'"'s location and orientation;

selecting from the one or more new images, using the one or more processors, a comparison image based on the likelihood values of the one or more new images;

comparing, using the one or more processors, text information and visual features of the reference image with text information and visual features of the comparison image;

determining, using the one or more processors, a text similarity score and a feature similarity score based on the comparison;

determining, using the one or more processors, a confidence level using the text similarity score and the feature similarity score, the confidence level indicating whether the business in the reference image is in the comparison image; and

updating, using the one or more processors, the business information of the business according to the confidence level.

2. The method of claim 1, further comprising:

receiving, using the one or more processors, a request for business information of the business; and

providing, using the one or more processors, the updated business information in response to the request.

3. The method of claim 1, wherein a likelihood value for a particular image is highest when the particular image is oriented along one of the rays and in the direction of the business location.

4. The method of claim 1, wherein selecting the comparison image further comprises:

determining whether the likelihood value of each new image of the one or more new images satisfies a likelihood value threshold; and

selecting a new image that both satisfies the likelihood value threshold and has a most recent timestamp of any other new images of the one or more new images that also satisfy the likelihood value as the comparison image.

5. The method of claim 1, further comprising:

identifying, by the one or more processors, an image region in the reference image and an image region in the comparison image, and

wherein comparing the text information and visual features of the reference image with text information and visual features of the comparison image comprises comparing text information and visual features from within the image regions.

6. The method of claim 1, wherein comparing the text information of the reference image with text information of the comparison image further comprises:

detecting words in the reference image and the comparison image; and

determining a percentage of the words detected in the comparison image that match the words detected in the reference image, wherein the text similarity score is determined further based on the percentage.

7. The method of claim 6, further comprising determining a number of uncommon words detected in the comparison image that match the words detected in the reference image, and wherein the text similarity score is based on at least the determined number of uncommon words.

8. The method of claim 1, wherein the feature similarity score is determined based on at least a percentage of visual features in the comparison image that match visual features in the reference image.

9. The method of claim 1, wherein updating the business information comprises, when the confidence level is above a confidence level threshold, storing the comparison image in association with the business.

10. The method of claim 9, wherein the confidence level threshold is a first confidence level threshold and updating the business information comprises, when the confidence level is below the first confidence level threshold and above a second confidence level threshold, flagging the comparison image for review.

11. The method of claim 1, wherein updating the business information comprises, when the confidence level is below a confidence level threshold, indicating the business is closed or replaced by another business.

12. The method of claim 1, wherein selecting the reference image further comprises checking for objects in one or more of the pre-stored images that are at least partially occluding the business location.

13. The method of claim 1, further comprising:

selecting, by the one or more processors, a plurality of businesses along a route; and

receiving, by the one or more processors, a series of new images from a device configured to collect images sequentially along the route;

wherein each business in the plurality of businesses is selected and updated sequentially along the route.

14. A system comprising:

a memory storing business information and one or more images in association with one or more businesses; and

one or more processors configured to;

select a business associated with stored business information;

select a reference image depicting a business location of the business, the reference image having a timestamp;

identify one or more new images based on characteristics of at least one of the business and the reference image, each of the one or more new images being associated with a timestamp that is more recent than the timestamp of the reference image and location and orientation information;

for each new image, determine a likelihood value indicative of the likelihood that the business location is shown in a given new image of the one or more new images by;

projecting one or more rays from the business location to the a location of the given new image, and

identifying an angular relationship between the one or more rays and the given new image'"'"'s location and orientation;

select from the one or more new images a comparison image based on the likelihood values of the one or more new images;

compare text information and visual features of the reference image with text information and visual features of the comparison image;

determine a text similarity score and a feature similarity score based on the comparison;

determine a confidence level using the text similarity score and the feature similarity score, the confidence level indicating whether the business in the reference image is in the comparison image; and

update the business information of the business according to the confidence level.

15. The system of claim 14, wherein the one or more processors are further configured to:

receive a request for business information of the business; and

provide the updated business information in response to the request.

16. The system of claim 14, wherein the one or more processors are further configured to select the comparison image by:

determining whether the likelihood value of each new image of the one or more new images satisfies a likelihood value threshold; and

selecting a new image that both satisfies the likelihood value threshold and has a most recent timestamp of any other new images of the one or more new images that also satisfy the likelihood value as the comparison image.

17. The system of claim 14, wherein the one or more processors are further configured to:

identify an image region in the reference image and an image region in the comparison image, and

wherein the text information and visual features of the reference image are compared with text information and visual features of the comparison image by comparing text information and visual features from within the image regions.

18. The system of claim 14, wherein the one or more processors are further configured to compare the text information of the reference image with text information of the comparison image by:

detecting words in the reference image and the comparison image;

determining a percentage of the words detected in the comparison image that match the words detected in the reference image, wherein the text similarity score is determined further based on the percentage; and

determining a number of uncommon words detected in the comparison image that match the words detected in the reference image.

19. The system of claim 14, wherein the one or more processors are further configured to select the reference image by checking for objects in one or more of the pre-stored images that are at least partially occluding the business location.

20. A non-transitory, tangible machine readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

selecting a business associated with stored business information;

selecting a reference image depicting a business location of the business, the reference image having a timestamp;

identifying one or more new images based on characteristics of at least one of the business and the reference image, each of the one or more new images being associated with a timestamp that is more recent than the timestamp of the reference image and location and orientation information;

for each new image, determining a likelihood value indicative of the likelihood that the business location is shown in a given new image of the one or more new images by;

projecting one or more rays from the business location to the a location of the given new image, and

identifying an angular relationship between the one or more rays and the given new image'"'"'s location and orientation;

selecting from the one or more new images a comparison image based on the likelihood values of the one or more new images;

comparing text information and visual features of the reference image with text information and visual features of the comparison image;

determining a text similarity score and a feature similarity score based on the comparison;

determining a confidence level using the text similarity score and the feature similarity score, the confidence level indicating whether the business in the reference image is in the comparison image; and

updating the business information of the business according to the confidence level.