Sequence transcription with deep neural networks

• US 8,965,112 B1
• Filed: 12/17/2013
• Issued: 02/24/2015
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ibarz, Julian, Bulatov, Yaroslav, Goodfellow, Ian
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US14/108,474
• Time in Patent Office: 434 Days
• US Class Current: 382/156
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Systems and methods for sequence transcription with neural networks are provided. More particularly, a neural network can be implemented to map a plurality of training images received by the neural network into a probabilistic model of sequences comprising P(S|X) by maximizing log P(S|X) on the plurality of training images. X represents an input image and S represents an output sequence of characters for the input image. The trained neural network can process a received image containing characters associated with building numbers. The trained neural network can generate a predicted sequence of characters by processing the received image.

Claims

1. A computer-implemented method comprising:

training, by one or more computing devices, a neural network implemented by one or more processors of a system to map a plurality of training images received by the neural network into a probabilistic model of sequences comprising P(S|X) by maximizing log P(S|X) on the plurality of training images, wherein the training images respectively contain a sequence of characters having a sequence length that is greater than 1, (and wherein each character in the sequence is a discrete variable having a finite number of multiple possible values,) wherein X represents an input image and S represents an output sequence of characters for the input image, and wherein each character in the sequence is a discrete variable having a finite number of multiple possible values, the neural network and the plurality of training images being configured to assume a predetermined maximum sequence length that is greater than 1, wherein each of the one or more computing devices comprises one or more processors;

receiving, by the one or more computing devices, an image containing characters associated with building numbers;

processing, by the one or more computing devices, with the trained neural network the received image containing characters associated with building numbers; and

generating, by the one or more computing devices, with the trained neural network a predicted sequence of characters based at least in part on the processing of the received image.

2. The computer-implemented method of claim 1, wherein at least one of the plurality of training images is preprocessed to subtract the mean of the respective image prior to being received by the neural network.

3. The computer-implemented method of claim 1, wherein when processing the received image, the trained neural network performs localization and segmentation on the received image.

4. The computer-implemented method of claim 1, wherein S is defined as S as a collection of N random variables S₁, . . . , S_N representing the elements of the sequence and an additional random variable L representing the length of the sequence so that a probability of a specific sequence s=s₁, . . . , s_n is given by P(S=s|X)=P(L=n midX)Π

_i=1ⁿP(S_i=s_i midX).

5. The computer-implemented method of claim 4, wherein L is less than or equal to 20.

6. The computer-implemented method of claim 4, wherein a softmax function is utilized to predict each random variable.

7. The computer-implemented method of claim 1, wherein the received image is a street level image.

8. The computer-implemented method of claim 1, further comprising storing an association between the predicted sequence of characters and the received image.

9. The computer-implemented method of claim 1, wherein the predicted sequence of characters from the received image comprises a probability of accuracy.

10. The computer-implemented method of claim 1, wherein the neural network is end-to-end trained by backpropagation.

11. A system comprising:

one or more processors;

one or more memories; and

machine-readable instructions stored in the one or more memories, that upon execution by the one or more processors cause the system to carry out operations comprising;

training a neural network implemented by one or more processors of a system to map a plurality of training images received by the neural network into a probabilistic model of sequences comprising P(S|X) by maximizing log P(S|X) on the plurality of training images, wherein X represents an input image and S represents an output sequence of characters for the input image, and wherein each character in the sequence is a discrete variable having a finite number of multiple possible values, the neural network and the plurality of training images being configured to assume a predetermined maximum sequence length that is greater than 1;

receiving, by the one or more computing devices, an image containing characters associated with building numbers;

processing, by the one or more computing devices, with the trained neural network the received image containing characters associated with building numbers; and

generating, by the one or more computing devices, with the trained neural network a predicted sequence of characters based at least in part on the processing of the received image.

12. The system of claim 11, wherein at least one of the plurality of training images is preprocessed to subtract the mean of the respective image prior to being received by the neural network.

13. The system of claim 11, wherein when processing the received image, the trained neural network performs localization and segmentation on the received image.

14. The system of claim 11, wherein S is defined as S as a collection of N random variables S₁, . . . , S_N representing the elements of the sequence and an additional random variable L representing the length of the sequence so that a probability of a specific sequence s=s₁, . . . , s_n is given by P(S=s|X)=P(L=n midX)Π

_i=1ⁿP(S_i=s_i midX).

15. The system of claim 14, wherein L is less than or equal to 20.

16. The system of claim 14, wherein L is less than or equal to 10.

17. The system of claim 11, wherein the received image is a street level image.

18. A non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

training a neural network implemented by one or more processors of a system to map a plurality of training images received by the neural network into a probabilistic model of sequences comprising P(S|X) by maximizing log P(S|X) on the plurality of training images, wherein X represents an input image and S represents an output sequence of characters for the input image, and wherein each character in the sequence is a discrete variable having a finite number of multiple possible values, the neural network and the plurality of training images being configured to assume a predetermined maximum sequence length that is greater than 1;

receiving, by the one or more computing devices, an image containing characters associated with building numbers;

processing, by the one or more computing devices, with the trained neural network the received image containing characters associated with building numbers; and

generating, by the one or more computing devices, with the trained neural network a predicted sequence of characters based at least in part on the processing of the received image.

19. The non-transitory computer-readable medium of claim 18, wherein a softmax function is utilized to predict each random variable.

20. The non-transitory computer-readable medium of claim 18, wherein the received image is a street level image.