Sequence transcription with deep neural networks

• US 9,454,714 B1
• Filed: 12/31/2014
• Issued: 09/27/2016
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ibarz, Julian, Bulatov, Yaroslav, Goodfellow, Ian
• Primary Examiner(s): Tsai, Tsung-Yin
• Application Number:
  US14/587,088
• Time in Patent Office: 636 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Systems and methods for sequence transcription with neural networks are provided. More particularly, a neural network can be implemented to map a plurality of training images received by the neural network into a probabilistic model of sequences comprising P(S|X) by maximizing log P(S|X) on the plurality of training images. X represents an input image and S represents an output sequence of characters for the input image. The trained neural network can process a received image containing characters associated with building numbers. The trained neural network can generate a predicted sequence of characters by processing the received image.

Claims

1. A computer-implemented method comprising:

receiving, by the one or more computing devices, an image containing characters associated with character sequences;

processing, by the one or more computing devices, the received image using a trained neural network, wherein

the trained neural network has been trained on a plurality of training images to predict character sequences in images by maximizing log P(S|X), wherein X represents an input image and S represents an output sequence of characters for the input image; and

the plurality of training images each contain a sequence of characters having a character sequence length that is greater than 1, wherein each character in the character sequence is a discrete variable having a finite number of multiple possible values; and

generating, by the one or more computing devices, with the trained neural network a predicted sequence of characters based at least in part on the processing of the received image.

2. The computer-implemented method of claim 1, wherein the neural network is configured to assume a predetermined maximum sequence length that is greater than 1.

3. The computer-implemented method of claim 1, wherein at least one of the plurality of training images is preprocessed to subtract the mean of the respective image prior to being received by the neural network.

4. The computer-implemented method of claim 1, wherein when processing the received image, the trained neural network performs localization and segmentation on the received image.

5. The computer-implemented method of claim 1, wherein S is defined as a collection of N random variables S₁, . . . , S_N representing the elements of the sequence and an additional random variable L representing the length of the sequence so that a probability of a specific sequence s=s₁, . . . , s_n is given by P(S=s|X)=P(L=n midX)Π

_i=1ⁿP(S_i=s_i midX).

6. The computer-implemented method of claim 4, wherein L is less than or equal to 20.

7. The computer-implemented method of claim 4, wherein a softmax function is utilized to predict each random variable.

8. The computer-implemented method of claim 1, wherein the received image is a street level image.

9. The computer-implemented method of claim 1, further comprising storing an association between the predicted sequence of characters and the received image.

10. The computer-implemented method of claim 1, wherein the neural network is end-to-end trained by backpropagation.

11. A system comprising:

one or more processors;

one or more memories; and

machine-readable instructions stored in the one or more memories, that upon execution by the one or more processors cause the system to carry out operations comprising;

receiving an image containing characters associated with character sequences;

processing the received image using a trained neural network, wherein

the trained neural network has been trained on a plurality of training images to predict character sequences in images by maximizing log P(S|X), wherein X represents an input image and S represents an output sequence of characters for the input image; and

the plurality of training images each contain a sequence of characters having a character sequence length that is greater than 1, wherein each character in the character sequence is a discrete variable having a finite number of multiple possible values; and

generating with the trained neural network a predicted sequence of characters based at least in part on the processing of the received image.

12. The system of claim 11, wherein at least one of the plurality of training images is preprocessed to subtract the mean of the respective image prior to being received by the neural network.

13. The system of claim 11, wherein when processing the received image, the trained neural network performs localization and segmentation on the received image.

14. The system of claim 11, wherein S is defined as a collection of N random variables S₁, . . . , S_N representing the elements of the sequence and an additional random variable L representing the length of the sequence so that a probability of a specific sequence s=s₁, . . . , s_n is given by P(S=s|X)=P(L=n midX)Π

_i=1ⁿP(S_i=s_i midX).

15. The system of claim 14, wherein L is less than or equal to 20.

16. The system of claim 14, wherein L is less than or equal to 10.

17. The system of claim 11, wherein the received image is a street level image.

18. A non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

receiving an image containing characters associated with character sequences;

processing the received image using a trained neural network, wherein

the trained neural network has been trained on a plurality of training images to predict character sequences in images by maximizing log P(S|X), wherein X represents an input image and S represents an output sequence of characters for the input image; and

the plurality of training images each contain a sequence of characters having a character sequence length that is greater than 1, wherein each character in the character sequence is a discrete variable having a finite number of multiple possible values; and

generating with the trained neural network a predicted sequence of characters based at least in part on the processing of the received image.

19. The non-transitory computer-readable medium of claim 18, wherein a softmax function is utilized to predict each random variable.

20. The non-transitory computer-readable medium of claim 18, wherein the received image is a street level image.