SYSTEM AND METHOD FOR TOY RECOGNITION

• US 20170304732A1
• Filed: 11/09/2015
• Published: 10/26/2017
• Est. Priority Date: 11/10/2014
• Status: Active Grant

• Current Assignee: Lego A/S (Kirkbi A/S)
• Original Assignee: Lego A/S (Kirkbi A/S)
• Inventors: VELIC, Marko, NOE, Karsten stergaard, MOSEGAARD, Jesper, CHRISTENSEN, Brian Bunch, RIMESTAD, Jens
• Granted Patent: US 10,213,692 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63H 33/08 provided with complementary holes, grooves, or protuberances, e.g. dovetails
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69

Abstract

System and method for automatic computer aided optical recognition of toys, for example, construction toy elements, recognition of those elements on digital images and associating the elements with existing information is presented. The method and system may recognize toy elements of various sizes invariant of toy element distance from the image acquiring device for example camera, invariant of rotation of the toy element, invariant of angle of the camera, invariant of background, invariant of illumination and without the need of predefined region where a toy element should be placed. The system and method may detect more than one toy element on the image and identify them. The system is configured to learn to recognize and detect any number of various toy elements by training a deep convolutional neural network.

Claims

1. A system for recognizing real-world toy objects from one or more images, the system comprising a model creation system and a recognition system;

wherein the model creation system comprises one or more processors and a training database and wherein the training database is configured to store annotated digital images, each annotated digital image depicting a real-world toy object and being annotated with an object identifier identifying the depicted real-world toy object;

wherein the model creation system is configured to;

train a convolutional classification model based on at least a subset of the annotated digital images to predict a matching object identifier when the convolutional classification model is presented with a digital image of a real-world toy object.

wherein the recognition system comprises an image capturing device and one or more processors and wherein the recognition system is configured to;

capture at least one image of a real-world toy object;

use the trained convolutional classification model to predict a matching object identifier from the captured image.

2. A system according to claim 1 wherein the recognition system is further configured to detect one or more real-world toy objects in the captured image and to identify respective object locations within the captured image, each object location corresponding to one of the detected real-world toy objects;

and to use the trained convolutional classification model to predict a respective object identifier for each of the detected real-world toy objects.

3. A system according to claim 1 or 2, wherein the real-world toy objects are toy construction elements of a toy construction system, each toy construction element comprising coupling members for detachably connecting the toy construction element with other toy construction elements of the toy construction system.

4. A system according to any one of the preceding claims;

wherein the recognition system is further configured to obtain, based on the predicted object identifier, further object information about the depicted real-world toy object.

5. A system according to claim 4;

wherein the further object information includes connectivity information indicative of how the real-world toy object can be detachably connected to toy construction elements of a toy construction system.

6. A system according to any one of the preceding claims;

wherein the model creation system is further configured to process at least a first digital image so as to create one or more processed versions of the first digital image; and

wherein training the convolutional classification model is based on one or more processed versions of the first digital image, each processed version being annotated with an object identifier associated with the first digital image.

7. A system according to claim 6;

wherein the model creation system is configured to identify an object portion and a background portion of the first digital image, the object portion depicting the real-world toy object; and

to create one or more processed images by replacing the identified background portion with one or more other background portions.

8. The system according to any one of the preceding claims, wherein the convolutional classification model is a convolutional neural network.

9. The system according to any one of the preceding claims, wherein the convolutional classification model is a deep convolutional classification model.

10. The system according to claim 9;

wherein the deep convolutional classification model comprises of one or more convolutional layers, one or more rectification layers, one or more normalization layers and one or more pooling layers of artificial neurons.

11. The system according to claim 9 or 10, wherein the deep convolutional classification model is a deep convolutional neural network comprising one or more fully connected layers of artificial neurons;

wherein each artificial neuron of a fully connected layer receives respective inputs from at least a majority of artificial neurons of a preceding layer of the deep convolutional neural network.

12. The system according to any one of the preceding claims wherein the model creation system comprises one or more graphics processing units (“

GPU”

) configured to execute a training process for training the convolutional classification model.

13. The system according to any one of the preceding claims, wherein the trained convolutional classification model is configured to output, when presented with a digital image, a prediction indicative of respective likelihoods that said digital image depicts respective real-world toy objects included in the said training database.

14. The system according to any one of the preceding claims, wherein the recognition system is implemented by a mobile device comprising a digital camera, a display and a processor.

15. The system according to any one of the preceding claims, wherein the recognition system is configured to:

create one or more processed versions of the captured image;

feed a plurality of images through the trained convolutional classification model to predict a corresponding plurality of matching object identifiers;

wherein the plurality of images are chosen from the captured image and the one or more processed images; and

to

compute an aggregated predicted object identifier from the plurality of matching object identifiers.

16. The system according to claim 15, wherein the recognition system is configured to:

detect a position of a real-world toy object within the captured image;

create a plurality of different cropped images from the captured image, each cropped image comprising the detected position;

feed the plurality of cropped images through the trained convolutional classification model to predict a corresponding plurality of matching object identifiers; and

compute an aggregated predicted object identifier from the plurality of matching object identifiers.

17. The system according to any one of the preceding claims;

wherein the model creation system is configured to;

receive image data indicative of digital images depicting real-world toy objects;

receive image annotations indicative of object identifiers associated with respective ones of the depicted real-world toy objects; and

to

store annotated digital images in the training database, each annotated digital image depicting a real-world object and being annotated with an object identifier identifying the depicted real-world toy object.

18. The system according to any one of the preceding claims;

wherein the model creation system comprises an image capturing device and a support member for receiving a real-world toy object;

wherein at least one of the support member and the image capturing device is movably arranged so as to vary a viewpoint of the image capturing device relative to the support member; and

wherein the model creation system is configured to capture multiple images from respective viewpoints of a real-world toy object positioned on the support member.

19. The system according to any one of the preceding claims;

wherein the recognition system is further configured to estimate one or more additional attributes of the real-world object depicted in the captured image, in addition to an identification of the real-world object.

20. The system according to claim 19;

wherein the convolutional classification model comprises a convolutional stage configured to extract one or more features from a digital image, and a classification stage configured to predict an object identifier based on the extracted one or more features;

wherein the model creation system is configured to store a plurality of reference representations of respective outputs of the convolutional stage of the trained convolutional classification model produced by the trained convolutional classification model when presented with respective ones of the annotated digital images, each stored reference representation being associated with the object identifier associated with the corresponding annotated digital image and with one or more corresponding reference attributes; and

wherein the recognition system is configured to estimate the one or more additional attributes of the real-world toy object depicted in the captured image by comparing an output of the convolutional stage of the trained convolutional classification model produced by the trained convolutional classification model based on the captured image with one or more of the stored reference representations associated with the predicted object identifier.

21. The system according to claim 19 or 20;

wherein the one or more additional attribute comprises a relative orientation of the real-world toy object relative to the image capturing device.

22. The system according to any one of the preceding claims;

wherein the convolutional classification model comprises at least an upstream and a downstream convolutional layer of feature-extracting kernels, the downstream layer receiving an input based on an output of the upstream layer; and

wherein the downstream layer has a number of kernels larger than the number of kernels of the upstream layer.

23. The system according to any one of the preceding claims;

wherein the convolutional classification model comprises at least 6 convolutional layers of feature-extracting kernels.

24. The system according to claim 22 or 23;

wherein the convolutional layers are arranged as a sequence of layers where information is past from an input layer in a downstream direction; and

where a number of feature-extracting kernels in each convolutional layer increases in the downstream direction.

25. The system according to claim 24;

wherein the convolutional classification model comprises one or more pooling layers, each pooling layer being adapted to receive input from a respective one of the convolutional layers; and

wherein the number of feature-extracting kernels in each convolutional layer downstream of a pooling layer is larger than the number of kernels of the convolutional layer upstream of the pooling layer.

26. The system according to any one of the preceding claims;

wherein the convolutional classification model comprises a plurality of convolutional layers and one or more pooling layers arranged as a sequence of layers where information is past from an input layer in a downstream direction to a final layer of said sequence;

wherein the final layer is a 1-dimensional layer.

38. A game system according to any one of claims 32 through 37;

comprising a recognition system having one or more of the features defined in any one of claim 1-26 or 40-41.

39. A game system according to any one of claims 32 through 38;

comprising a model creation system having one or more of the features defined in any one of claims 1 through 26.

42. A recognition system according to any one of claims 40 through 41;

comprising a recognition system having one or more of the features defined in any one of claims 1 through 26.

27. A computer implemented method for creating a recognition model for use in a system for detecting and recognizing real-world toy objects from one or more captured images, the method comprising:

receiving image data indicative of digital images depicting real-world toy objects;

receiving image annotations indicative of an object identifier of each of the depicted real-world toy objects;

storing annotated digital images in the training database, each annotated digital image depicting a real-world object and being annotated with an object identifier associated with the depicted real-world toy object;

training a convolutional classification model based on at least a subset of the annotated digital images to predict a matching object identifier when the convolutional classification model is presented with a digital image of a real-world toy object.

29. A data processing system configured to perform the steps of the method according to any one of claims 27 through 28.

30. A computer program product;

comprising program code configured, when executed by a data processing system, to cause the data processing system to perform the steps of the method according to any one of claims 27 through 28.

31. A computer program product according to claim 30, embodied as a computer-readable medium having stored thereon said program code.

28. A computer implemented method for recognizing real-world toy objects from one or more captured images, the method comprising:

receiving a captured image of a real-world toy object;

using a trained convolutional classification model to predict a matching object identifier.

32. A game system, comprising:

one or more real-world toy objects;

a database of real-world toy objects available for inclusion as virtual objects in a virtual environment, each virtual object being associated with an object identifier of a corresponding real-world toy object and with accompanying information about one or more virtual properties of the virtual object;

an image capturing device; and

a data processing system configured to;

receive a captured image of a real-world toy object in a real-world scene;

use a trained convolutional classification model to predict a matching object identifier;

add one or more virtual objects associated with the matched object identifier to a virtual environment; and

affect a gameplay experience based on the accompanying information.

33. A game system according to claim 32;

wherein the data processing system is further configured to;

use the trained convolutional classification model to create a list of candidate object identifiers and associated likelihood scores indicative of a likelihood of the respective candidate object identifiers matching the real-world toy object;

for each of the candidate object identifiers and based on the captured image, estimate a placement of the corresponding virtual object in a virtual scene corresponding to the real-world scene;

compute a respective correspondence score for each candidate object identifier based on a correlation of the estimated placement with at least the captured image;

select a candidate object identifier and a corresponding placement based at least on the computed correspondence scores.

34. A game system according to claim 32 or 33;

wherein the data processing system is further configured to;

receive depth information of the real-world scene including the real-world toy object;

for at least one matching object identifier and based on the received depth information, estimate a placement of the corresponding virtual object in a virtual scene corresponding to the real-world scene;

35. A game system according to any one of claims 32 through 34;

wherein the data processing system is further configured to;

receive depth information of the real-world scene including the real-world toy object;

use the trained convolutional classification model to create a list of candidate object identifiers and associated likelihood scores indicative of a likelihood of the candidate object identifiers matching the real-world toy object;

for each of the candidate object identifiers and based on the received depth information, estimate a placement of the corresponding virtual object in a virtual scene corresponding to the real-world scene;

compute a respective correspondence score for each candidate identifier based on a correlation of the estimated placement with at least the depth information;

select a candidate object identifier and a corresponding placement based at least on the computed correspondence scores.

36. A game system according to any one of claims 34 through 35;

wherein estimating a placement in a virtual scene is based at least on received depth information or colour information or both.

37. A game system according to any one of claims 34 through 36;

wherein the real-world toy objects are toy construction elements of a toy construction system; and

wherein estimating a placement in a virtual scene is based at least on known physical constraints imposed by the toy construction system when interconnecting toy construction elements with each other.

40. A recognition system for recognizing real-world toy objects from one or more images, wherein the recognition system comprises an image capturing device and one or more processors and wherein the recognition system is configured to:

capture at least one image of a real-world toy object;

create one or more processed versions of the captured image;

use a classification model to predict a respective matching object identifier from each of a plurality of input images;

wherein the plurality of input images are chosen from the captured image and the one or more processed images; and

to

compute an aggregated predicted object identifier from the predicted matching object identifiers.

41. The recognition system according to claim 40, wherein the recognition system is configured to:

detect a position of a toy object within the captured image;

create a plurality of different cropped images from the captured image, each cropped image comprising the detected position;

feed the plurality of cropped images through a classification model to predict a corresponding plurality of matching object identifiers; and

compute an aggregated predicted object identifier from the plurality of matching object identifiers.