Systems and methods for activities solver development in augmented reality applications

• US 9,189,892 B2
• Filed: 07/01/2011
• Issued: 11/17/2015
• Est. Priority Date: 07/01/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Palm, Leon, Adam, Hartwig
• Primary Examiner(s): Ma, Tize
• Application Number:
  US13/175,252
• Publication Number:
  US 20150170417A1
• Time in Patent Office: 1,600 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/5375 for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2003/0418 with a grid, e.g. 'Sudoku'-type games
• CPC Class Codes: A63F 2009/2425 Scanners, e.g. for scanning regular characters
• CPC Class Codes: A63F 2009/2435 using a video camera
• CPC Class Codes: A63F 2009/2486 the computer being an accessory to a board game
• CPC Class Codes: A63F 3/00094 Ticktacktoe
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Systems and methods for generating an augmented reality interface for generics activities are disclosed. The systems and methods may be directed to creating an augmented reality display for an activity performed on a surface. Given an image of the activity, an activity solver library and associated configuration information for the activity may be selected. The surface of the activity from the image may be rectified, forming a rectified image, from which activity state information may be extracted using the configuration information. The activity state information may be provided to the activity solver library to generate solution information, and elements indicating the solution information may be rendered in a perspective of the original image. By providing the configuration information associated with an activity solver library, an augmented reality interface can be generated for an activity by capturing an image of the activity.

Claims

1. A computer-implemented method comprising:

obtaining an image of a game playing surface, and of one or more game pieces on the game playing surface;

generating, by one or more computers and based on the image, a rectified representation of the game playing surface;

determining, based on the image and from among multiple games of different types, a game of a particular type being played in the image;

determining, by the one or more computers and based on a relative arrangement of the game pieces on the rectified representation of the game playing surface, a game play state of the game being played in the image;

selecting, based on the image and from among a set of game solvers that each provides solutions for a game of a different type, a particular game solver that provides solutions for the game of the particular type being played in the image;

providing data indicating the game play state to the particular game solver that provides solutions for the game being played in the image;

in response to providing data indicating the game play state to the particular game solver, receiving a solution for the game being played in the image; and

providing, for display, a representation of the solution for the game being played in the image.

2. The method of claim 1, wherein determining the game play state of the game being played in the image comprises:

after determining the game being played in the image, identifying a surface associated with the game in the image;

generating a rectified image based on the surface; and

determining the game play state based on the rectified image.

3. The method of claim 2, wherein identifying a surface associated with the game in the image comprises:

receiving configuration data associated with the game being played in the image; and

identifying the surface associated with the game based on the configuration data.

4. The method of claim 2, wherein generating a rectified image of the surface comprises:

identifying a first perspective associated with the surface in the image;

determining a second perspective associated with the rectified image of the surface, wherein the first perspective is different from the second perspective; and

generating the rectified image of the surface using the second perspective.

5. The method of claim 2, wherein identifying a surface associated with the game in the image comprises identifying the surface using quadrilateral rectification.

6. The method of claim 2, wherein determining the game play state based on the rectified image comprises:

dividing the rectified image into a plurality of image regions;

for each respective image region of the plurality of image regions, determining state data associated with the respective image region; and

determining the game play state of the game being played in the image based on the state data associated with the plurality of image regions.

7. The method of claim 2, comprising:

detecting a relative arrangement of features within the image based on one or more training images associated with the game; and

determining a mapping between the image and the rectified image.

8. The method of claim 1, wherein determining the game play state of the game being played in the image comprises determining the game play state using optical character recognition.

9. The method of claim 1, wherein determining the game play state of the game being played in the image comprises matching a portion of the image with one or more training images associated with the game.

10. A non-transitory computer-readable medium storing software having stored thereon instructions, which, when executed by one or more computers, cause the one or more computers to perform operations of:

obtaining an image of a game playing surface, and of one or more game pieces on the game playing surface;

generating, by one or more computers and based on the image, a rectified representation of the game playing surface;

determining, based on the image and from among multiple games of different types, a game of a particular type being played in the image;

determining, by the one or more computers and based on a relative arrangement of the game pieces on the rectified representation of the game playing surface, a game play state of the game being played in the image;

selecting, based on the image and from among a set of game solvers that each provides solutions for a game of a different type, a particular game solver that provides solutions for the game of the particular type being played in the image;

providing data indicating the game play state to the particular game solver that provides solutions for the game being played in the image;

in response to providing data indicating the game play state to the particular game solver, receiving a solution for the game being played in the image; and

providing, for display, a representation of the solution for the game being played in the image.

11. The computer-readable medium of claim 10, wherein determining the game play state of the game being played in the image comprises:

after determining the game being played in the image, identifying a surface associated with the game in the image;

generating a rectified image based on the surface; and

determining the game play state based on the rectified image.

12. The computer-readable medium of claim 11, wherein identifying a surface associated with the game in the image comprises:

receiving configuration data associated with the game being played in the image; and

identifying the surface associated with the game based on the configuration data.

13. The computer-readable medium of claim 11, wherein generating a rectified image of the surface comprises:

identifying a first perspective associated with the surface in the image;

determining a second perspective associated with the rectified image of the surface, wherein the first perspective is different from the second perspective; and

generating the rectified image of the surface using the second perspective.

14. The computer-readable medium of claim 11, wherein determining the game play state based on the rectified image comprises:

dividing the rectified image into a plurality of image regions;

for each respective image region of the plurality of image regions, determining state data associated with the respective image region; and

determining the game play state of the game being played in the image based on the state data associated with the plurality of image regions.

15. The computer-readable medium of claim 11, comprising:

detecting a relative arrangement of features within the image based on one or more training images associated with the game; and

determining a mapping between the image and the rectified image.

16. A system comprising:

one or more processors and one or more computer storage media storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising;

obtaining an image of a game playing surface, and of one or more game pieces on the game playing surface;

generating, by one or more computers and based on the image, a rectified representation of the game playing surface;

determining, based on the image and from among multiple games of different types, a game of a particular type being played in the image;

determining, by the one or more computers and based on a relative arrangement of the game pieces on the rectified representation of the game playing surface, a game play state of the game being played in the image;

selecting, based on the image and from among a set of game solvers that each provides solutions for a game of a different type, a particular game solver that provides solutions for the game of the particular type being played in the image;

providing data indicating the game play state to the particular game solver that provides solutions for the game being played in the image;

in response to providing data indicating the game play state to the particular game solver, receiving a solution for the game being played in the image; and

providing, for display, a representation of the solution for the game being played in the image.

17. The system of claim 16, wherein determining the game play state of the game being played in the image comprises:

after determining the game being played in the image, identifying a surface associated with the game in the image;

generating a rectified image based on the surface; and

determining the game play state based on the rectified image.

18. The system of claim 17, wherein identifying a surface associated with the game in the image comprises:

receiving configuration data associated with the game being played in the image; and

identifying the surface associated with the game based on the configuration data.

19. The system of claim 17, wherein generating a rectified image of the surface comprises:

identifying a first perspective associated with the surface in the image;

determining a second perspective associated with the rectified image of the surface, wherein the first perspective is different from the second perspective; and

generating the rectified image of the surface using the second perspective.

20. The system of claim 17, wherein determining the game play state based on the rectified image comprises:

dividing the rectified image into a plurality of image regions;

for each respective image region of the plurality of image regions, determining state data associated with the respective image region; and

determining the game play state of the game being played in the image based on the state data associated with the plurality of image regions.