Image processing apparatus, image processing method, and medium to display augmented reality objects

• US 9,355,496 B2
• Filed: 08/15/2012
• Issued: 05/31/2016
• Est. Priority Date: 10/03/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kasahara, Shunichi
• Primary Examiner(s): Chen, Wenpeng
• Application Number:
  US14/235,830
• Publication Number:
  US 20140193038A1
• Time in Patent Office: 1,385 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments

Abstract

An information processing system acquires an image captured by an image pickup unit and one or more templates where each template includes one or more fields. The acquired image is then compared to the templates and a result based on the comparison is generated such that the result indicates whether recognition of the fields of the templates was successful.

Claims

1. An information processing system comprising:

one or more processing circuits configured to;

acquire an image captured by an image pickup unit;

acquire one or a plurality of templates each including one or a plurality of fields;

perform a comparison of the image to the one or plurality of templates; and

output augmented reality content related to a result based on the comparison, the result indicating whether recognition of each of the one or plurality of fields of the one or plurality of templates was successful,

wherein the result that is output based on the comparison includes controlling a display to display one or a plurality of fields for which recognition was successful differently from one or a plurality of fields for which recognition was not successful.

2. The information processing system of claim 1, wherein the image includes an object and the one or more processing circuits are configured to recognize a structure of the object.

3. The information processing system of claim 2, wherein the one or more processing circuits are configured to recognize a structure of the object by detecting one or more fields included in the object.

4. The information processing system of claim 1, wherein the one or more processing circuits are configured to extract edges from the image.

5. The information processing system of claim 4, wherein the one or more processing circuits are configured to compare the image to the one or plurality of templates by determining a degree of matching between the edges extracted from the image and frames of fields included in the one or plurality of templates.

6. The information processing system of claim 4, wherein the one or more processing circuits are configured to compare the image to the one or plurality of templates by determining a degree of matching between the edges extracted from the image and frames of fields included in the one or plurality of templates while changing a position of the one or plurality of templates.

7. The information processing system of claim 4, wherein the one or more processing circuits are configured to compare the image to the one or plurality of templates by determining a degree of matching between the edges extracted from the image and frames of fields included in the one or plurality of templates while changing a posture of the one or plurality of templates.

8. The information processing system of claim 1, wherein the one or more processing circuits are configured to recognize information associated with at least one field included in a matching template from the image.

9. The information processing system of claim 8, wherein the at least one field is a field selected based on recognizing the matching template.

10. The information processing system of claim 8, wherein the one or more processing circuits are configured to control a display to superimpose the recognized information on the image.

11. The information processing system of claim 1, wherein the one or more processing circuits are configured to recognize information associated with at least one field included in a matching template from the image by performing optical character recognition (OCR) on a portion of the image associated with the at least one field.

12. The information processing system of claim 1, wherein the one or more processing circuits are configured to recognize information associated with at least one field included in a matching template from the image by matching portions of the image associated with the at least one field with stored image information.

13. The information processing system of claim 1, wherein the image associated with at least one field is an image of a face, and the image of the face is matched with stored face image information.

14. The information processing system of claim 1, wherein the one or more processing circuits are configured to generate assistance information for guiding a user of the image pickup unit to adjust the image pickup unit based on a result of the comparison.

15. The information processing system of claim 14, wherein the assistance information instructs the user to change an orientation of the image pickup unit.

16. The information processing system of claim 14, wherein the assistance information instructs the user to change at least one of a position and setting of the image pickup unit to change a size of the image captured by the image pickup unit.

17. The information processing system of claim 1, wherein the captured image and the one or plurality of templates correspond to a web page.

18. The information processing system of claim 17, wherein the one or more processing circuits are configured to recognize information associated with at least one field included in a matching template corresponding to the web page from the image.

19. The information processing system of claim 18, wherein the one or more processing circuits are configured to control a process of accessing the web page based on the recognized information.

20. The information processing system of claim 18, wherein the recognized information corresponds to login information for the web page.

21. The information processing system of claim 1, wherein the information processing system is a mobile phone and the one or more processing units are a processor of the mobile phone.

22. The information processing system of claim 1, wherein the information processing system includes a client and a server.

23. The information processing system of claim 22, wherein the client includes

the image pickup unit;

and

a first communication interface configured to send the image to the server; and

the server includes

a second communication interface configured to receive the image from the client;

at least one of the one or more processing circuits are configured to acquire the one or a plurality of templates from a memory, compare the image to the one or plurality of templates, and control the second communication interface to output the result based on the comparison to the client.

24. An information processing method performed by an information processing system, the method comprising:

acquiring an image captured by an image pickup unit;

acquiring one or a plurality of templates each including one or a plurality of fields;

performing a comparison of the image to the one or plurality of templates; and

outputting augmented reality (AR) content related to a result based on the comparison, the result indicating whether recognition of each of the one or plurality of fields of the one or plurality of templates was successful,

wherein the result that is output based on the comparison includes controlling a display to display one or a plurality of fields for which recognition was successful differently from one or a plurality of fields for which recognition was not successful.

25. A non-transitory computer-readable medium including a set of computer-executable instructions, which when executed by an information processing system, cause the information processing system to perform a method, the method comprising:

acquiring an image captured by an image pickup unit;

acquiring one or a plurality of templates each including one or a plurality of fields;

performing a comparison of the image to the one or plurality of templates; and

outputting augmented reality content related to a result based on the comparison, the result indicating whether recognition of each of the one or plurality of fields of the one or plurality of templates was successful,

wherein the result that is output based on the comparison includes controlling a display to display one or a plurality of fields for which recognition was successful differently from one or a plurality of fields for which recognition was not successful.