Image processing device, image processing method, and program

• US 10,037,628 B2
• Filed: 09/25/2017
• Issued: 07/31/2018
• Est. Priority Date: 02/02/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Suto, Yasuhiro, Fukuchi, Masaki, Ooi, Kenichirou, Guo, Jingjing, Matsuda, Kouichi
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US15/713,801
• Publication Number:
  US 20180012415A1
• Time in Patent Office: 309 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

There is provided an image processing device including: a data storage unit storing feature data indicating a feature of appearance of one or more physical objects; an environment map building unit for building an environment map based on an input image obtained by imaging a real space and the feature data, the environment map representing a position of a physical object present in the real space; a control unit for acquiring procedure data for a set of procedures of operation to be performed in the real space, the procedure data defining a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed; and a superimposing unit for generating an output image by superimposing the direction for each procedure at a position in the input image determined based on the environment map and the position information, using the procedure data.

Claims

1. An apparatus comprising:

one or more processors configured to

control to display first information associated with a first procedure ID,

determine whether a first condition is satisfied based upon first conditional data associated with the first procedure ID and a captured image,

control to display second information associated with a second procedure ID upon determining that the first condition is satisfied,

determine whether a second condition is satisfied based upon second conditional data associated with the second procedure ID and a captured image, and

control to display third information associated with a third procedure ID upon determining that the second condition is satisfied,

wherein the first information, the second information, and the third information are defined to be displayed in order.

2. The apparatus according to claim 1, wherein

the first information is displayed at a location based upon first position information associated with the first procedure ID,

the second information is displayed at a location based upon second position information associated with the second procedure ID, and

third information is displayed at a location based upon third position information associated with the third procedure ID.

3. The apparatus according to claim 2, wherein the first position information, the second position information, and the third position information each correspond to location of one of more physical objects.

4. The apparatus according to claim 1,

wherein a real object in a real world is indicated by one or more selected from the group consisting of the first position information, the second position information, and the third position information, and

wherein the one or more processors is further configured to control to display, on or in vicinity of a location corresponding to the real object, one or more selected from the group consisting of the first information, the second information, and the third information.

5. The apparatus according to claim 4, wherein the one or more selected from the group consisting of the first information, the second information, and the third information is displayed to be visually superimposed upon the real object or in the vicinity of the location corresponding to the real object.

6. The apparatus according to claim 1, wherein one or more selected from the group consisting of the first information, the second information, and the third information includes a text description to be displayed.

7. The apparatus according to claim 1, wherein the one or more processors is further configured to

generate a 3-D map based upon one or more captured images, and

identify a position and attitude of the apparatus based upon the one or more captured images.

8. The apparatus according to claim 1, wherein the first conditional data represents a progress condition of a procedure associated with the first procedure ID, and the second conditional data represents a progress condition of a procedure associated with the second procedure ID.

9. The apparatus according to claim 1, wherein the captured image based upon which the determination made as to whether the first condition is satisfied is different than the captured image based upon which the determination made as to whether the second condition is satisfied.

10. The apparatus according to claim 1, wherein the first information, the second information, and the third information are defined to be displayed in a sequential order representing chronological steps a user performs.

12. The method according to claim 1, wherein

the first information is displayed at a location based upon first position information associated with the first procedure ID,

the second information is displayed at a location based upon second position information associated with the second procedure ID, and

third information is displayed at a location based upon third position information associated with the third procedure ID.

13. The method according to claim 12, wherein the first position information, the second position information, and the third position information each correspond to location of one of more physical objects.

19. The method according to claim 1, wherein the captured image based upon which the determination made as to whether the first condition is satisfied is different than the captured image based upon which the determination made as to whether the second condition is satisfied.

11. A method comprising:

displaying first information associated with a first procedure ID;

determining whether a first condition is satisfied based upon first conditional data associated with the first procedure ID and a captured image;

displaying second information associated with a second procedure ID upon determining that the first condition is satisfied;

determining whether a second condition is satisfied based upon second conditional data associated with the second procedure ID and a captured image; and

displaying third information associated with a third procedure ID upon determining that the second condition is satisfied,

wherein the first information, the second information and the third information are defined to be displayed in order.

14. The method according to claim 11, further comprising:

displaying, on or in vicinity of a location corresponding to the real object, one or more selected from the group consisting of the first information, the second information, and the third in formation,

wherein a real object in a real world is indicated by one or more selected from the group consisting of the first position information, the second position information, and the third position information.

15. The method according to claim 14, wherein the one or more selected from the group consisting of the first information, the second information, and the third information is displayed to be visually superimposed upon the real object or in the vicinity of the location corresponding to the real object.

16. The method according to claim 11, wherein one or more selected from the group consisting of the first information, the second information, and the third information includes a text description to be displayed.

17. The method according to claim 11, further comprising:

generating a 3-D map based upon one or more captured images; and

identifying a position and attitude of the apparatus based upon the one or more captured images.

18. The method according to claim 11, wherein the first conditional data represents a progress condition of a procedure associated with the first procedure ID, and the second conditional data represents a progress condition of a procedure associated with the second procedure ID.

20. The method according to claim 11, wherein the first information, the second information and the third information are defined to be displayed in a sequential order representing chronological steps a user performs.

21. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

displaying first information associated with a first procedure ID;

determining whether a first condition is satisfied based upon first conditional data associated with the first procedure ID and a captured image;

displaying second information associated with a second procedure ID upon determining that the first condition is satisfied;

determining whether a second condition is satisfied based upon second conditional data associated with the second procedure ID and a captured image; and

displaying third information associated with a third procedure ID upon determining that the second condition is satisfied,

wherein the first information, the second information and the third information are defined to be displayed in order.