Image processing device, image processing method, and program

• US 10,515,488 B2
• Filed: 01/31/2019
• Issued: 12/24/2019
• Est. Priority Date: 02/02/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Suto, Yasuhiro, Fukuchi, Masaki, Ooi, Kenichirou, Guo, Jingjing, Matsuda, Kouichi
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US16/263,524
• Publication Number:
  US 20190164348A1
• Time in Patent Office: 327 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

There is provided an image processing device including: a data storage unit storing feature data indicating a feature of appearance of one or more physical objects; an environment map building unit for building an environment map based on an input image obtained by imaging a real space and the feature data, the environment map representing a position of a physical object present in the real space; a control unit for acquiring procedure data for a set of procedures of operation to be performed in the real space, the procedure data defining a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed; and a superimposing unit for generating an output image by superimposing the direction for each procedure at a position in the input image determined based on the environment map and the position information, using the procedure data.

Claims

1. An image processing device comprising:

a processor; and

a non-transitory computer-readable medium encoded with instructions which, when executed by the processor, cause the image processing device to;

acquire procedure data for a set of procedures of operation to be performed in a real space and which associates a procedure of the set of procedures with a physical object and a progress condition;

identify a first physical object present in the real space by using a camera to capture an image of the real space;

control, based on the acquired procedure data and the identified first physical object, a display of first procedure image data by superimposing a first procedure of the set of procedures at a position associated with the first physical object in the real space;

detect a first progress condition associated with the first procedure when the first procedure image data is displayed;

identify a second physical object present in the real space by using the camera; and

control, based on the acquired procedure data and the identified second physical object and when the first progress condition is detected, a display of second procedure image data by superimposing a second procedure of the set of procedures at a position associated with the second physical object in the real space.

2. The image processing device according to claim 1,

wherein the procedure data defines a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed or a condition for progressing displaying of each direction.

3. The image processing device according to claim 2,

wherein the condition for progressing displaying of each direction includes a condition according to a position or a posture of the physical object.

4. The image processing device according to claim 2,

wherein the displaying of the direction for each procedure included in the set of procedures is controlled according to the procedure data.

5. The image processing device according to claim 2,

wherein the position information is obtained based on one or more feature points in the captured image of the real space.

6. The image processing device according to claim 1,

wherein the image processing device further comprises a detector configured to dynamically detect a position in the real space of the image processing device based on the captured image of the real space and feature data that indicates features of appearance of physical objects in the real space, and

wherein the positions at which the first procedure and the second procedure are superimposed is determined according to the position in the real space of the image processing device.

7. The image processing device according to claim 1,

wherein the progress condition comprises at least one of a state of the physical object, passage of a given time, an external event, or a combination thereof.

8. The image processing device according to claim 1,

wherein the first procedure image data is displayed prior to the second procedure image data being displayed.

9. The image processing device according to claim 1,

wherein the physical object is identified by analyzing one or more feature points in the captured image of the real space.

10. The image processing device according to claim 9,

wherein the non-transitory computer-readable medium is further configured to store feature data that indicates features of appearance of physical objects in the real space, and

wherein the first physical object and the second physical object are identified based on correspondence between the one or more feature points and the stored feature data.

11. The image processing device according to claim 1,

wherein the set of procedures comprises more than two procedures.

12. The image processing device according to claim 11, wherein the image processing device is further caused to:

detect a second progress condition associated with the second procedure when the second procedure image data is displayed;

identify a third physical object present in the real space by using the camera; and

control, based on the acquired procedure data and the identified third physical object and when the second progress condition is detected, a display of third procedure image data by superimposing a third procedure of the set of procedures at a position associated with the third physical object in the real space,

wherein the first procedure image data is displayed prior to the second procedure image data being displayed, and the second procedure image data is displayed prior to the third procedure image data being displayed.

13. The image processing device according to claim 1,

wherein the set of procedures comprises text based information.

14. The image processing device according to claim 1,

wherein the camera comprises a CMOS image sensor.

15. An image processing method comprising:

acquiring procedure data for a set of procedures of operation to be performed in a real space and which associates a procedure of the set of procedures with a physical object and a progress condition;

identifying a first physical object present in the real space by using a camera to capture an image of the real space;

controlling, based on the acquired procedure data and the identified first physical object, a display of first procedure image data by superimposing a first procedure of the set of procedures at a position associated with the first physical object in the real space;

detecting a first progress condition associated with the first procedure when the first procedure image data is displayed;

identifying a second physical object present in the real space by using the camera; and

controlling, based on the acquired procedure data and the identified second physical object and when the first progress condition is detected, a display of second procedure image data by superimposing a second procedure of the set of procedures at a position associated with the second physical object in the real space.

16. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

acquiring procedure data for a set of procedures of operation to be performed in a real space and which associates a procedure of the set of procedures with a physical object and a progress condition;

identifying a first physical object present in the real space by using a camera to capture an image of the real space;

controlling, based on the acquired procedure data and the identified first physical object, a display of first procedure image data by superimposing a first procedure of the set of procedures at a position associated with the first physical object in the real space;

detecting a first progress condition associated with the first procedure when the first procedure image data is displayed;

identifying a second physical object present in the real space by using the camera; and

controlling, based on the acquired procedure data and the identified second physical object and when the first progress condition is detected, a display of second procedure image data by superimposing a second procedure of the set of procedures at a position associated with the second physical object in the real space.