Image processing device, image processing method, and program

• US 10,223,837 B2
• Filed: 07/03/2018
• Issued: 03/05/2019
• Est. Priority Date: 02/02/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Suto, Yasuhiro, Fukuchi, Masaki, Ooi, Kenichirou, Guo, Jingjing, Matsuda, Kouichi
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US16/026,763
• Publication Number:
  US 20180315249A1
• Time in Patent Office: 245 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

There is provided an image processing device including: a data storage unit storing feature data indicating a feature of appearance of one or more physical objects, an environment map building unit for building an environment map based on an input image obtained by imaging a real space and the feature data, the environment map representing a position of a physical object present in the real space; a control unit for acquiring procedure data for a set of procedures of operation to be performed in the real space, the procedure data defining a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed; and a superimposing unit for generating an output image by superimposing the direction for each procedure at a position in the input image determined based on the environment map and the position information, using the procedure data.

Claims

1. An imaging system comprising:

an imaging device configured to generate image data associated with a real space; and

a processor configured to

obtain position information of a physical object in the image data,

correlate the position information with a first proposed action stored in a memory, the first proposed action comprising one procedure of a set of procedures,

generate a first output image data by superimposing the one procedure of the set of procedures at a position associated with the physical object in the image data,

output the first output image data to a display unit,

correlate the position information with a second proposed action based on a progress condition, the second proposed action comprising another procedure of the set of procedures,

generate a second output image data by superimposing the another procedure of the set of procedures at a position associated with the physical object in the image data, and

output the second output image data to the display unit.

2. The imaging system according to claim 1,

wherein the progress condition comprises at least one of a state of the physical object, passage of a given time, an external event, or a combination thereof.

3. The imaging system according to claim 1,

wherein the first output image data is output prior to the second output image data being output.

4. The imaging system according to claim 1,

wherein the processor is further configured to identify the physical object by analyzing one or more feature points in the image data.

5. The imaging system according to claim 4,

wherein the imaging device is further configured to generate a plurality of image data, and

wherein the processor is further configured to track the one or more feature points between the plurality of image data and identify the physical object at each of the plurality of image data.

6. The imaging system according to claim 4,

wherein the processor is further configured to correlate the position information with the first proposed action based on prediction conditions,

wherein one of the prediction conditions is that a position of the one or more feature points does not change, and

wherein another one of the prediction conditions is that motion of the imaging device does not change.

7. The imaging system according to claim 4,

wherein the processor is further configured to correlate the position information with the second proposed action based on prediction conditions,

wherein one of the prediction conditions is that a position of the one or more feature points does not change, and

wherein another one of the prediction conditions is that motion of the imaging device does not change.

8. The imaging system according to claim 4,

wherein the memory is further configured to store feature data of the physical object, and

wherein the processor is further configured to identify the physical object based on a correspondence between the one or more feature points and the stored feature data.

9. The imaging system according to claim 4,

wherein the processor is further configured to obtain the position information based on the one or more feature points.

10. The imaging system according to claim 4,

wherein the processor is further configured to calculate a distance between the imaging device and position in the real space of the one or more feature points,

wherein the processor is further configured to correlate the position information with the first proposed action based on the calculated distance.

11. The imaging system according to claim 4,

wherein the processor is further configured to calculate a distance between the imaging device and position in the real space of the one or more feature points,

wherein the processor is further configured to correlate the position information with the second proposed action based on the calculated distance.

12. The imaging system according to claim 1,

wherein the set of procedures comprises more than two procedures.

13. The imaging system according to claim 12, wherein the processor is further configured to

correlate the position information with a third proposed action based on the progress condition, the third proposed action comprising a third procedure of the set of procedures,

generate a third output image data by superimposing the third procedure of the set of procedures at a position associated with the physical object in the image data, and

output the third output image data to the display unit,

wherein the first output image data is output prior to the second output image data being output, and the second output image data is output prior to the third output image data being output.

14. The imaging system according to claim 1,

wherein the set of procedures comprises text based information.

15. The imaging system according to claim 1,

wherein the imaging device comprises a CMOS image sensor.

16. An imaging method comprising:

generating image data associated with a real space;

obtaining position information of a physical object in the image data;

correlating the position information with a first proposed action stored in a memory, the first proposed action comprising one procedure of a set of procedures;

generating a first output image data by superimposing the one procedure of the set of procedures at a position associated with the physical object in the image data;

outputting the first output image data to a display unit;

correlating the position information with a second proposed action based on a progress condition, the second proposed action comprising another procedure of the set of procedures;

generating a second output image data by superimposing the another procedure of the set of procedures at a position associated with the physical object in the image data; and

outputting the second output image data to the display unit.

17. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

generating image data associated with a real space;

obtaining position information of a physical object in the image data;

correlating the position information with a first proposed action stored in a memory, the first proposed action comprising one procedure of a set of procedures;

generating a first output image data by superimposing the one procedure of the set of procedures at a position associated with the physical object in the image data;

outputting the first output image data to a display unit;

correlating the position information with a second proposed action based on a progress condition, the second proposed action comprising another procedure of the set of procedures;

generating a second output image data by superimposing the another procedure of the set of procedures at a position associated with the physical object in the image data; and

outputting the second output image data to the display unit.