Image processing device and image processing method

• US 9,741,171 B2
• Filed: 06/25/2015
• Issued: 08/22/2017
• Est. Priority Date: 08/15/2014
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Hara, Nobuyuki
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Wu, Yanna
• Application Number:
  US14/750,738
• Publication Number:
  US 20160049009A1
• Time in Patent Office: 789 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

An image processing device includes memory; and a processor configured to execute a plurality of instructions stored in the memory, the instructions comprising: recognizing a target object recognized from a first image, which is a captured image, including the target object in a real world; controlling a second image, which is an augmented image, including information of the target object from the first image, and a third image which is an augmented image of the second image and to be formed so as to inscribe an outer surrounding the second image and covers a center of visual field of a user relative to the second image; and displaying, in a state where the user directly visually recognizes the target object in the real world, the second image and the third image such that the second image and the third image are caused to correspond to a position.

Claims

1. An image processing device comprising:

a memory; and

a processor configured to execute a plurality of instructions stored in the memory, the instructions comprising;

recognizing a target object recognized from a first image, which is a captured image, including the target object in a real world;

controlling a second image, which is an augmented image, including information of the target object from the first image, the second image being out of range of a focus distance of the first image;

controlling a third image, which is an augmented image of the second image, to be formed so as to inscribe an outer surrounding of the second image and cover a center of visual field of a user relative to the second image, the third image being located on a boundary between the target object and the second image which is out of range of the focus distance of the first image; and

displaying, in a state where the user directly visually recognizes the target object in the real world, the second image and the third image such that the second image and the third image are caused to correspond to a position of the target object in the real world,

wherein, in the displaying, the second image which is out of range of the focus distance of the first image is displayed with the third image to be at the focus distance of the first image.

2. The device according to claim 1,

wherein the controlling controls an outer edge of the third image based on a focal distance of the user relative to the second image and an angle relative to a vertical line of a fovea of the user which is defined based on the center of the visual field of the user.

3. The device according to claim 1,

wherein the controlling controls the third image in such a manner that a brightness of the third image is greater than a brightness of the target object in the real world.

4. The device according to claim 1,

wherein the controlling defines

the outer edge of the second image based on a length of an outer edge of the target object in the first image, and

an outer edge of the third image by extending the length based on the focal distance and an tangent of the angle in a direction in which the third image includes the second image from the outer edge of the second image.

5. The device according to claim 1,

wherein the recognizing

accesses to a storage storing a first feature that corresponds to the target object, and

recognizes the target object included in the first image by extracting a second feature from the first image and associating the first feature and the second feature to each other.

6. The device according to claim 1,

wherein the first image further includes an operation site of the user,

wherein the recognizing recognizes the operation site from the first image,

wherein the instructions further comprising:

calculating a motion vector of the operation site from multiple ones of the first image, and

wherein the controlling controls display positions of the second image and the third image based on the position of the target object and the motion vector.

7. The device according to claim 6,

wherein, the controlling controls the display position of the second image so as to be superimposed on the target object, when the motion vector is less than a first threshold.

8. The device according to claim 7,

wherein the controlling controls, when the motion vector is more than the first threshold, the display positions of the second image and the third image to move within a predetermined direction from the motion vector, a distance between a reference position of the target object and a reference position of the second image is at a smallest distance.

9. The device according to claim 7,

wherein the controlling stops displaying of the second image and the third image, when the motion vector is equal to the first threshold or more.

10. An image processing method comprising:

recognizing a target object recognized from a first image, which is a captured image, including the target object in a real world;

controlling, by a computer processor, a second image, which is an augmented image, including information of the target object from the first image, the second image being out of range of a focus distance of the first image;

controlling a third image, which is an augmented image of the second image, to be formed so as to inscribe an outer surrounding of the second image and cover a center of visual field of a user relative to the second image, the third image being located on a boundary between the target object and the second image which is out of range of the focus distance of the first image; and

displaying, in a state where the user directly visually recognizes the target object in the real world, the second image and the third image such that the second image and the third image are caused to correspond to a position of the target object in the real world,

wherein, in the displaying, the second image which is out of range of the focus distance of the first image is displayed with the third image to be at the focus distance of the first image.

11. The method according to claim 10,

wherein the controlling controls an outer edge of the third image based on a focal distance of the user relative to the second image and an angle relative to a vertical line of a fovea of the user which is defined based on the center of the visual field of the user.

12. The method according to claim 10,

wherein the controlling controls the third image in such a manner that a brightness of the third image is greater than a brightness of the target object in the real world and/or the third image other than with a texture feature amount.

13. The method according to claim 10,

wherein the controlling defines

the outer edge of the second image based on a length of an outer edge of the target object in the first image, and

an outer edge of the third image by extending the length based on the focal distance and an tangent of the angle in a direction in which the third image includes the second image from the outer edge of the second image.

14. The method according to claim 10,

wherein the recognizing

accesses to a storage storing a first feature that corresponds to the target object, and

recognizes the target object included in the first image by extracting a second feature from the first image and associating the first feature and the second feature to each other.

15. The method according to claim 10,

wherein the first image further includes an operation site of the user,

wherein the recognizing recognizes the operation site from the first image,

wherein the instructions further comprising:

calculating a motion vector of the operation site from multiple ones of the first image, and

wherein the controlling controls display positions of the second image and the third image based on the position of the target object and the motion vector.

16. The method according to claim 15,

wherein, the controlling controls the display position of the second image so as to be superimposed on the target object, when the motion vector is less than a first threshold.

17. The method according to claim 15,

wherein the controlling controls, when the motion vector is more than the first threshold, the display positions of the second image and the third image in such a manner that a position in which the target object is located in the motion vector, the second image, and the third image, and a position in which a distance between a reference position of the target object and a reference position of the second image is at a smallest distance.

18. The method according to claim 15,

wherein the controlling stops displaying of the second image and the third image, when the motion vector is equal to the first threshold or more.

19. A non-transitory computer-readable medium that store an image processing program for causing a computer to execute a process comprising:

recognizing a target object recognized from a first image, which is a captured image, including the target object in a real world;

controlling a second image, which is an augmented image, including information of the target object from the first image, the second image being out of range of a focus distance of the first image;

controlling a third image, which is an augmented image of the second image, to be formed so as to inscribe an outer surrounding of the second image and cover a center of visual field of a user relative to the second image, the third image being located on a boundary between the target object and the second image which is out of range of the focus distance of the first image; and

displaying, in a state where the user directly visually recognizes the target object in the real world, the second image and the third image such that the second image and the third image are caused to correspond to a position of the target object in the real world,

wherein, in the displaying, the second image which is out of range of the focus distance of the first image is displayed with the third image to be at the focus distance of the first image.