Image processing apparatus displaying image of virtual object and method of displaying the same

• US 10,453,235 B2
• Filed: 11/22/2017
• Issued: 10/22/2019
• Est. Priority Date: 12/01/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Hayashi, Masayuki
• Primary Examiner(s): Thompson, James A
• Application Number:
  US15/821,442
• Publication Number:
  US 20180158222A1
• Time in Patent Office: 699 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

An apparatus includes: a detection unit that detects a position and an orientation of a real object with respect to a viewpoint of a user; a determination unit that determines whether the user is holding the real object; and a display control unit that displays a virtual object corresponding to the real object based on the position and the orientation of the real object. When the user is not holding the real object, the display control unit displays the virtual object in a first form in which a shape of the real object can be visually recognized. When the user is holding the real object, the display control unit displays the virtual object in a second form in which the shape of the real object on which the virtual object is superimposed cannot be visually recognized.

Claims

1. An apparatus comprising:

at least one processor; and

a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, perform operations as;

a detection unit configured to detect a position and an orientation of a real object with respect to a viewpoint of a user;

a determination unit configured to determine whether a part of the body of the user and the real object are in contact with each other; and

a display control unit configured to display a virtual object corresponding to the real object based on the position and the orientation of the real object,

wherein when it is determined that the part of the body of the user and the real object are not in contact with each other, the display control unit displays the virtual object in a first form in which a shape of the real object can be visually recognized, and

when it is determined that the part of the body of the user and the real object are in contact with each other, the display control unit displays the virtual object in a second form in which the shape of the real object on which the virtual object is superimposed cannot be visually recognized.

2. The apparatus according to claim 1, wherein the first form is one of a form in which the virtual object is displayed translucently, a form in which an outline of the virtual object is displayed and a form in which a portion in which the virtual object and the real object overlap each other is displayed in a specified color or pattern.

3. The apparatus according to claim 1, wherein the second form is a form in which the virtual object is displayed opaquely.

4. The apparatus according to claim 1, further comprising a calculation unit configured to calculate a distance between the real object and the part of the body of the user,

wherein when the distance is less than a threshold, the determination unit determines that the real object and the part of the body of the user are in contact with each other.

5. The apparatus according to claim 1, wherein the determination unit determines whether the part of the body of the user and the real object are in contact with each other based on whether the real object is moving.

6. The apparatus according to claim 5, wherein when the position and the orientation of the real object are changing or when a moving speed of the real object is equal to or greater than a threshold, the determination unit determines that the real object is moving.

7. The apparatus according to claim 1, wherein the display control unit switches a display form of the virtual object when a predetermined time period elapses after a determination result made by the determination unit changes.

8. An image processing method comprising:

detecting a position and an orientation of a real object with respect to a viewpoint of a user;

determining whether a part of the body of the user and the real object are in contact with each other; and

displaying a virtual object corresponding to the real object based on the position and the orientation of the real object,

wherein displaying the virtual object includes;

when it is determined that the part of the body of the user and the real object are not in contact with each other, displaying the virtual object in a first form in which a shape of the real object can be visually recognized; and

when it is determined that the part of the body of the user and the real object are in contact with each other, displaying the virtual object in a second form in which the shape of the real object on which the virtual object is superimposed cannot be visually recognized.

9. The method according to claim 8, wherein the first form is one of a form in which the virtual object is displayed translucently, a form in which an outline of the virtual object is displayed and a form in which a portion in which the virtual object and the real object overlap each other is displayed in a specified color or pattern.

10. The method according to claim 8, wherein the second form is a form in which the virtual object is displayed opaquely.

11. The method according to claim 8, further comprising calculating a distance between the real object and the part of the body of the user,

wherein when the distance is less than a threshold, the determining determines that the real object and the part of the body of the user are in contact with each other.

12. The method according to claim 8, wherein the determining determines whether the part of the body of the user and the real object are in contact with each other based on whether the real object is moving.

13. A non-transitory computer-readable storage medium storing a program for causing a computer to function as:

a detection unit configured to detect a position and an orientation of a real object with respect to a viewpoint of a user;

a determination unit configured to determine whether a part of the body of the user and the real object are in contact with each other; and

a display control unit configured to display a virtual object corresponding to the real object based on the position and the orientation of the real object,

wherein when the determination unit determines that the part of the body of the user and the real object are not in contact with each other, the display control unit displays the virtual object in a first form in which a shape of the real object can be visually recognized, and

when the determination unit determines that the part of the body of the user and the real object are in contact with each other, the display control unit displays the virtual object in a second form in which the shape of the real object on which the virtual object is superimposed cannot be visually recognized.

14. The non-transitory computer-readable storage medium according to claim 13, wherein the first form is one of a form in which the virtual object is displayed translucently, a form in which an outline of the virtual object is displayed and a form in which a portion in which the virtual object and the real object overlap each other is displayed in a specified color or pattern.

15. The non-transitory computer-readable storage medium according to claim 13, wherein the second form is a form in which the virtual object is displayed opaquely.

16. The non-transitory computer-readable storage medium according to claim 13, wherein the determination unit determines whether the part of the body of the user and the real object are in contact with each other based on whether the real object is moving.

17. The non-transitory computer-readable storage medium according to claim 13, wherein the display control unit switches a display form of the virtual object when a predetermined time period elapses after a determination result made by the determination unit changes.