Floating image interaction device and its program

• US 8,985,774 B2
• Filed: 03/31/2008
• Issued: 03/24/2015
• Est. Priority Date: 03/30/2007
• Status: Active Grant

• Current Assignee: National Institute of Information and Communications Technology
• Original Assignee: National Institute of Information and Communications Technology
• Inventors: Maekawa, Satoshi
• Primary Examiner(s): Chowdhury, Sultan
• Assistant Examiner(s): Owens, Danell L
• Application Number:
  US12/593,977
• Publication Number:
  US 20100110384A1
• Time in Patent Office: 2,549 Days
• Field of Search: 353/7, 353/10, 359/727, 359/627, 356/435, 356/432
• US Class Current: 353/10
• CPC Class Codes: G02B 30/56 by projecting aerial or floating images
• CPC Class Codes: G02B 5/124 plural reflecting elements forming part of a unitary plate or sheet
• CPC Class Codes: G02B 5/136 plural reflecting elements forming part of a unitary body G02B5/124 takes precedence
• CPC Class Codes: G03B 15/00 Special procedures for taking photographs; Apparatus therefor
• CPC Class Codes: G03B 35/18 by simultaneous viewing
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A floating image interaction device includes an imaging optical system projecting a floating real image of an object and an user object detecting means for obtaining information about the user accessing the floating real image; means to realize interactive response by the real image or by haptic feedback to actions by the user; and related computer programs.

Claims

1. A floating image interaction device, comprising:

distortion-free imaging optical system including a surface at which paths of light rays are bent by means of mirror surface reflection; and

a user object detecting means,

wherein for an object to be projected placed to one side of said surface, said imaging optical system forms a real image of said object on the opposite side of said surface with front and back being reversed,

the imaging optical system is for forming a two-dimension real image of a two-dimension object to be projected, and a three-dimensional real image of a three-dimension object to be projected;

the real image can be observed from oblique direction of the surface;

and said user object detecting means capable of detecting the position, or the position and shape of user objects approaching said real image.

2. A floating image interaction device according to claim 1, further comprising:

a real image modifying means wherein said position, or position and shape of said objects as detected by said user object detecting means is used to modify said real image.

3. A floating image interaction device according to claim 1, further comprising an object actuator means, wherein said position, or position and shape of said user objects as detected by said user object detecting means is used to exert force on said user object.

4. A floating image interaction device according to claim 3, wherein said object actuator means mechanically exert force.

5. A floating image interaction device according to claim 3, wherein said object actuator means generates an air flow.

6. A floating image interaction device according to claim 1, wherein said imaging optical system is realized by an imaging optical device comprising a plurality of unit optical elements, said unit optical elements comprising of at least one specular surface reflecting light and arranged perpendicularly or nearly perpendicularly to an optical device plane corresponding to said surface, whereas for an object to be projected placed to one side of said optical device plane, its real image will be formed on the opposite side of said optical device plane by said plurality of unit optical elements.

7. A floating image interaction device according to claim 6, wherein said imaging optical device comprises a plurality of unit optical elements realized by dihedral corner reflectors consisting of pairs of mutually perpendicular specular surfaces, thereby functioning as a dihedral corner reflector array.

8. A floating image interaction device according to claim 1, wherein said imaging optical system comprises a half-mirror having a half-mirror surface corresponding to said surface, and a retroreflector array comprising of a set of unit retroreflectors, whereas said retroreflectors are located so as to recursively reflect light reflected by, or transmitted through, said half-mirror.

9. A floating image interaction device according to claim 1, wherein said user object detecting means is located in a fixed position with respect to said surface.

10. A floating image interaction device according to claim 1, wherein said user object detecting means comprises at least one camera, and an image processing device capable of determining the position, or the position and shape of said object from images recorded by said camera.

11. A floating image interaction device according to claim 10, wherein said camera is located on the same side of said surface as said object to be projected.

12. A floating image interaction device according to claim 11, wherein said camera is located in such a position so that it can observe said object to be projected directly through said surface.

13. A floating image interaction device according to claim 11, wherein a half-mirror is placed between said surface and said object to be projected, and furthermore said camera is located in such a position so that it can observe the real image of said object to be projected as formed by said imaging optical system, and reflected by said half-mirror.

14. A floating image interaction device according to claim 6, wherein said user object detecting means comprises at least one camera, and an image processing device capable of determining the position, or the position and shape of said user object from images recorded by said camera, and said camera is located in such a position so that it can observe light emitted from said user object and reflected only once by said unit optical elements.

15. A floating image interaction device according to claim 10, wherein said camera is located in such a position so that it can observe directly said user object without the light passing through said surface.

16. A floating image interaction device according to claim 10, wherein said camera is located on the same side of said surface as said real image.

17. A floating image interaction device according to claim 10, wherein at least one of said cameras is a distance detecting camera.

18. A non-transitory computer readable medium storing computer program to be used by the floating image interaction device according to claim 1, comprising:

a user object detection procedure; and

a user object information generating procedure,

wherein said user object detection procedure detects an information of the position, or position and shape of said user object by using said user object detecting means, and determines the position or the position and shape of said user object based on said detected information,

and wherein said user object information generating procedure receives said information from said user object detection procedure about the position, or the position and shape of said user object, and generates information for an appropriate interaction program in order to control said object to be

projected according to the position or the position and shape of said user object.

19. A floating image interaction device according to claim 7, wherein each of the dihedral corner reflectors is provided by an inner surface of a hole penetrating the distortion-free imaging optical system, wherein the mirror surface reflection occurs at the dihedral corner reflectors when the light rays pass through the hole.

20. A floating image interaction device, comprising:

a substrate including a plurality of dihedral corner reflectors, the substrate transmitting light rays therethrough in a distortion-free manner such that the light rays are bent by the dihedral corner reflectors by means of mirror surface reflection, the substrate having a first surface and a second surface, wherein when an object to be projected is placed at a side of the first surface, an real image is formed reversely at a side of the second surface; and

a user object detecting means for detecting position or shape of the object.