Input/output device, input/output program, and input/output method that provide visual recognition of object to add a sense of distance

• US 10,171,800 B2
• Filed: 02/19/2013
• Issued: 01/01/2019
• Est. Priority Date: 02/19/2013
• Status: Active Grant

• Current Assignee: Chikara Minami
• Original Assignee: Miramar Services Inc.
• Inventors: Sugimoto, Yoshihiko
• Primary Examiner(s): Geroleo, Francis
• Application Number:
  US14/768,238
• Publication Number:
  US 20150381970A1
• Time in Patent Office: 2,142 Days
• Field of Search: 348 53
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/371 for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis

Abstract

An I/O device, an I/O program, and an I/O method that can add a sense of distance from the position of visually recognizing eyes to a stereoscopic image that is a virtual image, through an object that is a real image and provide visual recognition as if the stereoscopic image were associated with the object are provided. An I/O device of the present invention includes a display device that can generate a stereoscopic image, a depth level sensor that measures a distance to an object, and an event generating unit that generates an event according to a mode of the object at least either in a stereoscopic region of the stereoscopic image generated by the display device or in a depth level detection region of the depth level sensor.

Claims

1. An input/output (I/O) device comprising:

a display device that can generate a stereoscopic image in a virtual image display region;

a depth level sensor that measures a distance to an object in a three-dimensional space detection region; and

a control unit that comprises a memory and a processor and performs functions including;

a depth map processing function that sets an overlapping space region of the three-dimensional space detection region, the virtual image display region and an arm movement region in which both hands can move horizontally and vertically with joints of both shoulders being a center of rotation as a manipulation region, and sets a portion other than the manipulation region in the three-dimensional space detection region as a gesture region;

a gesture recognition function that recognizes a gesture of the hands in the gesture region; and

a graphics processing function that manipulates the stereoscopic image based on an output of the depth level sensor in the manipulation region or based on the gesture recognized in the gesture region; and

a calibration processing function that determines display position of the virtual image display region and adjusts automatically correlation of the three-dimensional space detection region and the virtual image display region.

2. The I/O device according to claim 1, the control unit further performing:

an event service function that carries out a predetermined event in accordance with the gesture recognized by the gesture recognition function.

3. The I/O device according to claim 2, wherein the predetermined event is displaying a shadow of the hand and changing the shadow of the hand according to a motion of the hand in a depth level direction.

4. The I/O device according to claim 1, wherein the display device is a head-mount display device.

5. An input/output (I/O) system comprising:

a memory to store computer executable instructions, and

a processor that executes the computer executable instructions to perform processes, comprising;

a display process of generating a stereoscopic image in a virtual image display region;

a depth level sensor process of measuring a distance to an object in a three-dimensional space detection region;

a depth map process of setting an overlapping space region of the three-dimensional space detection region, the virtual image display region and an arm movement region in which both hands can move horizontally and vertically with joints of both shoulders being a center of rotation as a manipulation region, and setting a portion other than the manipulation region in the three-dimensional space detection region as a gesture region;

a gesture recognition process of recognizing a gesture of the hands in the gesture region;

a graphics process of manipulating the stereoscopic image based on an output of the depth level sensor in the manipulation region or based on the gesture recognized in the gesture region; and

a calibration process of determining display position of the virtual image display region and automatically adjusting correlation of the three-dimensional space detection region and the virtual image display region.

6. The I/O system according to claim 5, further comprising:

an event service process of carrying out a predetermined event in accordance with the gesture recognized in the gesture recognition process.

7. The I/O system according to claim 6, wherein the predetermined event is displaying a shadow of the hand and changing the shadow of the hand according to a motion of the hand in a depth level direction.

8. An input/output (I/O) method comprising:

a display step of generating a stereoscopic image in a virtual image display region;

a depth level sensor step of measuring a distance to an object in a three-dimensional space detection region;

a depth map processing step of setting an overlapping space region of the three-dimensional space detection region, the virtual image display region and an arm movement region in which both hands can move horizontally and vertically with joints of both shoulders being a center of rotation as a manipulation region, and setting a portion other than the manipulation region in the three-dimensional space detection region as a gesture region;

a gesture recognition step of recognizing a gesture of the hands in the gesture region;

a graphics processing step of manipulating the stereoscopic image based on an output of the depth level sensor in the manipulation region or based on the gesture recognized in the gesture region; and

a calibration processing step of determining display position of the virtual image display region and automatically adjusting correlation of the three-dimensional space detection region and the virtual image display region.

9. The I/O method according to claim 8, further comprising:

an event service step of carrying out a predetermined event in accordance with the gesture recognized in the gesture recognition step.

10. The I/O method according to claim 9, wherein the predetermined event is displaying a shadow of the hand and changing the shadow of the hand according to a motion of the hand in a depth level direction.