Interface control system, interface control apparatus, interface control method, and program

• US 10,698,535 B2
• Filed: 04/18/2016
• Issued: 06/30/2020
• Est. Priority Date: 05/21/2015
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Norieda, Shin, Yoshimoto, Makoto, Saida, Yoshinori
• Primary Examiner(s): Marinelli, Patrick F
• Application Number:
  US15/575,186
• Publication Number:
  US 20180150186A1
• Time in Patent Office: 1,534 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

An interface control system (2000) includes a first marker detection unit (2020) and a display control unit (2040). The first marker detection unit (2020) detects a first marker (50) attached to the body of a user based on a captured image obtained from a camera (20) worn by the user. The display control unit (2040) displays an operation image on a display screen (32) of a head mounted display 30 mounted on the user based on the first marker detected by the first marker detection unit (2020). The display control unit (2040) displays the operation image so that the operation image is viewed to be superimposed on the arm portion of the user when viewed with the eyes of the user.

Claims

1. An interface control system comprising:

memory configured to store instructions; and

a processor configured to execute the instructions to;

detect a first marker attached to the body of a user based on an image obtained from a camera attached to the user;

display an operation image on a display screen of a head mounted display mounted on the user based on the detected first marker so that the operation image is superimposed on the body of the user and is visually recognized by the user; and

determine whether or not the operation image is operated by the user,

wherein the first marker is an image displayed on a display screen of a device attached to the arm portion of the user, and

wherein the processor is further configured to extract a feature of the background in an image obtained from the camera, and determine a shape, a pattern, or a color of the first marker displayed on the display screen of the device based on the extracted feature.

2. The interface control system according to claim 1, wherein the processor is further configured to:

determine a position, a size, or a posture of the detected first marker; and

determine a position, a size, or a posture of the operation image in the display screen of the head mounted display based on the position, the size, or the posture of the detected first marker.

3. The interface control system according to claim 1, wherein the processor is further configured to determine the operation image to be displayed based on a type of first marker captured in an image obtained from the camera.

4. The interface control system according to claim 1, wherein the processor is further configured to:

change types of first markers displayed on the display screen of the device in response to a user operation on the device;

determine the type of first marker detected from the image obtained from the camera, and

change a currently displayed operation image to an operation image corresponding to the determined type of first marker.

5. The interface control system according to claim 1, wherein the processor is further configured to change the operation image to be displayed based on an operation performed on the operation image by the user.

6. The interface control system according to claim 1, wherein the processor is further configured to:

display a second marker on the display screen of the head mounted display; and

display the operation image in a case where a positional relationship between the first marker visually recognized by the user through the head mounted display and the second marker becomes a predetermined positional relationship.

7. The interface control system according to claim 6, wherein the processor is further configured to detect the first marker from a partial region corresponding to the second marker displayed at a partial predetermined position on the display screen of the head mounted display.

8. The interface control system according to claim 1,

wherein the first marker is an image displayed on a display screen of a device attached to the arm portion of the user,

wherein the processor is further configured to:

perform predetermined display on the display screen of the device in a case where predetermined vibration is detected by a sensor detecting vibration of the arm portion of the user; and

determine that the operation image is operated by the user in a case where the predetermined display is imaged by the camera.

9. The interface control system according to claim 1, wherein the processor is further configured to determine that the operation image is operated by the user in a case where predetermined vibration is detected by a sensor detecting vibration of the arm portion of the user.

10. The interface control system according to claim 1, wherein the processor is further configured to detect an operation body from a part of regions defined based on the first marker among regions included in an image obtained from the camera, and determine that the operation image is operated by the user in a case where the operation body is detected from the part of regions.

11. The interface control system according to claim 1, wherein the processor is further configured to determine an operation target which overlaps the hand of the user with an area of a predetermined proportion or higher as an operation target operated by the user among a plurality of operation targets, in a case where the plurality of operation targets are included in the operation image, and where the hand of the user performing an operation on the operation image overlaps the plurality of operation targets.

12. The interface control system according to claim 1, wherein the processor is further configured to determine an operation target which is located on a leftmost part or a rightmost part as an operation target operated by the user among a plurality of operation targets, in a case where the plurality of operation targets are included in the operation image, and where the hand of the user performing an operation on the operation image overlaps the plurality of operation targets.

13. The interface control system according to claim 1, wherein the processor is further configured to, in a case where the operation image visually recognized by the user through the head mounted display overlaps the hand of the user operating the operation image, display an operation image in which an image of the hand is superimposed on a position where the hand of the user is located, on the display screen of the head mounted display.

14. The interface control system according to claim 1, wherein the processor is further configured to display only a part of the operation image according to a posture of the body of the user.

15. An interface control apparatus comprising the processor and the memory according to claim 1.

16. An interface control method executed by a computer, the method comprising:

detecting a first marker attached to the body of a user based on an image obtained from a camera attached to the user;

displaying an operation image on a display screen of a head mounted display mounted on the user based on the detected first marker so that the operation image is superimposed on the body of the user and is visually recognized by the user; and

determining whether or not the operation image is operated by the user,

wherein the first marker is an image displayed on a display screen of a device attached to the arm portion of the user, and

wherein the method further comprises extracting a feature of the background in an image obtained from the camera, and determine a shape, a pattern, or a color of the first marker displayed on the display screen of the device based on the extracted feature.

17. A non-transitory computer-readable storage medium storing a program causing a computer to execute each step according to claim 16.