REMOTE CONTROL APPARATUS, REMOTE CONTROL METHOD, REMOTE CONTROL SYSTEM, AND PROGRAM

• US 20180217671A1
• Filed: 02/10/2017
• Published: 08/02/2018
• Est. Priority Date: 02/23/2016
• Status: Abandoned Application

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Okamoto, Hideo, Nakagawa, Takaaki, Doi, Hiromasa, Kondo, Kazuhiro
• Application Number:
  US15/747,250
• Publication Number:
  US 20180217671A1
• Time in Patent Office: Days
• CPC Class Codes: B25J 11/0085 Cleaning
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/30 User interface
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: H04Q 9/00 Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom

Abstract

The present disclosure relates to a remote control apparatus, a remote control method, a remote control system, and a program that are capable of achieving remote control by an intuitive motion using only a hand and an eye.

A plurality of imaging units capture images of a space in which an eye and a hand of a user and an operation target device are present. Positions of the eye and the hand of the user and the operation target device are specified on the basis of the captured images. A line of sight of the user is specified on the basis of a positional relationship between the eye of the user and the operation target device. A motion of the user that is made while the hand is located on a straight line being the line of sight is assumed as an operation instruction given to the operation target device. The operation instruction is transmitted to the operation target device to cause the operation target device to operate. The present disclosure can be applied to a remote control apparatus.

Claims

1. A remote control apparatus, comprising:

a position detection unit that detects positions of a device to be operated, and an eye and a hand of a user;

a user motion recognition unit that recognizes a motion of the user when a positional relationship between the eye, the hand, and the device maintains a predetermined positional relationship;

an operation instruction specifying unit that specifies an operation instruction for the device to be operated, the operation instruction corresponding to the motion of the user; and

a drive instruction unit that drives the device to be operated on the basis of the operation instruction specified by the operation instruction specifying unit.

2. The remote control apparatus according to claim 1, wherein

the user motion recognition unit specifies a line of sight of the user from the eye to the device on the basis of the positional relationship between the eye, the hand, and the device, and recognizes the motion of the user when a positional relationship in which the hand is located on a straight line being the line of sight is maintained.

3. The remote control apparatus according to claim 2, wherein

the user motion recognition unit specifies the line of sight of the user from the eye to the device on the basis of the positional relationship between the eye, the hand, and the device, and recognizes a motion of the hand on the straight line being the line of sight as the motion of the user when a positional relationship in which a predetermined finger of the hand is located on the straight line being the line of sight is maintained.

4. The remote control apparatus according to claim 1, wherein

the device to be operated further includes a display unit that displays a UI (User Interface) image for controlling an operation of the device, and

the user motion recognition unit specifies a line of sight of the user from the eye to a predetermined UI image on the display unit of the device on the basis of the positional relationship between the eye, the hand, and the device, and recognizes the motion of the user, which is made when a positional relationship in which the hand is located on a straight line being the line of sight is maintained, as a motion of the user to the UI image.

5. The remote control apparatus according to claim 4, wherein

the display unit is a see-through HMD (Head Mount Display) that the user wears, and

the user motion recognition unit specifies the line of sight of the user from the eye to a predetermined UI image on the HMD serving as the display unit of the device on the basis of the positional relationship between the eye, the hand, and the device, and recognizes the motion of the user, which is made when the positional relationship in which the hand is located on the straight line being the line of sight is maintained at a forward position relative to the HMD when viewed from the user, as a motion of the user to the UI image.

6. The remote control apparatus according to claim 1, wherein

the position detection unit estimates, when the position detection unit fails to detect the position of at least any one of the eye and the hand, the positions of the eye and the hand on the basis of another body part of the detected user, and

the user motion recognition unit recognizes the motion of the user when an estimated positional relationship between the eye, the hand, and the device maintains a predetermined positional relationship.

7. The remote control apparatus according to claim 1, wherein

the position detection unit includes a plurality of imaging units that capture images in a plurality of imaging directions, and detects the positions of the device to be operated, and the eye and the hand of the user on the basis of the images captured with the plurality of imaging units.

8. The remote control apparatus according to claim 1, wherein

the user motion recognition unit recognizes the motion of the user when any one of a first positional relationship between a right eye of the eyes, the hand, and the device, a second positional relationship between a left eye of the eyes, the hand, and the device, and a third positional relationship between a middle position of the left and right eyes of the eyes, the hand, and the device maintains a predetermined positional relationship.

9. The remote control apparatus according to claim 1, further comprising

a presentation unit that presents an operation instruction for the device to be operated to the user, the operation instruction corresponding to the motion of the user.

10. The remote control apparatus according to claim 9, wherein

the presentation unit includes a display unit that displays a UI corresponding to the operation instruction, a light-emitting unit that emits light corresponding to the operation instruction, a sound output unit that outputs sound corresponding to the operation instruction, an odor generation unit that generates an odor corresponding to the operation instruction, or a reaction or an operation of the device to be operated, the reaction or the operation corresponding to the operation instruction.

11. A remote control method, comprising the steps of:

detecting positions of a device to be operated, and an eye and a hand of a user;

recognizing a motion of the user when a positional relationship between the eye, the hand, and the device maintains a predetermined positional relationship;

specifying an operation instruction for the device to be operated, the operation instruction corresponding to the motion of the user; and

driving the device to be operated on the basis of the specified operation instruction.

12. A program causing a computer to function as:

a position detection unit that detects positions of a device to be operated, and an eye and a hand of a user;

a user motion recognition unit that recognizes a motion of the user when a positional relationship between the eye, the hand, and the device maintains a predetermined positional relationship;

an operation instruction specifying unit that specifies an operation instruction for the device to be operated, the operation instruction corresponding to the motion of the user; and

a drive instruction unit that drives the device to be operated on the basis of the operation instruction specified by the operation instruction specifying unit.

13. A remote control system, comprising

a plurality of devices to be operated that include at least one master device and at least one different slave device,

the master device including

a position detection unit that detects positions of the master device, the slave device, and an eye and a hand of a user,

a user motion recognition unit that recognizes a motion of the user when a positional relationship between the eye, the hand, and the master device or a positional relationship between the eye, the hand, and the slave device maintains a predetermined positional relationship,

an operation instruction specifying unit that specifies an operation instruction for the master device or the slave device, the operation instruction corresponding to the motion of the user,

a master drive instruction unit that drives the master device on the basis of the operation instruction, and

a master communication unit that transmits the operation instruction to the slave device,

the slave device including

a slave communication unit that receives the operation instruction for the slave device, the operation instruction being transmitted from the master device, and

a slave drive instruction unit that drives the slave device on the basis of the operation instruction.

14. The remote control system according to claim 13, wherein

when the remote control system includes a first operation device and a second operation device, the first operation device being the master device or the slave device and having a display function, the second operation device being the master device or the slave device and being to be operated,

the position detection unit detects the positions of the first operation device, the second operation device, and the eye and the hand of the user,

the user motion recognition unit recognizes a first predetermined motion of the user and recognizes the first operation device as a target device for displaying a UI that controls the second operation device, when the positional relationship between the eye, the hand, and the first operation device maintains a predetermined first positional relationship,

the operation instruction specifying unit specifies an operation instruction causing the first operation device to display the UI that controls the second operation device in accordance with the first motion of the user,

the master drive instruction unit drives the first operation device on the basis of the operation instruction for displaying the UI that controls the second operation device,

the user motion recognition unit recognizes a second predetermined motion of the user and recognizes the second operation device as a target device for performing an operation corresponding to an operation of the UI displayed by the first operation device, when the positional relationship between the eye, the hand, and the second operation device maintains a predetermined second positional relationship,

the operation instruction specifying unit specifies an operation instruction causing the second operation device to perform the operation corresponding to the operation of the UI displayed by the first operation device in accordance with the second motion of the user, and

the master drive instruction unit drives the second operation device on the basis of the operation instruction corresponding to the operation of the UI displayed by the first operation device.

15. The remote control system according to claim 13, wherein

the slave device is a curtain opening and closing apparatus.

16. The remote control system according to claim 13, wherein

the slave device is a reclining apparatus for a rear seat of a vehicle.

17. The remote control system according to claim 16, wherein

the user motion recognition unit recognizes the motion of the user on the basis of the positional relationship between the eye, the hand, and the slave device when a line of sight of the user is a line of sight to the reclining apparatus for a rear seat through a rearview mirror of the vehicle, the reclining apparatus being the slave device, and when a positional relationship in which the hand is located on a straight line being the line of sight is maintained.

18. The remote control system according to claim 13, wherein

the slave device includes a flying object that flies with a rotary wing or a fixed wing, various robotic vacuum cleaners, a radio control device, a baggage sorting apparatus, and a virtual object.