System, method, and apparatus for man-machine interaction

• US 10,585,488 B2
• Filed: 09/09/2016
• Issued: 03/10/2020
• Est. Priority Date: 01/04/2016
• Status: Active Grant

• Current Assignee: BOE Technology Group Co., Ltd.
• Original Assignee: BOE Technology Group Co., Ltd.
• Inventors: Lu, Yongchun
• Primary Examiner(s): Butcher, Brian M
• Application Number:
  US15/519,689
• Publication Number:
  US 20180101237A1
• Time in Patent Office: 1,278 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A man-machine interaction system, method, and apparatus, the man-machine interaction system includes a wearable device and a display device. The wearable device includes an image acquiring module, a memory, a processor, an image projecting module, and an information transmission interface. The image acquiring module acquires a current gesture image of a user and a display image on the display device; the processor matches the current gesture image with gestures in a gesture database pre-stored in the memory, determines a gesture category of a matched gesture, determines a control instruction corresponding to the gesture category, and generates a virtual tool image corresponding to the control instruction based on the control instruction; the image projecting module projects a virtual image that includes the virtual tool image and the display image; and the information transmission interface transmits the control instruction to the display device.

Claims

1. A man-machine interaction system, comprising a wearable device and a display device;

wherein the wearable device comprises an image sensor, a memory, a processor, an image projector, and an information transmission interface;

the image sensor is connected to the processor, and configured to acquire a current gesture image of a user and a display image on the display device;

the processor is connected to the memory, and configured to match the current gesture image with gestures pre-stored in a gesture database in the memory, determine a gesture category of a matched gesture, determine a control instruction corresponding to the gesture category, and generate a virtual tool image corresponding to the control instruction based on the control instruction;

the image projector is connected to the processor, and configured to project a virtual image that includes the virtual tool image and the display image;

the information transmission interface is connected to the processor, and configured to transmit the control instruction to the display device; and

the display device is configured to execute the control instruction upon receipt of the control instruction.

2. The man-machine interaction system of claim 1, wherein the wearable device is smart glasses comprising lenses and a glasses frame for supporting the lenses, and the image projector is integrated in the lenses;

the image sensor is arranged on the glasses frame.

3. A man-machine interaction method, comprising:

acquiring, by a wearable device, a current gesture image of a user and a display image on a display device;

matching the current gesture image with gestures pre-stored in a gesture database, determining a gesture category of a matched gesture, and determining a control instruction corresponding to the gesture category;

generating a virtual tool image corresponding to the control instruction based on the control instruction;

projecting a virtual image that includes the virtual tool mage and the display image on the display device; and

transmitting the control instruction to the display device, so that the display device executes the control instruction.

4. The man-machine interaction method of claim 3, wherein acquiring the current gesture image of the user and the display image on the display device comprises:

photographing or video-shooting the current gesture image of the user and the display image on the display device to acquire a mixed image that includes the current gesture image and the display image on the display device; and

recognizing the current gesture image and the display image on the display device from the mixed image, respectively.

5. The man-machine interaction method of claim 3, wherein determining a control instruction corresponding to the gesture category comprises:

looking up and finding the control instruction corresponding to the gesture category based on pre-stored correspondence relationship between the gesture category and the control instruction.

6. The man-machine interaction method of claim 3, wherein matching the current gesture image with the gestures pre-stored in the gesture database, and determining the gesture category of the matched gesture comprises:

obtaining a gesture edge contour and a gesture movement trajectory of the current gesture image based on the current gesture image;

determining whether the gesture edge contour and the gesture movement trajectory of the current gesture image match a gesture edge contour and a gesture movement trajectory of a pre-stored gesture;

if there is a match, determining that a gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, determining that a gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

7. The man-machine interaction method of claim 3, wherein matching the current gesture image with the gestures pre-stored in the gesture database, and determining the gesture category of the matched gesture comprises:

obtaining the gesture edge contour and the gesture movement trajectory of the current gesture image based on the current gesture image to establish a feature vector of the gesture edge contour and a feature vector of the gesture movement trajectory, respectively;

extracting an eigenvalue of the feature vector corresponding to the gesture edge contour and an eigenvalue of the feature vector corresponding to the gesture movement trajectory;

determining whether the eigenvalue corresponding to the gesture edge contour and the eigenvalue corresponding to the gesture movement trajectory match an eigenvalue of the gesture edge contour and an eigenvalue of the gesture movement trajectory of a pre-stored gesture, respectively;

if there is a match, determining that a gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, determining that a gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

13. The man-machine interaction method of claim 4, wherein determining a control instruction corresponding to the gesture category comprises:

looking up and finding the control instruction corresponding to the gesture category based on pre-stored correspondence relationship between the gesture category and the control instruction.

14. The man-machine interaction method of claim 4, wherein matching the current gesture image with the gestures pre-stored in the gesture database, and determining the gesture category of the matched gesture comprises:

obtaining a gesture edge contour and a gesture movement trajectory of the current gesture image based on the current gesture image;

determining whether the gesture edge contour and the gesture movement trajectory of the current gesture image match a gesture edge contour and a gesture movement trajectory of a pre-stored gesture;

if there is a match, determining that a gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, determining that a gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

15. The man-machine interaction method of claim 4, wherein matching the current gesture image with the gestures pre-stored in the gesture database, and determining the gesture category of the matched gesture comprises;

obtaining the gesture edge contour and the gesture movement trajectory of the current gesture image based on the current gesture image to establish a feature vector of the gesture edge contour and a feature vector of the gesture movement trajectory, respectively;

extracting an eigenvalue of the feature vector corresponding to the gesture edge contour and an eigenvalue of the feature vector corresponding to the gesture movement trajectory;

determining whether the eigenvalue corresponding to the gesture edge contour and the eigenvalue corresponding to the gesture movement trajectory match an eigenvalue of the gesture edge contour and an eigenvalue of the gesture movement trajectory of a pre-stored gesture, respectively;

if there is a match, determining that a gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, determining that a gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

8. A man-machine interaction apparatus, comprising:

a sensor configured to acquire a current gesture image of a user and a display image on a display device;

a processor configured to match the current gesture image with gestures pre-stored in a gesture database, determine a gesture category of a matched gesture, and determine a control instruction corresponding to the gesture category, and generate a virtual tool image corresponding to the control instruction based on the control instruction;

a projector configured to project a virtual image that includes the virtual tool image and the display image on the display device; and

an information transmission interface configured to transmit the control instruction to the display device so that the display device executes the control instruction.

9. The man-machine interaction apparatus of claim 8, wherein the sensor is configured to:

photograph or video-shoot the current gesture image of the user and the display image on the display device to acquire a mixed image that includes the current gesture image and the display image on the display device; and

recognize the current gesture image and the display image on the display device from the mixed image, respectively.

10. The man-machine interaction apparatus of claim 8, wherein the processor is configured to look up and find a control instruction corresponding to the gesture category based on pre-stored correspondence relationship between the gesture category and the control instruction.

11. The man-machine interaction apparatus of claim 8, wherein the processor configured to:

obtain a gesture edge contour and a gesture movement trajectory of the current gesture image based on the current gesture image;

determine whether the gesture edge contour and the gesture movement trajectory of the current gesture image match a gesture edge contour and a gesture movement trajectory of a pre-stored gesture;

if there is a match, determine that the gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, determine that the gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

12. The man-machine interaction apparatus of claim 8, wherein the processor is configured to:

obtain a gesture edge contour and a gesture movement trajectory of the current gesture image based on the current gesture image to establish a feature vector of the gesture edge contour and a feature vector of the gesture movement trajectory, respectively;

extract an eigenvalue of the feature vector corresponding to the gesture edge contour and an eigenvalue of the feature vector corresponding to the gesture movement trajectory;

determine whether the eigenvalue corresponding to the gesture edge contour and the eigenvalue corresponding to the gesture movement trajectory match an eigenvalue of the gesture edge contour and an eigenvalue of the gesture movement trajectory of a pre-stored gesture, respectively;

if there is a match, the processor is configured to determine that the gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, the processor is configured to determine that the gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

16. The man-machine interaction apparatus of claim 9, wherein the processor is configured to look up and find a control instruction corresponding to the gesture category based on pre-stored correspondence relationship between the gesture category and the control instruction.

17. The man-machine interaction apparatus of claim 9, wherein the processor configured to:

obtain a gesture edge contour and a gesture movement trajectory of the current gesture image based on the current gesture image;

determine whether the gesture edge contour and the gesture movement trajectory of the current gesture image match a gesture edge contour and a gesture movement trajectory of a pre-stored gesture;

if there is a match, determine that the gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, determine that the gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.

18. The man-machine interaction apparatus of claim 9, wherein the processor is configured to:

obtain a gesture edge contour and a gesture movement trajectory of the current gesture image based on the current gesture image to establish a feature vector of the gesture edge contour and a feature vector of the gesture movement trajectory, respectively;

extract an eigenvalue of the feature vector corresponding to the gesture edge contour and an eigenvalue of the feature vector corresponding to the gesture movement trajectory;

determine whether the eigenvalue corresponding to the gesture edge contour and the eigenvalue corresponding to the gesture movement trajectory match an eigenvalue of the gesture edge contour and an eigenvalue of the gesture movement trajectory of a pre-stored gesture, respectively;

if there is a match, the processor is configured to determine that the gesture category of the pre-stored gesture as the gesture category to which the current gesture image corresponds;

if there is no match, the processor is configured to determine that the gesture category of the pre-stored gesture is not the gesture category to which the current gesture image corresponds.