POINTING INTERACTION METHOD, APPARATUS, AND SYSTEM

• US 20170024015A1
• Filed: 10/03/2016
• Published: 01/26/2017
• Est. Priority Date: 04/03/2014
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.), University of Electronic Science and Technology of China
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.), University of Electronic Science and Technology of China
• Inventors: SONG, Xingguang, CHENG, Hong, JI, Yanli, ZHAO, Yang, YE, Guo
• Granted Patent: US 10,466,797 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

Embodiments of the present invention provide a pointing interaction method, apparatus, and system. The method includes: obtaining a hand image and an arm image; determining spatial coordinates of a fingertip according to the hand image, and determining spatial coordinates of an arm key portion according to the arm image; and performing converged calculation on the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, to determine two-dimensional coordinates, on a display screen, of an intersection point between fingertip pointing and the display screen. Therefore, the pointing interaction apparatus can implement high-precision pointing only by using the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, and the pointing has good realtimeness.

Claims

1. A pointing interaction method, comprising:

obtaining a hand image and an arm image;

determining spatial coordinates of a fingertip according to the hand image, and determining spatial coordinates of an arm key portion according to the arm image; and

performing convergence calculation on the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, to determine two-dimensional coordinates, on a display screen, of an intersection point between a fingertip pointing and the display screen.

2. The method according to claim 1, wherein the obtaining a hand image and an arm image comprises:

obtaining a depth map acquired by a depth camera; and

extracting the hand image and the arm image in the depth map according to a specified threshold of the depth map.

3. The method according to claim 2, after the obtaining a depth map acquired by a depth camera, further comprising:

performing denoising processing on the depth map.

4. The method according to claim 1, wherein the determining spatial coordinates of a fingertip according to the hand image comprises:

extracting a hand contour according to the hand image; and

if the fingertip is not perpendicular to the display screen, determining the spatial coordinates of the fingertip by performing curvature recognition on the hand contour;

or if the fingertip is perpendicular to the display screen, calculating the spatial coordinates of the fingertip by performing Gaussian distribution on the hand contour.

5. The method according to claim 1, wherein the performing the convergence calculation on the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, to determine two-dimensional coordinates, on a display screen, of an intersection point between the fingertip pointing and the display screen comprises:

determining the fingertip pointing according to the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion;

calculating spatial coordinates of the intersection point between the fingertip pointing and the display screen; and

converting the spatial coordinates of the intersection point into the two-dimensional coordinates on the display screen according to a correspondence between a spatial coordinate system of the depth camera and a coordinate system of the display screen.

6. A pointing interaction apparatus, comprising:

an obtaining module, configured to obtain a hand image and an arm image;

a determining module, configured to determine spatial coordinates of a fingertip according to the hand image, and determine spatial coordinates of an arm key portion according to the arm image; and

a calculation module, configured to perform a convergence calculation on the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, to determine two-dimensional coordinates, on a display screen, of an intersection point between a fingertip pointing and the display screen.

7. The apparatus according to claim 6, wherein the obtaining module is configured to:

obtain a depth map acquired by a depth camera, and extract the hand image and the arm image in the depth map according to a specified threshold of the depth map.

8. The apparatus according to claim 7, wherein after obtaining the depth map acquired by the depth camera, the obtaining module is further configured to perform denoising processing on the depth map.

9. The apparatus according to claim 6, wherein the determining module is configured to:

extract a hand contour according to the hand image; and

if the fingertip is not perpendicular to the display screen, determine the spatial coordinates of the fingertip by performing curvature recognition on the hand contour;

or if the fingertip is perpendicular to the display screen, calculate the spatial coordinates of the fingertip by performing Gaussian distribution on the hand contour.

10. The apparatus according to claim 6, wherein the calculation module is configured to:

determine the fingertip pointing according to the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion;

calculate spatial coordinates of the intersection point between the fingertip pointing and the display screen; and

convert the spatial coordinates of the intersection point into the two-dimensional coordinates on the display screen according to a correspondence between a spatial coordinate system of the depth camera and a coordinate system of the display screen.

11. A pointing interaction system, comprising:

a depth camera, the pointing interaction apparatus according to claim 6, a projector connected to the pointing interaction apparatus, and a display screen, wherein

the depth camera is configured to collect a hand image and an arm image; and

the projector is configured to display a intersection point in a position, corresponding to two-dimensional coordinates, on the display screen.