Pointing interaction method, apparatus, and system

• US 10,466,797 B2
• Filed: 10/03/2016
• Issued: 11/05/2019
• Est. Priority Date: 04/03/2014
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.), University of Electronic Science and Technology of China
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Song, Xingguang, Cheng, Hong, Ji, Yanli, Zhao, Yang, Ye, Guo
• Primary Examiner(s): Leiby, Christopher E
• Application Number:
  US15/284,273
• Publication Number:
  US 20170024015A1
• Time in Patent Office: 1,128 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

Embodiments of the present invention provide a pointing interaction method, apparatus, and system. The method includes: obtaining a hand image and an arm image; determining spatial coordinates of a fingertip according to the hand image, and determining spatial coordinates of an arm key portion according to the arm image; and performing converged calculation on the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, to determine two-dimensional coordinates, on a display screen, of an intersection point between fingertip pointing and the display screen. Therefore, the pointing interaction apparatus can implement high-precision pointing only by using the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion, and the pointing has good realtimeness.

Claims

1. A pointing interaction method comprising:

obtaining a hand image and an arm image;

determining (x₁,y₁,z₁) spatial coordinates of a fingertip according to the hand image by performing curvature recognition on a hand contour or performing Gaussian distribution of the hand contour, and

determining (x₂,y₂,z₂) spatial coordinates of an arm key portion according to the arm image;

determining a fingertip pointing direction by applying a spatial linear relationship to the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion;

determining (x,y,z) spatial coordinates, on a display screen, of an intersection point between the fingertip pointing direction and the display screen by applying the spatial linear expression describing the fingertip pointing direction and a planar expression describing a surface of the display screen to determine the intersection point, wherein the spatial linear relationship applied for determining the fingertip pointing direction is a spatial linear expression defined as

2. The method according to claim 1, wherein obtaining the hand and arm images comprises:

obtaining a depth map acquired by the depth camera; and

extracting the hand and arm images in the depth map according to a threshold of the depth map.

3. The method according to claim 2, wherein after obtaining the depth map acquired by the depth camera, the method further comprises performing denoising processing on the depth map.

4. The method according to claim 1, wherein determining spatial coordinates of the fingertip according to the hand image comprises:

extracting a hand contour according to the hand image; and

if the fingertip is not perpendicular to the display screen, determining the spatial coordinates of the fingertip by performing curvature recognition on the hand contour;

or

if the fingertip is perpendicular to the display screen, calculating the spatial coordinates of the fingertip by performing the Gaussian distribution on the hand contour.

5. The method according to claim 3, wherein after performing the denoising processing on the depth map, the method further comprises:

segmenting the hand image;

obtaining a binary image from the hand image; and

extracting a hand contour according to the hand image.

6. The method according to claim 5, wherein extracting the hand contour according to the hand image comprises:

emptying inner points of the binary image using a contour extraction algorithm; and

retaining only points on edges of the binary image using the contour extraction algorithm, wherein an edge width of the edges of the binary image obtained using the contour extraction algorithm is a single pixel.

7. The method according to claim 5, wherein the extracting a hand contour according to the hand image comprises:

scanning each pixel of the binary image, the scanning comprising, according to a contour extraction rule;

(a) if a central pixel of the binary image is has a value of 1 and eight pixels adjacent to the central pixel each have values of 1, setting the central pixel of the binary image to 0;

(b) if, regardless of any of the values of the eight pixels adjacent to the central pixel, the central pixel has a value of 0, retaining the value of 0 for the central pixel; and

(c) if condition (a) is not met or if condition (b) is not met, setting the values of all central pixels to 1,

wherein, after the scanning ends, remaining points in the binary image are a continuous contour of the hand image.

8. The method according to claim 4, wherein the determining the spatial coordinates of the fingertip by performing curvature recognition of the hand contour when the fingertip is not perpendicular to the display screen comprises:

calculating values of K₁(p) of all points on the hand contour according to a curvature calculation formula defined as;

9. The method according to claim 8, wherein the determining spatial coordinates of a fingertip according to the hand image further comprises:

determining the point P as the fingertip if the point P on the contour meets both of two conditions;

(i) K₁(p) exceeds a threshold in at least one scale; and

(ii) K₁(p) is a local maximum value in a neighborhood of the contour, wherein different detection scales are independent of each other.

10. A pointing interaction apparatus comprising:

a processor in communication with a non-transitory storage medium storing program instructions for execution by the processor, which when executed by the processor, cause the processor to be configured to provide the following operations;

obtain a hand image and an arm image;

determine (x₁,y₁,z₁) spatial coordinates of a fingertip according to the hand image by performing curvature recognition on a hand contour or performing Gaussian distribution of the hand contour, and

determine (x₂,y₂,z₂) spatial coordinates of an arm key portion according to the arm image;

determine a fingertip pointing direction by applying a spatial linear relationship to the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion;

determine (x,y,z) spatial coordinates, on a display screen, of an intersection point between the fingertip pointing direction and the display screen by applying the spatial linear expression describing the fingertip pointing direction and a planar expression describing a surface of the display screen to determine the intersection point, wherein the spatial linear relationship applied for determining the fingertip pointing direction is a spatial linear expression defined as

11. The apparatus according to claim 10, wherein the processor is further configured to:

obtain a depth map acquired by the depth camera, and extract the hand image and the arm image in the depth map according to a specified threshold of the depth map.

12. The apparatus according to claim 11, wherein after obtaining the depth map acquired by the depth camera, the processor is further configured to perform denoising processing on the depth map.

13. The apparatus according to claim 10, wherein the processor is further configured to:

extract a hand contour according to the hand image; and

if the fingertip is not perpendicular to the display screen, determine the spatial coordinates of the fingertip by performing curvature recognition on the hand contour;

or

if the fingertip is perpendicular to the display screen, calculate the spatial coordinates of the fingertip by performing the Gaussian distribution on the hand contour.

14. The apparatus according to claim 13, wherein, to determine the spatial coordinates of the fingertip according to the hand image by performing curvature recognition on the hand contour when the fingertip is not perpendicular to the display screen, the processor is further configured to:

calculate values of K₁(p) of all points on the hand contour according to a curvature calculation formula defined as;

15. A pointing interaction system comprising:

a depth camera;

a pointing interaction apparatus for

obtaining a hand image and an arm image;

determining (x₁,y₁,z₁) spatial coordinates of a fingertip according to the hand image by performing curvature recognition on a hand contour or performing Gaussian distribution of the hand contour,

determining (x₂,y₂,z₂) spatial coordinates of an arm key portion according to the arm image;

determining a fingertip pointing direction by applying a spatial linear relationship to the spatial coordinates of the fingertip and the spatial coordinates of the arm key portion;

determining (x,y,z) spatial coordinates of an intersection point between the fingertip pointing direction and a display screen by applying the spatial linear expression describing the fingertip pointing direction and a planar expression describing a surface of the display screen to determine the intersection point, wherein the spatial linear relationship applied for determining the fingertip pointing direction is a spatial linear expression defined as