DETECTION OF USER GESTURES

• US 20170322633A1
• Filed: 07/24/2017
• Published: 11/09/2017
• Est. Priority Date: 05/14/2013
• Status: Active Application

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Shen, Zhen, Li, Qi, Song, Xuefeng
• Granted Patent: US 10,268,279 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

In one example, a system may include a display including a display screen; a display controller configured to display visual information on the display screen of the display screen, control at least portions of one or more pixels corresponding to the display screen to render the display screen transparent; and a depth camera, positioned behind the display screen, that is configured to obtain depth image information of one or more objects in front of the display screen when at least portions of the one or more pixels are rendered transparent.

Claims

1. A method, comprising:

halting intermittently, at particular points in time, a transmission of interactive media content that is to be displayed on a display screen, to render one or more regions of the display screen transparent for particular time periods, wherein a number of the particular points in time is determined based, at least in part, on an anticipated rate of user interaction with the interactive media content;

obtaining depth image information that includes a number of distance values regarding at least one object located in front of the display screen when the one or more regions of the display screen are rendered transparent;

determining one or more trigger points based on the depth image information;

reconciling the obtained number of distance values and the one or more determined trigger points to determine a user input; and

executing one or more computer commands that correspond to the determined user input.

2. The method of claim 1, wherein the determining the one or more trigger points includes:

measuring the distance values of the at least one object when the transmission of the interactive media content is intermittently halted;

selecting the one or more trigger points from the at least one object;

outputting the number of distance values of the one or more trigger points to a processor;

determining a movement associated with the one or more trigger points as the user input, wherein the one or more trigger points are indicative of the user input that is to be integrated into the displayed interactive media content; and

associating the movement with the user input relative to the interactive media content presently displayed on the display screen.

3. The method of claim 2, wherein the measuring the distance values includes comparing a reference image, which is captured after a camera is initiated, to an image that is captured through the camera when the one or more regions of the display screen are rendered transparent.

4. The method of claim 1, wherein the halting intermittently the transmission of the interactive media content includes halting intermittently the transmission of the interactive media content to render the one or more regions of the display screen transparent for the particular time periods such that the transparency of the display screen is undetectable to a human eye.

5. The method of claim 1, further comprises capturing an image of the at least one object when the one or more regions of the display screen are rendered transparent.

6. The method of claim 1, wherein the halting intermittently the transmission of interactive media content includes halting intermittently a transmission of a portion of the interactive media content.

7. The method of claim 1, wherein the halting intermittently the transmission of interactive media content includes halting intermittently a transmission of all the interactive media content.

8. A system, comprising:

a display that includes a display screen;

a display controller that is communicatively coupled to the display, wherein the display controller is configured to;

transmit interactive media content to display the interactive media content on the display screen;

halt intermittently, at particular points in time, the transmission of a portion of the interactive media content, to render one or more portions of the display screen transparent for particular time periods; and

transmit transparency instructions to indicate the one or more portions of the display screen that are rendered transparent;

a camera controller, communicatively coupled to the display controller, configured to;

obtain the transparency instructions to control a depth camera to capture an image of one or more objects that are located in front of the display screen; and

obtain depth image information of the one or more objects in the captured image; and

the depth camera, deployed behind the display screen and communicatively coupled to the camera controller, configured to capture the image of the one or more objects when the one or more portions of the display screen are rendered transparent.

9. The system of claim 8, wherein the camera controller is further configured to measure distance values and identify location information, relative to the display, for one or more trigger points that correspond to at least one of the one or more objects.

10. The system of claim 9, wherein the camera controller is further configured to reconcile data that includes:

the measured distance values and the identified location information for the one or more trigger points, the captured image of the one or more objects, and the received transparency instructions to determine whether a trigger point gesture is interpreted as a user input.

11. The system of claim 10, wherein the camera controller is further configured to determine the user input from the reconciled data, in response to a presence of the trigger point gesture at one of the particular time periods when the one or more portions of the display screen are rendered transparent.

12. The system of claim 10, further comprising a processor configured to obtain the reconciled data from the camera controller.

13. The system of claim 11, wherein the processor is further configured to match the user input to a stored user command.

14. The system of claim 13, wherein the processor is further configured to execute, in response to the match, the stored user command in a present context of the displayed interactive media content.

15. The system of claim 9, wherein the camera controller is configured to measure the distance values by comparison of a reference image, which is captured after the depth camera is initiated, with the captured image.

16. A device, comprising;

a display;

a processor configured to be in communication with the display, wherein the processor is effective to;

transmit interactive media content to display the interactive media content on the display, wherein the interactive media content includes programmable code that facilitates user interaction;

halt intermittently, at particular points in time, the transmission of the interactive media content to control one or more pixels that correspond to the display, to render at least a portion of the display transparent for particular time periods;

obtain depth image information that includes a number of distance values regarding at least one object that is located in front of the display when the interactive media content is intermittently halted;

obtain at least one gesture that is identified based on the number of distance values; and

execute one or more commands that correspond to the at least one gesture; and

a camera configured to be in communication with the processor, wherein the camera is effective to capture one or more images or one or more video streams of the at least one object when at least the portion of the one or more pixels of the display are rendered transparent.

17. The device of claim 16, wherein the processor is further effective to match the at least one gesture that corresponds to a user input with a stored user command, and wherein the execution of the one or more commands includes execution of the stored user command, matched to the at least one identified gesture, in a present context of the interactive media content that is displayed on the display.

18. The device of claim 16, wherein the particular time periods are rendered transparent such that the transparency of the display is undetectable to a human eye.

19. The device of claim 16, wherein the at least one object includes a person, wherein the at least one gesture includes a user gesture or a movement of the person, and wherein the user gesture or the movement is rendered using body parts of the person in front of the display.

20. The device of claim 16, wherein the at least one object includes one or more inanimate objects controlled by a user of the device.