System and method for combining three-dimensional tracking with a three-dimensional display for a user interface
First Claim
Patent Images
1. A method comprising:
- acquiring depth data of a subject with a depth sensor, including acquiring depth data of at least one of the subject'"'"'s hands;
tracking the subject'"'"'s movements using the acquired depth data, including;
applying a classifier to the acquired depth data to segment a group of pixels corresponding to the subject from the acquired depth data to form segmented depth data;
identifying a plurality of features of the subject in segmented depth data;
obtaining three-dimensional positions corresponding to the identified plurality of features from segmented depth data; and
constructing a three-dimensional model of the plurality of features from the three-dimensional positions;
constructing a three-dimensional scene including a plurality of virtual objects;
inserting the three-dimensional model into the three-dimensional scene;
determining whether a distance between the subject and a screen is greater than a an average virtual distance between a virtual object and the screen;
causing to be displayed in a three-dimensional display the subject'"'"'s movements interacting with the virtual object in the three-dimensional scene upon a determination that the distance between the subject and the screen is greater than an average virtual distance between a virtual object and the screen; and
using the acquired depth data of the at least one of the subject'"'"'s hands, dynamically rendering a topmost one of the plurality of virtual objects caused to be displayed in the three-dimensional display to be progressively transparent in proportion to a distance between the at least one of the subject'"'"'s hands and the screen as the at least one hand moves closer to the screen and progressively visible as the at least one hand moves away from the screen, wherein the progressively transparent topmost virtual object reveals other visible objects of the plurality of virtual objects previously obscured by the topmost one of the plurality of virtual objects before becoming progressively transparent.
3 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods for combining three-dimensional tracking of a user'"'"'s movements with a three-dimensional user interface display is described. A tracking module processes depth data of a user performing movements, for example, movements of the user'"'"'s hand and fingers. The tracked movements are used to animate a representation of the hand and fingers, and the animated representation is displayed to the user using a three-dimensional display. Also displayed are one or more virtual objects with which the user can interact. In some embodiments, the interaction of the user with the virtual objects controls an electronic device.
182 Citations
19 Claims
-
1. A method comprising:
-
acquiring depth data of a subject with a depth sensor, including acquiring depth data of at least one of the subject'"'"'s hands; tracking the subject'"'"'s movements using the acquired depth data, including; applying a classifier to the acquired depth data to segment a group of pixels corresponding to the subject from the acquired depth data to form segmented depth data; identifying a plurality of features of the subject in segmented depth data; obtaining three-dimensional positions corresponding to the identified plurality of features from segmented depth data; and constructing a three-dimensional model of the plurality of features from the three-dimensional positions; constructing a three-dimensional scene including a plurality of virtual objects; inserting the three-dimensional model into the three-dimensional scene; determining whether a distance between the subject and a screen is greater than a an average virtual distance between a virtual object and the screen; causing to be displayed in a three-dimensional display the subject'"'"'s movements interacting with the virtual object in the three-dimensional scene upon a determination that the distance between the subject and the screen is greater than an average virtual distance between a virtual object and the screen; and using the acquired depth data of the at least one of the subject'"'"'s hands, dynamically rendering a topmost one of the plurality of virtual objects caused to be displayed in the three-dimensional display to be progressively transparent in proportion to a distance between the at least one of the subject'"'"'s hands and the screen as the at least one hand moves closer to the screen and progressively visible as the at least one hand moves away from the screen, wherein the progressively transparent topmost virtual object reveals other visible objects of the plurality of virtual objects previously obscured by the topmost one of the plurality of virtual objects before becoming progressively transparent. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
-
-
16. A system comprising:
-
a sensing module configured to acquire a sequence of depth images of one or more body parts of a subject;
a processor to implement a tracking module configured to track movements of the one or more body parts in the sequence of depth images, including applying a classifier to the acquired depth data to segment a group of pixels corresponding to the subject from the acquired depth data to form segmented depth data, identifying a plurality of features of the subject in segmented depth data, obtaining three-dimensional positions corresponding to the identified plurality of features from segmented depth data and constructing a three-dimensional model of the plurality of features from the three-dimensional positions, constructing a three-dimensional scene including a plurality of virtual objects, and inserting the three-dimensional model into the three-dimensional scene and determine whether a distance between the subject and a screen is greater than a an average virtual distance between a virtual object and the screen;a display configured to provide feedback images perceived as three-dimensional to the subject, wherein the feedback images include a first virtual object animated with movements corresponding to the tracked movements of the one or more body parts, and further wherein the feedback images include a second virtual object with which the subject can interact upon a determination that the distance between the subject and the screen is greater than an average virtual distance between a virtual object and the screen; and upon a determination that the distance between the subject and the screen is decreasing, dynamically rendering the second virtual object with which the subject can interact progressively visible, and upon a determination that the distance between the subject and the screen is increasing, dynamically rendering the second virtual object with which the subject can interact progressively transparent. - View Dependent Claims (17, 18)
-
-
19. A non-transitory computer readable medium having instructions which, when executed by a processor, cause the processor to perform operations comprising:
-
acquiring depth data of a subject with a depth sensor; tracking the subject'"'"'s movements using the acquired depth data, including; applying a classifier to the acquired depth data to segment a group of pixels corresponding to the subject from the acquired depth data to form segmented depth data; identifying a plurality of features of the subject in segmented depth data; obtaining three-dimensional positions corresponding to the identified plurality of features from segmented depth data; and constructing a three-dimensional model of the plurality of features from the three-dimensional positions; constructing a three-dimensional scene including a plurality of virtual objects; inserting the three-dimensional model into the three-dimensional scene; determining whether a distance between the subject and a screen is greater than a an average virtual distance between a virtual object and the screen; causing to be displayed in a three-dimensional display the subject'"'"'s movements interacting with the virtual object in the three-dimensional scene upon a determination that the distance between the subject and the screen is greater than an average virtual distance between a virtual object and the screen; and using the acquired depth data of the at least one of the subject'"'"'s hands, dynamically rendering a topmost one of the plurality of virtual objects caused to be displayed in the three-dimensional display to be progressively transparent in proportion to a distance between the at least one of the subject'"'"'s hands and the screen as the at least one hand moves closer to the screen and progressively visible as the at least one hand moves away from the screen, wherein the progressively transparent topmost virtual object reveals other visible objects of the plurality of virtual objects previously obscured by the topmost one of the plurality of virtual objects before becoming progressively transparent.
-
Specification