Augmented reality with direct user interaction
First Claim
1. An augmented reality system, comprising:
- a user-interaction region;
an image sensor arranged to capture images of a user-controlled object in the user-interaction region;
a display device which is at least partially transparent and arranged to combine a displayed virtual environment with a view of the user-interaction region, such that both the virtual environment and the object are concurrently visible to a viewing user on the display device;
a processor arranged to receive the images and track movement of the object in the user-interaction region, calculate a corresponding movement within the virtual environment that is visually aligned with the movement of the object when viewed by the user on the display device, and control the display device to update the virtual environment based on the corresponding movement; and
at least one of;
the display device comprising a plurality of partitioned sections, the processor being further arranged to selectively control the display device to reduce the light emitted from at least one partition of the display device at a location coincident with a user'"'"'s hand, such that at least one partition of the display device emits less light than at least one other partition of the display device and a portion of the user-interaction region corresponding to the at least one partition of the display device with reduced light emission is more visible through the display device than surrounding portions;
orthe processor being further arranged to selectively control the display device to render black pixels at locations in the virtual environment that are coincident with the user'"'"'s hands.
2 Assignments
0 Petitions
Accused Products
Abstract
Augmented reality with direct user interaction is described. In one example, an augmented reality system comprises a user-interaction region, a camera that captures images of an object in the user-interaction region, and a partially transparent display device which combines a virtual environment with a view of the user-interaction region, so that both are visible at the same time to a user. A processor receives the images, tracks the object'"'"'s movement, calculates a corresponding movement within the virtual environment, and updates the virtual environment based on the corresponding movement. In another example, a method of direct interaction in an augmented reality system comprises generating a virtual representation of the object having the corresponding movement, and updating the virtual environment so that the virtual representation interacts with virtual objects in the virtual environment. From the user'"'"'s perspective, the object directly interacts with the virtual objects.
49 Citations
20 Claims
-
1. An augmented reality system, comprising:
-
a user-interaction region; an image sensor arranged to capture images of a user-controlled object in the user-interaction region; a display device which is at least partially transparent and arranged to combine a displayed virtual environment with a view of the user-interaction region, such that both the virtual environment and the object are concurrently visible to a viewing user on the display device; a processor arranged to receive the images and track movement of the object in the user-interaction region, calculate a corresponding movement within the virtual environment that is visually aligned with the movement of the object when viewed by the user on the display device, and control the display device to update the virtual environment based on the corresponding movement; and at least one of; the display device comprising a plurality of partitioned sections, the processor being further arranged to selectively control the display device to reduce the light emitted from at least one partition of the display device at a location coincident with a user'"'"'s hand, such that at least one partition of the display device emits less light than at least one other partition of the display device and a portion of the user-interaction region corresponding to the at least one partition of the display device with reduced light emission is more visible through the display device than surrounding portions;
orthe processor being further arranged to selectively control the display device to render black pixels at locations in the virtual environment that are coincident with the user'"'"'s hands. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
-
14. A computer-implemented method of direct user-interaction in an augmented reality system, comprising:
-
controlling, using a processor, an at least partially transparent display device to display a virtual environment combined with a view of a user-interaction region, such that both the virtual environment and an object are concurrently visible and visually aligned to a viewing user on the display device; tracking movement of a user-controlled object located in the user-interaction region using an image sensor; generating a virtual representation of the object having a corresponding movement in the virtual environment that is visually aligned with the movement of the object when viewed by a user on the display device; updating the virtual environment on the display device such that the virtual representation interacts with one or more virtual objects in the virtual environment, such that, from the perspective of the user, the movement of the object viewed through the display device directly interacts with the virtual objects in the virtual environment; and at least one of; the at least partially transparent display device comprising a plurality of partitioned sections, selectively controlling the at least partially transparent display device to reduce the light emitted from at least one partition of the display device at a location coincident with a user'"'"'s hand, such that at least one partition of the display device emits less light than at least one other partition of the display device and a corresponding portion of the user-interaction region is more visible through the least partially transparent display device than surrounding portions;
orselectively controlling the at least partially transparent display device to render black pixels at locations in the virtual environment that are coincident with the user'"'"'s hands, the black pixels making the user'"'"'s hands more visible in the user-interaction region. - View Dependent Claims (15, 16, 17, 18, 19)
-
-
20. An augmented reality system, comprising:
-
a display screen arranged to display a three-dimensional virtual environment comprising one or more virtual objects; a user-interaction region; a depth camera arranged to capture depth images of a user'"'"'s hand located within the user-interaction region; a half-silvered mirror positioned to reflect light from the display screen and transmit light from within the user-interaction region, such that both the virtual environment and the user'"'"'s hand are concurrently visible to the user on a surface of the half-silvered mirror; a processor arranged to;
control the display screen to display the virtual environment such that the virtual environment is visually aligned with the user-interaction region on a surface of the half-silvered mirror from the perspective of the user;
receive the depth images and track movement of the user'"'"'s hand in three dimensions in the user-interaction region;
generate a virtual representation of the user'"'"'s hand having a corresponding movement in the virtual environment that is visually aligned with the movement of the user'"'"'s hand when viewed by the user on the half-silvered mirror; and
update the virtual environment on the display screen such that the virtual representation interacts with the one or more virtual objects in the virtual environment, such that, from the perspective of the user, the movement of the user'"'"'s hand viewed through the half-silvered mirror directly interacts with the virtual objects in the virtual environment; andthe display screen comprising a plurality of partitioned sections, the processor being further arranged to selectively control the display device to reduce the light emitted from at least one partition of the display device at a location coincident with a user'"'"'s hand, such that at least one partition of the display device emits less light than at least one other partition of the display device and a portion of the user-interaction region corresponding to the at least one partition of the display device with reduced light emission is more visible through the display device than surrounding portions.
-
Specification