APPARATUS AND METHOD OF AUGMENTED REALITY INTERACTION

• US 20130249944A1
• Filed: 03/01/2013
• Published: 09/26/2013
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardayal, Sharwin Winesh
• Granted Patent: US 9,135,753 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method of augmented reality interaction for repositioning a virtual object on an image of a surface comprises capturing successive video images of the surface and first and second control objects and defining an interaction start area over the surface with respect to the virtual object. The method detects the control objects in successive video images, detects whether the control objects are brought together over the interaction start area, and if so, analyses a region of successive video images using optical flow analysis to determine the overall direction of motion of the control objects and augmenting the video image to show the virtual object being held by the control objects. Augmenting the video image itself comprises superposing a graphical effect on the video image prior to superposition of the virtual object, such that the graphical effect visually disconnects the virtual object from the video image in the resulting augmented image.

Claims

1. A method of augmented reality interaction for repositioning a virtual object on an image of a surface, the method comprising:

capturing successive video images of the surface and first and second control objects;

defining an interaction start area over the surface with respect to the virtual object;

detecting the first and second control objects in successive video images;

detecting whether the first and second control objects are brought together over the interaction start area;

and when it is detected that the first and second control objects are brought together over the interaction start area,

analysing a region of successive video images using optical flow analysis to determine an overall direction of motion of the first and second control objects; and

augmenting the video image to show the virtual object being held by the first and second control objects;

and in which

the step of augmenting the video image to show the virtual object being held by the first and second control objects comprises the step of superposing a graphical effect on the video image prior to superposition of the virtual object, such that the graphical effect visually disconnects the virtual object from the video image in the resulting augmented image.

2. A method according to claim 1, in which the first and second control objects are one selected from a list consisting of:

i. a user'"'"'s hands;

ii. two of a user'"'"'s digits; and

iii. two control devices.

3. A method according to claim 1, in which the step of detecting whether the first and second control objects are brought together over the interaction start area comprises the step of detecting whether the respective motions of the first and second control objects are substantially toward the centre of the start area, from substantially opposite directions.

4. A method according to claim 3 in which the step of detecting whether the first and second control objects are brought together over the interaction start area comprises the step of detecting whether the first and second control objects occupy a threshold amount of a predetermined region at or adjacent to the centre of the start area.

5. A method according to claim 3, in which the step of detecting whether the first and second control objects are brought together over the interaction start area comprises a predetermined tolerance to asymmetry of actions by the first and second control objects with respect to the centre of the start area.

6. A method according to claim 1, in which only motion directions from sub-regions of the analysed region corresponding to a characteristic of the control object are used to determine the overall direction of motion.

7. A method according to claim 1, in which the step of augmenting the video image to show the virtual object being held by the first and second control objects is made subject to a predetermined time limit, after which the virtual object is no longer shown being held by the first and second control objects.

8. A method according to claim 1, in which the overall direction of motion is obtained for sub regions in respective left and right halves of the analysed region, and if the horizontal component of motion between the overall motions of the left and right halves differs by more than a threshold amount, then the virtual object is no longer shown being held by the first and second control objects.

9. A method according to claim 1, comprising the step of:

positioning the virtual object back on the image of the surface responsive to the position of the first and second controllers in the captured video image and also their relative change in size compared to when they were first detected to be brought together.

10. A method according to claim 1, in which the surface is a book.

11. A method according to claim 1, in which the virtual object represents a narrator of the book, and repositioning the narrator produces a corresponding shift in an audio narration of the book.

12. A non-transitory computer program product storing computer implementable instructions thereon, the instructions, when executed by a computer, implement the steps of:

capturing successive video images of the surface and first and second control objects;

defining an interaction start area over the surface with respect to the virtual object;

detecting the first and second control objects in successive video images;

detecting whether the first and second control objects are brought together over the interaction start area;

and when it is detected that the first and second control objects are brought together over the interaction start area,

analysing a region of successive video images using optical flow analysis to determine an overall direction of motion of the first and second control objects; and

augmenting the video image to show the virtual object being held by the first and second control objects;

and in which

the step of augmenting the video image to show the virtual object being held by the first and second control objects comprises the step of superposing a graphical effect on the video image prior to superposition of the virtual object, such that the graphical effect visually disconnects the virtual object from the video image in the resulting augmented image.

13. An entertainment device for interacting with a virtual object in an augmented reality environment, the device comprising:

video input means for receiving successive captured video images of a surface and first and second control objects;

processing means operable to define an interaction start area over the surface with respect to the virtual object;

the processing means being operable to detect whether the first and second control objects are brought together over the interaction start area;

analysis means operable to analyse a region of successive video images using optical flow analysis to determine the overall direction of motion of the first and second control objects; and

augmentation means operable to augment the video image to show the virtual object being held by the first and second control objects, in the event that the first and second control objects are detected as being brought together over the interaction start area,

and in which

the augmentation means is operable to superpose a graphical effect on the video image prior to superposition of the virtual object, such that the graphical effect visually disconnects the virtual object from the video image in the resulting augmented image.