Method and device for detecting in real time interactions between a user and an augmented reality scene

• US 20100002909A1
• Filed: 06/30/2009
• Published: 01/07/2010
• Est. Priority Date: 06/30/2008
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Total Immersion
• Inventors: Pasquier, Thomas, LEFEVRE, Valentin, Livet, Nicolas
• Application Number:
  US12/495,402
• Publication Number:
  US 20100002909A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The invention consists in a system for detection in real time of interactions between a user and an augmented reality scene, the interactions resulting from the modification of the appearance of an object present in the image. After having created (110) and processed (115) a reference model in an initialization phase (100), the pose of the object in the image is determined (135) and a comparison model is extracted from the image (160). The reference and comparison models are compared (170), as a function of said determined pose of the object, and, in response to the comparison step, the interactions are detected.

Claims

1. Method of detection in real time of at least one interaction between a user and an augmented reality scene in at least one image from a sequence of images, said at least one interaction resulting from the modification of the appearance of the representation of at least one object present in said at least one image, this method being characterized in that it comprises the following steps:

in an initialization phase,

creation of at least one reference model associated with said at least one object;

processing of said at least one reference model;

in a use phase;

determination of the pose of said at least one object in said at least one image;

extraction of at least one comparison model from said at least one image according to said at least one object; and

comparison of said at least one processed reference model and said at least one comparison model as a function of said determined pose and detection of said at least one interaction in response to said comparison step.

2. Method according to claim 1, wherein said step of determination of the pose of said at least one object in said at least one image uses an object tracking algorithm.

3. Method according to claim 1, wherein said steps of the use phase are repeated at least once on at least one second image from said sequence of images.

4. Method according to claim 3, further comprising a step of recursive processing of the result of said comparison step.

5. Method according to claim 1, wherein said step of processing said at least one reference model comprises a step of definition of at least one active area in said at least one reference model, said method further comprising a step of determination of said at least one active area in said comparison model during said use phase, said comparison step being based on said active areas.

6. Method according to claim 5, further comprising the determination of at least one reference point in said at least one active area of said at least one reference model, said comparison step comprising a step of location of said at least one reference point in said active area of said comparison model.

7. Method according to claim 1, wherein said comparison step comprises a step of evaluation of the correlation of at least one part of said at least one processed reference model and at least one part of said at least one comparison model.

8. Method according to claim 1, wherein said comparison step comprises a step of evaluation of the difference of at least one part of said at least one processed reference model from at least one part of said at least one comparison model.

9. Method according to claim 1, wherein said step of creation of said reference model comprises a step of geometric transformation of homographic type of a representation of said at least one object.

10. Method according to claim 1, wherein said at least one comparison model is determined according to said pose of said at least one object, said step of extraction of said comparison model comprising a step of geometric transformation of homographic type of at least one part of said at least one image.

11. Method according to claim 1, wherein said step of creation of said at least one reference model comprises a step of determination of at least one Gaussian model representing the distribution of at least one parameter of at least one element of said reference model.

12. Method according to claim 11, wherein said comparison step comprises a step of evaluation of a measurement of a distance between at least one point of said comparison model corresponding to said at least one point of said reference model and said Gaussian model associated with said at least one point of said reference model.

13. Method according to claim 1, wherein said reference model comprises a three-dimensional geometric model and an associated texture, the method further comprising a step of determination, during said use phase, of a representation of said reference model according to said pose of said at least one object and according to said geometric model and said texture.

14. Method according to claim 1, further comprising a step of activation of at least one action in response to said detection of said at least one interaction.

15. Method according to claim 1, wherein said modification of the appearance of said at least one object results from the presence of an object, referred to as the second object, different from said at least one object, between said at least one object and the source of said at least one image.

16. Method according to claim 1, wherein said modification of the appearance of said at least one object results from a modification of said at least one object.

17. Computer program comprising instructions adapted to execute each of the steps:

in an initialization phase,

creation of at least one reference model associated with said at least one object;

processing of said at least one reference model;

in a use phase;

determination of the pose of said at least one object in said at least one image;

extraction of at least one comparison model from said at least one image according to said at least one object; and

comparison of said at least one processed reference model and said at least one comparison model as a function of said determined pose and detection of said at least one interaction in response to said comparison step.

18. Computer program according to claim 17, further comprising instructions adapted to execute each of the steps of definition of at least one active area in said at least one reference model and of determination of said at least one active area in said comparison model during said use phase, said comparison step being based on said active areas.

19. Computer program according to claim 17, further comprising instructions adapted to execute the step of evaluation of the correlation of at least one part of said at least one processed reference model and at least one part of said at least one comparison model.

20. Computer program according to claim 17, further comprising instructions adapted to execute each of the steps of determination of at least one Gaussian model representing the distribution of at least one parameter of at least one element of said reference model and of evaluation of a measurement of a distance between at least one point of said comparison model corresponding to said at least one point of said reference model and said Gaussian model associated with said at least one point of said reference model.

21. Computer program according to claim 17, further comprising instructions adapted to execute the step of activation of at least one action in response to said detection of said at least one interaction.

22. Device comprising means adapted to execute each of the steps of,

in an initialization phase,

creation of at least one reference model associated with said at least one object;

processing of said at least one reference model;

in a use phase;

determination of the pose of said at least one object in said at least one image;

extraction of at least one comparison model from said at least one image according to said at least one object; and

comparison of said at least one processed reference model and said at least one comparison model as a function of said determined pose and detection of said at least one interaction in response to said comparison step.

23. Device according to claim 22, further comprising means adapted to execute each of the steps of definition of at least one active area in said at least one reference model and of determination of said at least one active area in said comparison model during said use phase, said comparison step being based on said active areas.

24. Device according to claim 22, further comprising means adapted to execute the step of evaluation of the correlation of at least one part of said at least one processed reference model and at least one part of said at least one comparison model.

25. Device according to claim 22, further comprising means adapted to execute each of the steps of determination of at least one Gaussian model representing the distribution of at least one parameter of at least one element of said reference model and of evaluation of a measurement of a distance between at least one point of said comparison model corresponding to said at least one point of said reference model and said Gaussian model associated with said at least one point of said reference model.

26. Device according to claim 22, further comprising means adapted to execute the step of activation of at least one action in response to said detection of said at least one interaction.