Method and system for virtual touch entertainment

• US 7,225,414 B1
• Filed: 08/05/2003
• Issued: 05/29/2007
• Est. Priority Date: 09/10/2002
• Status: Active Grant

• Current Assignee: F Poszat Hu LLC (Intellectual Ventures LLC)
• Original Assignee: Videomining Corporation
• Inventors: Jung, Namsoon, Sharma, Rajeev, Schapira, Emilio
• Primary Examiner(s): Lo; Weilun
• Assistant Examiner(s): BECKER, SHASHI KAMALA
• Application Number:
  US10/634,962
• Time in Patent Office: 1,393 Days
• Field of Search: 715/863
• US Class Current: 715/863
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

The present invention is a method and apparatus for attracting the attention of people in public places and engaging them in a touch-free interaction with a multimedia display using an image-capturing system and a set of Computer Vision algorithms as a means of informing the public as well as collecting data about/from the users. The invention is named, Virtual Touch Entertainment (VTE) Platform. The VTE Platform comprises of a series of interaction states, such as the Wait State, the Attraction State, the User Engagement State, the User Interaction State, and the Interaction Termination State. The modules in these interaction states handle complicated tasks assigned to them, such as attracting the users, training the users, providing the multimedia digital content to the users, and collecting the user data and statistics, in an efficient and intelligent manner. The user is able to experience a whole new way of interaction paradigm while getting information and entertainment through the rich digital multimedia. The system operates automatically and dynamically in real-time throughout the whole interaction process.

Claims

1. A method for attracting attention of people in public places and engaging them in an interaction with a system for getting information and entertainment, comprising the following steps of:

a) executing a wait state module playing multimedia on the system in a wait state when there are no people nearby the system,

b) executing an attraction state module attracting the people when they appear nearby the system and encouraging them for an engaged interaction with the system by making a transition from the wait state to an attraction state,

c) executing an user engagement state module helping a user who decided to use the system to learn how to use the system at an user engagement state,

d) executing an user interaction state module allowing the user to select and use the content of the information and entertainment of the system at an user interaction state, and

e) executing an user terminations state module finalizing the user interaction at an optional interaction termination state, to which the system can make a transition at the end of the user interaction from the user interaction state,

wherein the step c) of helping a user at the user engagement state can be skipped if the user does not need the content of the user engagement state,

wherein the step d) further allows the user to navigate from one content of the information and entertainment to another content of the information and entertainment at the user interaction state and whereby the steps are repeated in a loop by going back to the wait state after the user interaction is terminate at the user interaction state or at the optional interaction termination state, and

f) executing an image capturing system and a set of computer vision algorithms for user sensing and hand motion sensing;

wherein said image-capturing system divides its field of view into a plurality of capture zones, and apply said set of computer vision algorithms within said capture zones to sense said user and said hand motion, and wherein said user is promoted or demoted depending on the coordinate of said user'"'"'s position in said plurality of capture zones of said image-capturing system.

2. The method according to claim 1, wherein each state comprises a plurality of modules,

whereby the module is defined as a standalone application or a collection of applications,

whereby said collection of applications is a container of sub-applications, and

whereby said collection of applications can also manage the execution of individual sub-applications.

3. The method according to claim 1, wherein the selection of said information and entertainment content is achieved by a touch-free interaction.

4. The method according to claim 3, wherein said touch-free interaction modality is fused with speech recognition input modality to provide multi-modality.

5. The method according to claim 1, wherein said information and entertainment content in a hierarchical structure is delivered by a multimedia display system.

6. The method according to claim 5, wherein said information and entertainment content is updated using a network.

7. The method according to claim 1, wherein said image-capturing system is able to control functionalities, including pan, tilt, zoom, focus, auto-exposure, and white balance, according to the applications of a specific embodiment in order to adapt to the environment and said user'"'"'s uniqueness.

8. The method according to claim 1, wherein the essentials of a background technology based on the computer vision algorithms further comprises 3-I technologies including:

the intelligence technologies, the interaction technologies, and the immersive technologies.

9. The method according to claim 8, wherein said intelligence technologies comprises means for data collection of said user, means for gathering usage statistics, means for getting user demographics, means for processing real-time personalization based on said demographics, means for processing security applications for authentication based on face recognition.

10. The method according to claim 8, wherein said interaction technologies comprises means for face/body detection, means for localization, means for tracking, means for gesture recognition and means for multi-modal integration to facilitate interaction with digital content, and means for appliances for information access and entertainment.

11. The method according to claim 8, wherein said immersive technologies comprises passive immersion, active immersion, and mixed immersion or augmented reality,

whereby said passive immersion integrates said user'"'"'s face and body into an application or video, while said user(s) face/body is immersed into selectable background, including group photos in college campus background,

whereby said active immersion allows said users to control avatars constructed using said user'"'"'s face image, and

whereby said mixed immersion or augmented reality allows said users to virtually interact with virtual objects.

12. The method according to claim 1, wherein said wait state further comprises:

means for playing video loop for advertising purpose and playing content intended to run in an introduction state.

13. The method according to claim 1, wherein said attraction state further comprises:

means for attracting people and engaging them to the interaction with an embodiment, whereby the means for attracting is an active and intelligent way of interrupting said user by graphical effects, sound effects, or mechanical effects, encouraging said user to engage in the interaction with said method.

14. The method according to claim 1, wherein said user engagement state further comprises:

means for helping said users to engage in an interaction with an embodiment smoothly by training them to know how to use said embodiment in graphical, vocal, and literal forms.

15. The method according to claim 1, wherein said user interaction state further comprises:

means for providing said information and entertainment content to said user.

16. The method according to claim 1, wherein said user interaction state further comprises:

means for providing the interaction not only to a single user but also to a plurality of users.

17. The method according to claim 1, wherein said interaction termination state further comprises:

means for collecting data about said user,

whereby the method uses explicit and implicit data collection about said user and stores the result in a database,

whereby said implicit data collection is done by the computer vision based technologies in the method automatically,

whereby said data collection method in said interaction termination state is an explicit data collection.

18. A system for attracting attention of people in public places and engaging them in an interaction with a system for getting information and entertainment, comprising the following steps of:

a) executing a wait state module playing multimedia on the system in a wait state when there are no people nearby the system,

b) executing an attraction state module attracting the people when they appear nearby the system and encouraging them for an engaged interaction with the system by making a transition from the wait state to an attraction state,

c) executing an user engagement state module helping a user who decided to use the system to learn how to use the system at an user engagement state,

d) executing an user interaction state module allowing the user to select and use the content of the information and entertainment of the system at an user interaction state, and

e) executing an user terminations state module finalizing the user interaction at an optional interaction termination state, to which the system can make a transition at the end of the user interaction from the user interaction state,

wherein the step c) of helping a user at the user engagement state can be skipped if the user does not need the content of the user engagement state,

wherein the step d) further allows the user to navigate from one content of the information and entertainment to another content of the information and entertainment at the user interaction state and whereby the steps are repeated in a loop by going back to the wait state after the user interaction is terminate at the user interaction state or at the optional interaction termination state, and

f) executing an image capturing system and a set of computer vision algorithms for user sensing and hand motion sensing;

wherein said image-capturing system divides its field of view into a plurality of capture zones, and apply said set of computer vision algorithms within said capture zones to sense said user and said hand motion, and wherein said user is promoted or demoted depending on the coordinate of said user'"'"'s position in said plurality of capture zones of said image-capturing system.

19. The system according to claim 18, wherein said wait state further comprises:

means for playing video loop for advertising purpose and playing content intended to run in an introduction state.

20. The apparatus according to claim 18, wherein said attraction state further comprises:

means for attracting people and engaging them to the interaction with an embodiment, whereby the means for attracting is an active and intelligent way of interrupting said user by graphical effects, sound effects, or mechanical effects, encouraging said user to engage in the interaction with said method.

21. The apparatus according to claim 18, wherein said user engagement state further comprises:

means for helping said users to engage in an interaction with an embodiment smoothly by training them to know how to use said embodiment in graphical, vocal, and literal forms.

22. The system according to claim 18, wherein said user interaction state further comprises:

means for providing said information and entertainment content to said user.

23. The apparatus according to claim 18, wherein said user interaction state further comprises:

means for providing the interaction not only to a single user but also to a plurality of users.

24. The system according to claim 18, wherein said interaction termination state further comprises:

means for collecting data about said user,

wherein the system uses explicit and implicit data collection about said user and stores the result in a database,

wherein said implicit data collection is done by said computer vision algorithms in the system automatically,

wherein a data collection method in said interaction termination state is an explicit data collection.