Virtual Photorealistic Digital Actor System for Remote Service of Customers

• US 20170308905A1
• Filed: 07/06/2017
• Published: 10/26/2017
• Est. Priority Date: 03/28/2014
• Status: Active Grant

• Current Assignee: Ratnakumar Navaratnam
• Original Assignee: Ratnakumar Navaratnam
• Inventors: Navaratnam, Ratnakumar
• Granted Patent: US 10,152,719 B2
• Time in Patent Office: Days
• CPC Class Codes: B25J 11/001 with emotions simulating means
• CPC Class Codes: B25J 11/0015 Face robots, animated artificial faces for imitating human expressions
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/637 Administration of user profiles, e.g. generation, initialization, adaptation or distribution
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: H04M 2203/1025 of avatars
• CPC Class Codes: H04M 3/5125 : with remote located operators
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 : defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A system for remote servicing of customers includes an interactive display unit at the customer location providing two-way audio/visual communication with a remote service/sales agent, wherein communication inputted by the agent is delivered to customers via a virtual Digital Actor on the display. The system also provides for remote customer service using physical mannequins with interactive capability having two-way audio visual communication ability with the remote agent, wherein communication inputted by the remote service or sales agent is delivered to customers using the physical mannequin. A web solution integrates the virtual Digital Actor system into a business website. A smart phone solution provides the remote service to customers via an App. In another embodiment, the Digital Actor is instead displayed as a 3D hologram. The Digital Actor is also used in an e-learning solution, in a movie studio suite, and as a presenter on TV, online, or other broadcasting applications.

Claims

1. A method of providing service to a customer from a remote service agent using a talking virtual digital actor displayed on a video display in real time;

said method comprising;

storing a database of recordings of different phonemes at a plurality of different speeds for a language;

detecting, by the remote service agent, when a customer is proximate to the video display;

speaking to the customer in real time, by the remote service agent;

identifying the real-time speech of the remote service agent comprising;

segmenting the real-time speech into a plurality of segments each being smaller than a minimum length of the different phonemes;

determining an average energy distribution of each said speech segment;

obtaining a derivative of said average energy distribution for said plurality of speech segments;

segregating the real-time speech into a plurality of corresponding phonemes by identifying locations of rapid changes in said average energy distributions for said plurality of speech segments; and

identifying each of said plurality of corresponding phonemes by comparing each to said database of recordings to find a respective matching phoneme;

creating verbal communication of said talking virtual digital actor using said matching phonemes.

2. The method according to claim 1 further comprising:

storing a plurality of viseme transitions in said database, each respectively transitioning between two visemes;

applying synchronized lip movements to the talking virtual digital actor using a phoneme-to-viseme-map, for each of said matching phonemes, and using said viseme transitions corresponding to each successive pair of said matching phonemes.

3. The method according to claim 2 further comprising:

storing selective portions of a plurality of video sequences and photographs of a real presenter into a face image database, an eye image database, a hand image database, and a head image database, respectively;

applying facial expressions, eye movements, body movements, and hand movements, respectively, to the talking virtual digital actor within a plurality of successive image frames by combining a plurality of separate parts of a physique of the real presenter stored in said video sequences and photographs in said face image database, said eye image database, said hand image database, and said head image database.

4. The method according to claim 3 further comprising creating said movements without noticeable discontinuities using image morphing by generating an intermediate image frame between each of two of said successive image frames to create a transition image between every two consecutive positions of the talking virtual digital actor.

5. The method according to claim 1, further comprising producing dynamic head movements for the digital actor by rotating the face in real-time based on a depth map of the digital actor face wherein the rotating is done by linearly moving each pixel of the face by an amount proportional to the corresponding pixel value of the depth image in the 2D direction it is needed for rotating the image.

6. The method according to claim 1, further comprising giving facial emotions to the digital actor by using map images for different regions of the face and moving only those regions relative to the current rotation of the face image.

7. The method according to claim 1, further comprising using a talk loop, idle loop and full idle loop and specific gestures together with transition to generate the behavioral aspect of the digital actor without any discontinuity.

8. The method in claim 7, wherein a talk loop image sequence created by reversing latter half or by making last frame equal to first frame using morphing techniques is subdivided to equal parts or unequal parts and morph transitions from the resting position to each of these junctions is created to be used when needed, where an image sequence which depicts hands moving from the idle position to the resting position is created to get a continuous sequence sync with speech of the operator.

9. The method according to claim 1 further comprising programming a command as input for controlling gestures, face emotions, and speech pauses of said digital actor.

10. The method according to claim 7 further comprising programming said command as being a command from the group of commands consisting of:

inputting a text, inputting a numeral, inputting a symbol, and inputting a combination thereof.

11. The method according to claim 1 further comprising dynamically changing, scaling, cropping, and/or changing a position of said digital actor automatically in real-time, according to a command, depending on a space requirement in the video display, including:

suddenly repositioning and/or resizing said digital actor in the video display according to a space required for one or more of advertisements, videos, and/or images used to promote a product displayed in the display screen, and wherein said repositioning and resizing of said digital actor comprises dynamically changing said digital actor as an animation effect occurring within a specified time and at a specified speed.

12. The method according to claim 1 further comprising integrating said digital actor into a company website for promoting products and answering questions from the customers by the remote service agent, by allocating a region of space on the website for placing said digital actor thereon.

13. The method according to claim 1 further comprising integrating said digital actor into a smart mobile phone app for contacting the customers, for promoting the products, and for guiding the customers.

14. The method according to claim 11 further comprising running the digital actor using an operator side computer using only symbols and/or image numbers relating to movements of a full body, a face, and a pair of eyes of said digital actor;

and syncing said verbal communication of said talking virtual digital actor with images of said talking virtual digital actor in the smart mobile phone app, while providing two way communication between the customer and the remote service agent.

15. The method according to claim 1 further comprising using said digital actor:

(i) for presenting tutorials, for providing e-learning, and for communicating as an interviewer,

(ii) for acting in a movie studio suite application, and

(iii) for broadcasting as an a presenter on TV, and online.

16. The method according to claim 1 further comprising integrating said digital actor into a standalone desktop application, said desk-top application being utilized for combining said verbal communication of said digital actor with gestures, emotions, speech pauses;

and for replacing a background, for scaling, for rotating, and for adding animations to said digital actor.

17. The method according to claim 1 further comprising integrating said digital actor into a web browser application, by controlling said digital actor using a remote server for providing images of said digital actor and verbal communication, and providing a download link in the web browser for downloading said image and verbal communication.

18. The method according to claim 1, further comprising providing images of said digital actor using a previously created presentation file based on a real person'"'"'s appearance, voice, and behavior, and providing informational slides, and formatting said verbal communication of said digital actor for describing each slide so that no real presenter is necessary.