AUDIO, VIDEO, AND ACTUATION (A/V/A) SYNCHRONIZATION FOR MIXED REALITY

• US 20190007726A1
• Filed: 06/30/2017
• Published: 01/03/2019
• Est. Priority Date: 06/30/2017
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Anderson, Glen J.
• Application Number:
  US15/639,921
• Publication Number:
  US 20190007726A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/442 Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk arrangements for monitoring broadcast services or broadcast-related services H04H60/29; arrangements for identifying or recognising characteristics with a direct linkage to broadcast information H04H60/35; monitoring of user activities for profile generation for accessing a video database G06F16/739; monitoring in wireless networks H04W24/00
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 21/8186 specially adapted to be executed by a peripheral of the client device, e.g. by a reprogrammable remote control
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Systems, apparatuses and methods for monitoring and adjusting audio/video/actuation (A/V/A) synchronization to ensure immersive mixed reality experiences. The method includes preparing the mixed reality content with timestamp metadata in the content stream to describe modality synchronization requirements. The mixed reality content is presented. Synchronization between components of the mixed reality content is monitored. The system determines whether the components of the mixed reality content are in synchronization. If they are not in synchronization, the system may take action to improve the perception of the A/V/A synchronization.

Claims

1. A computing system comprising:

network interface circuitry;

a processor coupled to the network interface circuitry; and

one or more memory devices coupled to the processor, the one or more memory devices including instructions, which when executed by the processor, cause the system to;

prepare mixed reality content with timestamp audio, video, and actuation metadata in a content stream to describe modality synchronization requirements, wherein the actuation metadata describes haptic output, projectile launcher output, and other mechanical stimulation output to provide a user with a sense of physical contact;

present the mixed reality content;

determine whether audio, video, and actuation components of the mixed reality content are in synchronization; and

improve perception of the mixed reality content when the audio, video, and actuation components of the mixed reality content are not in synchronization.

2. The computing system of claim 1, wherein the instructions, which when executed by the processor, cause the system to monitor synchronization between the audio video, and actuation components of the mixed reality content.

3. The computing system of claim 1, wherein the instructions, which when executed by the processor, cause the system to predict synchronization between the audio, video, and actuation components of the mixed reality content.

4. The computing system of claim 1, wherein when the audio video, and actuation components of the mixed reality content are in synchronization, the instructions, which when executed by the processor, cause the system to:

track successful haptic output sync for specific users, across users, and specific devices; and

continue to present the mixed reality content.

5. The computing system of claim 1, wherein instructions to improve perception of the mixed reality content comprises instructions, which when executed by the processor, cause the system to perform one or more of to:

pre-power motors to enable fast response times for actuations;

or

monitor for user distraction and create additional audio/video complexity to distract the user.

7. The computing system of claim 5, wherein if when the actuation needs to sync quickly, further instructions, which when executed by the processor, cause the system to respond to sync timestamps by pre-powering actuation motors to enable fast response times.

26. The computing system of claim 1, wherein mixed reality content includes augmented reality and virtual reality.

30. The computing system of claim 1, wherein instructions to improve perception of the mixed reality content comprises instructions, which when executed by the processor, cause the system to send a flurry of haptic responses consistent with a storyline to cover excessive de-sync.

31. The computing system of claim 1, wherein instructions to improve perception of the mixed reality content comprises instructions, which when executed by the processor, cause the system to remove actuation commands in the content stream and, when necessary, change the storyline to cover the removal of the actuation commands.

6. (canceled)

8. An apparatus comprising:

a substrate; and

logic coupled to the substrate, wherein the logic includes one or more of configurable logic or fixed-functionality hardware logic, the logic coupled to the substrate to;

prepare the mixed reality content with timestamp audio, video, and actuation metadata in a content stream to describe modality synchronization requirements, wherein the actuation metadata describes haptic output, projectile launcher output, and other mechanical stimulation output to provide a user with a sense of physical contact;

present the mixed reality content;

determine whether audio, video, and actuation components of the mixed reality content are in synchronization; and

improve perception of the mixed reality content when the audio, video, and actuation components of the mixed reality content are not in synchronization.

9. The apparatus of claim 8, further comprising logic coupled to the substrate to monitor synchronization between the audio, video, and actuation components of the mixed reality content.

10. The apparatus of claim 8, wherein when the audio, video, and actuation components of the mixed reality content are in synchronization, the logic coupled to the substrate to:

track successful haptic output sync for specific users, across users, and specific devices; and

continue to present the mixed reality content.

11. The apparatus of claim 8, further comprising logic coupled to the substrate to predict synchronization between the audio, video, and actuation components of the mixed reality content.

12. The apparatus of claim 8, wherein to improve perception of the mixed reality content comprises logic coupled to the substrate to perform one or more of:

pre-powering motors to enable fast response times for actuations;

sending a flurry of haptic responses consistent with a storyline to cover excessive de-sync;

removing actuation commands in the content stream and, when necessary, changing the storyline to cover the removal of the actuation commands;

or

monitoring for user distraction and creating additional audio/video complexity to distract the user.

14. The apparatus of claim 12, wherein when the actuation needs to sync quickly, the logic coupled to the substrate to respond to sync timestamps by pre-powering actuation motors for fast response times.

27. The apparatus of claim 8, wherein mixed reality content includes augmented reality and virtual reality.

13. (canceled)

15. A method of synchronizing mixed reality content, comprising:

preparing the mixed reality content with timestamp audio, video, and actuation metadata in a content stream to describe modality synchronization requirements;

presenting the mixed reality content;

monitoring synchronization between audio, video, and actuation components of the mixed reality content;

determining whether the audio, video, and actuation components of the mixed reality content are in synchronization, wherein the actuation metadata describes haptic output, projectile launcher output, and other mechanical stimulation output to provide a user with a sense of physical contact; and

improving perception of the mixed reality content when the audio, video, and actuation components of the mixed reality content are not in synchronization.

16. The method of claim 15, wherein when the audio, video, and actuation components of the mixed reality content are in synchronization, tracking successful haptic output sync for specific users, across users, and specific devices and continuing to present the mixed reality content.

17. The method of claim 15, wherein improving perception of the mixed reality content comprises one or more of:

pre-powering motors to increase response for actuations;

sending a flurry of haptic responses consistent with a storyline to cover excessive de-sync;

removing actuation commands in the content stream and, when necessary, changing the storyline to cover the removal of the actuation commands;

or

monitoring for user distraction and creating additional audio/video complexity to distract the user.

19. The method of claim 17, wherein when the actuation needs to sync quickly, responding to sync timestamps by pre-powering actuation motors for fast response times.

28. The method of claim 15, wherein mixed reality content includes augmented reality and virtual reality.

18. (canceled)

20. At least one non-transitory computer readable storage medium comprising a set of instructions, which when executed by a computing system, cause the computing system to:

prepare the mixed reality content with timestamp audio, video, and actuation metadata in a content stream to describe modality synchronization requirements, wherein the actuation metadata describes haptic output, projectile launcher output, and other mechanical stimulation output to provide a user with a sense of physical contact;

present the mixed reality content;

monitor synchronization between audio, video, and actuation components of the mixed reality content;

determine whether the audio, video, and actuation components of the mixed reality content are in synchronization; and

improve perception of the mixed reality content when the audio, video, and actuation components of the mixed reality content are not in synchronization.

21. The at least one non-transitory computer readable storage medium of claim 20, wherein when the audio, video, and actuation components of the mixed reality content are in synchronization, the instructions, when executed, cause the computing system to track successful haptic output sync for specific users, across users, and specific devices and continue to present the mixed reality content.

22. The at least one non-transitory computer readable storage medium of claim 20, wherein instructions to improve perception of the mixed reality content further comprise instructions, which when executed by the computing system, cause the computing system to perform one or more of to:

pre-power motors to enable fast response times for actuations;

send a flurry of haptic responses consistent with a storyline to cover excessive de-sync;

remove actuation commands in the content stream and, when necessary, change the storyline to cover the removal of the actuation commands;

or

monitor for user distraction and create additional audio/video complexity to distract the user.

25. The at least one non-transitory computer readable storage medium of claim 22, wherein when the actuation needs to sync quickly, the instructions, when executed by the computing system, cause the computing system to respond to sync timestamps by pre-powering actuation motors to enable fast response times.

29. The at least one non-transitory computer readable storage medium of claim 20, wherein mixed reality content includes augmented reality and virtual reality.

23. (canceled)

24. (canceled)