System and method for processing an audio and video input in a point of view program for haptic delivery

• US 10,466,790 B2
• Filed: 03/01/2017
• Issued: 11/05/2019
• Est. Priority Date: 03/17/2015
• Status: Active Grant

• Current Assignee: Whirlwind VR, Inc.
• Original Assignee: Whirlwind VR, Inc.
• Inventors: Sun, Timothy Chia-Chieh, Gilg, Matthew Ryan
• Primary Examiner(s): Lin, Jason
• Application Number:
  US15/446,803
• Publication Number:
  US 20170177085A1
• Time in Patent Office: 979 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems

Abstract

The present embodiments disclose apparatus, systems and methods for allowing users to receive targeted delivery of haptic effects—air flow of variable intensity and temperature—from a single tower or surround tower configuration. The haptic tower may have an enclosed, modular assembly that manipulates air flow, fluid flow, scent, or any other haptic or sensation, for an immersed user. Moreover, the system has an application of sensor technology to capture data regarding a user'"'"'s body positioning and orientation in the real environment. This data and, or data from a program coupled to the system, and, or audio-video data corresponding to a user in a virtual environment, is relayed to a haptic engine; recognized; scored along a plurality of parameters; and converted into a haptic output command for haptic output expression corresponding to the user in the virtual environment.

Claims

1. A system for processing an audio and video input in a point of view program for haptic delivery, said system comprising:

at least one modular haptic tower capable of air displacement with at least one of variable air flow and latency control;

a processor;

a memory element coupled to the processor;

a haptic engine comprising;

an audio and video (a/v) buffer recognition block;

a program executable by the haptic engine and configured to;

recognize at least one of the audio and video input from a virtual environment comprising the user by the a/v buffer recognition block, and determine for at least one tagged event;

a pixel color score of the tagged event by capturing a screen buffer, wherein the pixel color scored is determined based on at least a calculated average hue score of the tagged event using pixel data in the screen buffer, a calculated average of red and blue channels in the screen buffer, and a calculated average luminescence in the screen buffer; and

convert the at least one pixel color scored tagged event into a haptic output command in response to the pixel color score being above a threshold, and based on the haptic output command, control at least one of an intensity of a motor coupled to a fan assembly and a brake coupled to the motor enabling latency control resulting in the air displacement corresponding to the virtual environment comprising the user.

2. The system of claim 1, wherein the audio and video input is from at least one device that outputs at least one of an audio or video signal during operation.

3. The system of claim 1, wherein the a/v buffer recognition block tags at least one event for scoring by at least one of, motion, color or sound.

4. The system of claim 1, wherein the a/v buffer recognition block determines a pixel proximity score of the tagged event by determining the distance from any one a selected target zone comprising the event and a selected destination zone comprising the user, within a matrix of zones occupying the entire field of view or sound.

5. The system of claim 1, wherein the a/v buffer recognition block determines a pixel color score of the tagged event by calculating an average hue score of the tagged event using pixel data in a screen buffer;

calculate a nearness coefficient;

calculate an average of red &

blue channels in the screen buffer;

calculate an offset coefficient;

calculate an average luminance in the screen buffer; and

deriving the average pixel score of the tagged event based on an aggregation of the coefficients.

6. The system of claim 1, wherein the a/v buffer recognition block determines a pixel velocity score of the tagged event by capturing a series of frames, and calculate a coefficient related to pixel velocity by testing the per-frame and per-range delta in any one of, or combination of, hue, luminance, brightness, saturation, or color value;

and deriving the pixel velocity score of the tagged event based on the coefficient.

7. The system of claim 1, wherein the a/v buffer recognition block determines an audio score of the tagged event by capturing an audio buffer;

calculating an Average Energy of the audio buffer, Immediate Energy of the audio buffer, Immediate Energy Delta of the audio buffer, and Immediate Energy Mean Deviation of the audio buffer, and calculating a coefficient related to changes in a frequency spectrum; and

deriving the audio score of the tagged event based on the coefficient.

8. The system of claim 1, further comprising a haptic conversion block applying a scoring rule, wherein a threshold-grade scored event is converted into a haptic output command by the haptic conversion block.

9. The system of claim 8, wherein the haptic conversion block is further coupled to a haptic controller, and said haptic controller processes the haptic output command for actuating a fan and, or temperature element disposed within the modular haptic tower.

10. The system of claim 1, further comprising a feed-forward and, or back-propagated neural network trained to trigger a haptic output based on any one of, or combination of, a stored data input, stored tagged event, stored coefficient value, stored event proximity score value, stored pixel color score value, stored pixel velocity score value, stored audio score value, or haptic output command.

11. The system of claim 10, wherein the feed-forward and, or back-propagated neural network uses a series of externally captured buffers containing known audio-visual sources to aid in real-time recognition of the audio and video input by using a probabilistic approach to determine presence in a captured buffer.

12. The system of claim 1, further comprising at least one of a haptic output command, actuation of the fan assembly, or temperature element that vary based on a program-defined or user-defined mode.

13. The system of claim 12, wherein the haptic output command controls an actuator coupled to any one of, or combination of, a fan assembly, air displacement, temperature element, mist spray unit, or tactile member.