Non-Interference Field-of-view Support Apparatus for a Panoramic Sensor

• US 20160255305A1
• Filed: 05/11/2016
• Published: 09/01/2016
• Est. Priority Date: 02/15/2006
• Status: Active Grant

• Current Assignee: Virtual Video Reality By Ritchey Limited Liability Corporation
• Original Assignee: Kenneth Ira Ritchey, Kurtis John Ritchey
• Inventors: Ritchey, Kurtis John, Ritchey, Kenneth Ira
• Granted Patent: US 10,447,966 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0156 with optionally usable elements
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04N 2007/145 Handheld terminals
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 23/50 Constructional details
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/142 : Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 : Conference systems

Abstract

A support apparatus with adjustable mechanisms hold a spherical field-of-regard image and audio sensor assembly located at its distal end in front of a user'"'"'s face such that the user'"'"'s fine focus field-of-view of the foreground is not blocked by the mast or the sensor assembly. Automated mechanisms rotate and extend the armature and sensor assembly into position for face-to-face panoramic hands-free video teleconferencing, gaming, or logging. A portable wired or wireless host electronic device with a personal assistant application and sensor correlation system interactively communicates with support apparatus circuitry and servos, imagery, audio, eye and ROI tracking, neural, and user input and feedback systems to orchestrate responses to a local remote user. A sensor assembly includes a VLSIC multi-ROI processing system with integrated camera and display that hides the assembly and provides information to the user or onlooker.

Claims

1. A compensation system for a facial sensor that blocks the field-of-view of the user comprising:

a physiological sensing subsystem;

a surrounding environment sensing subsystem;

an automated sensor support subsystem;

a command and control subsystem;

said subsystems being configured as a user portable system with at least one a user mounted housing, electrical power source, and circuitry to communicatively connect said command and control subsystem with said sensing subsystems and said automated sensor support subsystem;

said command and control subsystem driving said sensing subsystems and said automated support subsystem based on user commands and signatures derived from said sensing subsystems;

said automated sensor support subsystem driving at least one armature connected to at least one sensor of said sensing subsystems to a position outside the fine focus field-of view of the user or to an alternate position.

2. The system according to claim 1, wherein:

said sensor support subsystem comprises an actuator attached to an adjustable armature to automatically position a surrounding environment sensing subsystem comprising a spherical field-of-view sensor assembly to a predetermined position forward of the face of a user or predefined alternate position;

said actuator responsive to command and control signals from a command and control subsystem comprising a host personal electronic device (PED), like a iPhone, that moves said sensor with spherical field-of-view FOV coverage forward of the users face at the initiation of a panoramic video telephone conversation;

said sensor assembly including an image capture capability to record a scene for display;

said actuator further responsive to the command and control signals derived by physiological sensing subsystem comprising an eye-tracking system and host PED which operates in conjunction on imagery captured by the sensor to move the sensor assembly and armature outside of the fine focus FOV of the user so not to block the fine focus FOV of the user but still leave the sensor within monitoring relationship to at least one of the eyes of the user being tracked; and

sensor electronics of at least one of said subsystems including image processing hardware and firmware that are simultaneously responsive to read out imagery for transmission and display of the users region-of-interest derived from the eye-tracking system and sensor assembly operating in conjunction with the PED, the face of the wearer captured by the sensor assembly, allow panning and zooming around the panoramic scene, and windowing or overlaying of imagery, graphics, or text or any combination thereof by the user or a person at a remote location involved in the panoramic video telephone conversation.

3. The system according to claim 1, wherein:

said sensor support subsystem comprises at least one automated servo for at least one extending, retracting, positioning, or rotating at least one armature or sensor into a position about the body of the user to facilitate capture of at least one signature from the sensing subsystems worn by the user;

said subsystems being configured as a user portable system with at least one user mounted housing, electrical power source, and circuitry to communicatively connect said command and control subsystem with said sensing subsystems and said automated sensor support subsystem;

said command and control subsystem driving said sensing subsystems and said automated sensor support subsystem based on user commands and signatures derived from said sensing subsystems to automatically move at least one said sensor subsystem in a hands-free manner about the body of said user.

4. The system according to claim 1, wherein:

said automated sensor support subsystem with an armature and sensor with a manual override that allows the user to at least one turn off or not use the automated feature of said mechanism of said support subsystem and manually position at least one the extending, retracting, positioning, or rotating of the armature and sensor about the body of the user to facilitate capture of a signature by the sensing subsystems worn by the user.

5. The system according to claim 1, wherein:

said physiological and surrounding environmental sensing subsystems comprise at least one on the following sensing subsystems;

a panoramic camera subsystem with at least one monocular, binocular, or stereoscopic image recording, playback, live, and live streaming capability;

said camera having at least one a still and motion imaging capability;

a Light Detection And Ranging (LIDAR) subsystem to at least one find range and/or other information of a distant target, record shape and motion of objects in the field-of-view of said system;

at least one tracking subsystem to include at least one eye (i.e. via optical and electrooculography eye tracking), target, gesture, region-of-interest, or feature tracking and optionally an identification subsystem;

a brain activity sensing subsystem (i.e. EEG, MRI, and EMG Electromyography electrical potential neurological activated, etc.) wherein said command and control subsystem operating on brain activity signatures derived from said physiological sensing subsystem consisting of said brain activity sensing subsystem;

a speech recognition subsystem;

said speech recognition subsystem receiving voice commands;

said speech recognition including a computer system that identifies words and phrases in spoken language and converts them to machine language;

a sub-vocalization recognition (SVR) subsystem wherein signatures from electrodes placed on the users throat produce electromyograms are pattern-matched to known words the user is saying;

said words being recognized by a computer without the user opening his or her mouth and uttering a sound;

an audio subsystem with at least one a mono, stereo, or directional sound recording for processing and playback by an audio subsystem;

said physiological and surrounding environmental sensing subsystems providing information to system to at least one position said sensor support armature out of the fine focus field-of-view of said user, to position other subsystem sensors for improved signature reception, to position peripheral devices like an armature mounted display in an improved position, to position said sensor support armature in an alternate position, to capture information for further processing, and to capture information for display for at least one the wearer of said system and a user at a remote location.

6. The system according to claim 1, wherein:

said sensing subsystems and command and control subsystem comprises at least the functionality of a portable personal electronic device (PED) (i.e. an iPhone) simultaneously responsive to read out imagery for at least one transmission or display of a region of interest of the user derived by at least one a tracking system and a spherical FOV camera system as the eye-tracking system tracks the at least one eye of said user, wherein at least one of said PED further comprising processing means for panning and zooming around the foreground scene, displaying a graphical user interface window via the display device, or overlaying secondary imagery over the modified imagery via the display device.

7. The system according to claim 1, wherein:

said command and control system includes a personal electronic device (PED) that includes a computer processing system with firmware that operates to filter and correlate data from multiple sensor assemblies of at least one a physiological or surrounding environment sensing subsystem that includes at least one a panoramic camera, voice recognition, or brain activity sensor system that drives said automated sensor support subsystem that includes an armature and sensor with at least one actuator to position at least one said sensor to a recording or alternate position.

8. The system according to claim 1, wherein:

said subsystem and components of said system are reconfigured into separate housings that are wirelessly connected and continue to function as said system, each said reconfigurable housed subsystem and component including electronic circuitry with a transceiver for interconnecting separated said subsystems and components and including an electric power unit in the form of a battery to power each said subsystem and components being mounted on or adjacent to a local user or a remote user.

9. The system according to claim 1, wherein:

a sensor, armature, support housing, or combination thereof in which at least one of the automated sensor support system includes a transceiver and battery unit that unplugs from the armature and is positioned in a remote location for wireless communication with at least one a PED, EMD, HMD, display integrated into a armature, mast, camera, integrated camera display or a combination thereof of said system.

10. The user portable system according to claim 1, wherein:

said surrounding environmental sensing subsystem comprises a panoramic camera system with at least one integrated electronic display device;

said display device integrated into at least one a sensor, sensor housing, or the support armature or mast supporting the sensor;

said display oriented in an outward facing manner from said support structure for view by said local or remote user.

11. The wearable armature for attaching a personal electronic device and portable spherical field-of-view camera to the head of a user according to claim 1, comprising:

a spherical field-of-view sensor assembly including a sensor housing, said camera system with a plurality of objective lenses facing outward to provide continuous spherical field-of-view coverage mounted within the sensor housing;

said camera having standalone capability except for a display device;

said camera having at least one a land line and wireless still image and video output capability for connecting to at least one a personal electronic device (PED);

said camera system mounted within the sensor housing in visual line of site to the face and at least one eye of the user;

a support housing being adjustably mounted to a support armature while the support housing is secured by a fastener relative to the head of the user;

said camera sensor housing connected by a fastener that attaches to the distal end of the support armature;

support housing including fasteners to secure the housing to the users head such that the housing, support armature, and sensor assembly are positioned by the user to be in front of the users face but not blocking the fine focus field-of-view of the user;

at least one of said armature and sensor assembly including fasteners for connecting said PED and portable spherical field-of-view camera to the head of said user;

said PED being portable and able to operate in a standalone manner;

said PED including circuitry provided output in communicating relationship to receive from said camera via a flexible cable and connector so that imagery is transmitted from said camera through said cable and connector;

said PED including a integrated display with a touch-screen at least one mounted on the armature or the sensor assembly forward and in communicating relationship to at least one of the eyes of the user;

images derived from the sensor assembly are presented on the display of a PED fastened to the armature or sensor housing allowing the user to view with near field-of-view optics mounted between users eyes and the display.

12. The wearable armature for attaching a PED and portable spherical field-of-view camera to the head of a user according to claim 1, comprising:

a spherical field-of-view sensor assembly including a sensor housing, said camera system with a plurality of objective lenses facing outward to provide continuous spherical field-of-view coverage mounted within the sensor housing;

said camera having standalone capability except for a display device;

said camera having at least one a land line and wireless still image and video output capability for connecting to at least one a personal electronic device (PED);

said camera system mounted within the sensor housing in visual line of site to the face and at least one eye of the user;

a support housing being adjustably mounted to a support armature while the support housing is secured by a fastener relative to the head of the user;

said camera sensor housing connected by a fastener that attaches to the distal end of the support armature;

support housing including fasteners to secure the housing to the users head such that the housing, support armature, and sensor assembly are positioned by the user to be in front of the users face but not blocking the fine focus field-of-view of the user;

at least one of said armature and sensor assembly including fasteners for connecting said PED and portable spherical field-of-view camera to the head of said user;

said PED being portable and able to operate in a standalone manner;

said PED having a wireless communicating relationship with said camera for receiving panoramic imagery and command and control operations;

said PED including a integrated display with a touch-screen at least one mounted on the armature or the sensor assembly forward and in communicating relationship to at least one of the eyes of the user;

images derived from the sensor assembly are presented on the display of a PED fastened to the armature or sensor housing allowing the user to view with near-field of view optics mounted between users eyes and the display.

13. The wearable armature for attaching to a portable spherical field-of-view camera to the head of a user according to claim 1, comprising:

a spherical field-of-view sensor assembly including a sensor housing, said camera system with a plurality of objective lenses facing outward to provide continuous spherical field-of-view coverage mounted within the sensor housing;

said camera having standalone capability except for a display device;

said camera having at least one a wireless still image and video output capability for connecting to at least one a LED/OLED display device;

said camera system mounted within the sensor housing in visual line of site to the face and at least one eye of the user;

a support housing being adjustably mounted to a support armature while the support housing is secured by a fastener relative to the head of the user;

said camera sensor housing connected by a fastener that attaches to the distal end of the support armature;

support housing including fasteners to secure the housing to the users head such that the sensor assembly with LED/OLED display device is positioned by the user to be in front of the users face but not blocking the fine focus field-of-view of the user;

at least one of said armature and sensor assembly including fasteners for connecting said LED/OLED display device;

said LED/OLED display device being portable and able to operate in a standalone manner;

said PED including circuitry to at least one communicate via wire or wirelessly via a transceiver with said LED/OLED display device;

said PED including a integrated display with a touch-screen at least one mounted on the armature or the sensor assembly forward and in communicating relationship to at least one of the eyes of the user;

images derived from the sensor assembly are presented on the display of a LED/OLED display device fastened to the armature or sensor housing allowing the user to view with near-field of view optics mounted between users eyes and the display.

14. The wearable armature for attaching a personal electronic device and portable spherical field-of-view camera to the head of a user according to claim 1, including:

an integrated camera and display comprising an LED/OLED array lens-less camera with touch-screen and user interface devices and associated processors;

said integrated camera and display incorporated for multiple uses in the present invention to include at least one as a spherical field-of-view camera, a camera for eye tracking, a display, and touch-screen embodiment, said embodiments including at least one an integrated eye-tracking camera and a head-mounted display screen;

or a display and touch-screen for a spherical FOV camera housing, armature, and sensor assembly.

15. The system according to claim 1, further including;

an image stabilization at least in one a spherical view camera comprising the a physiological sensing subsystem and a surrounding environment sensing subsystem and a automated sensor support subsystem.

16. The wearable system for attaching a portable spherical field-of-view camera and display to the head of a user according to claim 1, comprising:

a spherical field-of-view sensor assembly including;

a sensor housing;

said camera system with a plurality of objective lenses facing outward to provide continuous spherical field-of-view coverage mounted within the sensor housing;

said camera having standalone capability except for a display device;

said camera having at least one video output capability via transceiver and output jack (i.e. HDMI);

a support housing being adjustably mounted to the support armature while the support housing is secured by a fastener relative to the head of the user;

said camera sensor housing connected by a fastener that attaches to the distal end of the support armature;

said camera system mounted within the sensor housing in visual line of site to the face and at least one eye of the user;

circuitry included in said housing miming through the support armature through at least one male and one female connecting jack between the armature and the camera;

said circuitry facilitating command and control, physiological sensing subsystem, surrounding environment sensing subsystem, a display device, other peripheral device in communication with said apparatus, transmission of electrical power to the camera, or alternatively to provide just structural support and positioning of the sensor assembly if the attached sensor assembly has a wireless standalone capability;

circuitry provided to connect in communicating relationship said camera through said support armature to a armature mounted camera for connection to a portable digital device via a flexible cable and connector;

at least one actuator attached to said support armature that at least one raises, lowers, extends, rotates, or retracts at least one the armature and sensor assembly to a predetermined positon forward of the face of the a wearer or alternate position;

said actuator responsive to command and control signals from said portable digital device and said sensor assembly such a that the sensor is moved forward of the users face at the initiation of a hands-free panoramic video teleconference conversation;

an integrated display with touch-screen and associated processors at least one mounted below the armature or the sensor assembly, images derived from the sensor assembly are presented on the display of a personal electronic device (PED) mounted fastened to the armature or sensor allowing the user to view with near field of view optics mounted between users eyes and the display, wherein at least one actuator attached to the adjustable armature positions the sensor assembly with derived from an eye tracking system to move the sensor assembly and armature outside of the mid and distant range fine focus field-of-view of the wearer so not to block the fine focus field-of-view of the wearers field-of-view of the foreground, but still position the display just below the find focus field-of-view and within reading distance of the users close in field-of-view such that the user looks over the armature and sensor assembly to view the foreground, glances down to read the display on the armature, and positions the armature selectively to look through the display for augmented reality applications,

support housing including fasteners to secure the housing to the users head such that the housing, support armature, and sensor assembly are positioned by the user to be in front of the users face but not blocking the fine focus field-of-view of the user.

17. A user portable system according to claim 1, further comprising:

an OLED display is mounted below an armature or the sensor assembly, images derived from the sensor assembly and from the personal digital assistant are presented to the user for viewing with near field of view optics mounted between users eyes and the display, wherein at least one actuator attached to the adjustable armature positions the sensor assembly derived from an eye tracking system to move the sensor assembly and armature outside of the mid and distant range fine focus field-of-view of the wearer so not to block the fine focus field-of-view of the wearers field-of-view of the foreground, but still position the display just below the find focus field-of-view and within reading distance of the users close in field-of-view such that the user looks over the armature and sensor assembly to view the foreground, glances down to read the display on the armature, and positions the armature selectively to look through the display for augmented reality applications.

18. A user portable system according to claim 1, further comprising:

said automated sensor support subsystem including a gimbal mechanism that rotates the optical axis of at closest one of the objective lenses of a spherical FOV camera in an orientation of the same optical axis as the objective lens to of at least one the Conscious Precept (CP) or the Region of Interest (ROI) defined by at least one the user, a remote user, or said system defined orientation or subject;

said rotation compensating for having a seam in the middle of said ROI or CP the user or the sensing system is focused upon and in order to make image processing for said system less difficult and achieve image continuity.

19. A system according to claim 1, further comprising:

a user unique relational database generated by said mobile system going about life outside a laboratory environment;

said database including at least one a surrounding environment information logged by said surrounding environmental sensing subsystem, a physiological information logged by said physiological sensing subsystem, and a correlated relational database generated by said command and control subsystem that functions as the central processing unit for said system for use in mechanical, biological, or bio-mechanical system;

said database providing a correlated historical accessible database for positioning said armature with sensor.

20. A display system for a portable spherical camera system for a comprising:

a portable spherical field-of-view camera including a compensation system for a facial sensor that blocks the field-of-view;

said compensation system comprising a support housing including fasteners to secure the housing to the users head such that the housing, support armature, and a spherical field-of-view sensor assembly positioned by the user to be in front of the users face but not blocking the fine focus field-of-view of the user;

said camera system and said user portable display system including at least one operating system software or firmware application to cooperatively process said spherical camera imagery and electrical power source to process said panoramic image for further processing or viewing while not blocking the sensors view of the users face or the users view of imagery and overlaid graphics and text on a display presented to a local or remote user;

said camera including a communication device to transmit camera imagery by the use of circuitry or wirelessly at least some portion of said derived image from said camera system to at least one a personal electronic device (PED) display, eye mounted device (EMD) display, head mounted display (HMD) device, spherical camera housing display, integrated LED/OLED camera/display with touch-screen and associated processors, camera armature integrated display, or camera mast integrated display.

21. A wearable apparatus for attaching a portable spherical field-of-view camera to the head of a user comprising:

a spherical field-of-view sensor assembly including;

a sensor housing,

a camera system with a plurality of objective lenses facing outward to provide continuous spherical field-of-view coverage mounted within the sensor housing;

a support housing being adjustably mounted to the support armature while the support housing is secured by a fastener relative to the head of the user;

said camera sensor housing connected by a fastener that attaches to the distal end of the support armature;

said camera system mounted within the sensor housing in visual line of site to the face and at least one eye of the user;

circuitry included in said housing miming through the support armature through at least one male and one female connecting jack between the armature and the camera;

said circuitry facilitating command and control, physiological sensing, a display device, other peripheral device in communication with said apparatus, transmission of electrical power to the camera, or alternatively to provide just structural support and positioning of the sensor assembly if the attached sensor assembly has a wireless standalone capability;

circuitry provided to connect in communicating relationship said camera through said support armature to a armature mounted camera for connection to a portable digital device via a flexible cable and connector;

at least one actuator attached to said support armature that at least one raises, lowers, extends, rotates, or retracts at least one the armature and sensor assembly to a predetermined positon forward of the face of the a wearer or alternate position;

said actuator responsive to command and control signals from said portable digital device and said sensor assembly such a that the sensor is moved forward of the users face at the initiation of a hands-free panoramic video teleconference conversation; and

support housing including fasteners to secure the housing to the users head such that the housing, s support armature, and sensor assembly are positioned by the user to be in front of the users face but not blocking the fine focus field-of-view of the user.