Vision based real-time object tracking system for robotic gimbal control

• US 10,095,942 B2
• Filed: 12/14/2015
• Issued: 10/09/2018
• Est. Priority Date: 12/15/2014
• Status: Active Grant

• Current Assignee: Reflex Robotics, Inc.
• Original Assignee: Reflex Robotics, Inc.
• Inventors: Mentese, Enes, Kim, Seongdo, Babukr, Abdo
• Primary Examiner(s): Pontius, James M
• Assistant Examiner(s): Lotfi, Kyle M
• Application Number:
  US14/968,892
• Publication Number:
  US 20160171330A1
• Time in Patent Office: 1,030 Days
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: H04N 23/685 performed by mechanical compensation

Abstract

Using various embodiments, methods, systems, and apparatuses for controlling a camera pivoting device (e.g., mechanical gimbal) are described. In one embodiment, the system comprises a main computing device, a gimbal stabilizer controller, and a computer vision camera, and/or a user camera. The system is able to track a target object using the computer vision camera even while the target object is moving, the base of the pivoting device is moving (e.g., when a user controlling the camera moves), or a combination of thereof. The camera pivoting device of the embodiments disclosed herein can be mounted on to any number of devices/objects that can provide mobility and/or transportation.

Claims

1. A real-time interactive display system for dynamic object selection and tracking, the system comprising:

a pivoting device, wherein the pivoting device includes a roll motor, pitch motor, and yaw motor, wherein actuating the roll motor, pitch motor, and yaw motor causes the pivoting device to be operable with at least three degrees of freedom;

a computer vision camera, wherein the computer vision camera is mounted on the pivoting device;

a stabilizing controller device coupled to the pivoting device, wherein the stabilizing controller device can actuate the roll, pitch, and yaw motors of the pivoting device; and

a computing device connected to the stabilizing controller device, wherein the computing device comprises a processing system having one or more cores, the processing system configured to;

periodically receive a frame from the computer vision camera;

receive information about a Region Of Interest (ROI) within the frame, wherein the ROI includes a target subject that is tracked in real-time by the computer vision camera, wherein the information includes at least one parameter that can be used to set one or more polygonal boundaries within the frame relative to the target subject, each of the one or more polygonal boundaries defining a region within the frame having an accuracy mode with which the target subject is tracked;

compute a location and size of the target subject based on the information;

determine that the target subject is within a first threshold defined by a first polygonal boundary, out of the one or more polygonal boundaries, within the frame;

in response to determining that the target subject is within the first threshold, set the accuracy mode to a first mode with which the target subject is tracked; and

transmit instructions to the stabilizing controller device to control the pitch and yaw motors of the pivoting device to adjust the pivoting device based on the first mode such that the subject target is maintained within the frame of the computer vision camera,

in response to determining that the target subject is outside of the first threshold defined by the first polygonal boundary, set the accuracy mode to a second mode with which target subject is tracked; and

transmit instructions to the stabilizing controller device to control the pitch and yaw motors of the pivoting device to adjust the pivoting device based on the second mode such that the subject target is maintained within the frame of the computer vision camera, wherein the second mode instructs the stabilizing controller device to move more rapidly to track the movement of the target subject than in the first mode.

2. The system of claim 1, wherein the stabilizing controller device comprises a microcontroller wherein the stabilizing controller device comprises brushless motor drivers.

3. The system of claim 2, wherein proportional-integral-derivative (PID) loops are executed on the microcontroller of the stabilizing controller, and wherein the PID loops control the roll, pitch, and yaw motors of the pivoting device.

4. The system of claim 1, wherein the pivoting device is mechanical gimbal.

5. The system of claim 1, wherein the pivoting device includes an inertial measurement unit (IMU), wherein the IMU can provide feedback to the stabilizing controller device using which the pivoting device is maintained in a stabilized and leveled manner with respect to the target subject, and wherein the stabilizing controller device directly controls the roll motor based on feedback received from the IMU.

6. The system of claim 1, wherein the information about the ROI is received using at least one of a vision based tracking algorithm or an infrared beacon based tracking algorithm.

7. The system of claim 1, wherein the ROI is received from a user operating a machine interface, the machine interface connected to the computing device.

8. A method to dynamically select and track a subject interactively, the method comprising:

periodically receiving, by a computing device, a frame from a computer vision camera;

receiving information about a Region Of Interest (ROI) within the frame, wherein the ROI includes a target subject that is tracked in real-time, wherein the information includes at least one parameter can be used to set one or more polygonal boundaries within the frame relative to the target subject, each of the one or more polygonal boundaries related to an accuracy mode with which the target subject is tracked;

computing a location and size of the target subject based on the information;

determining that the target subject is within a first threshold defined by a first polygonal boundary,

out of the one or more polygonal boundaries, within the frame;

in response to determining that the target subject is within the first threshold, setting the accuracy mode to a first mode with which the target subject is tracked; and

transmitting instructions to a stabilizing controller device to control at least one motor of a pivoting device to adjust the pivoting device such that the subject target is maintained within the frame of the computer vision camera in real-time;

in response to determining that the target subject is outside of the first threshold defined by the first polygonal boundary, setting the accuracy mode to a second mode with which target subject is tracked; and

transmitting instructions to the stabilizing controller device to control the pitch and yaw motors of the pivoting device to adjust the pivoting device based on the second mode such that the subject target is maintained within the frame of the computer vision camera, wherein the second mode instructs the stabilizing controller device to move more rapidly to track the movement of the target subject than in the first mode;

wherein the pivoting device includes a roll motor, pitch motor, and yaw motor, and wherein actuating the roll motor, pitch motor, and yaw motor causes the pivoting device to be operable with at least three degrees of freedom, and wherein the computer vision camera is mounted on the pivoting device, and wherein the stabilizing controller device is coupled to the pivoting device, and wherein the stabilizing controller device can actuate the roll, pitch, and yaw motors of the pivoting device, and wherein the computing device is connected to the stabilizing controller device.

9. The method of claim 8, wherein the stabilizing controller device comprises a microcontroller and brushless motor drivers.

10. The method of claim 9, wherein proportional-integral-derivative (PID) loops are executed on the microcontroller of the stabilizing controller, and wherein the PID loops control the roll, pitch, and yaw motors of the pivoting device.

11. The method of claim 8, wherein the pivoting device is a mechanical gimbal.

12. The method of claim 8, wherein the pivoting device includes an inertial measurement unit (IMU), wherein the IMU can provide feedback to the stabilizing controller device using which the pivoting device is maintained in a stabilized and leveled manner with respect to the target subject, and wherein the stabilizing controller device directly controls the roll motor based on feedback received from the IMU.

13. The method of claim 8, wherein the information about the ROI is received using at least one of a vision based tracking algorithm or an infrared beacon based tracking algorithm.

14. The method of claim 8, wherein the ROI is received from a user operating a machine interface, the machine interface connected to the computing device.

15. A non-transitory computer readable medium comprising instructions which when executed by a processing system executes a method to dynamically select and track a subject interactively, the method comprising:

periodically receiving, by a computing device, a frame from a computer vision camera;

receiving information about a Region Of Interest (ROI) within the frame, wherein the ROI includes a target subject that is tracked in real-time, wherein the information includes at least one parameter can be used to set one or more polygonal boundaries within the frame relative to the target subject, each of the one or more polygonal boundaries related to an accuracy mode with which the target subject is tracked;

computing a location and size of the target subject based on the information;

determining that the target subject is within a first threshold defined by a first polygonal boundary,

out of the one or more polygonal boundaries, within the frame;

in response to determining that the target subject is within the first threshold, setting the accuracy mode to a first mode with which the target subject is tracked; and

transmitting instructions to a stabilizing controller device to control at least one motor of a pivoting device to adjust the pivoting device such that the subject target is maintained within the frame of the computer vision camera in real-time;

in response to determining that the target subject is outside of the first threshold defined by the first polygonal boundary, set the accuracy mode to a second mode with which target subject is tracked; and

transmit instructions to the stabilizing controller device to control the pitch and yaw motors of the pivoting device to adjust the pivoting device based on the second mode such that the subject target is maintained within the frame of the computer vision camera, wherein the second mode instructs the stabilizing controller device to move more rapidly to track the movement of the target subject than in the first mode;

wherein the pivoting device includes a roll motor, pitch motor, and yaw motor, and wherein actuating the roll motor, pitch motor, and yaw motor causes the pivoting device to be operable with at least three degrees of freedom, and wherein the computer vision camera is mounted on the pivoting device, and wherein the stabilizing controller device is coupled to the pivoting device, and wherein the stabilizing controller device can actuate the roll, pitch, and yaw motors of the pivoting device, and wherein the computing device is connected to the stabilizing controller device.

16. The non-transitory computer readable medium of claim 15, wherein the stabilizing controller device comprises a microcontroller and brushless motor drivers.

17. The non-transitory computer readable medium of claim 16, wherein proportional-integral-derivative (PID) loops are executed on the microcontroller of the stabilizing controller, and wherein the PID loops control the roll, pitch, and yaw motors of the pivoting device.

18. The non-transitory computer readable medium of claim 15, wherein the pivoting device is a mechanical gimbal.

19. The non-transitory computer readable medium of claim 15, wherein the pivoting device includes an inertial measurement unit (IMU), wherein the IMU can provide feedback to the stabilizing controller device using which the pivoting device is maintained in a stabilized and leveled manner with respect to the target subject, and wherein the stabilizing controller device directly controls the roll motor based on feedback received from the IMU.

20. The non-transitory computer readable medium of claim 15, wherein the information about the ROI is received using at least one of a vision based tracking algorithm or an infrared beacon based tracking algorithm.