METHOD AND APPARATUS FOR CONTROLLING SURVEILLANCE SYSTEM WITH GESTURE AND/OR AUDIO COMMANDS

• US 20150074532A1
• Filed: 09/10/2013
• Published: 03/12/2015
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Corporation (Motorola Solutions, Inc.)
• Inventors: Adam, Matthew, Lapidot, Uriel, Rushton, Elliot
• Granted Patent: US 9,766,855 B2
• Time in Patent Office: Days
• US Class Current: 715/719
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08B 13/1963 Arrangements allowing camera rotation to change view, e.g. pivoting camera, pan-tilt and zoom [PTZ]
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A computer-implemented method controls aspects of a surveillance system using gestures and/or voice commands, and comprises: receiving one or both of an operator'"'"'s skeleton input data and voice input data from a gesture detection camera and a microphone; using a processor, matching one or both of the received skeleton input data with a gesture stored in a database and the received voice input data with a text string stored in the database; matching one or both of the gesture and text string to a corresponding video management program command stored on the database; and transmitting the one or more video management program commands to a video management program of the surveillance system.

Claims

1. A computer-implemented method for controlling aspects of a surveillance system, comprising:

(a) receiving one or both of an operator'"'"'s skeleton input data and voice input data;

(b) matching one or both of the received skeleton input data with a gesture stored in a database and the received voice input data with a text string stored in the database, using a processor;

(c) matching one or both of the gesture and text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the one or more video management program commands to a video management program of a surveillance system.

2. A method as claimed in claim 1 wherein the step of matching the receiving skeleton data with a gesture comprises storing frames of skeleton input data received over a defined time frame in a gesture list, then determining whether the stored frames match a set of parameters that define the gesture.

3. A method as claimed in claim 2 wherein the database comprises multiple gestures comprising discrete physical gestures and continuous physical gestures, and the method further comprises determining an operating state of the video management program, then matching the received skeleton data with a gesture selected from a group of discrete physical gestures or a group of continuous physical gestures depending on the determined operating mode.

4. A method as claimed in claim 3 wherein the discrete physical gestures comprise “

swipe left”

, “

swipe right”

, “

swipe up”

, and “

swipe down”

, and wherein the corresponding video management program command to the “

swipe left”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the left, the corresponding video management program command to the “

swipe right”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the right, the “

swipe up”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel above; and

the “

swipe down”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel below.

5. A method as claimed in claim 3 wherein the continuous physical gestures comprise “

TimeLine Scrubbing” and

“

Video Panning and Zooming” and

wherein the corresponding video management program command to the “

TimeLine Scrubbing”

gesture causes a cursor of a displayed timeline to be moved in the direction of the TimeLine Scrubbing gesture, and wherein the corresponding video management program command to the “

Video Panning and Zooming”

gesture causes a part of a video stream to be moved about a panel in the direction of the Video Panning and Zooming gesture.

6. A method as claimed in claim 1 wherein the step of matching the received voice input data with a text string stored in the database comprises using a speech recognition engine to convert the voice input data into a text string, then determining if the converted text string matches a recognized text string stored on the database.

13. A system as claimed in claim 6 wherein the step of matching the received voice input data with a text string stored in the database performed by the interpreting software component program, comprises using a speech recognition engine to convert the voice input data into a text string, then determining if the converted text string matches a recognized text string stored on the database.

7. A surveillance system comprising:

(a) a surveillance camera;

(b) one or both of a gesture capture device and a microphone;

(c) a computer communicative with the surveillance camera and with one or both of the gesture capture device and the microphone, and comprising a processor, and a computer readable medium having stored thereon a video management program and an interpreting software component program, wherein the interpreting software component program comprises a database and program code executable by the processor to perform a method comprising;

(i) receiving one or both of an operator'"'"'s skeleton input data from the gesture capture device and voice input data from the microphone;

(ii) matching one or both of the received skeleton input data with a gesture stored in the database and the received voice input data with a text string stored in the database, using the processor;

(iii) matching one or both of the gesture and text string to a corresponding video management program command stored on the database, using the processor; and

(iv) transmitting the one or more video management program commands to the video management program.

8. A system as claimed in claim 7 wherein the interpreting software component program comprises a speech recognition application program interface comprising an algorithm that converts the voice input data into a text string.

9. A system as claimed in claim 7 wherein the step of matching the receiving skeleton data with a gesture performed by the interpreting software component program comprises storing frames of skeleton input data received over a defined time frame in a gesture list, then determining whether the stored frames match a set of parameters that define the gesture.

10. A system as claimed in claim 7 wherein the database of the interpreting software component program comprises multiple gestures comprising discrete physical gestures and continuous physical gestures, and the method performed by the interpreting software component program further comprises determining an operating state of the video management program, then matching the received skeleton data with a gesture selected from a group of discrete physical gestures or a group of continuous physical gestures depending on the determined operating mode.

11. A system as claimed in claim 10 wherein the discrete physical gestures comprise “

swipe left”

, “

swipe right”

, “

swipe up”

, and “

swipe down”

, and wherein the corresponding video management program command to the “

swipe left”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the left, the corresponding video management program command to the “

swipe right”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the right, the “

swipe up”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel above; and

the “

swipe down”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel below.

12. A system as claimed in claim 10 wherein the continuous physical gestures comprise “

TimeLine Scrubbing” and

“

Video Panning and Zooming” and

wherein the corresponding video management program command to the “

TimeLine Scrubbing”

gesture causes a cursor of a displayed timeline to be moved in the direction of the TimeLine Scrubbing gesture, and wherein the corresponding video management program command to the “

Video Panning and Zooming”

gesture causes a part of a video stream to be moved about a panel in the direction of the Video Panning and Zooming gesture.

14. A computer readable medium having stored thereon an interpreting software program comprising a database and program code executable by a processor to perform a method comprising:

(a) receiving one or both of an operator'"'"'s skeleton input data and voice input data;

(b) matching one or both of the received skeleton input data with a gesture stored on the database and the received voice input data with a text string stored on the database, using the processor;

(c) matching one or both of the gesture and text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the one or more video management program commands to a video management program of a surveillance system.

15. A computer-readable medium as claimed in claim 14 wherein the interpreting software component program comprises a speech recognition application program interface comprising an algorithm that converts the voice input data into a text string.

16. A computer-readable medium as claimed in claim 14 wherein the database comprises multiple gestures comprising discrete physical gestures and continuous physical gestures, and the method performed by the interpreting software component program further comprises determining an operating state of the video management program, then matching the received skeleton data with a gesture selected from a group of discrete physical gestures or a group of continuous physical gestures depending on the determined operating mode.

17. A computer-readable medium as claimed in claim 16 wherein the discrete physical gestures comprise “

swipe left”

, “

swipe right”

, “

swipe up”

, and “

swipe down”

, and wherein the corresponding video management program command to the “

swipe left”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the left, the corresponding video management program command to the “

swipe right”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the right, the “

swipe up”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel above; and

the “

swipe down”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel below.

18. A computer-readable medium as claimed in claim 16 wherein the continuous physical gestures comprise “

TimeLine Scrubbing” and

“

Video Panning and Zooming” and

wherein the corresponding video management program command to the “

TimeLine Scrubbing”

gesture causes a cursor of a displayed timeline to be moved in the direction of the TimeLine Scrubbing gesture, and wherein the corresponding video management program command to the “

Video Panning and Zooming”

gesture causes a part of a video stream to be moved about a panel in the direction of the Video Panning and Zooming gesture.

19. A computer-readable medium as claimed in claim 14 wherein the step of matching the received voice input data with a text string stored in the database performed by the interpreting software component program, comprises using a speech recognition engine to convert the voice input data into a text string, then determining if the converted text string matches a recognized text string stored on the database.