Method and apparatus for controlling surveillance system with gesture and/or audio commands

• US 9,766,855 B2
• Filed: 09/10/2013
• Issued: 09/19/2017
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Corporation (Motorola Solutions, Inc.)
• Inventors: Lapidot, Uriel, Rushton, Elliot, Adam, Matthew
• Primary Examiner(s): Bashore, William
• Assistant Examiner(s): Distefano, Gregory A
• Application Number:
  US14/022,947
• Publication Number:
  US 20150074532A1
• Time in Patent Office: 1,470 Days
• Field of Search: 715719
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08B 13/1963 Arrangements allowing camera rotation to change view, e.g. pivoting camera, pan-tilt and zoom [PTZ]
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A computer-implemented method controls aspects of a surveillance system using gestures and/or voice commands, and comprises: receiving one or both of an operator'"'"'s skeleton input data and voice input data from a gesture detection camera and a microphone; using a processor, matching one or both of the received skeleton input data with a gesture stored in a database and the received voice input data with a text string stored in the database; matching one or both of the gesture and text string to a corresponding video management program command stored on the database; and transmitting the one or more video management program commands to a video management program of the surveillance system.

Claims

1. A computer-implemented method for controlling aspects of a surveillance system, comprising:

(a) receiving an operator'"'"'s skeleton input data;

(b) matching the received skeleton input data with a gesture stored in a database, using a processor;

(c) matching the gesture to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command to a video management program of a surveillance system,

wherein the database comprises multiple gestures comprising discrete physical gestures and continuous physical gestures, and the method further comprises determining an operating state of the video management program, then matching the received skeleton data with a gesture selected from either a group of discrete physical gestures or a group of continuous physical gestures depending on the determined operating state.

2. A method as claimed in claim 1 wherein the step of matching the received skeleton data with a gesture comprises storing frames of skeleton input data received over a defined time frame in a gesture list, then determining whether the stored frames match a set of parameters that define the gesture.

3. A method as claimed in claim 2 wherein the discrete physical gestures comprise “

swipe left”

, “

swipe right”

, “

swipe up”

, and “

swipe down”

, and wherein the corresponding video management program command to the “

swipe left”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the left, the corresponding video management program command to the “

swipe right”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the right, the “

swipe up”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel above; and

the “

swipe down”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel below.

4. A method as claimed in claim 3 wherein the “

swipe left”

gesture is triggered by a swipe left of a hand of an operator of the system, the “

swipe right”

gesture is triggered by a swipe right of the hand, the “

swipe up”

gesture is triggered by a swipe up of the hand, and the “

swipe down”

gesture is triggered by a swipe down of the hand.

5. A method as claimed in claim 4 wherein determining whether the operator has gestured a swipe of the hand comprises:

(a) obtaining a latest location of the hand and of a chest of the operator;

(b) adding the latest location of the hand to the gesture list;

(c) determining a distance between the hand and the chest;

(d) obtaining all locations of the hand from the gesture list that occurred within a last defined time frame;

(e) determining whether a threshold number of locations of the hand have been obtained for the last defined time frame;

(f) comparing a latest location of the hand with all other locations of the hand that have occurred within the last defined time frame;

(g) determining whether all the locations of the hand in the last defined time frame are following a direction of the swipe without going in an opposite direction or deviating to a perpendicular direction by more than a first threshold;

(h) determining whether a distance between a latest location of the hand and an oldest location of the hand during the defined time frame is over a second threshold; and

(i) when

(i) the distance between the hand and the chest exceeds a minimum distance;

(ii) the threshold number of locations have been obtained for the last defined time frame;

(iii) all the locations of the hand in the last defined time frame are following the direction of the swipe without going in the opposite direction or deviating to the perpendicular direction by more than the first threshold; and

(iv) the distance between the latest location of the hand and the oldest location of the hand during the defined time frame is over the second threshold,

determining that the operator has gestured the swipe.

6. A method as claimed in claim 2 wherein the continuous physical gestures comprise “

TimeLine Scrubbing” and

“

Video Panning and Zooming” and

wherein the corresponding video management program command to the “

TimeLine Scrubbing”

gesture causes a cursor of a displayed timeline to be moved in the direction of the TimeLine Scrubbing gesture, and wherein the corresponding video management program command to the “

Video Panning and Zooming”

gesture causes a part of a video stream to be moved about a panel in the direction of the Video Panning and Zooming gesture.

7. A method as claimed in claim 2 wherein the gesture list comprises joint locations and times for each location.

8. A method as claimed in claim 2 further comprising, when the operating state is commanded by the discrete physical gestures:

(a) prior to matching the received skeleton input data with the discrete physical gesture, determining whether the gesture list contains a complete discrete physical gesture;

(b) when the gesture list contains the complete discrete physical gesture, proceeding to match the received skeleton input data with the discrete physical gesture; and

(c) when the gesture list does not contain the complete discrete physical gesture, receiving new skeleton input data.

9. A method as claimed in claim 1 further comprising:

(a) receiving the operator'"'"'s voice input data;

(b) matching the received voice input data with a text string stored in the database, using the processor;

(c) matching the text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command matched to the text string to the video management program,

wherein the step of matching the received voice input data with a text string stored in the database comprises using a speech recognition engine to convert the voice input data into a converted text string, then determining if the converted text string matches the text string stored in the database.

10. A method as claimed in claim 1 wherein matching the received skeleton data with a gesture comprises storing frames of skeleton input data received over multiple defined time frames in a gesture list, and wherein the discrete physical gestures each begins and ends in one of the defined time frames.

11. A method as claimed in claim 10 wherein the continuous physical gestures each begins in one of the defined time frames and continues continuously beyond the end of the one of the defined time frames.

12. A method as claimed in claim 11 wherein at least one of the continuous physical gestures comprises movement of a hand and ends when the hand is lowered.

13. A surveillance system comprising:

(a) a surveillance camera;

(b) a gesture capture device;

(c) a computer communicative with the surveillance camera and with the gesture capture device, and comprising a processor, and a computer readable medium having stored thereon a video management program and an interpreting software component program, wherein the interpreting software component program comprises a database and program code executable by the processor to perform a method comprising;

(i) receiving an operator'"'"'s skeleton input data from the gesture capture device;

(ii) matching the received skeleton input data with a gesture stored in the database, using the processor;

(iii) matching the gesture to a corresponding video management program command stored on the database, using the processor; and

(iv) transmitting the video management program command to the video management program,

wherein the database of the interpreting software component program comprises multiple gestures comprising discrete physical gestures and continuous physical gestures, and the method performed by the interpreting software component program further comprises determining an operating state of the video management program, then matching the received skeleton data with a gesture selected from either a group of discrete physical gestures or a group of continuous physical gestures depending on the determined operating state.

14. A system as claimed in claim 13 further comprising a microphone communicative with the computer, and wherein the method further comprises:

(a) receiving the operator'"'"'s voice input data from the microphone;

(b) matching the received voice input data with a text string stored in the database, using the processor;

(c) matching the text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command matched to the text string to the video management program,

wherein the interpreting software component program comprises a speech recognition application program interface comprising an algorithm that converts the voice input data into a converted text string.

15. A system as claimed in claim 13 wherein the step of matching the received skeleton data with a gesture performed by the interpreting software component program comprises storing frames of skeleton input data received over a defined time frame in a gesture list, then determining whether the stored frames match a set of parameters that define the gesture.

16. A system as claimed in claim 15 wherein the discrete physical gestures comprise “

swipe left”

, “

swipe right”

, “

swipe up”

, and “

swipe down”

, and wherein the corresponding video management program command to the “

swipe left”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the left, the corresponding video management program command to the “

swipe right”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the right, the “

swipe up”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel above; and

the “

swipe down”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel below.

17. A system as claimed in claim 16 wherein the “

swipe left”

gesture is triggered by a swipe left of a hand of an operator of the system, the “

swipe right”

gesture is triggered by a swipe right of the hand, the “

swipe up”

gesture is triggered by a swipe up of the hand, and the “

swipe down”

gesture is triggered by a swipe down of the hand.

18. A system as claimed in claim 17 wherein determining whether the operator has gestured a swipe of the hand comprises:

(a) obtaining a latest location of the hand and of a chest of the operator;

(b) adding the latest location of the hand to the gesture list;

(c) determining a distance between the hand and the chest;

(d) obtaining all locations of the hand from the gesture list that occurred within a last defined time frame;

(e) determining whether a threshold number of locations of the hand have been obtained for the last defined time frame;

(f) comparing a latest location of the hand with all other locations of the hand that have occurred within the last defined time frame;

(g) determining whether all the locations of the hand in the last defined time frame are following a direction of the swipe without going in an opposite direction or deviating to a perpendicular direction by more than a first threshold;

(h) determining whether a distance between a latest location of the hand and an oldest location of the hand during the defined time frame is over a second threshold; and

(i) when

(i) the distance between the hand and the chest exceeds a minimum distance;

(ii) the threshold number of locations have been obtained for the last defined time frame;

(iii) all the locations of the hand in the last defined time frame are following the direction of the swipe without going in the opposite direction or deviating to the perpendicular direction by more than the first threshold; and

(iv) the distance between the latest location of the hand and the oldest location of the hand during the defined time frame is over the second threshold,

determining that the operator has gestured the swipe.

19. A system as claimed in claim 15 wherein the continuous physical gestures comprise “

TimeLine Scrubbing” and

“

Video Panning and Zooming” and

wherein the corresponding video management program command to the “

TimeLine Scrubbing”

gesture causes a cursor of a displayed timeline to be moved in the direction of the TimeLine Scrubbing gesture, and wherein the corresponding video management program command to the “

Video Panning and Zooming”

gesture causes a part of a video stream to be moved about a panel in the direction of the Video Panning and Zooming gesture.

20. A system as claimed in claim 15 wherein the gesture list comprises joint locations and times for each location.

21. A system as claimed in claim 15 further comprising, when the operating state is commanded by the discrete physical gestures:

(a) prior to matching the received skeleton input data with the discrete physical gesture, determining whether the gesture list contains a complete discrete physical gesture;

(b) when the gesture list contains the complete discrete physical gesture, proceeding to match the received skeleton input data with the discrete physical gesture; and

(c) when the gesture list does not contain the complete discrete physical gesture, receiving new skeleton input data.

22. A system as claimed in claim 13 further comprising a microphone communicative with the computer, and wherein the method further comprises:

(a) receiving the operator'"'"'s voice input data from the microphone;

(b) matching the received voice input data with a text string stored in the database, using the processor;

(c) matching the text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command matched to the text string to the video management program,

wherein the step of matching the received voice input data with a text string stored in the database performed by the interpreting software component program, comprises using a speech recognition engine to convert the voice input data into a converted text string, then determining if the converted text string matches the text string stored in the database.

23. A system as claimed in claim 13 wherein matching the received skeleton data with a gesture comprises storing frames of skeleton input data received over multiple defined time frames in a gesture list, and wherein the discrete physical gestures each begins and ends in one of the defined time frames.

24. A system as claimed in claim 23 wherein the continuous physical gestures each begins in one of the defined time frames and continues continuously beyond the end of the one of the defined time frames.

25. A system as claimed in claim 24 wherein at least one of the continuous physical gestures comprises movement of a hand and ends when the hand is lowered.

26. A non-transitory computer readable medium having stored thereon an interpreting software component program comprising a database and program code executable by a processor to perform a method comprising:

(a) receiving an operator'"'"'s skeleton input data;

(b) matching the received skeleton input data with a gesture stored on the database, using the processor;

(c) matching the gesture to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command to a video management program of a surveillance system,

wherein the database comprises multiple gestures comprising discrete physical gestures and continuous physical gestures, and the method performed by the interpreting software component program further comprises determining an operating state of the video management program, then matching the received skeleton data with a gesture selected from either a group of discrete physical gestures or a group of continuous physical gestures depending on the determined operating state.

27. A computer-readable medium as claimed in claim 26 wherein the method further comprises:

(a) receiving the operator'"'"'s voice input data;

(b) matching the received voice input data with a text string stored in the database, using the processor;

(c) matching the text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command matched to the text string to the video management program,

wherein the interpreting software component program comprises a speech recognition application program interface comprising an algorithm that converts the voice input data into a converted text string.

28. A computer-readable medium as claimed in claim 27 wherein the discrete physical gestures comprise “

swipe left”

, “

swipe right”

, “

swipe up”

, and “

swipe down”

, and wherein the corresponding video management program command to the “

swipe left”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the left, the corresponding video management program command to the “

swipe right”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel to the right, the “

swipe up”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel above; and

the “

swipe down”

gesture causes a selected panel in a grid of panels displayed by the video management program to switch places a panel below.

29. A computer-readable medium as claimed in claim 28 wherein the “

swipe left”

gesture is triggered by a swipe left of a hand of an operator of the system, the “

swipe right”

gesture is triggered by a swipe right of the hand, the “

swipe up”

gesture is triggered by a swipe up of the hand, and the “

swipe down”

gesture is triggered by a swipe down of the hand.

30. A computer-readable medium as claimed in claim 29 wherein determining whether the operator has gestured a swipe of the hand comprises:

(a) obtaining a latest location of the hand and of a chest of the operator;

(b) adding the latest location of the hand to the gesture list;

(c) determining a distance between the hand and the chest;

(d) obtaining all locations of the hand from the gesture list that occurred within a last defined time frame;

(e) determining whether a threshold number of locations of the hand have been obtained for the last defined time frame;

(f) comparing a latest location of the hand with all other locations of the hand that have occurred within the last defined time frame;

(g) determining whether all the locations of the hand in the last defined time frame are following a direction of the swipe without going in an opposite direction or deviating to a perpendicular direction by more than a first threshold;

(h) determining whether a distance between a latest location of the hand and an oldest location of the hand during the defined time frame is over a second threshold; and

(i) when

(i) the distance between the hand and the chest exceeds a minimum distance;

(ii) the threshold number of locations have been obtained for the last defined time frame;

(iii) all the locations of the hand in the last defined time frame are following the direction of the swipe without going in the opposite direction or deviating to the perpendicular direction by more than the first threshold; and

(iv) the distance between the latest location of the hand and the oldest location of the hand during the defined time frame is over the second threshold,

determining that the operator has gestured the swipe.

31. A computer-readable medium as claimed in claim 27 wherein the continuous physical gestures comprise “

TimeLine Scrubbing” and

“

Video Panning and Zooming” and

wherein the corresponding video management program command to the “

TimeLine Scrubbing”

gesture causes a cursor of a displayed timeline to be moved in the direction of the TimeLine Scrubbing gesture, and wherein the corresponding video management program command to the “

Video Panning and Zooming”

gesture causes a part of a video stream to be moved about a panel in the direction of the Video Panning and Zooming gesture.

32. A computer-readable medium as claimed in claim 26 wherein the method further comprises:

(a) receiving the operator'"'"'s voice input data;

(b) matching the received voice input data with a text string stored in the database, using the processor;

(c) matching the text string to a corresponding video management program command stored on the database, using the processor; and

(d) transmitting the video management program command matched to the text string to the video management program,

wherein the step of matching the received voice input data with a text string stored in the database performed by the interpreting software component program, comprises using a speech recognition engine to convert the voice input data into a converted text string, then determining if the converted text string matches the text string stored in the database.

33. A computer-readable medium as claimed in claim 26 wherein the step of matching the received skeleton data with a gesture comprises storing frames of skeleton input data received over a defined time frame in a gesture list, then determining whether the stored frames match a set of parameters that define the gesture.

34. A computer-readable medium as claimed in claim 33 wherein the gesture list comprises joint locations and times for each location.

35. A computer-readable medium as claimed in claim 33 wherein the method further comprises, when the operating state is commanded by the discrete physical gestures:

(a) prior to matching the received skeleton input data with the discrete physical gesture, determining whether the gesture list contains a complete discrete physical gesture;

(b) when the gesture list contains the complete discrete physical gesture, proceeding to match the received skeleton input data with the discrete physical gesture; and

(c) when the gesture list does not contain the complete discrete physical gesture, receiving new skeleton input data.

36. A computer-readable medium as claimed in claim 26 wherein matching the received skeleton data with a gesture comprises storing frames of skeleton input data received over multiple defined time frames in a gesture list, and wherein the discrete physical gestures each begins and ends in one of the defined time frames.

37. A computer-readable medium as claimed in claim 36 wherein the continuous physical gestures each begins in one of the defined time frames and continues continuously beyond the end of the one of the defined time frames.

38. A computer-readable medium as claimed in claim 37 wherein at least one of the continuous physical gestures comprises movement of a hand and ends when the hand is lowered.