SPEECH INTERFACE FOR VISION-BASED MONITORING SYSTEM

• US 20180246964A1
• Filed: 02/28/2017
• Published: 08/30/2018
• Est. Priority Date: 02/28/2017
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Lighthouse AI, Inc.
• Inventors: Teichman, Alexander William, Sperling, Karsten, Dahlkamp, Hendrik, Dall, Andreas Ubbe, Griffiths, Andy, Hassanpur, Miraj
• Application Number:
  US15/445,501
• Publication Number:
  US 20180246964A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method for natural language-based interaction with a vision-based monitoring system. The method includes obtaining a request input from a user, by the vision-based monitoring system. The request input is directed to an object detected by a classifier of the vision-based monitoring system. The method further includes obtaining an identifier associated with the request input, identifying a site of the vision-based monitoring system from a plurality of sites, based on the identifier, generating a database query, based on the request input and the identified site, and obtaining, from a monitoring system database, video frames that relate to the database query. The video frames include the detected object. The method also includes providing the video frames to the user.

Claims

1. A method for natural language-based interaction with a vision-based monitoring system, the method comprising:

obtaining a request input from a user, by the vision-based monitoring system, wherein the request input is directed to a site-specific object detected by a classifier of the vision-based monitoring system and associated with a site-specific identifier;

obtaining the identifier associated with the request input;

identifying a site of the vision-based monitoring system from a plurality of sites, based on the identifier;

generating a database query, based on the request input and the identified site;

obtaining, from a monitoring system database, video frames that relate to the database query, wherein the video frames comprise the detected object; and

providing the video frames to the user.

2. The method of claim 1, wherein the request input comprises text obtained from a user.

3. The method of claim 2, wherein obtaining the text from the user comprises obtaining a spoken user request and converting the spoken user request to text.

4. The method of claim 1, wherein the request input is obtained using a remote computing device that is accessed by the user.

5. The method of claim 1, wherein the identifier comprises one selected from a group consisting of a hardware ID, an authentication key and credentials.

6. The method of claim 1, wherein generating the database query comprises:

identifying, in the request input, a plurality of distinct filtering intents;

validating the plurality of filtering intents; and

composing the database query from the validated plurality of filtering intents.

7. The method of claim 6, wherein validating the plurality of filtering intents comprises:

making a determination that at least one of the plurality of filtering intents is unknown; and

based on the determination;

resolving the unknown filtering intent using site-specific data of the monitoring system database.

8. The method of claim 6, wherein validating the plurality of filtering intents comprises:

making a determination that at least one of the plurality of filtering intents is unknown; and

based on the determination;

submitting a clarification request to the user;

obtaining a user response; and

resolving the unknown filtering intent based on the obtained user response.

9. The method of claim 8, wherein the user response is a spoken clarification, by the user.

10. The method of claim 8, wherein the user response is a selection in a video frame, made by the user.

11. The method of claim 1, wherein obtaining, from the monitoring system database, video frames that relate to the query, comprises:

identifying, in site-specific data of a metadata archive of the monitoring system database, tags that relate to the query,

wherein the tags label occurrences of at least one selected from a group consisting of the object and an action involving the object,

wherein the tags identify the video frames that relate to the query; and

retrieving the video frames that relate to the query from a video archive of the monitoring system.

12. The method of claim 11,

wherein the video frames that relate to the queries are video frames of archived video streams, stored in the video archive, and

wherein the tags of the video frames label content, detected by the vision-based monitoring system.

13. The method of claim 1, further comprising:

receiving a modification input after receiving the request input

modifying, in response to receiving the modification input, the database query to obtain a modified database query;

obtaining, from the monitoring system database, additional video frames that relate to the modified database query; and

providing the additional video frames to the user.

14. The method of claim 1, further comprising, prior to providing the video frames to the user:

augmenting the video frames by adding a halo to highlight the detected object.

15. The method of claim 1, wherein the video frames provided to the user comprise instructions to enable the user'"'"'s portable device to augment the video frames by adding a halo to highlight the detected object.

16. The method of claim 1, wherein the object detection by the classifier is performed based on the detected object matching information stored in a moving object definition.

17. The method of claim 16, wherein the information stored in the moving object definition comprises at least one selected from a group consisting of visual characteristics of the object and an identifier of the portable computing device associated with the object.

18. A non-transitory computer readable medium comprising instructions that enable a vision-based monitoring system to:

obtain a request input from a user, by the vision-based monitoring system, wherein the request input is directed to a site-specific object detected by a classifier of the vision-based monitoring system and associated with a site-specific identifier;

obtain the identifier associated with the request input;

identify a site of the vision-based monitoring system from a plurality of sites, based on the identifier;

generate a database query, based on the request input and the identified site;

obtain, from a monitoring system database, video frames that relate to the database query, wherein the video frames comprise the detected object; and

provide the video frames to the user.

19. The non-transitory computer readable medium of claim 18, wherein the request input comprises text obtained from a user.

20. The non-transitory computer readable medium of claim 19, wherein obtaining the text from the user comprises obtaining a spoken user request and converting the spoken user request to text.

21. The non-transitory computer readable medium of claim 18, wherein the request input is obtained using a remote computing device that is accessed by the user.

22. The non-transitory computer readable medium of claim 18, wherein the instructions further enable the vision-based monitoring system to, in order to generate the database query:

identify, in the request input, a plurality of distinct filtering intents;

validate the plurality of filtering intents; and

compose the database query from the validated plurality of filtering intents.

23. The non-transitory computer readable medium of claim 22, wherein the instructions further enable the vision-based monitoring system to, in order to validate the plurality of filtering intents comprises:

make a determination that at least one of the plurality of filtering intents is unknown; and

based on the determination;

resolve the unknown filtering intent using site-specific data of the monitoring system database.

24. The non-transitory computer readable medium of claim 22, wherein the instructions further enable the vision-based monitoring system to, in order to validate the plurality of filtering intents:

make a determination that at least one of the plurality of filtering intents is unknown; and

based on the determination;

submit a clarification request to the user;

obtain a user response; and

resolve the unknown filtering intent based on the obtained user response.

25. The non-transitory computer readable medium of claim 23, wherein the user response is a spoken clarification, by the user.

26. The non-transitory computer readable medium of claim 23, wherein the user response is a selection in a video frame, made by the user.

27. The non-transitory computer readable medium of claim 18, wherein the instructions further enable the vision-based monitoring system to, in order to obtain, from the monitoring system database, video frames that relate to the query:

identify, in site-specific data of a metadata archive of the monitoring system database, tags that relate to the query,

wherein the tags label occurrences of at least one selected from a group consisting of the object and an action involving the object,

wherein the tags identify the video frames that relate to the query; and

retrieve the video frames that relate to the query from a video archive of the monitoring system.

28. The non-transitory computer readable medium of claim 27,

wherein the video frames that relate to the queries are video frames of archived video streams, stored in the video archive, and

wherein the tags of the video frames label content, detected by the vision-based monitoring system.

29. The non-transitory computer readable medium of claim 18, wherein the instructions further enable the vision-based monitoring system to, in order to:

receive a modification input after receiving the request input

modify, in response to receiving the modification input, the database query to obtain a modified database query;

obtain, from the monitoring system database, additional video frames that relate to the modified database query; and

provide the additional video frames to the user.

30. The non-transitory computer readable medium of claim 18, wherein the object detection by the classifier is performed based on the detected object matching information stored in a moving object definition.