An engine that provides an natural language interface to any database. Essentially it converts user's natural language question into a database compliant query.
<TITLE></TITLE> <STYLE TYPE="text/css"> </STYLE>Chapter - 1
INTRODUCTION
Interaction with computers has been a problem for most people that have had a chance to do so. Most work done on such systems is repetitive and specialized which results in the creation of small one – off programs that result in the wastage of programming effort. Moreover, end users (developers in this case) are burdened with the task of learning a large number of domain specific languages which further increases the effort and money spent on training.
The goal of program synthesis is to construct automatically a program that provably satisfies a given high level specification. Previously implemented program synthesis techniques use complete specifications of the language but it is often difficult to check without user interference that the specification is been followed by the synthesized program. Also such specifications are hard to write. More recent implementations have started to use incomplete specifications that are easier to write. Such systems are known as PBEs (Programming by Example). We will try to use both techniques and point out their pros and cons.
In this project we will address the problem of synthesizing programs based on an underlying DSL (domain specific language) from NL(natural language). Since NL is inherently imprecise, so we won’t always be able to generate a perfectly accurate query. Instead we will generate a ranked set of queries and ask the user to select the one he/she thinks is the most appropriate. This will also allow us to view how our product is working and what it’s ranking efficiency is.
Requirement Analysis
Software Requirements:
1. Speech Recognition Tool
2. NLP Tools
3. Operating System
4. DBMS.
1.1.2 Hardware Requirements:
1. Client System
2. Microphone
3. Database Server
Objectives
1. Speech to Text Integration
2. Keyword extraction from textual data (using NLP tools)
3. Recommendation system to rank queries
4. Front end to show generated ranked queries to user
5. Interfacing with Database/Program
1.2.1 Sub Objectives
Integrating Speech to text as a JS script
Connecting controlling to model
Integrating NLP POS Tagger
Extracting only noun keywords
Querying database for keyword relations
Generating trivial queries
Combining trivial queries to complex queries
Generating UI for output webpage
Connecting model and view
Chapter - 2
SYSTEM ANALYSIS
2.1
Existing System
A
recent study on the topic of Program Synthesis was published in a
research paper named Compositional
Program Synthesis from Natural Language and Example in
January, 2015 where they described a domain-agnostic program
synthesis framework and algorithm, with which complex tasks can be
accomplished by providing input in a compositional manner. Other
papers related to this topic also propose and give examples for
automatically generating programs from a compositional input of
natural language and examples.
These theories and implementations used a specific database with an inbuilt schema to generate queries for. Token generation is hardwired to this schema in the sense that keywords are identified from the input statement only if the keyword occurs somewhere in the schema declaration or the entries. These tokens are then passed to the synthesizer to generate a meaningful list of programs that fully represent the input of the user and maintain the compliance with the inbuilt database.
Other popular artificial intelligence systems such as Google Voice and Alexa translate natural language to queries for the search engine system which is not useful for enterprise applications since most enterprises use a private database whose schema is not available until the program is deployed on their system / server.
Moreover most of these systems are either theories or large scale deployments which are not open source or available for public use either. The implementations made for the related papers only focus on the particular of domain they are solving and also are hard to salvage from the documents of concerned organizations / colleges. The projects deployed at a large scale are also not available for in-depth research or for personal or professional deployments.
2.2 Motivation
The motivation for this project is the need for an open source system that can be deployed in small environments and can also be scaled to bigger environments. Most of the existing systems listed above share this problem. As more and more enterprises start to realize the importance of big data, they are starting to embrace the fact that they need to store and index huge amounts of data. This data then needs to be processed by efficient systems which use the big data approach to organize data in a human comprehensible format. However such big data applications lack the key approach of interacting with the user in a more user friendly and natural way so as to allow non-technical personnel available in such enterprises to also effectively make use of the huge amount of data collected and stored.
Also, with the evolving nature of artificial intelligence domain, more and more applications are now focussed on providing an assistant to the user which handles all the hard work of understanding various versions of input to the system and providing the desired output in a way that is more easily grasped and can easily be used by the users available in various demographics. This project would thus be an addition to such systems where our system would allow the user to run queries on any enterprise based database using just natural questions and queries instead of the long process of grasping and having a good understanding of the DSL before understanding the required result format and then using such information to build a program / query to run on such data.
2.3 Proposed System
The system is built on the MVC (Model View Controller) architecture where the controller of the system is a JavaScript script which allows the user’s speech to be converted to text on the user’s system itself. Then this text is passed on to the model which is the query synthesizer comprising of a set of java files which do all the work from extracting keywords, to comparing them against the database schema and finally ranking the generated queries. The ranked query strings are then exported to the view of the system which is a webpage which allows the user to view and select the desired queries.
The selected query is returned to the servlet controller which passes the query to a model class which accesses the database and return the desired result to the view which is a web page which is populated with the result and then served to the user. Thus the system shall be deployable to small and big environments as a server application on the local server used by the enterprise. This will also be beneficial since most enterprises host their database onto the serve and so our system will easily be able to interface with the database and generate the required result.
Chapter – 3
DESIGN
3.1 Methodology
Fig.3 1:Activity Diagram
In voice base query implementation the client will speak in normal language then JavaScript will allow the user’s speech to be converted to text on the user’s system itself with the help of web speech API. Then this text is passed to query synthesizer comprising of a set of java files which do all the work of extracting keywords with the help of Stanford Parser Tool.
After this, the extracted keywords are matched with the database schema and finally ranking is done on the generated queries by using various algorithms. Then the ranked query strings are run on the database and the output is visible to the user on a web page.
3.2 Use Case Diagram
Title: Natural Language
Purpose: To provide easy access to Enterprise Database.
Overview: User speaks in natural language and that natural language helps in providing easy access to Enterprise Database.
Title: Parser.
Purpose: To extract parts of speech from the given input sentence.
Overview: Parser reads the input sentence and processes it and extracts the parts of speech from the sentence.
Title: Query.
Purpose: To retrieve data from the database.
Overview: After the system generates a query, it is then passed in the database to fetch data.
Title: Ranking Query.
Purpose: To find the best fitting query.
Overview: System generates multiple queries, and then by ranking them we decide which one satisfies the user requirements the best and then we put that query in the database.
Fig.3 2: Use Case Diagram
3.3 Class Diagram
Fig.3 3:Class diagram
The system comprises of 5 bean classes namely Tables, Columns, Values, ForeignKey, and Query which are used to save the data about the tables and the keywords used to identify them. The DBHelper class allows us to handle all the comparison and interfacing required with the database. It gets a metadata object of the database information and uses it to compare keywords against the Database schema. The keyword extractor class uses the POS tagger libraries and generates the tagged sentence from the user’s input while the QueryGenrator class handles all the generation processes from the tables and columns identified from the keywords.
3.4 Context Level data flow diagram
Fig.3 4: Level 0 DFD
|
|
|
Fig.3 5: Level 1 DFD
The whole flow of the process starts with input processing as the name suggests is for taking input from the user and processes it using the parser. Then comes Query synthesizer phase which consist of query generation, ranking the query and applying it on the database. The final Output phase is just for the display of the result fetched from the database. The result will be put in an html page and displayed on a web platform. |
3.5 Component Diagram
|
Fig.3 6 : Component Diagram |
The core component of the system are Database, UI, Query Synthesizer and Speech to Text.
In this the Speech to Text is used so that users can give input in form of a voice based instruction which can later be processed as a query by extracting the important parts of speech. The query synthesizer is used to convert the user input in the form of a query which can be applied on a database. Database is used on which query will be run down to extract information. UI is for the users to interact with system easily which uses a voice based input system.
3.6 Deployment Diagram
The whole deployment is divided into three phases which are Input Processing, Query Synthesizer and displaying the output to user. Input Processing as the name suggests is for taking input from the user and process it using the parser. Query synthesizer phase consist of query generation, ranking the query and applying it on the database. The final Output phase is just for the display of the result fetched from the database. The result will be put in an html page and displayed on a web platform.
|
Fig.3 7: Deployment diagram |
Chapter - 4
IMPLEMENTATION
Controller
The controller in our system is a snippet of JavaScript code which leverages the power of modern web browsers and accept the user inputs as an audio and converts it to text. This is helpful since our system is built to run as a server application and so the speech to text processing can easily be outsourced to the clients device. A major feature of the web speech API that we used in our system is that it supports various accents for English and other languages, this not only saves us the work of training our controller but also allows our project to be extended in the future to even more languages. The web speech API is integrated into a user input web page where once the user is satisfied with his input he can click the submit button and send the input to the model; this is necessary since speech to text can sometimes be spotty in its output. The reset button thus allows the user to restate the input in a better formatted way.
Model
The model consist of a package of java classes and some libraries stored on a J2EE deployment war file. So that it can be easily deployed on enterprise servers. The libraries consist of POS(Part of Speech), tagging JAR files which we obtain from the Stanford NLP Project. The advantages of this library is that it handles all the Natural Language Processing efficiently and provides a string output where each word in the sentence is tagged with an identifier. The proper nouns are tagged with a /NN identifier. These proper nouns are taken as keywords for our projects engine, the rest of the identifiers can be used in the future to provide a structure score for the generated queries
The keywords are then piped to an algorithm which compares the keywords against the database schema and the values stored in the database to provide a general sense about what the keyword means, if the keyword of its synonyms are:-
similar to a tables name the user’s desired result might be the tables primary key or any of its column the query generated with the result as the primary key is ranked higher then the rest of the queries generated by this process
similar to a columns name in which case either the columns entire content are assumed to be the desired output or a set of values based on the constraints from the other keywords are extracted from the columns.
The same as a value in the database in which case the keyword is taken as a constraint such as where a=b; a>b; a<b.
The tables collected from the above process are then ranked by the number of relations among them identified using foreign keys, if the foreign key of a table from our collections points to another table in our collection it is ranked higher then the tables with the foreign keys pointing outside the collection. Finally, the tables and column names except the constrains table are added to a select statement and the constraints are appended to it. Each combination of such query is displayed to the user in a ranked manned so as to allow the user to select the desired query before the rank.
View
The output to the user is displayed in the form of webpage which contains the ranked set of queries from the model and allows the user to select the desired queries, if the user is unaware of the DSL the first query is automatically send to the database, since the queries are ranked there is a good probability that the query which has been run will produce the desired result.
The selected query is sent to the database using a small java file in the model where the database name and the connection to it can be changed as per the environment requirements. The output of the query is again added to the webpage in the view and displayed to the user. If the user is unsatisfied with the result he is asked to reframe the input and start the process all over again.
Chapter – 5
RESULT
The database we choose for testing our system/platform was a pokemon database which consisted of every kind of detail about all the pokemon’s from across the generations, we choose this database because it has huge amount of data and a large number of relations due to which it resembles an enterprise deployed database. The database we selected followed all the normal form of definitions which is a norm in enterprise application
Fig.5 1:Controller
We modified the Web speech API Demo given by Google as a sample on the Google chrome browser product page to include a submit button which sends the users input to our model
Fig.5 2:Model
Stanford NLP POS tagger libraries were added to provide identifiers for keyword in the users input from the controller
Fig.5 3:Ranked Queries
The queries were generated and ranked based on the number of relations in the used table.
Fig.5 4:Query output
The query was run on the used database and the output was 7.
Chapter - 6
LIMITATIONS AND FUTURE ENHANCEMENTS
6.1 Limitations
The query generation process takes a lot of time to compare the keywords and this time required increases with the size of the database.
The engine only works for simple queries. For any complex queries to be generated, the description provided by user should be more descriptive.
The system does not currently support multiple sentences of natural language as an input.
The structure of the user’s input is not taken into count to generate queries with more specificity.
6.2 Future Prospects
The database can be indexed to replace the keyword comparison time for only the initialization of the system.
Complex queries can be processed by using machine learning to ensure better generation of queries.
The structure of the user’s input when taken into count shall be able to include multiple sentences of input and map query keywords to database variables.
A REST API can be developed to allow further integration with future applications based on our system
Chapter - 7
CONCLUSION
Availability of good speech to text APIs are sparse. Google had developed a multi lingual API but now has stopped support for it and it has since been deprecated. Other popularly used APIs such as the Voce library do not support multiple accents and hence could not be used in our platform without training the library for Indian accent. The Web Speech API is the one we used in our platform because it fit our requirement of performing the transformation of speech to text on the user’s device. It however is not supported on normal java platforms which is a drawback we did not face.
The algorithms provided in papers and from literature review of previously implemented systems showed that most of such systems are based on the assumption that the database schema will not change with propagation from one environment to the other. We thus, developed a lightweight platform which allowed the system to deployed on all types of environments or
Our system is currently based on simple algorithm of keyword to database comparison which can be extended to include indexes for database schema and state (with values). Also, the system can be improved by using machine learning and generating structure scores for the generated queries to allow the system to rank the desired query higher than the others. Also since our system’s main usage is to be used in various environments, an API to access the functionality will allow our system to be extended to various new use cases.
References
[1] Welcome to Apache OpenNLP, Available from: <https://opennlp.apache.org/>
[2] Voce: Open Source Speech Interaction, Available from: http://voce.sourceforge.net/
[3] Zohar Manna, Richard J. Wildinger, “Toward automatic program synthesis”, Communications of the ACM, vol 14 issue 3, pp 151-165, March 1971
[4] Mohammad Raza, Sumit Gulwani, Natasa Milic-Frayling, “Compositional Program Synthesis from Natural Language and Examples”, Microsoft Research Publications, March 2015
10