Update of Data Generation to accept file-based input data (#11)

* Added input-file based utternaces and entities

* Updated Data Generation Notebook to use csv-based input data

* Fixed comment from review, added ps1 to run scenarios for testing, updated documentation to use example assets in sample commands

* Updated sampel config

Author: goergenj

.gitignore

@@ -112,6 +112,8 @@ config.ini
 # Working directories
 input/
 output/
+output_files/
+assets/customer_data/
 
 # Virtual envs
 .venv/

GET_YOUR_KEYS.md

@@ -10,6 +10,7 @@ In this section, we describe how to set up ffmpeg in case you do TTS.
 #### Windows
 - Download the latest version of [ffmpeg](https://ffmpeg.org/download.html#build-windows).
 - Extract the archive locally and copy the file `bin/ffpmeg.exe` to a location of your choice, e.g. to the `assets` folder of GLUE.
+- Add the location to the system environment variables "PATH" setting.
 - Finally, copy the path and insert it in your config.ini as below.
 ```
 [driver]

README.md

@@ -166,7 +166,7 @@ This scenario describes how you can batch-transcribe audio files using GLUE. A p
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --audio C:/audio_files/ --do_transcribe
+python src/glue.py --audio assets\examples\input_files\audio --do_transcribe
 ```
 4. Wait for the run to finish.
 
@@ -190,7 +190,7 @@ This scenario describes how you can batch-synthesize text data using GLUE. A pot
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --input ../input-files/text.csv --do_synthesize
+python src/glue.py --input assets\examples\input_files\example_tts.csv --do_synthesize
 ```
 4. Wait for the run to finish.
 
@@ -227,7 +227,7 @@ This scenario shows how you can use GLUE to batch-score textual data on a LUIS-e
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --input /home/files/luis-utterances.csv --do_scoring
+python src/glue.py --input assets\examples\input_files\example_luis.csv --do_scoring
 ```
 4. Wait for the run to finish and see the command line outputs.
 
@@ -263,7 +263,7 @@ This scenario describes how you can compare already existing recognitions with a
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --input ../input/transcriptions.txt --do_evaluate
+python src/glue.py --input assets\examples\output_files\example_transcriptions_full.csv --do_evaluate
 ```
 4. Wait for the run to finish and see the command line outputs.
 
@@ -292,7 +292,7 @@ This scenario describes how you can batch-transcribe audio files and compare the
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --audio C:/audio_files/ --input C:/audio_files/transcriptions.txt --do_transcribe --do_evaluate
+python src/glue.py --audio assets\examples\input_files\audio --input assets\examples\input_files\example_stt_eval.csv --do_transcribe --do_evaluate
 ```
 4. Wait for the run to finish and see the command line outputs.
 
@@ -322,7 +322,7 @@ This scenario describes how you can batch-transcribe audio files, compare these
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --audio C:/audio_files/ --input C:/audio_files/transcriptions.txt --do_transcribe --do_evaluate --do_scoring
+python src/glue.py --audio assets\examples\input_files\audio --input assets\examples\input_files\example_stt_eval_luis.csv --do_transcribe --do_evaluate --do_scoring
 ```
 4. Wait for the run to finish and see the command line outputs.
 
@@ -363,7 +363,7 @@ This scenario describes how you can batch-transcribe audio files and score both
 2. Make sure your `.venv` is activated.
 3. Run the following command:
 ```bash
-python src/glue.py --audio C:/audio_files/ --input C:/audio_files/transcriptions.txt --do_transcribe --do_scoring
+python src/glue.py --audio assets\examples\input_files\audio --input assets\examples\input_files\example_luis.csv --do_transcribe --do_scoring
 ```
 4. Wait for the run to finish and see the command line outputs.
 

assets/examples/input_files/example-luis-app-with-generated-data.lu

@@ -0,0 +1,47 @@
+intent;text
+BookFlight;a ticket to {Airport} please
+BookFlight;could you book a flight to {Airport} on 2nd of april for me?
+BookFlight;i want a flight to {Airport}
+BookFlight;i would like to book a flight to {Airport}.
+BookFlight;no, i want to fly to {Airport}, not to {Airport}.
+BookSeat;book a seat as close as possible to the galley on my flight to {Airport}
+BookSeat;book a seat for me on my preferred seat.
+BookSeat;i need a special seat on my flight
+BookSeat;i want to have a window seat
+BookSeat;i would like to book a seat on my flight to {Airport}
+CancelFlight;cancel flight to {Airport}
+CancelFlight;cancellation of flight to {Airport}.
+CancelFlight;i need to cancel my flight to {Airport}
+CancelFlight;i want to cancel my journey to {Airport}.
+CancelFlight;please cancel my flight to {Airport}
+ChangeFlight;change flight to {Airport}.
+ChangeFlight;change my flight from {Airport} to {Airport}
+ChangeFlight;i need to change my flight from {Airport} to {Airport}
+ChangeFlight;i would like to change my flight.
+ChangeFlight;please rebook my flight to {Airport}, please.
+DepartureInfo;could you check whether my flight has a delay?
+DepartureInfo;i need the status of my flight {FlightNumber}
+DepartureInfo;is my flight on time?
+DepartureInfo;what is the status of my flight {FlightNumber}
+DepartureInfo;when does {FlightNumber} depart?
+GetEntities;{first_name} {last_name}
+GetEntities;my name is {first_name} {last_name}
+GetEntities;i am {first_name} {last_name}
+GetEntities;my firstname is {first_name}
+GetEntities;my lastname is {last_name}
+GetEntities;It's {last_name}
+GetEntities;it is {first_name}
+GetEntities;{first_name} {last_name}.
+GetEntities;my name is {first_name} {last_name}.
+GetEntities;i am {first_name} {last_name}.
+GetEntities;my firstname is {first_name}.
+GetEntities;my lastname is {last_name}.
+GetEntities;It's {last_name}.
+GetEntities;it is {first_name}.
+None;What are you talking about?
+None;You are dumb!
+None;Do you like candy?
+None;how are you doing
+None;i want to speak to a human
+None;what can you do
+None;whats the weather like today

assets/examples/input_files/example_baseutterances.csv

@@ -0,0 +1,36 @@
+entity_name;entity_value
+first_name;Peter
+first_name;Paul
+first_name;Mary
+first_name;Margret
+first_name;Theo
+first_name;Tom
+first_name;Maria
+first_name;Linda
+first_name;Lisa
+last_name;Coskun
+last_name;Delaunay
+last_name;Maier
+last_name;Ebner
+last_name;Inhestern
+last_name;Rasztawicki
+last_name;Reichenbach
+last_name;Kistner
+last_name;Mahoney
+last_name;Müller
+FlightNumber;AB1234
+FlightNumber;LH7721
+FlightNumber;AA1834556
+FlightNumber;CP9919223
+FlightNumber;LH1346
+FlightNumber;AA1994557
+FlightNumber;Ab9164
+Airport;Berlin
+Airport;Tokyo
+Airport;New York
+Airport;Tel Aviv
+Airport;London
+Airport;Paris
+Airport;Los Angeles
+Airport;Toronto
+Airport;Seattle