Recipe las-pyramidal.yaml does not work

[SrutiBh]

Hi @neubig,

I have installed all the requirements. But whenever I am trying to install xnmt by "python setup.py install". It is giving an error as "fatal: not a git repository" The error is as follows.

(base) D:\anaconda_install_3\new path\pkgs\xnmt-master>python setup.py install
checking git revision in setup.py
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
File "setup.py", line 36, in
open("./xnmt/git_rev.py", "w").write("CUR_GIT_REVISION = "" + get_git_revision() + "" # via setup.py")
TypeError: must be str, not NoneType

(base) D:\anaconda_install_3\new path\pkgs\xnmt-master>git init
Initialized empty Git repository in D:/anaconda_install_3/new path/pkgs/xnmt-master/.git/

(base) D:\anaconda_install_3\new path\pkgs\xnmt-master>python setup.py install
checking git revision in setup.py
fatal: Needed a single revision
Traceback (most recent call last):
File "setup.py", line 36, in
open("./xnmt/git_rev.py", "w").write("CUR_GIT_REVISION = "" + get_git_revision() + "" # via setup.py")
TypeError: must be str, not NoneType

My conda version is 4.6.7. Please let me know if you require any additional information. I need to fix this as soon as possible.
Thanks,
Sruti

[neubig]

@philip30 was the one who created the setup.py prompt I believe, so maybe he can help?

[philip30]

That line was actually written by @msperber.

To quickly avoid some trivial bugs, @SrutiBh can you change that line 36 to:

open("./xnmt/git_rev.py", "w").write("CUR_GIT_REVISION = \"" + str(get_git_revision()) + "\" # via setup.py")

[SrutiBh]

@philip30 @neubig @msperber The line still showing the same problem. and in next lines coming as error. I think you missed ')' after write("CUR_GIT_REVISION = "" and in python '#' is comment which is showing as error while closing bracket at 'via setup.py")'

[philip30]

Ah yes I forgot the closing bracket. Maybe the problem is because open() returns None? perhaps you can comment the entire line as that line is just for INFO only. When not using conda, xnmt setup.py should be run on the root of XNMT directory.

[SrutiBh]

@philip30 @neubig this is coming...

(base) E:\xnmt-master\examples>python -m xnmt.xnmt_run_experiments examples/01_standard.yaml
[dynet] random seed: 1416409111
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
Traceback (most recent call last):
File "e:\xnmt-master\xnmt\persistence.py", line 837, in experiment_names_from_file
with open(filename) as stream:
FileNotFoundError: [Errno 2] No such file or directory: 'examples/01_standard.yaml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\install_Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "D:\install_Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "e:\xnmt-master\xnmt\xnmt_run_experiments.py", line 123, in
sys.exit(main())
File "e:\xnmt-master\xnmt\xnmt_run_experiments.py", line 56, in main
config_experiment_names = YamlPreloader.experiment_names_from_file(args.experiments_file)
File "e:\xnmt-master\xnmt\persistence.py", line 840, in experiment_names_from_file
raise RuntimeError(f"Could not read configuration file {filename}: {e}")
RuntimeError: Could not read configuration file examples/01_standard.yaml: [Errno 2] No such file or directory: 'examples/01_standard.yaml'

(base) E:\xnmt-master\examples>

[philip30]

Hi Sruti,

If you spend a little time to read the error message, it clearly says that "FileNotFoundError: [Errno 2] No such file or directory: 'examples/01_standard.yaml'"

It is because you are running the program from "examples" directory, and the program tries to open "examples/01_standard.yaml" from there, and of course "examples/examples/01_standard.yaml" does not exists. can you try to do cd .. and run the same command again?

Thank you

[SrutiBh]

@philip30 @neubig
Hi Philip,
Thanks. It worked 👍 👍

Some other problems are occuring now. Whenever I am trying to run config.las-pyramidal.yaml from recipe it is giving error as follows. I am using different dataset and want to train my data with the same model given here using las. please have a look. I wonder if it is error from yaml file structure or my file directories or something else. Here is it showing AutoRegression error. In this case I basically want to classify data types that I have and see the accuracy.
[dynet] memory allocation done.
fatal: Not a git repository (or any of the parent directories): .git
running XNMT revision None on SRUTI on 2019-07-12 01:21:12
fatal: Not a git repository (or any of the parent directories): .git
=> Running las-pyramidal
Traceback (most recent call last):
File "D:\install_anaconda3\Scripts\xnmt-script.py", line 11, in
load_entry_point('xnmt', 'console_scripts', 'xnmt')()
File "e:\my xnmt\xnmt-master\xnmt\xnmt_run_experiments.py", line 108, in main
raise e
File "e:\my xnmt\xnmt-master\xnmt\xnmt_run_experiments.py", line 96, in main
experiment = initialize_if_needed(uninitialized_exp_args)
File "e:\my xnmt\xnmt-master\xnmt\persistence.py", line 1411, in initialize_if_needed
return _YamlDeserializer().initialize_if_needed(root)
File "e:\my xnmt\xnmt-master\xnmt\persistence.py", line 1141, in initialize_if_needed
else: return self.initialize_object(deserialized_yaml_wrapper=obj)
File "e:\my xnmt\xnmt-master\xnmt\persistence.py", line 1173, in initialize_object
self.check_args(self.deserialized_yaml)
File "e:\my xnmt\xnmt-master\xnmt\persistence.py", line 1190, in check_args
_check_serializable_args_valid(node)
File "e:\my xnmt\xnmt-master\xnmt\persistence.py", line 479, in _check_serializable_args_valid
f"'{name}' is not a accepted argument of {type(node).name}.init(). Valid are {list(init_args.keys())}")
ValueError: 'rnn_layer' is not a accepted argument of AutoRegressiveDecoder.init(). Valid are ['input_dim', 'trg_embed_dim', 'input_feeding', 'bridge', 'rnn', 'transform', 'scorer', 'truncate_dec_batches']

[philip30]

Hi Sruti,

The error was because that the 'rnn_layers' fields are not found in the constructor of AutoRegressiveDecoder object. Can you reflect each object from the yaml file to the code and debug it? I believe if you are using a specific version of the yaml file which is working and written by @msperber, you should also checkout to the branch that the code was working. (I believe it is now somewhat diverges...)

[msperber]

Hi Sruti,

it seems that this recipe was not adjusted to the latest change in how decoders are specified. I can't send code right now unfortunately, but if you look in the examples (e.g. here: https://github.com/neulab/xnmt/blob/master/examples/01_standard.yaml ), you can see that all that needs to happen is to move a few things inside the decoder object.

-- Matthias
(this comment is "not a contribution")

[SrutiBh]

Hi @msperber, @philip30
Can you specifically tell me where should I edit exactly and what should I include? I am scared that I am confused. I tried some of the example yaml files initially to test xnmt but they were not working so I wrote my own to make h5 files. As @philip30 told, rnn_layers' are not in the constructor of AutoRegressiveDecoder object. So where should I edit to to fix it as well? And @philip30 I am not getting how should I debug? Are you asking me to debug las-pyramidal's every line differently? If so can you give me an example? I am not sure how to do so.
Here is the yaml file code I am using

las-pyramidal: !Experiment
exp_global: !ExpGlobal
dropout: 0.2
default_layer_dim: 512
placeholders:
DATA_DIR: E:\my xnmt\xnmt-master\recipes\mande
preproc: !PreprocRunner
overwrite: False
tasks:
- !PreprocExtract
in_files:
- '{DATA_DIR}/for_auditory_roughness/db/train.yaml'
- '{DATA_DIR}/for_auditory_roughness/db/test.yaml'
- '{DATA_DIR}/for_auditory_roughness/db/train.yaml'
out_files:
- '{DATA_DIR}/for_auditory_roughness/feat/devout.h5'
- '{DATA_DIR}/for_auditory_roughness/feat/testout.h5'
- '{DATA_DIR}/for_auditory_roughness/feat/trainout.h5'
specs: !MelFiltExtractor {}
model: !DefaultTranslator
src_embedder: !NoopEmbedder
emb_dim: 40
encoder: !ModularSeqTransducer
modules:
- !PyramidalLSTMSeqTransducer
layers: 4
reduce_factor: 2
downsampling_method: concat
input_dim: 40
hidden_dim: 512
attender: !MlpAttender
hidden_dim: 128
trg_embedder: !SimpleWordEmbedder
emb_dim: 64
word_dropout: 0.1
fix_norm: 1
decoder: !AutoRegressiveDecoder
rnn_layer: !UniLSTMSeqTransducer
layers: 1
hidden_dim: 512
input_feeding: True
bridge: !CopyBridge {}
scorer: !Softmax
label_smoothing: 0.1
src_reader: !H5Reader
transpose: true
trg_reader: !PlainTextReader
vocab: !Vocab
vocab_file: '{EXP_DIR}/vocab.char'
output_proc: join-char
train: !SimpleTrainingRegimen
src_file: '{DATA_DIR}/for_auditory_roughness/feat/trainout.h5'
trg_file: '{DATA_DIR}/for_auditory_roughness/transcript/train.char'
max_src_len: 1500
max_trg_len: 350
run_for_epochs: 500
batcher: !WordSrcBatcher
avg_batch_size: 24
pad_src_to_multiple: 8
trainer: !AdamTrainer
alpha: 0.001
lr_decay: 0.5
lr_decay_times: 3
patience: 8
initial_patience: 15
dev_every: 0
restart_trainer: True
dev_tasks:
- !AccuracyEvalTask
eval_metrics: wer,cer
src_file: &dev_src '{DATA_DIR}/for_auditory_roughness/feat/devout.h5'
ref_file: '{DATA_DIR}/for_auditory_roughness/transcript/dev.words'
hyp_file: '{EXP_DIR}/logs/{EXP}.dev_hyp'
inference: !AutoRegressiveInference
max_src_len: 1500
post_process: join-char
search_strategy: !BeamSearch
max_len: 500
beam_size: 20
len_norm: !PolynomialNormalization
apply_during_search: true
m: 1.5
- !LossEvalTask
max_src_len: 1500
src_file: *dev_src
ref_file: '{DATA_DIR}/for_auditory_roughness/transcript/dev.char'
evaluate:
- !AccuracyEvalTask
eval_metrics: wer,cer
src_file: '{DATA_DIR}/for_auditory_roughness/feat/testout.h5'
ref_file: '{DATA_DIR}/for_auditory_roughness/transcript/test.words'
hyp_file: '{EXP_DIR}/logs/{EXP}.test_hyp'
inference: !AutoRegressiveInference
max_src_len: 1500
post_process: join-char
search_strategy: !BeamSearch
max_len: 500
beam_size: 20
len_norm: !PolynomialNormalization
apply_during_search: true
m: 1.5
Thanking you,
Sruti

[philip30]

@SrutiBh it is helpful to read the documentation https://xnmt.readthedocs.io/en/latest/getting_started.html so you won't need my help for future implementation if you find any problem. (If you plan to use the code, I think it is essential to understand how xnmt works). And I am sorry, I can't help with the debuging, as the logic of that particular code is not written by me (and I might make mistake in correcting that, which risk I don't want to take). BTW, as it says, rnn_layers might refer to the layer numbers of the RNN, so if you wanna fix, you might want to correctly reflect that logic to the code.

[SrutiBh]

@philip30 the rnn bugs are still there. I tried to find out if there is any attribute name changes or anything but I did not found any problem. Can you tell me who wrote las-pyramidal so that I can discuss with him? However I see in place of rnn-layer it should be rnn...and I think for Listen-attend-spell paper architecture we need to use BiLSTMSeqTransducer but in the las-pyramidal it is structured for UniLSTMSeqTransducer. Do both of them refer to pyramidal lstm?

[msperber]

@SrutiBh you are correct that the rnn_layer parameter needs to be renamed. The config specifies a (bidirectional) pyramidal LSTM as encoder, and a unidirectional LSTM as decoder. This is correct, as on the decoder side only unidirectional LSTMs can be used. Hope this helps!

-- Matthias
(this comment is "not a contribution")