Neural Net structures

Convolutional Neural Networks

CNLessPad

Layer	In	Out	Kernel size	Params
Conv1d	1	32	14	left_padding=10
ReLU
MaxPool1d			2	padding=1
Conv1d	32	64	8	left_padding=6
ReLU
MaxPool1d			2	padding=0
Conv1d	64	128	5	left_padding=1
ReLU
MaxPool1d			2	padding=0
Flatten
Dropout				0.5
Linear	2*32	pred_len

CNLong

Layer	In	Out	Kernel size	Params
Conv1d	1	64	12	left_padding=6
ReLU
MaxPool1d			2	padding=0
Conv1d	64	256	16	left_padding=8
ReLU
MaxPool1d			2	padding=0
GaussianNoise				0.02
Flatten
Linear	6*256	256
Dropout				0.5
ReLU
Linear	256	pred_len

Long Short-Term Memory Network

General

Layer	In	Out	Layers	Bidirectional	Params
LSTM	features * t	hidden_size * bidirectional(1 or 2) * num_layers	num_layers	bidirectional?	dropout
Flatten
GaussianNoise					noise
Dropout					dropout
Linear	hidden_size * bidirectional(1 or 2) * num_layers	pred_len

Temporal Convolutional Networks

Temporal Block

Layer	In	Out	Kernel size	Dilation	Params
Conv1D	in_channels	out_channels	kernel_size	dilation
LeftPad					kernel_size-1
ReLU
Dropout					dropout
Conv1D	in_channels	out_channels	kernel_size	dilation
LeftPad					kernel_size-1
ReLU
Dropout					dropout
DownSample	in_channels	out_channels	1		only if in_channels != out_channels

TemporalConvNet

Temporal networks are built via looping over a given amount of channels, let's name a single one nc[i]

Repeat	Layer	In	Out	Kernel size	Dilation	Params
i times	Temporal Block	nc[i-1]	nc[i]	kernel_size	2^i	dropout
1	GaussianNoise					noise
1	TakeLast					Takes last block output
1	Linear	nc[-1]	pred_len

GRU encoder-decoder

Seq2seq

GRU encoder

Layer	In	Out	Layers	Bidirectional	Params	Notes
GRU	features	-	num_layers	bidirectional?	dropout	inits hidden state

GRU decoder

Layer	In	Out	Layers	Bidirectional	Params	Notes
GRU	1	1	num_layers	bidirectional?	dropout	predicts 1 time-step
Linear	embedding_size * num_layer * bidirectional?	1

GRU decoder-encoder

Layer	Params	Notes
GRU encoder	specified above	inits hidden state
GRU decoder	specified above	uses hidden state

Decoding is repeated as many times as we want to predict forward, the hidden state used always comes from the encoder.

GRU

Layer	In	Out	Layers	Bidirectional	Params
GRU	features * t	hidden_size * bidirectional(1 or 2) * num_layers	num_layers	bidirectional?	dropout
Flatten
GaussianNoise					noise
Dropout					dropout
Linear	hidden_size * bidirectional(1 or 2) * num_layers	pred_len

ConvNetForRecursion

Layer	In	Out	Kernel size	Params
Conv1d	1	channels[0]	kernel_sizes[0]	left_padding=kernel_size//2
ReLU
MaxPool1d			2	padding=1
Conv1d	channels[0]	channels[1]	kernel_sizes[1]	left_padding=kernel_size//2
ReLU
MaxPool1d			2	padding=0
GaussianNoise				noise
Flatten
Dropout				dropout
Linear	calculated	pred_len