Adding experiments, new README, and new experiments accompanying README.

Almost done attending to #16

README.md

@@ -1,30 +1,81 @@
 # Creative Adversarial Networks
-![collage](assets/collage.png)
+![collage](assets/256_external_collage.png)
 
-*128x128 pixel Samples from CAN train on WikiART.*
+*256x256 samples directly from CAN (no cherry picking) with fixed classification network trained on WikiArt*
 
-A WIP implementation of [CAN: Creative Adversarial Networks, Generating "Art" 
-by Learning About Styles and Deviating from Style Norms](https://arxiv.org/abs/1706.07068). 
-Repo bases DCGAN implementation on [DCGAN-tensorflow](https://github.com/carpedm20/DCGAN-tensorflow) 
-with modifications to reduce checkerboard artifacts according to [this 
-distill article](https://distill.pub/2016/deconv-checkerboard/)
 
-The paper authors basically modified the GAN objective to encourage the network to deviate away from art norms.
+An implementation of [CAN: Creative Adversarial Networks, Generating "Art" 
+by Learning About Styles and Deviating from Style Norms](https://arxiv.org/abs/1706.07068) with a variation that improves sample variance and quality significantly.
+
+Repo based on [DCGAN-tensorflow](https://github.com/carpedm20/DCGAN-tensorflow).
+
+<!-- with modifications to reduce checkerboard artifacts according to [this --> 
+<!-- distill article](https://distill.pub/2016/deconv-checkerboard/) -->
+
 
 ## Getting the Dataset
-We used this compiled [wikiart](https://www.wikiart.org/) dataset 
+We used the [wikiart](https://www.wikiart.org/) dataset 
 [available here](https://github.com/cs-chan/ICIP2016-PC/tree/f5d6f6b58a6d8a4bd05aaaedd9688d08c02df8f2/WikiArt%20Dataset). 
-Using the dataset 
-is subject to wikiart's [terms of use](https://www.wikiart.org/en/terms-of-use)
+Using the dataset is subject to wikiart's [terms of use](https://www.wikiart.org/en/terms-of-use)
 
-Extract the dataset, then set the path in `train.sh`
+```
+mkdir data
+cd data
+wget http://www.cs-chan.com/source/ICIP2017/wikiart.zip
+unzip wikiart.zip
+```
 
-## Training a DCGAN model
-Edit the parameters of train.sh then
+## Getting pretrained models
+We uploaded all of our models to this [google drive folder](https://drive.google.com/open?id=1FNDxvpb_UY5MZ3zBnOOfGDQCXzeE7hbs)
+
+## Training a CAN model from scratch (architecture used in the paper)
+```
+bash experiments/train_can_paper.sh # must run from the root directory of the project
+```
+## Evaluating an existing CAN model
+```
+# make sure that load_dir acts correctly
+bash experiments/eval_can_paper.sh
+```
+
+# External Style Classification network
+We ran an experiment where we trained an inception resnet to classify style (60% accuracy) 
+and then used this for the style classification loss, removing the need to learn the layers
+in the discriminator. We hold the style classification network constant, so the style distribution
+doesn't change as the generator improves. We found that this improved the quality and diversity
+of our samples.
+
+## Training CAN with External Style Network
+```
+# make sure that `style_net_checkpoint` is set correctly, or you will error out
+bash experiment/train_can_external_style.sh
+```
+
+## Training the (ImageNet pre-trained) Inception Resnet
+Everything you need should be included in the script. The gist is that it converts the wikiart images into tf records
+trains the last layer of the model on these images, then fine-tunes the entire model for 100 epochs, at the end of which
+you should get roughlyy 60% validation accuracy. Since we're looking to generate artwork, this gives us a
+level of accuracy that is sufficient to try and generate new artwork.
+```
+cd slim/
+vim finetune_inception_resnet_v2_on_wikiart.sh # edit INPUT_DATASET_DIR to match the location of where you downloaded wikiart
+bash finetune_inception_resnet_v2_on_wikiart.sh
+```
+## Evaluating CAN with External Style Network
 ```
-bash train.sh
+# make sure that `style_net_checkpoint` and `load_dir` point to the downloaded models. 
+bash eval_can_external_style.sh
 ```
+
+## Experiments
+We have run a variety of experiments, all of which are available in the `experiments/` directory.
+## Authors 
+[Phillip Kravtsov](https://github.com/phillip-kravtsov)
+
+[Phillip Kuznetsov](https://github.com/philkuz)
+
 ## Citation
+
 If you use this implementation in your own work please cite the following
 ```
 @misc{2017cans,
@@ -35,12 +86,3 @@ If you use this implementation in your own work please cite the following
   note = {commit xxxxxxx}
 }
 ```
-## Authors 
-[Phillip Kravtsov](https://github.com/phillip-kravtsov)
-
-[Phillip Kuznetsov](https://github.com/philkuz)
-
-
-
-
-

experiments/eval_can_external_style.sh

@@ -0,0 +1,24 @@
+# trains gan with an outside can network instead of having the discriminator learn style classification
+export PYTHONPATH="slim/:$PYTHONPATH"
+export CUDA_VISIBLE_DEVICES=0
+BATCH_SIZE=16
+python3 main.py \
+--epoch 25 \
+--learning_rate .0001 \
+--beta 0.5 \
+--batch_size $BATCH_SIZE \
+--sample_size $BATCH_SIZE \
+--input_height 256 \
+--output_height 256 \
+--lambda_val 1.0 \
+--smoothing 1.0 \
+--use_resize True \
+--dataset wikiart \
+--input_fname_pattern */*.jpg \
+--crop False \
+--visualize False \
+--use_s3 False \
+--can True \
+--train \
+--style_net_checkpoint "slim/logs/wikiart/inception_resnet_v2/all/bs=16,lr=0.0001,epochs=100/smol_adam_fixedLR"
+# --style_net_checkpoint "logs/inception_resnet_v2/"

experiments/eval_can_paper.sh

@@ -0,0 +1,20 @@
+# export CUDA_VISIBLE_DEVICES=0 # edit this if you want to limit yourself to GPU
+export PYTHONPATH="slim/:$PYTHONPATH"
+python3 main.py \
+--epoch 25 \
+--learning_rate .0001 \
+--beta 0.5 \
+--batch_size 16 \
+--sample_size 16 \
+--input_height 256 \
+--output_height 256 \
+--lambda_val 1.0 \
+--smoothing 1.0 \
+--use_resize True \
+--dataset wikiart \
+--input_fname_pattern */*.jpg \
+--load_dir "logs/can_paper"
+--crop False \
+--visualize False \
+--can True \
+--train False

experiments/train_can_external_style.sh

@@ -0,0 +1,24 @@
+# trains gan with an outside can network instead of having the discriminator learn style classification
+export PYTHONPATH="slim/:$PYTHONPATH"
+export CUDA_VISIBLE_DEVICES=0
+BATCH_SIZE=16
+python3 main.py \
+--epoch 25 \
+--learning_rate .0001 \
+--beta 0.5 \
+--batch_size $BATCH_SIZE \
+--sample_size $BATCH_SIZE \
+--input_height 256 \
+--output_height 256 \
+--lambda_val 1.0 \
+--smoothing 1.0 \
+--use_resize True \
+--dataset wikiart \
+--input_fname_pattern */*.jpg \
+--crop False \
+--visualize False \
+--use_s3 False \
+--can True \
+--train False \
+--load_dir "logs/can_external_style" \ 
+--style_net_checkpoint "logs/inception_resnet_v2/"

experiments/train_can_paper.sh

@@ -0,0 +1,19 @@
+# export CUDA_VISIBLE_DEVICES=0 # edit this if you want to limit yourself to GPU
+export PYTHONPATH="slim/:$PYTHONPATH"
+python3 main.py \
+--epoch 25 \
+--learning_rate .0001 \
+--beta 0.5 \
+--batch_size 16 \
+--sample_size 16 \
+--input_height 256 \
+--output_height 256 \
+--lambda_val 1.0 \
+--smoothing 1.0 \
+--use_resize True \
+--dataset wikiart \
+--input_fname_pattern */*.jpg \
+--crop False \
+--visualize False \
+--can True \
+--train \

experiments/wiki_can_256.sh

@@ -1,9 +1,9 @@
-export CUDA_VISIBLE_DEVICES=1
+# export CUDA_VISIBLE_DEVICES=0 # edit this if you want to limit yourself to GPU
 python3 main.py \
 --epoch 25 \
 --learning_rate .0001 \
 --beta 0.5 \
---batch_size 15 \
+--batch_size 16 \
 --sample_size 16 \
 --input_height 256 \
 --output_height 256 \

experiments/wiki_external_can.sh

@@ -21,3 +21,4 @@ python3 main.py \
 --can True \
 --train \
 --style_net_checkpoint "slim/logs/wikiart/inception_resnet_v2/all/bs=16,lr=0.0001,epochs=100/smol_adam_fixedLR"
+# --style_net_checkpoint "logs/inception_resnet_v2/"

main.py

@@ -28,6 +28,7 @@
 flags.DEFINE_string("log_dir", 'logs', "Directory to store logs [logs]")
 flags.DEFINE_string("checkpoint_dir", None, "Directory name to save the checkpoints [<FLAGS.log_dir>/checkpoint]")
 flags.DEFINE_string("sample_dir", None, "Directory name to save the image samples [<FLAGS.log_dir>/samples]")
+flags.DEFINE_string("load_dir", None, "Directory that specifies checkpoint to load")
 flags.DEFINE_boolean("train", False, "True for training, False for testing [False]")
 flags.DEFINE_boolean("crop", False, "True for training, False for testing [False]")
 flags.DEFINE_boolean("visualize", False, "True for visualizing, False for nothing [False]")

model.py

@@ -559,8 +559,10 @@ def get_parent_path(path):
 
         checkpoint_dir  = os.path.join(path, last_, 'checkpoint')
       else:
-        checkpoint = None
+        checkpoint_dir = None
 
+    if config.load_dir:
+      checkpoint_dir = config.load_dir
 
     ckpt = tf.train.get_checkpoint_state(checkpoint_dir)
     if ckpt and ckpt.model_checkpoint_path:
@@ -575,6 +577,8 @@ def get_parent_path(path):
         print(" [*] Failed to find a sample_z")
         sample_z = None
       return True, counter, sample_z
+    elif config.load_dir:
+      raise ValueError(" [*] Failed to find the load_dir")
     else:
       print(" [*] Failed to find a checkpoint")
       return False, 0, None

slim/finetune_inception_resnet_v2_on_wikiart.sh

@@ -49,74 +49,74 @@ if [ ! -f ${PRETRAINED_CHECKPOINT_DIR}/${MODEL_NAME}.ckpt ]; then
 fi
 
 # # Download the dataset
-# python download_and_convert_data.py \
-#   --dataset_name=wikiart \
-#   --dataset_dir=${DATASET_DIR}
-#   --input_dataset_dir=${INPUT_DATASET_DIR}
+python download_and_convert_data.py \
+  --dataset_name=wikiart \
+  --dataset_dir=${DATASET_DIR}
+  --input_dataset_dir=${INPUT_DATASET_DIR}
 
 # @philkuz I use this to create a nice initialization - haven't tried random
 # TODO try out if your'e curious to see whether random initialization of last
 # layer makes sense in this case.
-# Fine-tune only the new layers for 1000 steps.
-# python3 train_image_classifier.py \
-#   --train_dir=${TRAIN_DIR} \
-#   --dataset_name=wikiart \
-#   --dataset_split_name=train \
-#   --dataset_dir=${DATASET_DIR} \
-#   --model_name=${MODEL_NAME} \
-#   --checkpoint_path=${PRETRAINED_CHECKPOINT_DIR}/${MODEL_NAME}.ckpt \
-#   --checkpoint_exclude_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \
-#   --trainable_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \
-#   --max_number_of_steps=10000 \
-#   --batch_size=32 \
-#   --learning_rate=0.01 \
-#   --learning_rate_decay_type=fixed \
-#   --save_interval_secs=300 \
-#   --save_summaries_secs=60 \
-#   --log_every_n_steps=200 \
-#   --optimizer=rmsprop \
-#   --train_image_size=256 \
-#   --weight_decay=0.00004
+# Fine-tune only the last layer for 1000 steps.
+python3 train_image_classifier.py \
+  --train_dir=${TRAIN_DIR} \
+  --dataset_name=wikiart \
+  --dataset_split_name=train \
+  --dataset_dir=${DATASET_DIR} \
+  --model_name=${MODEL_NAME} \
+  --checkpoint_path=${PRETRAINED_CHECKPOINT_DIR}/${MODEL_NAME}.ckpt \
+  --checkpoint_exclude_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \
+  --trainable_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \
+  --max_number_of_steps=10000 \
+  --batch_size=32 \
+  --learning_rate=0.01 \
+  --learning_rate_decay_type=fixed \
+  --save_interval_secs=300 \
+  --save_summaries_secs=60 \
+  --log_every_n_steps=200 \
+  --optimizer=rmsprop \
+  --train_image_size=256 \
+  --weight_decay=0.00004
 
 # Run evaluation.
-# python3 eval_image_classifier.py \
-#   --checkpoint_path=${TRAIN_DIR} \
-#   --eval_dir=${TRAIN_DIR} \
-#   --dataset_name=wikiart \
-#   --dataset_split_name=validation \
-#   --dataset_dir=${DATASET_DIR} \
-#   --model_name=${MODEL_NAME} \
-#   --eval_image_size=256
+python3 eval_image_classifier.py \
+  --checkpoint_path=${TRAIN_DIR} \
+  --eval_dir=${TRAIN_DIR} \
+  --dataset_name=wikiart \
+  --dataset_split_name=validation \
+  --dataset_dir=${DATASET_DIR} \
+  --model_name=${MODEL_NAME} \
+  --eval_image_size=256
 
 # Fine-tune all the new layers for 500 steps.
 NUM_EPOCHS=100
 BATCH_SIZE=16
-EXPERIMENT_NAME=smol_adam_fixedLR
+EXPERIMENT_NAME=inception_resnet_v2
 LR=0.0001 \
 
 TRAIN_DIR=logs/wikiart/inception_resnet_v2/experiments/${EXPERIMENT_NAME}/bs=${BATCH_SIZE},lr=${LR},epochs=${NUM_EPOCHS}/
 
-# python3 train_image_classifier.py \
-#   --train_dir=${TRAIN_DIR}/all \
-#   --dataset_name=wikiart \
-#   --dataset_split_name=train \
-#   --dataset_dir=${DATASET_DIR} \
-#   --model_name=${MODEL_NAME} \
-#   --checkpoint_path=${TRAIN_DIR} \
-#   --batch_size=${BATCH_SIZE} \
-#   --learning_rate=${LR} \
-#   --learning_rate_decay_type=fixed \
-#   --save_interval_secs=300 \
-#   --save_summaries_secs=60 \
-#   --num_epochs_per_decay=1 \
-#   --log_every_n_steps=200 \
-#   --optimizer=adam \
-#   --weight_decay=0.00004 \
-#   --experiment_name=${EXPERIMENT_NAME} \
-#   --num_epochs=${NUM_EPOCHS} \
-#   --train_image_size=256 \
-#   --continue_training False \
-#   --experiment_numbering # TODO flag to flip on experiment numbering independent of experiement name arg existing
+python3 train_image_classifier.py \
+  --train_dir=${TRAIN_DIR}/all \
+  --dataset_name=wikiart \
+  --dataset_split_name=train \
+  --dataset_dir=${DATASET_DIR} \
+  --model_name=${MODEL_NAME} \
+  --checkpoint_path=${TRAIN_DIR} \
+  --batch_size=${BATCH_SIZE} \
+  --learning_rate=${LR} \
+  --learning_rate_decay_type=fixed \
+  --save_interval_secs=300 \
+  --save_summaries_secs=60 \
+  --num_epochs_per_decay=1 \
+  --log_every_n_steps=200 \
+  --optimizer=adam \
+  --weight_decay=0.00004 \
+  --experiment_name=${EXPERIMENT_NAME} \
+  --num_epochs=${NUM_EPOCHS} \
+  --train_image_size=256 \
+  --continue_training False \
+  # --experiment_numbering # TODO flag to flip on experiment numbering independent of experiement name arg existing
 # # TODO catch the naming convention
 
 # Run evaluation.