diff --git a/README.md b/README.md index 20d8de4..b486489 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,81 @@ # Creative Adversarial Networks -![collage](assets/collage.png) +![collage](assets/256_external_collage.png) -*128x128 pixel Samples from CAN train on WikiART.* +*256x256 samples directly from CAN (no cherry picking) with fixed classification network trained on WikiArt* -A WIP implementation of [CAN: Creative Adversarial Networks, Generating "Art" -by Learning About Styles and Deviating from Style Norms](https://arxiv.org/abs/1706.07068). -Repo bases DCGAN implementation on [DCGAN-tensorflow](https://github.com/carpedm20/DCGAN-tensorflow) -with modifications to reduce checkerboard artifacts according to [this -distill article](https://distill.pub/2016/deconv-checkerboard/) -The paper authors basically modified the GAN objective to encourage the network to deviate away from art norms. +An implementation of [CAN: Creative Adversarial Networks, Generating "Art" +by Learning About Styles and Deviating from Style Norms](https://arxiv.org/abs/1706.07068) with a variation that improves sample variance and quality significantly. + +Repo based on [DCGAN-tensorflow](https://github.com/carpedm20/DCGAN-tensorflow). + + + + ## Getting the Dataset -We used this compiled [wikiart](https://www.wikiart.org/) dataset +We used the [wikiart](https://www.wikiart.org/) dataset [available here](https://github.com/cs-chan/ICIP2016-PC/tree/f5d6f6b58a6d8a4bd05aaaedd9688d08c02df8f2/WikiArt%20Dataset). -Using the dataset -is subject to wikiart's [terms of use](https://www.wikiart.org/en/terms-of-use) +Using the dataset is subject to wikiart's [terms of use](https://www.wikiart.org/en/terms-of-use) -Extract the dataset, then set the path in `train.sh` +``` +mkdir data +cd data +wget http://www.cs-chan.com/source/ICIP2017/wikiart.zip +unzip wikiart.zip +``` -## Training a DCGAN model -Edit the parameters of train.sh then +## Getting pretrained models +We uploaded all of our models to this [google drive folder](https://drive.google.com/open?id=1FNDxvpb_UY5MZ3zBnOOfGDQCXzeE7hbs) + +## Training a CAN model from scratch (architecture used in the paper) +``` +bash experiments/train_can_paper.sh # must run from the root directory of the project +``` +## Evaluating an existing CAN model +``` +# make sure that load_dir acts correctly +bash experiments/eval_can_paper.sh +``` + +# External Style Classification network +We ran an experiment where we trained an inception resnet to classify style (60% accuracy) +and then used this for the style classification loss, removing the need to learn the layers +in the discriminator. We hold the style classification network constant, so the style distribution +doesn't change as the generator improves. We found that this improved the quality and diversity +of our samples. + +## Training CAN with External Style Network +``` +# make sure that `style_net_checkpoint` is set correctly, or you will error out +bash experiment/train_can_external_style.sh +``` + +## Training the (ImageNet pre-trained) Inception Resnet +Everything you need should be included in the script. The gist is that it converts the wikiart images into tf records +trains the last layer of the model on these images, then fine-tunes the entire model for 100 epochs, at the end of which +you should get roughlyy 60% validation accuracy. Since we're looking to generate artwork, this gives us a +level of accuracy that is sufficient to try and generate new artwork. +``` +cd slim/ +vim finetune_inception_resnet_v2_on_wikiart.sh # edit INPUT_DATASET_DIR to match the location of where you downloaded wikiart +bash finetune_inception_resnet_v2_on_wikiart.sh +``` +## Evaluating CAN with External Style Network ``` -bash train.sh +# make sure that `style_net_checkpoint` and `load_dir` point to the downloaded models. +bash eval_can_external_style.sh ``` + +## Experiments +We have run a variety of experiments, all of which are available in the `experiments/` directory. +## Authors +[Phillip Kravtsov](https://github.com/phillip-kravtsov) + +[Phillip Kuznetsov](https://github.com/philkuz) + ## Citation + If you use this implementation in your own work please cite the following ``` @misc{2017cans, @@ -35,12 +86,3 @@ If you use this implementation in your own work please cite the following note = {commit xxxxxxx} } ``` -## Authors -[Phillip Kravtsov](https://github.com/phillip-kravtsov) - -[Phillip Kuznetsov](https://github.com/philkuz) - - - - - diff --git a/assets/collage.png b/assets/collage.png deleted file mode 100644 index 5f14b20..0000000 Binary files a/assets/collage.png and /dev/null differ diff --git a/experiments/eval_can_external_style.sh b/experiments/eval_can_external_style.sh new file mode 100644 index 0000000..1d685ec --- /dev/null +++ b/experiments/eval_can_external_style.sh @@ -0,0 +1,24 @@ +# trains gan with an outside can network instead of having the discriminator learn style classification +export PYTHONPATH="slim/:$PYTHONPATH" +export CUDA_VISIBLE_DEVICES=0 +BATCH_SIZE=16 +python3 main.py \ +--epoch 25 \ +--learning_rate .0001 \ +--beta 0.5 \ +--batch_size $BATCH_SIZE \ +--sample_size $BATCH_SIZE \ +--input_height 256 \ +--output_height 256 \ +--lambda_val 1.0 \ +--smoothing 1.0 \ +--use_resize True \ +--dataset wikiart \ +--input_fname_pattern */*.jpg \ +--crop False \ +--visualize False \ +--use_s3 False \ +--can True \ +--train \ +--style_net_checkpoint "slim/logs/wikiart/inception_resnet_v2/all/bs=16,lr=0.0001,epochs=100/smol_adam_fixedLR" +# --style_net_checkpoint "logs/inception_resnet_v2/" diff --git a/experiments/eval_can_paper.sh b/experiments/eval_can_paper.sh new file mode 100644 index 0000000..8c9ab7e --- /dev/null +++ b/experiments/eval_can_paper.sh @@ -0,0 +1,20 @@ +# export CUDA_VISIBLE_DEVICES=0 # edit this if you want to limit yourself to GPU +export PYTHONPATH="slim/:$PYTHONPATH" +python3 main.py \ +--epoch 25 \ +--learning_rate .0001 \ +--beta 0.5 \ +--batch_size 16 \ +--sample_size 16 \ +--input_height 256 \ +--output_height 256 \ +--lambda_val 1.0 \ +--smoothing 1.0 \ +--use_resize True \ +--dataset wikiart \ +--input_fname_pattern */*.jpg \ +--load_dir "logs/can_paper" +--crop False \ +--visualize False \ +--can True \ +--train False diff --git a/experiments/train_can_external_style.sh b/experiments/train_can_external_style.sh new file mode 100644 index 0000000..9e4836c --- /dev/null +++ b/experiments/train_can_external_style.sh @@ -0,0 +1,24 @@ +# trains gan with an outside can network instead of having the discriminator learn style classification +export PYTHONPATH="slim/:$PYTHONPATH" +export CUDA_VISIBLE_DEVICES=0 +BATCH_SIZE=16 +python3 main.py \ +--epoch 25 \ +--learning_rate .0001 \ +--beta 0.5 \ +--batch_size $BATCH_SIZE \ +--sample_size $BATCH_SIZE \ +--input_height 256 \ +--output_height 256 \ +--lambda_val 1.0 \ +--smoothing 1.0 \ +--use_resize True \ +--dataset wikiart \ +--input_fname_pattern */*.jpg \ +--crop False \ +--visualize False \ +--use_s3 False \ +--can True \ +--train False \ +--load_dir "logs/can_external_style" \ +--style_net_checkpoint "logs/inception_resnet_v2/" diff --git a/experiments/train_can_paper.sh b/experiments/train_can_paper.sh new file mode 100644 index 0000000..aaa8f54 --- /dev/null +++ b/experiments/train_can_paper.sh @@ -0,0 +1,19 @@ +# export CUDA_VISIBLE_DEVICES=0 # edit this if you want to limit yourself to GPU +export PYTHONPATH="slim/:$PYTHONPATH" +python3 main.py \ +--epoch 25 \ +--learning_rate .0001 \ +--beta 0.5 \ +--batch_size 16 \ +--sample_size 16 \ +--input_height 256 \ +--output_height 256 \ +--lambda_val 1.0 \ +--smoothing 1.0 \ +--use_resize True \ +--dataset wikiart \ +--input_fname_pattern */*.jpg \ +--crop False \ +--visualize False \ +--can True \ +--train \ diff --git a/experiments/wiki_can_256.sh b/experiments/wiki_can_256.sh index f0e6b88..96c13b4 100644 --- a/experiments/wiki_can_256.sh +++ b/experiments/wiki_can_256.sh @@ -1,9 +1,9 @@ -export CUDA_VISIBLE_DEVICES=1 +# export CUDA_VISIBLE_DEVICES=0 # edit this if you want to limit yourself to GPU python3 main.py \ --epoch 25 \ --learning_rate .0001 \ --beta 0.5 \ ---batch_size 15 \ +--batch_size 16 \ --sample_size 16 \ --input_height 256 \ --output_height 256 \ diff --git a/experiments/wiki_external_can.sh b/experiments/wiki_external_can.sh index cdd3533..1d685ec 100644 --- a/experiments/wiki_external_can.sh +++ b/experiments/wiki_external_can.sh @@ -21,3 +21,4 @@ python3 main.py \ --can True \ --train \ --style_net_checkpoint "slim/logs/wikiart/inception_resnet_v2/all/bs=16,lr=0.0001,epochs=100/smol_adam_fixedLR" +# --style_net_checkpoint "logs/inception_resnet_v2/" diff --git a/main.py b/main.py index a2fe1a1..e032dda 100644 --- a/main.py +++ b/main.py @@ -28,6 +28,7 @@ flags.DEFINE_string("log_dir", 'logs', "Directory to store logs [logs]") flags.DEFINE_string("checkpoint_dir", None, "Directory name to save the checkpoints [/checkpoint]") flags.DEFINE_string("sample_dir", None, "Directory name to save the image samples [/samples]") +flags.DEFINE_string("load_dir", None, "Directory that specifies checkpoint to load") flags.DEFINE_boolean("train", False, "True for training, False for testing [False]") flags.DEFINE_boolean("crop", False, "True for training, False for testing [False]") flags.DEFINE_boolean("visualize", False, "True for visualizing, False for nothing [False]") diff --git a/model.py b/model.py index a2e55f9..7b1a61d 100644 --- a/model.py +++ b/model.py @@ -559,8 +559,10 @@ def get_parent_path(path): checkpoint_dir = os.path.join(path, last_, 'checkpoint') else: - checkpoint = None + checkpoint_dir = None + if config.load_dir: + checkpoint_dir = config.load_dir ckpt = tf.train.get_checkpoint_state(checkpoint_dir) if ckpt and ckpt.model_checkpoint_path: @@ -575,6 +577,8 @@ def get_parent_path(path): print(" [*] Failed to find a sample_z") sample_z = None return True, counter, sample_z + elif config.load_dir: + raise ValueError(" [*] Failed to find the load_dir") else: print(" [*] Failed to find a checkpoint") return False, 0, None diff --git a/slim/finetune_inception_resnet_v2_on_wikiart.sh b/slim/finetune_inception_resnet_v2_on_wikiart.sh index d3ca90a..9456161 100644 --- a/slim/finetune_inception_resnet_v2_on_wikiart.sh +++ b/slim/finetune_inception_resnet_v2_on_wikiart.sh @@ -49,74 +49,74 @@ if [ ! -f ${PRETRAINED_CHECKPOINT_DIR}/${MODEL_NAME}.ckpt ]; then fi # # Download the dataset -# python download_and_convert_data.py \ -# --dataset_name=wikiart \ -# --dataset_dir=${DATASET_DIR} -# --input_dataset_dir=${INPUT_DATASET_DIR} +python download_and_convert_data.py \ + --dataset_name=wikiart \ + --dataset_dir=${DATASET_DIR} + --input_dataset_dir=${INPUT_DATASET_DIR} # @philkuz I use this to create a nice initialization - haven't tried random # TODO try out if your'e curious to see whether random initialization of last # layer makes sense in this case. -# Fine-tune only the new layers for 1000 steps. -# python3 train_image_classifier.py \ -# --train_dir=${TRAIN_DIR} \ -# --dataset_name=wikiart \ -# --dataset_split_name=train \ -# --dataset_dir=${DATASET_DIR} \ -# --model_name=${MODEL_NAME} \ -# --checkpoint_path=${PRETRAINED_CHECKPOINT_DIR}/${MODEL_NAME}.ckpt \ -# --checkpoint_exclude_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \ -# --trainable_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \ -# --max_number_of_steps=10000 \ -# --batch_size=32 \ -# --learning_rate=0.01 \ -# --learning_rate_decay_type=fixed \ -# --save_interval_secs=300 \ -# --save_summaries_secs=60 \ -# --log_every_n_steps=200 \ -# --optimizer=rmsprop \ -# --train_image_size=256 \ -# --weight_decay=0.00004 +# Fine-tune only the last layer for 1000 steps. +python3 train_image_classifier.py \ + --train_dir=${TRAIN_DIR} \ + --dataset_name=wikiart \ + --dataset_split_name=train \ + --dataset_dir=${DATASET_DIR} \ + --model_name=${MODEL_NAME} \ + --checkpoint_path=${PRETRAINED_CHECKPOINT_DIR}/${MODEL_NAME}.ckpt \ + --checkpoint_exclude_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \ + --trainable_scopes=InceptionResnetV2/Logits,InceptionResnetV2/AuxLogits \ + --max_number_of_steps=10000 \ + --batch_size=32 \ + --learning_rate=0.01 \ + --learning_rate_decay_type=fixed \ + --save_interval_secs=300 \ + --save_summaries_secs=60 \ + --log_every_n_steps=200 \ + --optimizer=rmsprop \ + --train_image_size=256 \ + --weight_decay=0.00004 # Run evaluation. -# python3 eval_image_classifier.py \ -# --checkpoint_path=${TRAIN_DIR} \ -# --eval_dir=${TRAIN_DIR} \ -# --dataset_name=wikiart \ -# --dataset_split_name=validation \ -# --dataset_dir=${DATASET_DIR} \ -# --model_name=${MODEL_NAME} \ -# --eval_image_size=256 +python3 eval_image_classifier.py \ + --checkpoint_path=${TRAIN_DIR} \ + --eval_dir=${TRAIN_DIR} \ + --dataset_name=wikiart \ + --dataset_split_name=validation \ + --dataset_dir=${DATASET_DIR} \ + --model_name=${MODEL_NAME} \ + --eval_image_size=256 # Fine-tune all the new layers for 500 steps. NUM_EPOCHS=100 BATCH_SIZE=16 -EXPERIMENT_NAME=smol_adam_fixedLR +EXPERIMENT_NAME=inception_resnet_v2 LR=0.0001 \ TRAIN_DIR=logs/wikiart/inception_resnet_v2/experiments/${EXPERIMENT_NAME}/bs=${BATCH_SIZE},lr=${LR},epochs=${NUM_EPOCHS}/ -# python3 train_image_classifier.py \ -# --train_dir=${TRAIN_DIR}/all \ -# --dataset_name=wikiart \ -# --dataset_split_name=train \ -# --dataset_dir=${DATASET_DIR} \ -# --model_name=${MODEL_NAME} \ -# --checkpoint_path=${TRAIN_DIR} \ -# --batch_size=${BATCH_SIZE} \ -# --learning_rate=${LR} \ -# --learning_rate_decay_type=fixed \ -# --save_interval_secs=300 \ -# --save_summaries_secs=60 \ -# --num_epochs_per_decay=1 \ -# --log_every_n_steps=200 \ -# --optimizer=adam \ -# --weight_decay=0.00004 \ -# --experiment_name=${EXPERIMENT_NAME} \ -# --num_epochs=${NUM_EPOCHS} \ -# --train_image_size=256 \ -# --continue_training False \ -# --experiment_numbering # TODO flag to flip on experiment numbering independent of experiement name arg existing +python3 train_image_classifier.py \ + --train_dir=${TRAIN_DIR}/all \ + --dataset_name=wikiart \ + --dataset_split_name=train \ + --dataset_dir=${DATASET_DIR} \ + --model_name=${MODEL_NAME} \ + --checkpoint_path=${TRAIN_DIR} \ + --batch_size=${BATCH_SIZE} \ + --learning_rate=${LR} \ + --learning_rate_decay_type=fixed \ + --save_interval_secs=300 \ + --save_summaries_secs=60 \ + --num_epochs_per_decay=1 \ + --log_every_n_steps=200 \ + --optimizer=adam \ + --weight_decay=0.00004 \ + --experiment_name=${EXPERIMENT_NAME} \ + --num_epochs=${NUM_EPOCHS} \ + --train_image_size=256 \ + --continue_training False \ + # --experiment_numbering # TODO flag to flip on experiment numbering independent of experiement name arg existing # # TODO catch the naming convention # Run evaluation.