Release 1.0.1

Author: IgnatovFedor

README.md

@@ -18,7 +18,7 @@ DeepPavlov is designed for
 * Issues [*github/issues/*](https://github.com/deeppavlov/DeepPavlov/issues)
 * Forum [*forum.deeppavlov.ai*](https://forum.deeppavlov.ai/)
 * Blogs [*medium.com/deeppavlov*](https://medium.com/deeppavlov)
-* Tutorials [*examples/*](examples) and [extended colab tutorials](https://github.com/deeppavlov/dp_tutorials)
+* [Extended colab tutorials](https://github.com/deeppavlov/dp_tutorials)
 * Docker Hub [*hub.docker.com/u/deeppavlov/*](https://hub.docker.com/u/deeppavlov/) 
     * Docker Images Documentation [*docs:docker-images/*](http://docs.deeppavlov.ai/en/master/intro/installation.html#docker-images)
 
@@ -34,13 +34,9 @@ Please leave us [your feedback](https://forms.gle/i64fowQmiVhMMC7f9) on how we c
 
 [Automatic Spelling Correction](http://docs.deeppavlov.ai/en/master/features/models/spelling_correction.html) | [Entity Linking](http://docs.deeppavlov.ai/en/master/features/models/entity_linking.html)
 
-[Russian SuperGLUE](http://docs.deeppavlov.ai/en/master/features/models/superglue.html)
-
-**Skills**
-
-[Open Domain Questions Answering](http://docs.deeppavlov.ai/en/master/features/skills/odqa.html)
+[Open Domain Questions Answering](http://docs.deeppavlov.ai/en/master/features/models/odqa.html) | [Frequently Asked Questions Answering](http://docs.deeppavlov.ai/en/master/features/models/faq.html)
 
-[Frequently Asked Questions Answering](http://docs.deeppavlov.ai/en/master/features/skills/faq.html)
+[Russian SuperGLUE](http://docs.deeppavlov.ai/en/master/features/models/superglue.html)
 
 **Embeddings**
 
@@ -97,24 +93,26 @@ evaluate and infer it:
 
 #### GPU requirements
 
-To run supported DeepPavlov models on GPU you should have [CUDA](https://developer.nvidia.com/cuda-toolkit) compatible
-with used GPU and [library PyTorch version](deeppavlov/requirements/pytorch.txt).
+By default, DeepPavlov installs models requirements from PyPI. PyTorch from PyPI could not support your device CUDA
+capability. To run supported DeepPavlov models on GPU you should have [CUDA](https://developer.nvidia.com/cuda-toolkit)
+compatible with used GPU and [PyTorch version](deeppavlov/requirements/pytorch.txt) required by DeepPavlov models.
+See [docs](https://docs.deeppavlov.ai/en/master/intro/quick_start.html#using-gpu) for details.
 
 ### Command line interface (CLI)
 
 To get predictions from a model interactively through CLI, run
 
 ```bash
-python -m deeppavlov interact <config_path> [-d]
+python -m deeppavlov interact <config_path> [-d] [-i]
 ```
 
-* `-d` downloads required data -- pretrained model files and embeddings
-  (optional).
+* `-d` downloads required data - pretrained model files and embeddings (optional).
+* `-i` installs model requirements (optional).
 
 You can train it in the same simple way:
 
 ```bash
-python -m deeppavlov train <config_path> [-d]
+python -m deeppavlov train <config_path> [-d] [-i]
 ```
 
 Dataset will be downloaded regardless of whether there was `-d` flag or not.
@@ -126,10 +124,11 @@ The data format is specified in the corresponding model doc page.
 There are even more actions you can perform with configs:
 
 ```bash
-python -m deeppavlov <action> <config_path> [-d]
+python -m deeppavlov <action> <config_path> [-d] [-i]
 ```
 
 * `<action>` can be
+    * `install` to install model requirements (same as `-i`),
     * `download` to download model's data (same as `-d`),
     * `train` to train the model on the data specified in the config file,
     * `evaluate` to calculate metrics on the same dataset,
@@ -140,6 +139,7 @@ python -m deeppavlov <action> <config_path> [-d]
       *<file_path>* if `-f <file_path>` is specified.
 * `<config_path>` specifies path (or name) of model's config file
 * `-d` downloads required data
+* `-i` installs model requirements
 
 
 ### Python
@@ -149,33 +149,26 @@ To get predictions from a model interactively through Python, run
 ```python
 from deeppavlov import build_model
 
-model = build_model(<config_path>, download=True)
+model = build_model(<config_path>, install=True, download=True)
 
 # get predictions for 'input_text1', 'input_text2'
 model(['input_text1', 'input_text2'])
 ```
-
-* where `download=True` downloads required data from web -- pretrained model
-  files and embeddings (optional),
-* `<config_path>` is path to the chosen model's config file (e.g.
-  `"deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"`) or
-  `deeppavlov.configs` attribute (e.g.
+where
+* `install=True` installs model requirements (optional),
+* `download=True` downloads required data from web - pretrained model files and embeddings (optional),
+* `<config_path>` is model name (e.g. `'ner_ontonotes_bert_mult'`), path to the chosen model's config file (e.g.
+  `"deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"`),  or `deeppavlov.configs` attribute (e.g.
   `deeppavlov.configs.ner.ner_ontonotes_bert_mult` without quotation marks).
 
 You can train it in the same simple way:
 
 ```python
 from deeppavlov import train_model 
 
-model = train_model(<config_path>, download=True)
+model = train_model(<config_path>, install=True, download=True)
 ```
 
-* `download=True` downloads pretrained model, therefore the pretrained
-model will be, first, loaded and then train (optional).
-
-Dataset will be downloaded regardless of whether there was ``-d`` flag or
-not.
-
 To train on your own data you need to modify dataset reader path in the
 [train config doc](http://docs.deeppavlov.ai/en/master/intro/config_description.html#train-config).
 The data format is specified in the corresponding model doc page. 
@@ -185,12 +178,9 @@ You can also calculate metrics on the dataset specified in your config file:
 ```python
 from deeppavlov import evaluate_model 
 
-model = evaluate_model(<config_path>, download=True)
+model = evaluate_model(<config_path>, install=True, download=True)
 ```
 
-
 ## License
 
 DeepPavlov is Apache 2.0 - licensed.
-
-##

deeppavlov/__init__.py

@@ -26,13 +26,16 @@
 
 
 # TODO: make better
-def train_model(config: [str, Path, dict], download: bool = False, recursive: bool = False) -> Chainer:
-    train_evaluate_model_from_config(config, download=download, recursive=recursive)
+def train_model(config: [str, Path, dict], install: bool = False,
+                download: bool = False, recursive: bool = False) -> Chainer:
+    train_evaluate_model_from_config(config, install=install, download=download, recursive=recursive)
     return build_model(config, load_trained=True)
 
 
-def evaluate_model(config: [str, Path, dict], download: bool = False, recursive: bool = False) -> dict:
-    return train_evaluate_model_from_config(config, to_train=False, download=download, recursive=recursive)
+def evaluate_model(config: [str, Path, dict], install: bool = False,
+                   download: bool = False, recursive: bool = False) -> dict:
+    return train_evaluate_model_from_config(config, to_train=False, install=install,
+                                            download=download, recursive=recursive)
 
 
 # check version

deeppavlov/_meta.py

@@ -1,4 +1,4 @@
-__version__ = '1.0.0'
+__version__ = '1.0.1'
 __author__ = 'Neural Networks and Deep Learning lab, MIPT'
 __description__ = 'An open source library for building end-to-end dialog systems and training chatbots.'
 __keywords__ = ['NLP', 'NER', 'SQUAD', 'Intents', 'Chatbot']

deeppavlov/core/commands/infer.py

@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import json
-import pickle
 import sys
 from itertools import islice
 from logging import getLogger
@@ -24,15 +23,18 @@
 from deeppavlov.core.common.params import from_params
 from deeppavlov.core.data.utils import jsonify_data
 from deeppavlov.download import deep_download
+from deeppavlov.utils.pip_wrapper import install_from_config
 
 log = getLogger(__name__)
 
 
 def build_model(config: Union[str, Path, dict], mode: str = 'infer',
-                load_trained: bool = False, download: bool = False) -> Chainer:
+                load_trained: bool = False, install: bool = False, download: bool = False) -> Chainer:
     """Build and return the model described in corresponding configuration file."""
     config = parse_config(config)
 
+    if install:
+        install_from_config(config)
     if download:
         deep_download(config)
 

deeppavlov/core/commands/train.py

@@ -24,6 +24,7 @@
 from deeppavlov.core.data.data_learning_iterator import DataLearningIterator
 from deeppavlov.core.data.utils import get_all_elems_from_json
 from deeppavlov.download import deep_download
+from deeppavlov.utils.pip_wrapper import install_from_config
 
 log = getLogger(__name__)
 
@@ -70,12 +71,15 @@ def train_evaluate_model_from_config(config: Union[str, Path, dict],
                                      iterator: Union[DataLearningIterator, DataFittingIterator] = None, *,
                                      to_train: bool = True,
                                      evaluation_targets: Optional[Iterable[str]] = None,
+                                     install: bool = False,
                                      download: bool = False,
                                      start_epoch_num: Optional[int] = None,
                                      recursive: bool = False) -> Dict[str, Dict[str, float]]:
     """Make training and evaluation of the model described in corresponding configuration file."""
     config = parse_config(config)
 
+    if install:
+        install_from_config(config)
     if download:
         deep_download(config)
 

deeppavlov/core/data/simple_vocab.py

@@ -109,7 +109,7 @@ def load(self):
         self.reset()
         if self.load_path:
             if self.load_path.is_file():
-                log.info("[loading vocabulary from {}]".format(self.load_path))
+                log.debug("[loading vocabulary from {}]".format(self.load_path))
                 tokens, counts = [], []
                 for ln in self.load_path.open('r', encoding='utf8'):
                     token, cnt = self.load_line(ln)

deeppavlov/core/models/torch_model.py

@@ -99,7 +99,7 @@ def __init__(self, device: str = "gpu",
         # we need to switch to eval mode here because by default it's in `train` mode.
         # But in case of `interact/build_model` usage, we need to have model in eval mode.
         self.model.eval()
-        log.info(f"Model was successfully initialized! Model summary:\n {self.model}")
+        log.debug(f"Model was successfully initialized! Model summary:\n {self.model}")
 
     def init_from_opt(self, model_func: str) -> None:
         """Initialize from scratch `self.model` with the architecture built in  `model_func` method of this class
@@ -150,22 +150,22 @@ def load(self, fname: Optional[str] = None, *args, **kwargs) -> None:
         model_func = getattr(self, self.opt.get("model_name", ""), None)
 
         if self.load_path:
-            log.info(f"Load path {self.load_path} is given.")
+            log.debug(f"Load path {self.load_path} is given.")
             if isinstance(self.load_path, Path) and not self.load_path.parent.is_dir():
                 raise ConfigError("Provided load path is incorrect!")
 
             weights_path = Path(self.load_path.resolve())
             weights_path = weights_path.with_suffix(f".pth.tar")
             if weights_path.exists():
-                log.info(f"Load path {weights_path} exists.")
-                log.info(f"Initializing `{self.__class__.__name__}` from saved.")
+                log.debug(f"Load path {weights_path} exists.")
+                log.debug(f"Initializing `{self.__class__.__name__}` from saved.")
 
                 # firstly, initialize with random weights and previously saved parameters
                 if model_func:
                     self.init_from_opt(model_func)
 
                 # now load the weights, optimizer from saved
-                log.info(f"Loading weights from {weights_path}.")
+                log.debug(f"Loading weights from {weights_path}.")
                 checkpoint = torch.load(weights_path, map_location=self.device)
                 model_state = checkpoint["model_state_dict"]
                 optimizer_state = checkpoint["optimizer_state_dict"]
@@ -181,10 +181,10 @@ def load(self, fname: Optional[str] = None, *args, **kwargs) -> None:
                 self.optimizer.load_state_dict(optimizer_state)
                 self.epochs_done = checkpoint.get("epochs_done", 0)
             elif model_func:
-                log.info(f"Init from scratch. Load path {weights_path} does not exist.")
+                log.debug(f"Init from scratch. Load path {weights_path} does not exist.")
                 self.init_from_opt(model_func)
         elif model_func:
-            log.info(f"Init from scratch. Load path {self.load_path} is not provided.")
+            log.debug(f"Init from scratch. Load path {self.load_path} is not provided.")
             self.init_from_opt(model_func)
 
     @overrides

deeppavlov/core/trainers/fit_trainer.py

@@ -63,7 +63,7 @@ def __init__(self, chainer_config: dict, *, batch_size: int = -1,
                  max_test_batches: int = -1,
                  **kwargs) -> None:
         if kwargs:
-            log.info(f'{self.__class__.__name__} got additional init parameters {list(kwargs)} that will be ignored:')
+            log.warning(f'{self.__class__.__name__} got additional init parameters {list(kwargs)} that will be ignored:')
         self.chainer_config = chainer_config
         self._chainer = Chainer(chainer_config['in'], chainer_config['out'], chainer_config.get('in_y'))
         self.batch_size = batch_size

deeppavlov/deep.py

@@ -41,6 +41,7 @@
 parser.add_argument("-b", "--batch-size", dest="batch_size", default=None, help="inference batch size", type=int)
 parser.add_argument("-f", "--input-file", dest="file_path", default=None, help="Path to the input file", type=str)
 parser.add_argument("-d", "--download", action="store_true", help="download model components")
+parser.add_argument("-i", "--install", action="store_true", help="install model requirements")
 
 parser.add_argument("--folds", help="number of folds", type=int, default=5)
 
@@ -67,6 +68,8 @@ def main():
     args = parser.parse_args()
     pipeline_config_path = find_config(args.config_path)
 
+    if args.install or args.mode == 'install':
+        install_from_config(pipeline_config_path)
     if args.download or args.mode == 'download':
         deep_download(pipeline_config_path)
 
@@ -95,8 +98,6 @@ def main():
                              rabbit_virtualhost=args.rabbit_virtualhost)
     elif args.mode == 'predict':
         predict_on_stream(pipeline_config_path, args.batch_size, args.file_path)
-    elif args.mode == 'install':
-        install_from_config(pipeline_config_path)
     elif args.mode == 'crossval':
         if args.folds < 2:
             log.error('Minimum number of Folds is 2')

deeppavlov/download.py

@@ -145,7 +145,7 @@ def download_resource(url: str, dest_paths: Iterable[Union[Path, str]], headers:
 
 def download_resources(args: Namespace) -> None:
     if not args.all and not args.config:
-        log.error('You should provide either skill config path or -all flag')
+        log.error('You should provide either model config path or -all flag')
         sys.exit(1)
     elif args.all:
         downloads = get_configs_downloads()