Improve module loading path handling at completion, liinting, and so on

README.rst

@@ -56,6 +56,98 @@ issue if you require assistance writing a plugin.
 Configuration
 -------------
 
+Location of Configuration File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are three configuration levels below.
+
+* user-level configuration (e.g. ~/.config/CONFIGFILE)
+* lsp-level configuration (via LSP didChangeConfiguration method)
+* project-level configuration
+
+The latter level has priority over the former.
+
+As project-level configuration, configurations are read in from files
+in the root of the workspace, by default. What files are read in is
+described after.
+
+At evaluation of python source file ``foo/bar/baz/example.py`` for
+example, if there is any configuration file in the ascendant directory
+(i.e. ``foo``, ``foo/bar`` or ``foo/bar/baz``), it is read in before
+evaluation. If multiple ascendant directories contain configuration
+files, files only in the nearest ascendant directory are read in.
+
+In some cases, automatically discovered files are exclusive with files
+in the root of the workspace.
+
+Syntax in Configuration File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Configuration file should be written in "INI file" syntax.
+
+Value specified in configuration file should be one of types below.
+
+* bool
+* int
+* string
+* list
+
+"List" value is string entries joined with comma. Both leading and
+trailing white spaces of each entries in a "list" value are trimmed.
+
+Source Roots
+~~~~~~~~~~~~
+
+"Source roots" is determined in the order below.
+
+1. if ``pyls.source_roots`` (described after) is specified, its value
+   is used as "source roots"
+2. if any of setup.py or pyproject.toml is found in the ascendant
+   directory of python source file at evaluation, that directory is
+   treated as "source roots"
+3. otherwise, the root of the workspace is treated as "source roots"
+
+"Source roots" is used as a part of sys path at evaluation of python
+source files.
+
+Python Language Server Specific Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+lsp-level and project-level configuration are supported for Python
+Language Server specific configuration.
+
+For project-level configuration, setup.cfg and tox.ini are read in.
+Configuration files discovered automatically at evaluation of python
+source file are **not** exclusive with configuration files in the root
+of the workspace. Files in both locations are read in, and a
+configuration in the former files has priority over one in the latter.
+
+Python Language Server specific configurations are show below.
+
+* ``pyls.source_roots`` (list) to specify source roots
+* ``pyls.plugins.jedi.extra_paths`` (list) to specify extra sys paths
+
+Relative path in these configurations is treated as relative to the
+directory, in which configuration file exists. For configuration via
+LSP didChangeConfiguration method, the root of the workspace is used
+as base directory.
+
+Path in ``pyls.source_roots`` is ignored, if it refers outside of the
+workspace.
+
+To make these configurations persisted into setup.cfg or tox.ini,
+describe them under ``[pyls]`` section like below.
+
+.. code-block:: ini
+
+    [pyls]
+    source_roots = services/foo, services/bar
+    plugins.jedi.extra_paths = ../extra_libs
+
+
+Configuration at Source Code Evaluation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 Configuration is loaded from zero or more configuration sources. Currently implemented are:
 
 * pycodestyle: discovered in ~/.config/pycodestyle, setup.cfg, tox.ini and pycodestyle.cfg.
@@ -67,6 +159,10 @@ order to respect flake8 configuration instead.
 Overall configuration is computed first from user configuration (in home directory), overridden by configuration
 passed in by the language client, and then overriden by configuration discovered in the workspace.
 
+Configuration files discovered in the workspace automatically at
+evaluation of python source file are exclusive with configuration
+files in the root of the workspace.
+
 To enable pydocstyle for linting docstrings add the following setting in your LSP configuration:
 ```
 "pyls.plugins.pydocstyle.enabled": true

pyls/_utils.py

@@ -83,6 +83,48 @@ def find_parents(root, path, names):
     return []
 
 
+def is_inside_of(path, root, strictly=True):
+    """Return whether path is inside of root or not
+
+    It is assumed that both path and root are absolute.
+
+    If strictly=False, os.path.normcase and os.path.normpath are not
+    applied on path and root for efficiency. This is useful if
+    ablsolute "path" is made from relative one and "root" (and already
+    normpath-ed), for example.
+    """
+    if strictly:
+        path = os.path.normcase(os.path.normpath(path))
+        root = os.path.normcase(os.path.normpath(root))
+
+    return path == root or path.startswith(root + os.path.sep)
+
+
+def normalize_paths(paths, basedir, inside_only):
+    """Normalize each elements in paths
+
+    Relative elements in paths are treated as relative to basedir.
+
+    This function yields "(path, validity)" tuple as normalization
+    result for each elements in paths. If inside_only is specified and path is
+    not so, validity is False. Otherwise, path is already normalized
+    as absolute path, and validity is True.
+    """
+    for path in paths:
+        full_path = os.path.normpath(os.path.join(basedir, path))
+        if (not inside_only or
+                # If "inside_only" is specified, path must (1) not be
+                # absolute (= be relative to basedir), (2) not have
+                # drive letter (on Windows), and (3) be descendant of
+                # the root (= "inside_only").
+                (not os.path.isabs(path) and
+                 not os.path.splitdrive(path)[0] and
+                 is_inside_of(full_path, inside_only, strictly=False))):
+            yield full_path, True
+        else:
+            yield path, False
+
+
 def match_uri_to_workspace(uri, workspaces):
     if uri is None:
         return None
@@ -132,6 +174,25 @@ def _merge_dicts_(a, b):
     return dict(_merge_dicts_(dict_a, dict_b))
 
 
+def get_config_by_path(settings, path, default_value=None):
+    """Get the value in settings dict at the given path.
+
+    If path is not resolvable in specified settings, this returns
+    default_value.
+    """
+    paths = path.split('.')
+    while len(paths) > 1:
+        settings = settings.get(paths.pop(0))
+        if not isinstance(settings, dict):
+            # Here, at least one more looking up should be available,
+            # but the last retrieved was non-dict. Therefore, path is
+            # not resolvable in specified settings.
+            return default_value
+
+    # Here, paths should have only one value
+    return settings.get(paths[0], default_value)
+
+
 def format_docstring(contents):
     """Python doc strings come in a number of formats, but LSP wants markdown.
 

pyls/config/config.py

@@ -40,6 +40,9 @@ def __init__(self, root_uri, init_opts, process_id, capabilities):
         except ImportError:
             pass
 
+        from .pyls_conf import PylsConfig
+        self._config_sources['pyls'] = PylsConfig(self._root_path)
+
         self._pm = pluggy.PluginManager(PYLS)
         self._pm.trace.root.setwriter(log.debug)
         self._pm.enable_tracing()
@@ -104,7 +107,7 @@ def settings(self, document_path=None):
         settings.cache_clear() when the config is updated
         """
         settings = {}
-        sources = self._settings.get('configurationSources', DEFAULT_CONFIG_SOURCES)
+        sources = self._settings.get('configurationSources', DEFAULT_CONFIG_SOURCES) + ['pyls']
 
         for source_name in reversed(sources):
             source = self._config_sources.get(source_name)
@@ -141,6 +144,13 @@ def plugin_settings(self, plugin, document_path=None):
 
     def update(self, settings):
         """Recursively merge the given settings into the current settings."""
+        sources = settings.get('configurationSources', DEFAULT_CONFIG_SOURCES) + ['pyls']
+        for source_name in reversed(sources):
+            source = self._config_sources.get(source_name)
+            if not source:
+                continue
+            source.lsp_config(settings)
+
         self.settings.cache_clear()
         self._settings = settings
         log.info("Updated settings to %s", self._settings)

pyls/config/pyls_conf.py

@@ -0,0 +1,93 @@
+import logging
+import os
+from pyls._utils import find_parents, get_config_by_path, merge_dicts, normalize_paths
+from .source import ConfigSource, _set_opt
+
+log = logging.getLogger(__name__)
+
+CONFIG_KEY = 'pyls'
+PROJECT_CONFIGS = ['setup.cfg', 'tox.ini']
+
+OPTIONS = [
+    ('source_roots', 'source_roots', list),
+
+    # inter-plugins configurations
+    ('plugins.jedi.extra_paths', 'plugins.jedi.extra_paths', list),
+]
+
+# list of (config_path, inside_only) tuples for path normalization
+NORMALIZED_CONFIGS = [
+    ('source_roots', True),
+    ('plugins.jedi.extra_paths', False),
+]
+
+
+class PylsConfig(ConfigSource):
+    """Parse pyls configuration"""
+
+    def __init__(self, root_path):
+        super(PylsConfig, self).__init__(root_path)
+
+        # If workspace URI has trailing '/', root_path does, too.
+        # (e.g. "lsp" on Emacs sends such URI). To avoid normpath at
+        # each normalization, os.path.normpath()-ed root_path is
+        # cached here.
+        self._norm_root_path = os.path.normpath(self.root_path)
+
+    def user_config(self):
+        # pyls specific configuration mainly focuses on per-project
+        # configuration
+        return {}
+
+    def project_config(self, document_path):
+        settings = {}
+        seen = set()
+
+        # To read config files in root_path even if any config file is
+        # found by find_parents() in the directory other than
+        # root_path, root_path is listed below as one of targets.
+        #
+        # On the other hand, "seen" is used to manage name of already
+        # evaluated files, in order to avoid multiple evaluation of
+        # config files in root_path.
+        for target in self.root_path, document_path:
+            # os.path.normpath is needed to treat that
+            # "root_path/./foobar" and "root_path/foobar" are
+            # identical.
+            sources = map(os.path.normpath, find_parents(self.root_path, target, PROJECT_CONFIGS))
+            files = []
+            for source in sources:
+                if source not in seen:
+                    files.append(source)
+                    seen.add(source)
+            if not files:
+                continue  # there is no file to be read in
+
+            parsed = self.parse_config(self.read_config_from_files(files),
+                                       CONFIG_KEY, OPTIONS)
+            if not parsed:
+                continue  # no pyls specific configuration
+
+            self.normalize(parsed, os.path.dirname(files[0]))
+
+            settings = merge_dicts(settings, parsed)
+
+        return settings
+
+    def lsp_config(self, config):
+        self.normalize(config, self._norm_root_path)
+
+    def normalize(self, config, basedir):
+        for config_path, inside_only in NORMALIZED_CONFIGS:
+            paths = get_config_by_path(config, config_path)
+            if not paths:
+                continue  # not specified (or empty)
+
+            normalized = []
+            for path, valid in normalize_paths(paths, basedir, inside_only and self._norm_root_path):
+                if valid:
+                    normalized.append(path)
+                else:
+                    log.warning("Ignoring path '%s' for pyls.%s", path, config_path)
+
+            _set_opt(config, config_path, normalized)

pyls/config/source.py

@@ -25,6 +25,16 @@ def project_config(self, document_path):
         """Return project-level (i.e. workspace directory) configuration."""
         raise NotImplementedError()
 
+    def lsp_config(self, config):
+        """Check configuration at updating.
+
+        Typical usecase is updating via LSP didChangeConfiguration
+        method.
+
+        Change config directly, if any changing like normalization is
+        needed. It is nested dictionay.
+        """
+
     @staticmethod
     def read_config_from_files(files):
         config = configparser.RawConfigParser()