Merge pull request #39 from score-p/issue-#34

Closes #34, preparation of v2.0 release

* separate subsystem generation and __main__ more clearly
* introduce the notion of "instrumenter" and refactor the code accordingly
* introduce context managers

Author: AndreasGocht

.travis.yml

@@ -1,4 +1,4 @@
-dist: xenial
+dist: bionic
 
 language: python
 python:

README.md

@@ -5,15 +5,16 @@ scorep is a module that allows tracing of python scripts using [Score-P](http://
 
 # Install
 You need at least Score-P 5.0, build with `--enable-shared`.
-Please make sure, that `scorep-config` is in your `PATH` variable.
+Please make sure that `scorep-config` is in your `PATH` variable.
+
+Then run
 
-Then simply run
 ```
 pip install .
 ```
 # Use
 
-To trace the full script you need to run
+To trace the full script, you need to run
 
 ```
 python -m scorep <script.py>
@@ -35,12 +36,6 @@ Since version 0.9 it is possible to pass the traditional Score-P commands to the
 python -m scorep --mpp=mpi --thread=pthread <script.py>
 ```
 
-To see all flags simply call:
-
-```
-scorep --help
-```
-
 ## MPI
 
 To use trace an MPI parallel application, please specify
@@ -50,40 +45,69 @@ python -m scorep --mpp=mpi <script.py>
 ```
 
 ## User instrumentation
+### User Regions
+Since version 2.0 the python bindings support context managers for user regions:
 
-In some cases, the user might want to define a region, log some parameters, or just disable tracing for a certain area. To do so the module implements a few functions:
+```
+with scorep.user.region("region_name"):
+    do_something()
+```
+
+The traditional calls to  define a region and log some parameters, still exists:
 
 ```
 scorep.user.region_begin(name)
+scorep.user.parameter_int(name, val)
+scorep.user.parameter_uint(name, val)
+scorep.user.parameter_string(name, string)
 scorep.user.region_end(name)
 ```
 
-These functions allow the definition of user regions. `name` defines the name of a region. Each `user.region_begin` shall have a corresponding call to `user.region_end`.    
+`name` defines the name of the parameter or region, while `val` or `string` represents the value that is passed to Score-P. 
+
+Disabeling the recording with Score-P is still also possilbe:
 
 ```
 scorep.user.enable_recording()
 scorep.user.disable_recording()
 ```
 
-These functions allow enabling and disabling of the tracing.
+However, please be aware that the runtime impact is rather small, as the instrumenter is still active. For details about the instrumenter, please see [Instrumenter](#Instrumenter).  
+
+### Instrumenter
+With version 2.0 of the python bindings, the term "instrumenter" is introduced. The instrumenter describes the class that maps the Python `trace` or `profile` events to Score-P. Please be aware, that `trace` and `profile` does not refer to the traditional Score-P terms of tracing and profiling, but to the Python functions [sys.settrace](https://docs.python.org/3/library/sys.html#sys.settrace) and [sys.setprofile](https://docs.python.org/3/library/sys.html#sys.setprofile).
+
+The instrumenter that shall be used for tracing can be specified using `--instrumenter-type=<type>`.
+Currently there are the following tacers available:
+ * `profile` (default) implements `call` and `return`  
+ * `trace` implements `call` and `return`
+ * `dummy` does nothing, can be used without `-m scorep` (as done by user instrumentation)
+
+The `profile` instrumenter should have a smaller overhead than `trace`. 
+
+Moreover it is possible to disable (and enable) the instrumenter in the sourcecode:
 
 ```
-scorep.user.parameter_int(name, val)
-scorep.user.parameter_uint(name, val)
-scorep.user.parameter_string(name, string)
+with scorep.instrumenter.disable():
+    do_something()
+
+with scorep.instrumenter.enable():
+    do_something()    
 ```
 
-These functions allow passing user parameters to Score-P. These parameters can be int, uint and string. `name` defines the name of the parameter, while `val` or `string` defines the value that is passed to Score-P. 
-Please be aware, that the scorep module still needs to be preloaded, i.e. by using `python -m scorep`.
+or during startup with `--noinstrumenter`. Please be aware that the function calls override the Flag.
 
-## Additional Flags
+## Overview about Flags
 
-The Score-P bindings now also support a `--nopython` flag, which disables the python instrumentation.
-This might be helpful, if only user instrumentation is required, or only some instrumented libraries shall be traced.
+The following flags are special to the python bindings:
+
+ * `--noinstrumenter` disables the instrumentation of python code. Usefull for user instrumentation and to trace only specific code regions using `scorep.instrumenter.enable`.
+ * `--instrumenter-type=<type>` choose an instrumenter. See  [Instrumenter](#Instrumenter).
+ * `--keep-files` temporary files are kept.
 
 ## Backward Compatibility
 
-In order to maintain backwards Compatibility, the following flags are set per default:
+To maintain backwards compatibility, the following flags are set per default:
 
 ```
 python -m scorep --compiler --thread=pthread <script.py>
@@ -95,7 +119,7 @@ The traditional `--mpi` does still work, and is similar to the following call:
 python -m scorep --compiler --thread=pthread --mpp=mpi <script.py>
 ```
 
-To disable compiler instrumentation please specify:
+To disable compiler instrumentation, please specify:
 
 ```
 python -m scorep --nocompiler <script.py>
@@ -119,7 +143,4 @@ Please be aware the `--user` is always passed to Score-P, as this is needed for
 
 ## Not Working
 * python multiprocessing
-    * Score-P does currently not support any non MPI or non SHMEM communication. So the different processes will not know from each other. You might want to take a look to https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html .
-
-# Tracing
-The tracing uses Score-P User instrumentation. The python trace module was reworked, to pass the region names to `SCOREP_USER_REGION_BEGIN` and `SCOREP_USER_REGION_END` instead of printing. All other features of the tracing module where striped.
+    * Score-P does currently only support MPI or SHMEM. Any other multiprocessing approach cannot be traced.

scorep/__init__.py

@@ -1 +1,3 @@
-__all__ = ["trace", "scorep_bindings", "user", "subsystem"]
+import scorep.user
+import scorep.instrumenter
+__all__ = ["user", "instrumenter"]

scorep/__main__.py

@@ -2,8 +2,7 @@
 import sys
 import importlib
 
-import scorep.trace
-import scorep.helper
+import scorep.instrumenter
 import scorep.subsystem
 
 
@@ -12,37 +11,6 @@ def _err_exit(msg):
     sys.exit(1)
 
 
-def set_init_environment(scorep_config=[], keep_files=False):
-    """
-    Set the inital needed environmet variables, to get everythin up an running.
-    As a few variables interact with LD env vars, the programms needs to be restarted after this.
-    The function set the env var `SCOREP_PYTHON_BINDINGS_INITALISED` to true, once it is done with
-    initalising.
-
-    @param scorep_config configuration flags for score-p
-    @param keep_files whether to keep the generated files, or not.
-    """
-
-    if ("LD_PRELOAD" in os.environ) and (
-            "libscorep" in os.environ["LD_PRELOAD"]):
-        raise RuntimeError(
-            "Score-P is already loaded. This should not happen at this point")
-
-    subsystem_lib_name, temp_dir = scorep.subsystem.generate(
-        scorep_config, keep_files)
-    scorep_ld_preload = scorep.helper.generate_ld_preload(scorep_config)
-
-    scorep.helper.add_to_ld_library_path(temp_dir)
-
-    preload_str = scorep_ld_preload + " " + subsystem_lib_name
-    if "LD_PRELOAD" in os.environ:
-        sys.stderr.write(
-            "LD_PRELOAD is already specified. If Score-P is already loaded this might lead to errors.")
-        preload_str = preload_str + " " + os.environ["LD_PRELOAD"]
-    os.environ["LD_PRELOAD"] = preload_str
-    os.environ["SCOREP_PYTHON_BINDINGS_INITALISED"] = "true"
-
-
 def scorep_main(argv=None):
     # print(sys.flags)
     if argv is None:
@@ -55,7 +23,8 @@ def scorep_main(argv=None):
     keep_files = False
     no_default_threads = False
     no_default_compiler = False
-    no_python = False
+    no_instrumenter = False
+    instrumenter_type = "profile"
 
     for elem in argv[1:]:
         if parse_scorep_commands:
@@ -70,7 +39,12 @@ def scorep_main(argv=None):
                 scorep_config.append(elem)
                 no_default_compiler = True
             elif elem == "--nopython":
-                no_python = True
+                no_instrumenter = True
+            elif elem == "--noinstrumenter":
+                no_instrumenter = True
+            elif "--instrumenter-type" in elem:
+                param = elem.split("=")
+                instrumenter_type = param[1]
             elif elem[0] == "-":
                 scorep_config.append(elem)
             else:
@@ -90,20 +64,22 @@ def scorep_main(argv=None):
 
     if ("SCOREP_PYTHON_BINDINGS_INITALISED" not in os.environ) or (
             os.environ["SCOREP_PYTHON_BINDINGS_INITALISED"] != "true"):
-        set_init_environment(scorep_config, keep_files)
+        scorep.subsystem.init_environment(scorep_config, keep_files)
+        os.environ["SCOREP_PYTHON_BINDINGS_INITALISED"] = "true"
 
         """
         python -m starts the module as skript. i.e. sys.argv will loke like:
         ['/home/gocht/Dokumente/code/scorep_python/scorep.py', '--mpi', 'mpi_test.py']
 
-        To restart python we need to remove this line, and add python -m scorep ... again
+        To restart python we need to remove this line, and add `python -m scorep ...` again
         """
         new_args = [sys.executable, "-m", "scorep"]
         for elem in sys.argv:
             if "scorep/__main__.py" in elem:
                 continue
             else:
                 new_args.append(elem)
+
         os.execve(sys.executable, new_args, os.environ)
 
     scorep_bindings = importlib.import_module("scorep.scorep_bindings")
@@ -113,11 +89,11 @@ def scorep_main(argv=None):
     progname = prog_argv[0]
     sys.path[0] = os.path.split(progname)[0]
 
-    tracer = scorep.trace.ScorepTrace(scorep_bindings, not no_python)
+    tracer = scorep.instrumenter.get_instrumenter(
+        scorep_bindings, not no_instrumenter, instrumenter_type)
     try:
         with open(progname) as fp:
             code = compile(fp.read(), progname, 'exec')
-            target_code = code
         # try to emulate __main__ namespace as much as possible
         globs = {
             '__file__': progname,
@@ -152,5 +128,5 @@ def main(argv=None):
 else:
     if ("SCOREP_PYTHON_BINDINGS_INITALISED" not in os.environ) or (
             os.environ["SCOREP_PYTHON_BINDINGS_INITALISED"] != "true"):
-        sys.stderr.write(
+        logging.warning(
             "scorep needs to be loaded using \"python -m scorep <script>\". Please be aware that scorep might not work correctly!")

scorep/instrumenter.py

@@ -0,0 +1,91 @@
+import scorep.instrumenters.dummy
+import scorep.instrumenters.scorep_profile
+import scorep.instrumenters.scorep_trace
+
+global_instrumenter = None
+
+
+def get_instrumenter(
+        bindings=None,
+        enable_instrumenter=False,
+        instrumenter_type="dummy"):
+    """
+    returns an instrumenter
+
+    @param bindings the c/c++ scorep bindings
+    @param enable_instrumenter True if the Instrumenter should be enabled when run is called
+    @param instrumenter_type which python tracing interface to use. Currently available: `profile` (default), `trace` and `dummy`
+    """
+    global global_instrumenter
+    if global_instrumenter is None:
+        if instrumenter_type == "profile":
+            global_instrumenter = scorep.instrumenters.scorep_profile.ScorepProfile(
+                bindings, enable_instrumenter)
+        elif instrumenter_type == "trace":
+            global_instrumenter = scorep.instrumenters.scorep_trace.ScorepTrace(
+                bindings, enable_instrumenter)
+        elif instrumenter_type == "dummy":
+            global_instrumenter = scorep.instrumenters.dummy.ScorepDummy(
+                enable_instrumenter)
+        else:
+            raise RuntimeError(
+                "instrumenter_type \"{}\" unkown".format(instrumenter_type))
+
+    return global_instrumenter
+
+
+def register():
+    """
+    Reenables the python-tracing.
+    """
+    get_instrumenter().register()
+
+
+def unregister():
+    """
+    Disables the python-tracing.
+    Disabling the python-tracing is more efficient than disable_recording, as python does not longer call the tracing module.
+    However, all the other things that are traced by Score-P will still be recorded.
+    Please call register() to enable tracing again.
+    """
+    get_instrumenter().unregister()
+
+
+class enable():
+    """
+    Context manager to enable tracing in a certain region:
+    ```
+    with enable():
+        do stuff
+    ```
+    This overides --no-instrumenter (--nopython leagacy)
+    """
+
+    def __init__(self):
+        pass
+
+    def __enter__(self):
+        self.tracer_registered = get_instrumenter().register()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.tracer_registered = get_instrumenter().unregister()
+
+
+class disable():
+    """
+    Context manager to disable tracing in a certain region:
+    ```
+    with disable():
+        do stuff
+    ```
+    This overides --no-instrumenter (--nopython leagacy)
+    """
+
+    def __init__(self):
+        pass
+
+    def __enter__(self):
+        self.tracer_registered = get_instrumenter().unregister()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.tracer_registered = get_instrumenter().register()

scorep/instrumenters/__init__.py

@@ -1,5 +1,8 @@
-class ScorepTraceDummy:
-    def __init__(self, scorep_bindings=None, trace=True):
+__all__ = ['ScorepDummy']
+
+
+class ScorepDummy:
+    def __init__(self, scorep_bindings=None, enable_instrumenter=True):
         pass
 
     def register(self):
@@ -8,6 +11,9 @@ def register(self):
     def unregister(self):
         pass
 
+    def get_registered(self):
+        return None
+
     def run(self, cmd):
         pass