update docu

.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 5cd933207eea9274583801743e7e6e0e
+config: b6dbb8ee18ae87680af2debbe4e55eeb
 tags: 645f666f9bcd5a90fca523b33c5a78b7

_sources/design.rst.txt

@@ -0,0 +1,298 @@
+NXDL extensions for data sources
+================================
+
+In order to describe various data sources `the NXDL
+standard <http://download.nexusformat.org/sphinx/nxdl.html>`__ has been
+extended by XML tags listed below. Thus, ``<strategy />`` and
+``<datasource />`` tags can be situated inside
+
+* ``<field />`` -- for its value
+* ``<attribute />`` -- for its value
+* ``<dim />`` -- for  its value (from nxswriter v1.7.0)
+* ``<link />`` -- for its target (from nxswriter v1.9.0)
+
+The other ones are nested inside ``<datasource/>`` tag.
+
+The ``<strategy>`` tag
+----------------------
+
+The **strategy** tag defines when and in which way the data is stored.
+An example of usage:
+
+.. code-block:: xml
+
+    <field name="energy" type="NX_FLOAT" units="GeV" >
+      <strategy mode="STEP" trigger="trigger1" />
+      <datasource type="CLIENT">
+        <record name="counter_1"/>
+      </datasource>
+    </field>
+
+The tag can have the following attributes:
+
+* **mode** specifies when the data is fetched, i.e.
+
+  * *INIT* during opening a new entry
+  * *STEP* when the *record()* command is performed
+  * *FINAL* at the time of closing the entry
+  * *POSTRUN* during post-processing stage
+  * *CONFIG* before opening a new entry (used only for dimensions)
+
+* **trigger** stands for the name of the related trigger in asynchronous *STEP* mode (optional)
+
+* **grows** selects which a field dimension grows of in the STEP mode. The default growing dimension is the first one, i.e. grows=1 (optional)
+
+* **compression** specifies filterID if data is compressed (optional)
+
+  * integer filterID
+  * *true* data going to be compressed with deflate
+  * *false* data stored without compression (default)
+
+* **compression_opts** integer compression options (optional)
+
+  * list of integer compression options separated by comma
+
+* **rate** compression rate for deflate (optional)
+
+  * from *0* to *9*
+
+* **shuffle** compression shuffle (optional)
+
+  * *true* shuffle enabled (default)
+  * *false* shuffle disabled
+
+* **canfail** specifies if during reading data exception should be thrown (optional)
+
+  * *false* on error exception is raised (default)
+  * *true* on error
+
+warning info is printed and the record is filled by a maximum value for
+the record type
+
+The **content** of the strategy tags is an label describing data merged
+into the H5 file by a post-processing program.
+
+An example of usage:
+
+.. code-block:: xml
+
+    <field name="energy" type="NX_FLOAT" units="GeV" >
+      <strategy mode="POSTRUN" >
+        http://haso.desy.de:/data/energy.dat
+      </strategy>
+    </field>
+
+The ``<datasource>`` tag
+------------------------
+
+The **datasource** tag specifies a type of the used data sources. They
+can be one of built types, i.e. *CLIENT*, *TANGO*, *DB*, *PYEVAL* or
+external ones -- defined in external python package and registered via
+JSON data.
+
+The ``<datasouce>`` tag acquires the following attributes:
+
+* **type** related to a type of data source with possible values:
+
+  * *CLIENT* for communication with client via JSON strings
+  * *TANGO* for taking data from Tango servers
+  * *DB* for fetching data from databases
+  * *PYEVAL* for evaluating data from other data sources by python script
+  * *other type name* of data source which has been registered via JSON data.
+
+* **name** datasource name (optional)
+
+*CLIENT* datasource
+-------------------
+
+The CLIENT datasource allows to read data from client JSON strings. It
+should contain a ``<record />`` tag. An example of usage:
+
+.. code-block:: xml
+
+    <datasource type="CLIENT" name="exp_c01">
+      <record name="counter_1"/>
+    </datasource>
+
+``<record>``
+~~~~~~~~~~~~
+
+The **record** tag defines the fetched data by its name. It has an
+attrbute
+
+* **name** which for the *CLIENT* data source type denotes a name of the data in the JSON string.
+
+An example of usage:
+
+.. code:: xml
+
+
+    <record name="Position"/>
+
+*TANGO* datasource
+------------------
+
+The TANGO datasource allows to read data from other TANGO devices. It
+should contain ``<device/>`` and ``<record/>`` tags. An example of
+usage:
+
+.. code-block:: xml
+
+    <datasource type="TANGO">
+      <device hostname="haso.desy.de" member="attribute" name="p09/motor/exp.01" port="10000" encoding="LIMA_VIDEO_IMAGE"/>
+      <record name="Position"/>
+    </datasource>
+
+``<device>``
+~~~~~~~~~~~~
+
+The **device** tag describes the Tango device which is used to get the
+data. It has the following attributes:
+
+* **name** corresponding to a name of the Tango device
+
+* **member** defining a type of the class member, i.e.
+
+  * *attribute* an attribute to read
+  * *command* a result of a command to take
+  * *property* a property to read
+
+* **hostname** a name of the host with the Tango device server (optional)
+
+* **port** a port number related to the Tango device server (optional)
+
+* **encoding** a label defining a required decoder for ``DevEncoded`` data (optional)
+
+* **group** tango group name (optional)
+
+If **group** attribute is defined data of the same group is read
+simultaneously and only ones during one experimental step.
+
+``<record>``
+~~~~~~~~~~~~
+
+The **record** tag defines the fetched data by its name. It has an
+attrbute
+
+* **name** which for the *TANGO* data source type a name of the tango class member.
+
+*DB* datasource
+---------------
+
+The DB datasource allows to read data from accessible databases. It
+should contain ``<database />`` and ``<query>`` tags. An example of
+usage:
+
+.. code-block:: xml
+
+    <datasource type="DB">
+      <database dbname="tango" dbtype="MYSQL" hostname="haso.desy.de"/>
+      <query format="SPECTRUM">
+        SELECT pid FROM device limit 10
+      </query>
+    </datasource>
+
+``<database>``
+~~~~~~~~~~~~~~
+
+The **database** tag specifies parameters to connect to the required
+database. It acquires the attirbutes
+
+* **dbtype** describing a type of the database, i.e.
+
+  * *ORACLE* an ORACLE database
+  * *MYSQL* a MySQL database
+  * *PGSQL* a PostgreSQL database
+
+* **dbname** denoting a name of the database (optional)
+
+* **hostname** being a name of the host with the database (optional)
+
+* **port** corresponding to a port number related to the database (optional)
+
+* **user** denoting a user name (optional)
+
+* **passwd** being a user password (optional)
+
+* **mycnf** defining a location of the my.cnf file with MySQL database access configuration (optional)
+
+* **node** corresponding to a node parameter for the ORACLE database(optional)
+
+The **content** of the database tag defines Oracle DSN string (optional)
+
+``<query>``
+~~~~~~~~~~~
+
+The **query** tag defines the database query which fetches the data. It
+has one attribute
+
+* **format** which specifies a dimension of the fetch data, i.e.
+
+  * *SCALAR* corresponds to 0-dimensional data, e.g. a separate numerical value or string
+  * *SPECTRUM* is related to 1-dimensional data, e.g. a list of numerical values or strings
+  * *IMAGE* describes 2-dimensional data, i.e. a table of specific type values, e.g. a table of strings
+
+The **content** of the query tags is the SQL query
+
+Another example of usage:
+
+.. code-block:: xml
+
+    <datasource type="DB">
+      <database dbname="mydb" dbtype="PGSQL"/>
+      <query format="IMAGE">
+        SELECT * FROM weather limit 3
+      </query>
+    </datasource>
+
+*PYEVAL* datasource
+-------------------
+
+The PYEVAL datasource allows to read data from other datasources and
+evaluate it by a user python script. An example of usage:
+
+.. code-block:: xml
+
+    <datasource type="PYEVAL">
+
+      <datasource type="TANGO" name="position">
+        <device hostname="haso.desy.de" member="attribute" name="p09/motor/exp.01" port="10000"/>
+        <record name="Position"/>
+      </datasource>
+
+      <datasource type="CLIENT" name="shift">
+        <record name="exp_c01"/>
+      </datasource>
+
+      <result name="finalposition">
+        ds.finalposition = ds.position + ds.shift
+      </result>
+
+    </datasource>
+
+``<datasource>``
+~~~~~~~~~~~~~~~~
+
+The PYEVAL datasource can contain other datasources. They have to have
+defined name attributes. Those names with additional prefix 'ds.'
+correspond to input variable names from the python script, i.e.
+ds.\ **name**.
+
+``<result>``
+~~~~~~~~~~~~
+
+The **result** contains python script which evaluates input data. It has
+the following attribute:
+* **name** corresponding to a result name. It
+is related to python script variable by *ds.\ **name***. The default
+value **name**\ ="result". (optional)
+
+
+Sequence diagrams
+=================
+
+.. image:: files/basic_server_client_communication.png
+
+.. image:: files/configuration.png
+
+.. image:: files/recording_data.png

_sources/index.rst.txt

@@ -0,0 +1,27 @@
+.. NeXDaTaS documentation master file, created by
+   sphinx-quickstart on Wed Jul  6 13:29:44 2016.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+.. include:: ../README.rst
+
+
+Contents:
+=========
+
+.. toctree::
+   :maxdepth: 1
+
+   project_goals
+   design
+
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

_sources/project_goals.rst.txt

@@ -0,0 +1,55 @@
+Project goals
+=============
+
+The aim of this project is to implement a Tango server that manages data
+IO for synchrotron (and maybe neutron) beamlines. The server should
+satisfy the following requirements
+
+* remove responsibility for data IO
+  from the beamline control client
+* provide a simple configuration mechanism via NXDL
+* read data from the following sources without client interaction
+* SQL databases (MySQL, Postgres, Oracle, DB2, ...)
+* other TANGO servers
+* JSON records (important for the interaction
+  with the client and SARDANA)
+* the first implementation of the server will be written in Python
+* the communication model of the first implementation will be strictly synchronous
+  (future version most probably will support other communication models too)
+* the control client software has full control over the behavior of the server via
+  TANGO commands
+* only low data-rate sources will be handled directly by the server.
+  High data-rate sources will write their data independently
+  and additional software will add this data to the Nexus file produced by
+  the server once the experiment is finished.
+
+The server should make it easy to implement control clients which write
+Nexus files as the entire Nexus logic is kept in the server. Clients
+only produce NXDL configurations or use third party tools for this job.
+The first Python implementation of this server will serve as a proof of
+concept.
+
+The importance of the client
+============================
+
+.. image:: files/system_layout.png
+
+All operations carried out on a beamline are orchestrated by the control
+client (CC), a software operated by the beamline-scientist and/or a
+user. Although the term client suggests that it is only a minor
+component aside from all the hardware control servers, databases, and
+whatever software is running on a beamline it is responsible for all the
+other components and tells them what to do at which point in time. In
+terms of an orchestra the CC is the director which tells each group of
+instruments or individual artist what to do at a certain point in time.
+
+It is important to understand the role of the CC in the entire software
+system on a beamline as it determines who is responsible for certain
+operations. The CC might be a simple single script running on the
+control PC which can is configured by the user before start or it might
+be a whole application of its own like SPEC or ONLINE. Historically it
+is the job of the CC to write the data recorded during the experiment
+(this is true at least for low rate data-sources). However, with the
+appearance of complex data formats like Nexus the IO code becomes more
+complex.
+