Skip to content
This repository was archived by the owner on Aug 26, 2020. It is now read-only.

Commit 03fed72

Browse files
luisfabibluisfabib
authored
Merged develop for release 0.9
Release 0.9
2 parents 846abe2 + 69ff9a8 commit 03fed72

File tree

787 files changed

+37178
-67299
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

787 files changed

+37178
-67299
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,13 @@ on:
55
branches:
66
- master
77
- release
8+
pull_request:
9+
branches:
10+
- master
811

912
jobs:
1013
build:
11-
name: CI
14+
name: CI (tests)
1215
runs-on: ubuntu-latest
1316
steps:
1417
- name: Checkout trigger commit
@@ -37,35 +40,35 @@ jobs:
3740
- name: Transfer source files to CI-server
3841
if: success()
3942
run: |
40-
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} -r ../DeerAnalysis2 ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/
41-
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} scp -r /home/lufa/scp_traffic/DeerAnalysis2 ${{ secrets.CISERVER_HOST }}:/home/deeranalysis/
42-
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f -r /home/lufa/scp_traffic/DeerAnalysis2"
43+
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} -r ../DeerLab ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/
44+
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} scp -r /home/lufa/scp_traffic/DeerLab ${{ secrets.CISERVER_HOST }}:~
45+
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f -r /home/lufa/scp_traffic/DeerLab"
4346
- name: Run test suite connected to CI-Server
4447
if: success()
4548
run: |
4649
ssh -tt -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} -t ${{ secrets.GATEWAY_HOST }} ssh ${{ secrets.CISERVER_HOST }} << EOF
4750
#Run test suite
48-
cd DeerAnalysis2/build
49-
matlab -nodesktop -r "datestsuite;exit"
51+
cd DeerLab/build
52+
matlab -nodesktop -r "dltestsuite;exit"
5053
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "coverage_badge.json" --bucket deershields
5154
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "testsuite_badge.json" --bucket deershields
5255
#Transfer log file to the gateway
53-
scp -o StrictHostKeyChecking=no -i ~/.ssh/gateway_key -P ${{ secrets.GATEWAY_PORT }} ./datestsuite.error ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/ || true
56+
scp -o StrictHostKeyChecking=no -i ~/.ssh/gateway_key -P ${{ secrets.GATEWAY_PORT }} ./dltestsuite.error ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/ || true
5457
#Delete the source files
55-
cd /home/deeranalysis/
56-
rm -f -r DeerAnalysis2
58+
cd ~
59+
rm -f -r DeerLab
5760
EOF
5861
- name: Disconnect CI-server
5962
if: success()
6063
run: |
6164
#Transfer log file to GitHub VM
62-
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/datestsuite.error ~/ || true
63-
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f /home/lufa/scp_traffic/datestsuite.error" || true
65+
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/dltestsuite.error ~/ || true
66+
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f /home/lufa/scp_traffic/dltestsuite.error" || true
6467
rm -f ~/sshkey
6568
- name: All tests passed successfully
6669
if: success()
6770
run: |
68-
! [ -f ~/datestsuite.error ] && echo "Test suite passed" || (echo "Test suite failed" && deerfailed)
71+
! [ -f ~/dltestsuite.error ] && echo "Test suite passed" || (echo "Test suite failed" && deerfailed)
6972
- name: Disconnect VPN connection
7073
if: always()
7174
run: |

‎.github/workflows/ci_dev.yml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Scheduled Continuous Integration
2+
3+
on:
4+
schedule:
5+
# Run every day at 6:30 AM
6+
- cron: '30 6 * * *'
7+
8+
jobs:
9+
build:
10+
name: Scheduled CI (dev)
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Checkout trigger commit
14+
uses: actions/checkout@v2
15+
with:
16+
ref: develop
17+
- name: Install dependencies
18+
run: |
19+
sudo apt-get install -y openvpn openconnect
20+
- name: Prepare VM
21+
if: success()
22+
run: |
23+
rm -f -r ./.git
24+
echo "${{ secrets.GATEWAY_KEY }}" > ~/sshkey
25+
sudo chmod 600 ~/sshkey
26+
- name: Establish VPN connection
27+
if: success()
28+
run: |
29+
# Force returning packets to be routed over public interface and not over the VPN
30+
sudo ip rule add from $(ip route get 1 | grep -Po '(?<=src )(\S+)') table 128
31+
sudo ip route add table 128 to $(ip route get 1 | grep -Po '(?<=src )(\S+)')/32 dev $(ip -4 route ls | grep default | grep -Po '(?<=dev )(\S+)')
32+
sudo ip route add table 128 default via $(ip -4 route ls | grep default | grep -Po '(?<=via )(\S+)')
33+
# Start network tunnel (TUN) device
34+
sudo openvpn --mktun --dev tun1
35+
sudo ifconfig tun1 up
36+
# Launch VPN connection
37+
echo "${{ secrets.VPN_PASSWD }}" | sudo openconnect ${{ secrets.VPN_SERVER }} --user=${{ secrets.VPN_USER }} --background --passwd-on-stdin
38+
- name: Transfer source files to CI-server
39+
if: success()
40+
run: |
41+
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} -r ../DeerLab ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/
42+
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} scp -r /home/lufa/scp_traffic/DeerLab ${{ secrets.CISERVER_HOST }}:~
43+
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f -r /home/lufa/scp_traffic/DeerLab"
44+
- name: Run test suite connected to CI-Server
45+
if: success()
46+
run: |
47+
ssh -tt -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} -t ${{ secrets.GATEWAY_HOST }} ssh ${{ secrets.CISERVER_HOST }} << EOF
48+
#Run test suite
49+
cd DeerLab/build
50+
matlab -nodesktop -r "dltestsuite;exit"
51+
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "coverage_badge.json" --bucket deershields
52+
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "testsuite_badge.json" --bucket deershields
53+
#Transfer log file to the gateway
54+
scp -o StrictHostKeyChecking=no -i ~/.ssh/gateway_key -P ${{ secrets.GATEWAY_PORT }} ./dltestsuite.error ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/ || true
55+
#Delete the source files
56+
cd ~
57+
rm -f -r DeerLab
58+
EOF
59+
- name: Disconnect CI-server
60+
if: success()
61+
run: |
62+
#Transfer log file to GitHub VM
63+
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/dltestsuite.error ~/ || true
64+
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f /home/lufa/scp_traffic/dltestsuite.error" || true
65+
rm -f ~/sshkey
66+
- name: All tests passed successfully
67+
if: success()
68+
run: |
69+
! [ -f ~/dltestsuite.error ] && echo "Test suite passed" || (echo "Test suite failed" && deerfailed)
70+
- name: Disconnect VPN connection
71+
if: always()
72+
run: |
73+
#Ensure the VPN is disconnected by killing openconnect
74+
sudo pkill openconnect
75+
#Remove TUN device
76+
sudo ifconfig tun1 down
77+
sudo openvpn --rmtun --dev tun1

‎.github/workflows/docbuilder.yml‎ renamed to ‎.github/workflows/deploy_ghpages.yml‎

Lines changed: 47 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,47 @@
1-
name: Webpage update
2-
3-
on:
4-
push:
5-
branches:
6-
- master
7-
paths:
8-
- 'docsrc/**'
9-
- '.github/**'
10-
11-
jobs:
12-
13-
build:
14-
name: Build
15-
runs-on: ubuntu-latest
16-
steps:
17-
- name: Checkout
18-
uses: actions/checkout@v1
19-
- name: Set up Python 3.7
20-
uses: actions/setup-python@v1
21-
with:
22-
python-version: 3.7
23-
- name: Install dependencies
24-
run: |
25-
python -m pip install --upgrade pip
26-
python -m pip install sphinx==1.8.0
27-
python -m pip install sphinx_rtd_theme
28-
python -m pip install sphinxcontrib-matlabdomain
29-
python -m pip install sphinxcontrib-httpdomain
30-
python -m pip install boto3
31-
sudo apt install texlive-extra-utils
32-
sudo apt-get install texlive-latex-extra
33-
34-
- name: Run Sphinx Builder
35-
run: |
36-
cd docsrc
37-
sphinx-build -E -b html ./source ../docs
38-
- name: Connect and update DeerAnalysis AWS S3 Bucket
39-
run: |
40-
cd build
41-
python -m synchS3 -k ${{ secrets.S3_ACCESS }} -s ${{ secrets.S3_SECRETACCESS }} -d "../docs" --bucket deeranalysis.org
1+
name: Docs Build & Deployment
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
- hotfixes
8+
paths:
9+
- 'build/**'
10+
- 'docsrc/**'
11+
12+
schedule:
13+
# Run once a week on Sunday at 12:00 PM
14+
- cron: '0 12 * * 0'
15+
16+
jobs:
17+
18+
deploy:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@v1
23+
- name: Set up Python 3.7
24+
uses: actions/setup-python@v1
25+
with:
26+
python-version: 3.7
27+
- name: Install dependencies
28+
run: |
29+
python -m pip install --upgrade pip
30+
python -m pip install sphinx==1.8.0
31+
python -m pip install sphinx_rtd_theme
32+
python -m pip install sphinxcontrib-matlabdomain
33+
python -m pip install sphinxcontrib-httpdomain
34+
sudo apt install texlive-extra-utils
35+
sudo apt-get install texlive-latex-extra
36+
37+
- name: Build multi-version docs
38+
run: |
39+
cd build
40+
python -m multiversedocs
41+
cd ..
42+
43+
- name: Deploy to GH-Pages
44+
uses: peaceiris/actions-gh-pages@v3
45+
with:
46+
github_token: ${{ secrets.GITHUB_TOKEN }}
47+
publish_dir: ./multidocs

‎.gitignore‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ docsrc/aws_access_keys.txt
1111
docsrc/__pycache__/
1212
build/.ssh/
1313
tutorials/**/*.pdf
14-
tutorials/**/*.m
14+
tutorials/**/*.mlx
1515
tests/_tutorials
16-
tests/_tutorials/**
16+
tests/_tutorials/**
17+
build/__pycache__/multiversedocs.cpython-37.pyc
18+
multidocs/**/*
19+
build/__pycache__/multiversedocs_testing.cpython-37.pyc

‎README.md‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,28 @@
11

22
<p align="center">
3-
<img src="./docsrc/source/logo_dark.png" alt="DeerLab Logo" width="60%"></img>
3+
<img src="./docsrc/source/_static/logo_dark.png" alt="DeerLab Logo" width="60%"></img>
44
</p>
55
</div>
66

77
<p align="center">
8-
<img src="https://img.shields.io/github/issues-raw/luisfabib/deerlab?style=flat"></img>
9-
<img src="https://img.shields.io/badge/MATLAB-R2016b--R2019b-brightgreen?style=flat"></img>
10-
<img src="https://img.shields.io/github/downloads/luisfabib/deerlab/total?style=flat"></img>
11-
<img src="https://github.com/luisfabib/deerlab/workflows/Webpage%20update/badge.svg?style=flat-square"></img>
8+
<img src="https://img.shields.io/github/issues-raw/JeschkeLab/DeerLab?style=flat"></img>
9+
<img src="https://img.shields.io/badge/MATLAB-R2017a--R2020a-brightgreen?style=flat"></img>
10+
<img src="https://img.shields.io/github/downloads/JeschkeLab/DeerLab/total?style=flat"></img>
1211
<img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fdeershields.s3.eu-central-1.amazonaws.com%2Fcoverage_badge.json"></img>
1312
<img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fdeershields.s3.eu-central-1.amazonaws.com%2Ftestsuite_badge.json"></img>
1413
</p>
1514

1615
### About
17-
The DeerLab software package is a MATLAB toolbox for the analysis of data from DEER (double electron-electron resonance) spectroscopy and similar dipolar spectroscopy techniques (DQC, RIDME, SIFTER). The main homepage can be found at www.deeranalysis.org. This is the GitHub repository of the DeerLab source code, including instructions for compiling and installing DeerLab.
16+
The DeerLab software package is a MATLAB toolbox for the analysis of data from DEER (double electron-electron resonance) spectroscopy and similar dipolar EPR spectroscopy techniques (RIDME, DQC, SIFTER). The main homepage can be found at [jeschkelab.github.io/DeerLab](https://jeschkelab.github.io/DeerLab/). This is the GitHub repository of the DeerLab source code, including instructions for compiling and installing DeerLab.
1817

19-
It consists of a collection of functions that perform single processing or fitting tasks. They can be combined in scripts to generate custom analysis workflows.
18+
It consists of a collection of functions that perform modelling, processing or fitting tasks. They can be combined in scripts to build custom data analysis workflows.
2019

21-
To model distance distributions, DeerLab supports two types of model classes and associated workflows: parameter-free models (as used in Tikhonov regularization) as well as a series of parameterized models (mutli-Gaussians etc). It also provides a selection of background models. There are functions for generating synthetic datasets as well as for fitting and analyzing experimental data sets.
20+
To model distance distributions, DeerLab supports two types of model classes and associated workflows: parameter-free models (as used in Tikhonov regularization) as well as a series of parameterized models (mutli-Gaussians etc). It also provides a selection of background and experiment models. There are functions for generating synthetic datasets as well as for fitting and analyzing experimental data sets.
2221

2322
### Requirements
24-
The application programming interface (API) of DeerLab requires the following products:
23+
DeerLab requires the following products:
2524

26-
* MATLAB (oldest version supported R2016b) (see <https://ch.mathworks.com/products/matlab.html>)
25+
* MATLAB (R2016b or newer) (see <https://ch.mathworks.com/products/matlab.html>)
2726

2827
Optional functionality may require the following products:
2928

@@ -48,20 +47,21 @@ In order for MATLAB to access the DeerLab API functions, the path to the DeerLab
4847
2) Add the following lines of code:
4948

5049
addpath('mypath/DeerLab/functions')
51-
addpath('mypath/DeerLab/functions/models')
5250

5351
3) Save ``startup.m`` and restart MATLAB.
5452

5553
### License
5654

57-
The DeerLab package is licensed under the MIT License. The "package" consists of the DOE ([functions/](https://github.com/luisfabib/deerlab/tree/master/functions)), documentation source ([docsrc/](https://github.com/luisfabib/deerlab/tree/master/docsrc)), tutorial scripts ([tutorials/](https://github.com/luisfabib/deerlab/tree/master/tutorials)), test suite ([tests/](https://github.com/luisfabib/deerlab/tree/master/tests)) and pipeline scripts ([.github/workflows](https://github.com/luisfabib/deerlab/tree/master/.github/workflows)). See below for exceptions.
58-
59-
Copyright (c) 2019: Luis Fabregas, Stefan Stoll, Gunnar Jeschke, and [other contributors](https://github.com/luisfabib/deerlab/contributors).
55+
The DeerLab toolbox is licensed under the MIT License. The complete toolbox consists of the functions ([functions/](https://github.com/JeschkeLab/DeerLab/tree/master/functions)), documentation source ([docsrc/](https://github.com/JeschkeLab/DeerLab/tree/master/docsrc)), tutorial scripts ([tutorials/](https://github.com/JeschkeLab/DeerLab/tree/master/tutorials)), test suite ([tests/](https://github.com/JeschkeLab/DeerLab/tree/master/tests)), and pipeline scripts ([.github/workflows](https://github.com/JeschkeLab/DeerLab/tree/master/.github/workflows)). See below for exceptions.
6056

6157
DeerLab includes code from the following projects, which have their own licenses:
6258
- [datahash.m](https://www.mathworks.com/matlabcentral/fileexchange/31272-datahash) (Hash-key generator by Jan Simon) [BSD]
6359
- [fresnelS.m, fresnelC.m](https://www.mathworks.com/matlabcentral/fileexchange/28765-fresnels-and-fresnelc) (Efficient and accurate Fresnel integrals by John D'Errico) [BSD]
6460
- [fminsearchcon.m](https://www.mathworks.com/matlabcentral/fileexchange/8277-fminsearchbnd-fminsearchcon) (Bound constrained optimization using fminsearch by John D'Errico) [BSD]
65-
- [nlsqbnd.m](https://www.mathworks.com/matlabcentral/fileexchange/8277-fminsearchbnd-fminsearchcon) (Non-linear least squares solver with box constraints by Alain Barraud) [BSD]
66-
61+
- [nlsqbnd.m](https://ch.mathworks.com/matlabcentral/fileexchange/23621-nlsqbnd) (Non-linear least squares solver with box constraints by Alain Barraud) [BSD]
62+
- [golden.m](https://www.mathworks.com/matlabcentral/fileexchange/25919-golden-section-method-algorithm) (Golden Section method algorithm by Katarzyna Zarnowiec) [BSD]
63+
- [jacobianest.m](https://www.mathworks.com/matlabcentral/fileexchange/13490-adaptive-robust-numerical-differentiation) (Adaptive Robust Numerical Differentiation by John D'Errico) [BSD]
64+
- [kde.m](https://ch.mathworks.com/matlabcentral/fileexchange/14034-kernel-density-estimator) (Kernel Density Estimator by Zdravko Botev) [BSD]
65+
- [LevenbergMarquardt.m, jacobiansimple.m](https://ch.mathworks.com/matlabcentral/fileexchange/53449-levenberg-marquardt-toolbox)(Levenberg-Marquardt & Jacobian toolbox by Alexander Dentler)[BSD]
6766

67+
Copyright (c) 2019-2020: Luis Fabregas, Stefan Stoll, Gunnar Jeschke, and [other contributors](https://github.com/JeschkeLab/DeerLab/contributors).

‎build/compiletutorials.m‎

Lines changed: 0 additions & 54 deletions
This file was deleted.

0 commit comments

Comments
 (0)