howto_new_distribution.txt

NOTES/CHECKLIST FOR CREATING AND DISTRIBUTION A NEW DISTRIBUTION


Working by default in ~/coding/pyimfit on Mac.
[mac] = macOS (either version)
[mac-x86] = Intel macOS
[mac-arm64] = arm64 (Apple silicon) macOS
[vm] = Linux VM on Intel macOS

[opt] = possibly optional (i.e., do only "IF NECESSARY")


GENERAL PREP

[opt]1.a Generate updated libimfit.a for macOS [x86-64], IF NECESSARY
    A. [mac-x86] cd imfit
    B. [mac-x86] git pull origin master   [if necessary]
    C. [mac-x86] scons --clang-openmp libimfit.a
    D. [mac-x86] cp libimfit.a ../prebuilt/macos
    # copy updated Imfit C++ header files
    E. [mamac-x86c] cd .. ; ./update_from_imfit.sh

[opt]1.b Generate updated libimfit.a for macOS [arm64], IF NECESSARY
    A. [mac-arm64] cd imfit
    B. [mac-arm64] git pull origin master   [if necessary]
    C. [mac-arm64]] scons --clang-openmp libimfit.a
    D. [mac-arm64]] cp libimfit.a ../prebuilt/macos
    # copy updated Imfit C++ header files
    E. [mac-arm64]] cd .. ; ./update_from_imfit.sh

[opt]2. Generate updated libimfit.a for Linux [VM: ubuntu64-18_iraf], IF NECESSARY
    A. [vm] cd /home/vagrant/build/pyimfit/imfit
    B. [vm] git pull origin master   [if necessary]
    C. [vm] scons libimfit.a
    D. [vm] cp libimfit.a /vagrant/transfer
    E. [mac] cp ~/vagrant/ubuntu64-18_iraf/transfer/libimfit.a ./prebuilt/linux64

3.A. Do new "develop" install and run tests on macOS-x86
    [mac-x86] $ pip3 install -e ./
    [mac-x86] $ cd pyimfit/tests ; pytest

3.B. Do new "develop" install and run tests on macOS-arm64
    [mac-arm64] $ pip3 install -e ./
    [mac-arm64] $ cd pyimfit/tests ; pytest

4. Do new "develop" install and run tests on Linux VM
    (note: currently we do this using the ubuntu64-16_dev VM and our conda installation,
    because that has the right Python installations, including numpy)
    [vm] $ cd ~/build
    [vm] $ git clone https://github.com/perwin/pyimfit.git
    [vm] $ cd pyimfit
    [vm] $ conda activate
    [vm] $ python -m pip install --user -e ./
    [vm] $ cd pyimfit/tests ; pytest

5. Update version numbers
    A. Update version number in pyimfit/__init__.py
    B. Update version number in docs/conf.py
    C. Update version number in conda/rattler_setup/recipe.yaml

6. Generate new version of docs [docs/howto_generate_docs.txt]
    A. cd docs
    (B. Update version number in conf.py -- if not done in stage 5)
    C. IF the Jupyter notebook files have been updated, generate markdown versions with
        [NOTE: this currently works on my Intel MBPro *if* I do it from the py312_working conda
        environment]
        $ jupyter nbconvert --to markdown pyimfit_emcee.ipynb
        $ jupyter nbconvert pyimfit_bootstrap_BtoT.ipynb --to markdown
    D. Generate updated documentation
	    $ ./make_docs.sh
	E. View resulting docs in a web browser pointed at
	    file:///Users/erwin/coding/pyimfit/docs/_build/html/index.html
    F. Commit the changes, upload to github
        $ git commit -a -m "Updated documentation"      [or something like that]
        $ git push

7. See if updated version passes Github Workflows CI tests


GENERATE WHEELS FOR PIP INSTALLATION:

8.A. Generate source distribution and Mac binary wheels on Intel Mac:
    [max-x86] $ ./make_upload.py x.x.x   [where "x.x.x" = version number]

8.B. Same, on arm64 Mac
    [mac-arm64] $ ./make_upload.py x.x.x   [where "x.x.x" = version number]

8.C. Same, on Linux VM
    [vm] $ git pull
    [vm] $ ./make_upload.py x.x.x   [where "x.x.x" = version number]

8.D. Copy wheels to one Mac (arm64 to Intel, or vice-versa)
    copy binary wheels to ~/coding/pyimfit/dist/fixed_wheels/

9. Upload wheels to PyPI
    $ ./make_upload.py x.x.x --skip-build --test-upload
    $ ./make_upload.py x.x.x --skip-build --upload

10. Test installation via pip
    pip3 install --extra-index-url https://test.pypi.org/simple pyimfit

    (Note: don't use --index-url or -i, because that tells Pip to *only* look at that
    URL, causing it to be unable to find things like setuptools)


GENERATE CONDA PACKAGES:

Note that we can in principle upload conda packages to anaconda from either Intel or arm64 mac.
The following steps assume we are doing the upload from Intel mac.
For the time being, it's probably best to focus on Intel mac, since we run the Linux VM there.

11. Generate conda packages on Intel Mac
    A. [mac-x86] cd conda
    B. [mac-x86] rattler-build build --recipe ./rattler_setup --variant-config ./rattler_setup/variant_config.yaml

12. Generate conda packages on arm64 Mac
    A. [mac-arm64] git pull   (to get e.g. updated version numbers)
    B. [mac-arm64] cd conda
    C. [mac-arm64] rattler-build build --recipe ./rattler_setup --variant-config ./rattler_setup/variant_config.yaml
    D. [if using Intel Mac for uploading:] Transfer conda packages to Intel Mac; place them in
        .../pyimfit/conda/output/osx-arm64/

13. Generate conda packages on Linux VM (Intel Mac: /Users/erwin/vagrant/iraf)
    A. Set VM memory to 4 GB (via VirtualBox)
    B. Start up VM
    C. [vm] cd /home/vagrant/build/pyimfit
    D. [vm] git pull   (to get e.g. updated version numbers)
    E. [vm] cd conda
    F. [vm] rattler-build build --recipe ./rattler_setup --variant-config ./rattler_setup/variant_config.yaml
    G. [vm] Transfer new package files to Mac
        $ cp output/linux-64/pyimfit-<VERSION_NUMBER>*.conda /vagrant/transfer
        H. [vm] tar -cf linux-64.tar linux-64 && gzip linux-64.tar
    H. [if using Intel Mac for uploading:] Transfer conda packages to Intel Mac:
        $ cp ~/vagrant/iraf/transfer/pyimfit-<VERSION_NUMBER>*.conda ~/coding/pyimfit/conda/output/linux-64/
    I. If everything went OK, shut down the VM and reset its memory allocation to 2 GB.

    ALTERNATIVE: Do this on ex-sol.mpe.mpg.de
    A. ssh erwin@ex-sol.mpe.mpg.de
    B. cd data2/build/pyimfit
    C. git pull
    D. cd conda
    E. rattler-build build --recipe ./rattler_setup --variant-config ./rattler_setup/variant_config.yaml
    F. Transfer newly-generated conda packages from conda/output/linux64/ using Transmit


UPLOAD CONDA PACKAGES:

14. Upload conda packages to anaconda [from Intel Mac]
    A. [mac-x86] cd conda
    B. [mac-x86] conda activate
    C. [mac-x86] anaconda login   [if necessary; not necessary unless starting conda from scratch]
    D. [mac-x86] anaconda upload output/osx-64/pyimfit-<VERSION>*.conda output/osx-arm64/pyimfit-<VERSION>*.conda output/linux-64/pyimfit-<VERSION>*.conda

15. Test conda installation
    A. [mac-x86] -- repeat for each of py310, py311, py312
        $ conda activate py310_test
        $ conda install -y -c conda-forge perwin::pyimfit
        $ cd pyimfit/conda
        $ ./test_pyimfit_install.py
        $ conda uninstall -y pyimfit
    B. [mac-arm64] -- same
    c. [vm] -- same