To make things easier for developers we provide a set automation scripts
implemented in invoke (similar to a makefile if you are familiar
with that).
So you will want to install invoke somehow to use this tooling. So make
sure you are in a suitable python environment (i.e. not the systems
python) and install the tools in the tools.requirements.txt file.
pip install -r tools.requirements.txt
This is for managing environments which are just for the purpose of developing this project, and not necessarily just for running it (as a user). See Managing Dependencies for details on managing dependencies of the installable project.
We also require a few shell environmental variables which are exported
in the env.bash file. Go ahead and source this before doing anything
else:
source env.bash
To create an env called jubeo.dev just run the env target from
invoke:
inv env
Then follow the activation instructions that are printed as different projects might use different backends.
If you ever have problems with an environment just rerun this to get a clean one. A practice we encourage to do frequently so that developers don’t diverge in their envs with local modifications. So while you can make your env, try to use this one unless you have problems.
We maintain a number of preconfigured environments in the envs
directory which are used for different purposes. Calling inv env -n
dev is the same as inv dev since it is the default, but any other
environment can be created by passing the matching name. For instance
there is an environment that mimics the user’s installation environment
so that we can test experiences upon install, to make sure we haven’t
accidentally depended on something in the dev env. For instance:
inv env -n base
If you want to make another environment it is straightforward to copy
the examples in the envs dir.
Reminder that there are two separate goals of managing dependencies and where they are managed:
These dependencies are managed in setup.py and in PyPI or other
indices.
These are dependencies managed in requirements.in and
requirements.txt and used for developer environments and
deployment environments.
So for the library aspect we use abstract requirements. These should
essentially be the same as requirements.in.
For the deployment side of things we use requirements.txt. Don’t
manually edit this. We use pip-tools to “compile” dependencies for
this.
To initially pin an environment or when you add requirements run this target:
inv deps-pin
To update it (should be accompanied by a reason why):
inv deps-pin-update
Editing and Building Docs
To compile and build the docs just run:
inv docs-build
Which will output them to a temporary build directory
_build/html.
You can clean this build with:
inv clean-docs
To view how the docs would look as a website you can point your
browser at the _build/html folder or run a python http web server
with this target:
inv docs-serve
Choosing a version number
Primordial Epoch
In the pre-covenant releases (think “1.0” release) we just release based on the date. So just make the date string and add any release candidate, dev, or postrelease additions.
Covenental Epoch
After the primordial epoch we have a more semantic meaning behind version numbers.
Following PEP 440 epochs these should all be prefixed by ‘1!’ for the second epoch.
Otherwise versions should be “canonical” strings following the guidelines in PEP 440.
That means we support roughly:
semver-ish major, minor, and patch parts
release candidates
dev
post release “patches”
However, we don’t prescribe explicit semantics to any of the parts as this is highly variable depending on the needs of a project and the forums by which it is published (e.g. PyPI, Github, etc.).
Writing and/or Generating the Changelog and Announcement
Changing the version number
You can check the current version number with this command:
inv version-which
Increase the version number we currently do it by hand (although an
automatic way would be nice and bumpversion wasn’t working for
us). So go in and manually edit them. For reference see PEP 440 for
valid ones.
The target locations for changing versions are in the following files:
[ ] setup.py
[ ] src/jubeo/__init__.py
[ ] sphinx/config.py
[ ] conda/conda-forge/meta.yaml
Making the release official
To make a release do some changes and make sure they are fully tested and functional and commit them in version control. At this point you will also want to do any rebasing or cleaning up the actual commits if this wasn’t already done in the feature branch.
If this is a ‘dev’ release and you just want to run a version control tag triggered CI pipeline go ahead and change the version numbers and commit. Then tag the ‘dev’ release.
If you intend to make a non-dev release you will first want to test it out a little bit with a release-candidate prerelease.
So do all the following bookeeping steps in a single but separate git commit from the actual changes to the code:
[ ] write the changelog
[ ] write the announcement (optional)
[ ] change the version numbers
[ ] build to test it out inv build
To test a build go ahead and run:
inv build
And then try to install it from an empty environment:
inv env -n test_install
Activate the environment e.g.:
source _venv/test_install/bin/activate
or
conda activate _conda_envs/test_install
then run it for each build, e.g.:
pip install dist/BUILD.tar.gz
They should all succeed.
When you change the version number put a ‘rc0’ at the end of the new intended (semantic) number.
Once you have built it and nothing is wrong go ahead and publish it to the test indexes (if available):
inv publish-test
You can test that it works from the index using the same
test_install environment above.
And install the package from the test repo with no dependencies:
pip install --index-url https://test.pypi.org/simple/ --no-deps jubeo
Then go ahead and commit the changes after that works. The message should follow a fixed form like
1.0.0rc0 release preparation
Then you can tag the release in the git commit history:
inv release
Publishing the results will vary but you can start with publishing the package to PyPI and the VCS hosts with the real publish target:
inv publish
These are tasks that should only be done once at the inception of the project but are described for posterity and completeness.
First we need to initialize the version control system (git):
inv vcs-init
If you want to go ahead and add the remote repositories for this
project. We don’t manage this explicitly since git is treated mostly
as first class for these kinds of tasks and is better left to special
purpose tools which are well integrated and developed.
Then add any extra dependencies you want to the development environment requirements.in file and then compile and pin them:
inv deps-pin
Then just create the virtual environment. For portability we use the
builin venv package, but this is customizable.
inv env
Then you can activate it with the instructions printed to the screen.
We use Github Pages by default since it is pretty easy. Because we don’t
want to clutter up the master branch with website build artifacts we use
the gh-pages branch approach.
If you just run the inv website-deploy target this will idempotently
take care of setting this up for you.
However, you will need to create it and push it before you can set this in the github settings for the page.