Notes for developers
Creating a Conda environment
To create a separate Conda environment with all packages that hi-ml
requires for running and testing,
use the provided environment.yml
file. Create a Conda environment called himl
from that via
conda env create --file environment.yml
conda activate himl
Using specific versions hi-ml
in your Python environments
If you’d like to test specific changes to the hi-ml
package in your code, you can use two different routes:
You can clone the
hi-ml
repository on your machine, and usehi-ml
in your Python environment via a local package install:
pip install -e <your_git_folder>/hi-ml
You can consume an early version of the package from
test.pypi.org
viapip
:
pip install --extra-index-url https://test.pypi.org/simple/ hi-ml==0.1.0.post165
If you are using Conda, you can add an additional parameter for
pip
into the Condaenvironment.yml
file like this:
name: foo
dependencies:
- pip=20.1.1
- python=3.7.3
- pip:
- --extra-index-url https://test.pypi.org/simple/
- hi-ml==0.1.0.post165
Common things to do
The repository contains a makefile with definitions for common operations.
make check
: Runflake8
andmypy
on the repository.make test
: Runflake8
andmypy
on the repository, then all tests viapytest
make pip
: Install all packages for running and testing in the current interpreter.make conda
: Update the hi-ml Conda environment and activate it
Building documentation
To build the sphinx documentation, you must have sphinx and related packages installed
(see build_requirements.txt
in the repository root). Then run:
cd docs
make html
This will build all your documentation in docs/build/html
.
Setting up your AzureML workspace
In the browser, navigate to the AzureML workspace that you want to use for running your tests.
In the top right section, there will be a dropdown menu showing the name of your AzureML workspace. Expand that.
In the panel, there is a link “Download config file”. Click that.
This will download a file
config.json
. Move that file to the root folder of yourhi-ml
repository. The file name is already present in.gitignore
, and will hence not be checked in.
Creating and Deleting Docker Environments in AzureML
Passing a
docker_base_image
intosubmit_to_azure_if_needed
causes a new image to be built and registered in your workspace (see docs for more information).To remove an environment use the az ml environment delete function in the AzureML CLI (note that all the parameters need to be set, none are optional).
Testing
For all of the tests to work locally you will need to cache your AzureML credentials. One simple way to do this is to
run the example in src/health/azure/examples
(i.e. run python elevate_this.py --message='Hello World' --azureml
or
make example
) after editing elevate_this.py
to reference your compute cluster.
When running the tests locally, they can either be run against the source directly, or the source built into a package.
To run the tests against the source directly in the local
src
folder, ensure that there is no wheel in thedist
folder (for example by runningmake clean
). If a wheel is not detected, then the localsrc
folder will be copied into the temporary test folder as part of the test process.To run the tests against the source as a package, build it with
make build
. This will build the localsrc
folder into a new wheel in thedist
folder. This wheel will be detected and passed to AzureML as a private package as part of the test process.