WaveQC
Description
Quality for seismologic data at Resif
Visuals
Installation
First, ensure that git, python3, postgresql and rabbitmq are installed
sudo apt install git-core postgresql python3 rabbitmq-server redis npm
(Note that the minimum python version required is python 3.11 due to the syntax of type hints in waveqc code)
Then, install https://docs.astral.sh/uv/getting-started/installation/[uv]. Uv is a tool for dependency management and packaging python applications.
curl -LsSf https://astral.sh/uv/install.sh | sh
echo 'eval "$(uv generate-shell-completion bash)"' >> ~/.bashrc
echo 'eval "$(uvx --generate-shell-completion bash)"' >> ~/.bashrc
Then, clone the project from the Gitlab repository:
git clone git@gricad-gitlab.univ-grenoble-alpes.fr:OSUG/RESIF/waveqc.git
cd waveqc
Install python dependencies :
uv sync --frozen
# show the dependency tree for the project
uv tree
Configuration
Environment Variables
Copy the dotenv sample and modify the credentials for PostgreSql and Amqp You may also change the directory where the mseed and stationxml files will be temporary stored DEBUG should be set to 0 (or False) in a production environment
cp .env.sample .env
PostgreSQL
Create a PostGreSQL role and database dedicated to WaveQC with the right credentials
su - postgres
createuser --pwprompt <waveqc_user>
createdb -O <waveqc_user> waveqc
Get a dump of staging database and restore it on your local machine
# on your locale machine as the postgres user
su - postgres
pg_dump waveqc -h resif-pgpreprod.u-ga.fr -U waveqc --file waveqc.dump --format c
pg_restore --clean -d waveqc waveqc.dump
Migrations
Once the database is created, we can run project migrations
# run migrations
python manage.py migrate
Static files
See Parcel docs Generate static files by running the following commands
# install npm dependencies
npm install
# generate static files
npm run build
RabbitMQ
See Celery Doc for rabbitmq (Installing a rabbitmq-server on your local machine is not mandatory. If one is allready available somewhere, define its credentials in .env file)
Create a RabbitMQ user, vhost, and permissions for that user on that vhost. Use the same credentials as the one you defined in AMQP_* variables in your .env file
sudo rabbitmqctl add_user <waveqc_user> <waveqc_password>
sudo rabbitmqctl add_vhost <waveqc>
sudo rabbitmqctl set_user_tags <waveqc_user> <waveqc_tag>
sudo rabbitmqctl set_permissions -p <waveqc> <waveqc_user> ".*" ".*" ".*"
# And restart the rabbitmq-server
sudo systemctl restart rabbitmq-server.service
Redis
See Celery Doc for redis (Installing a redis-server on your local machine is not mandatory. If one is allready available somewhere, define its dsn in .env file)
sudo apt install redis
Celery
Using bash
For celery:
uv run celery -A waveqc worker -l INFO
For celery-beat
uv run celery -A waveqc beat -l INFO
Note that using this method, you'll need to run these commands each time you restart your computer In order to start celery and celery-beat along the system startup, you'll need to daemonize celery
Daemonization (caution, this section may not be up-to-date)
In order to use Celery tasks, we need to run it as a system service. See Celery Daemonization Doc
First, we need to create all necessary directories
sudo cp conf/tmpfiles/celery.conf /etc/tmpfiles.d/
sudo systemd-tmpfiles --create
Move configuration files from local repository to their destination
sudo cp conf/celery /etc/conf.d/
sudo cp conf/systemd/celery.service /etc/systemd/system/
sudo cp conf/systemd/celerybeat.service /etc/systemd/system/
You'll need to edit those configuration files:
- in
/etc/conf.d/celery
replace CELERY_BIN with the path of the celery bin (inside the virtualenv created by uv) - in
/etc/systemd/system/celery.service
replace :- WorkingDirectory with the path where you cloned the waveqc repository
- USER and GROUP with your user and your group
- in
/etc/systemd/system/celerybeat.service
replace :- WorkingDirectory with the path where you cloned the waveqc repository
- USER and GROUP with your user and your group
- in
/etc/tmpfiles.d/celery.conf
replace :- USER and GROUP with your user and your group
Finally, don't forget to enable and start celery daemon
sudo systemctl daemon-reload
sudo systemctl enable celery.service
sudo systemctl enable celerybeat.service
sudo systemctl start celery.service
sudo systemctl start celerybeat.service
Usage
Local server
In order to launch WaveQC in your browser :
python manage.py runserver
The application will be available at this url : http://localhost:8000/
Django Shell
A python shell is also available loaded with a bunch of modules among:
- models (Network, Station, etc...)
- usefull django stuff (settings, timezone, Session, etc...)
- some ORM functions (Min, Max, Avg, Prefetch, Subquery, etc...)
In order to launch Django shell :
python manage shell_plus
Admin user creation
To access the admin interface available at http://localhost:8000/admin/ use the following script
python manage.py createsuperuser
On production, only read access is possible with this admin interface. On development read/write rights are provided. Use this admin interface with caution !
Buildah/Podman
Once buildah and podman installed, you can build the waveqc docker image with the following command
buildah bud -t waveqc --layers --build-arg SENTRY_RELEASE=123456789 --network=host
... and run it with the following command (make sure you have the appropriate .env.docker env file)
In this .env.docker file, localhost
is replaced by host.containers.internal
to allow access to local database
podman run -p 8000:8000 --env-file=".env.docker" --rm waveqc sh -c "python manage.py collectstatic; gunicorn waveqc.wsgi --bind :8000"
Support
Our issue tracker is available here
Roadmap
Roadmap for this project is available in the original specifications
Contributing
If you're willing to contribute to this project, here are some tools that we are using to lint, test and format code.
Pre-commit hooks
First of, install the pre-commit hooks.
To do so, activate the virtual env with the following command and run the install command for pre-commit:
uv run pre-commit install
Now, each time you'll want to commit, several tasks will be run to ensure certain standards of code quality:
- ruff : a linter and formatter for python files
- djLint : a formatter for django templates
- django-upgrade : a tool that automatically upgrade jjango project code.
Configuration for pre-commit is located in .pre-commit-config.yaml
file
Gitlab Ci jobs
In addition to pre-commit hooks, the following jobs are run after each push:
Each of these tools can be run locally with the following commands : pytest
and mypy
Two more jobs may also be run:
- secret-detection : detects if password, secrets, ssh keys are present in repository
- semgrep-sast : Static Application Security Testing
For those two last jobs, reports are available in each merge requests as a json file
Tooling configuration
If you are curious about the configuration of all of these tools, please consult the pyproject.toml
file
Authors and acknowledgment
- Philippe Bollard (Project manager)
- Simon Panay (Developments)
- Laurent Stehly (Scientific overview)
WaveQC is funded by ISTerre
License
This software is licenced under the AGPLv3 Licence
Project status
In Development