Skip to content
Snippets Groups Projects
Select Git revision
13 results Searching

waveqc

  • Clone with SSH
  • Clone with HTTPS
  • WaveQC

    Description

    Quality for seismologic data at Resif

    Visuals

    Installation

    First, ensure that git, python3, postgresql and rabbitmq are installed

    sudo apt install git-core postgresql python3 rabbitmq-server redis npm

    (Note that the minimum python version required is python 3.11 due to the syntax of type hints in waveqc code)

    Then, install https://docs.astral.sh/uv/getting-started/installation/[uv]. Uv is a tool for dependency management and packaging python applications.

    curl -LsSf https://astral.sh/uv/install.sh | sh
    echo 'eval "$(uv generate-shell-completion bash)"' >> ~/.bashrc
    echo 'eval "$(uvx --generate-shell-completion bash)"' >> ~/.bashrc

    Then, clone the project from the Gitlab repository:

    git clone git@gricad-gitlab.univ-grenoble-alpes.fr:OSUG/RESIF/waveqc.git
    cd waveqc

    Install python dependencies :

    uv sync --frozen
    
    # show the dependency tree for the project
    uv tree

    Configuration

    Environment Variables

    Copy the dotenv sample and modify the credentials for PostgreSql and Amqp You may also change the directory where the mseed and stationxml files will be temporary stored DEBUG should be set to 0 (or False) in a production environment

    cp .env.sample .env

    PostgreSQL

    Create a PostGreSQL role and database dedicated to WaveQC with the right credentials

    su - postgres
    createuser --pwprompt <waveqc_user>
    createdb -O <waveqc_user> waveqc

    Get a dump of staging database and restore it on your local machine

    # on your locale machine as the postgres user
    su - postgres
    pg_dump waveqc -h resif-pgpreprod.u-ga.fr -U waveqc --file waveqc.dump --format c
    pg_restore --clean -d waveqc waveqc.dump

    Migrations

    Once the database is created, we can run project migrations

    # run migrations
    python manage.py migrate

    Static files

    See Parcel docs Generate static files by running the following commands

    # install npm dependencies
    npm install
    
    # generate static files
    npm run build

    RabbitMQ

    See Celery Doc for rabbitmq (Installing a rabbitmq-server on your local machine is not mandatory. If one is allready available somewhere, define its credentials in .env file)

    Create a RabbitMQ user, vhost, and permissions for that user on that vhost. Use the same credentials as the one you defined in AMQP_* variables in your .env file

    sudo rabbitmqctl add_user <waveqc_user> <waveqc_password>
    sudo rabbitmqctl add_vhost <waveqc>
    sudo rabbitmqctl set_user_tags <waveqc_user> <waveqc_tag>
    sudo rabbitmqctl set_permissions -p <waveqc> <waveqc_user> ".*" ".*" ".*"
    
    # And restart the rabbitmq-server
    sudo systemctl restart rabbitmq-server.service

    Redis

    See Celery Doc for redis (Installing a redis-server on your local machine is not mandatory. If one is allready available somewhere, define its dsn in .env file)

    sudo apt install redis

    Celery

    Using bash

    For celery:

    uv run celery -A waveqc worker -l INFO

    For celery-beat

    uv run celery -A waveqc beat -l INFO

    Note that using this method, you'll need to run these commands each time you restart your computer In order to start celery and celery-beat along the system startup, you'll need to daemonize celery

    Daemonization (caution, this section may not be up-to-date)

    In order to use Celery tasks, we need to run it as a system service. See Celery Daemonization Doc

    First, we need to create all necessary directories

    sudo cp conf/tmpfiles/celery.conf /etc/tmpfiles.d/
    sudo systemd-tmpfiles --create

    Move configuration files from local repository to their destination

    sudo cp conf/celery /etc/conf.d/
    sudo cp conf/systemd/celery.service /etc/systemd/system/
    sudo cp conf/systemd/celerybeat.service /etc/systemd/system/

    You'll need to edit those configuration files:

    • in /etc/conf.d/celery replace CELERY_BIN with the path of the celery bin (inside the virtualenv created by uv)
    • in /etc/systemd/system/celery.service replace :
      • WorkingDirectory with the path where you cloned the waveqc repository
      • USER and GROUP with your user and your group
    • in /etc/systemd/system/celerybeat.service replace :
      • WorkingDirectory with the path where you cloned the waveqc repository
      • USER and GROUP with your user and your group
    • in /etc/tmpfiles.d/celery.conf replace :
      • USER and GROUP with your user and your group

    Finally, don't forget to enable and start celery daemon

    sudo systemctl daemon-reload
    sudo systemctl enable celery.service
    sudo systemctl enable celerybeat.service
    sudo systemctl start celery.service
    sudo systemctl start celerybeat.service

    Usage

    Local server

    In order to launch WaveQC in your browser :

    python manage.py runserver

    The application will be available at this url : http://localhost:8000/

    Django Shell

    A python shell is also available loaded with a bunch of modules among:

    • models (Network, Station, etc...)
    • usefull django stuff (settings, timezone, Session, etc...)
    • some ORM functions (Min, Max, Avg, Prefetch, Subquery, etc...)

    In order to launch Django shell :

    python manage shell_plus

    Admin user creation

    To access the admin interface available at http://localhost:8000/admin/ use the following script

    python manage.py createsuperuser

    On production, only read access is possible with this admin interface. On development read/write rights are provided. Use this admin interface with caution !

    Buildah/Podman

    Once buildah and podman installed, you can build the waveqc docker image with the following command

    buildah bud -t waveqc --layers --build-arg SENTRY_RELEASE=123456789 --network=host

    ... and run it with the following command (make sure you have the appropriate .env.docker env file) In this .env.docker file, localhost is replaced by host.containers.internal to allow access to local database

    podman run -p 8000:8000 --env-file=".env.docker" --rm waveqc  sh -c "python manage.py collectstatic; gunicorn waveqc.wsgi --bind :8000"

    Support

    Our issue tracker is available here

    Roadmap

    Roadmap for this project is available in the original specifications

    Contributing

    If you're willing to contribute to this project, here are some tools that we are using to lint, test and format code.

    Pre-commit hooks

    First of, install the pre-commit hooks.

    To do so, activate the virtual env with the following command and run the install command for pre-commit:

    uv run pre-commit install

    Now, each time you'll want to commit, several tasks will be run to ensure certain standards of code quality:

    • ruff : a linter and formatter for python files
    • djLint : a formatter for django templates
    • django-upgrade : a tool that automatically upgrade jjango project code.

    Configuration for pre-commit is located in .pre-commit-config.yaml file

    Gitlab Ci jobs

    In addition to pre-commit hooks, the following jobs are run after each push:

    • pytest : unit tests
    • mypy : a type checker for python files

    Each of these tools can be run locally with the following commands : pytest and mypy

    Two more jobs may also be run:

    For those two last jobs, reports are available in each merge requests as a json file

    Tooling configuration

    If you are curious about the configuration of all of these tools, please consult the pyproject.toml file

    Authors and acknowledgment

    • Philippe Bollard (Project manager)
    • Simon Panay (Developments)
    • Laurent Stehly (Scientific overview)

    WaveQC is funded by ISTerre

    License

    This software is licenced under the AGPLv3 Licence

    Project status

    In Development