3. Install Remote Workers

cyberCommons can scale horizontally by allowing remote workers to take on tasks and execute them on remote systems. The following describes how to setup a remote Celery worker for use with cyberCommons. Celery is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

3.1. Requirements

  • PIP - Install

  • Copies of client certificates and credentials to communicate with central cyberCommons server:

    • MongoDB

      • dc_config/ssl/backend/client/mongodb.pem

      • dc_config/ssl/testca/cacert.pem

    • RabbitMQ

      • dc_config/ssl/backend/client/key.pem

      • dc_config/ssl/backend/client/cert.pem

      • dc_config/ssl/testca/cacert.pem

  • RabbitMQ and MongoDB ports are open by default:

    • RabbitMQ port 5671

    • MongoDB port 27017

3.2. Install Celery

  1. Create virtual environment and activate

    python -m venv virtpy
source virtpy/bin/activate

  2. Install Celery

    (virtpy) $ pip install Celery

3.3. Configuration

3.3.1. Get Config Files and Certificates

  1. Download example celeryconfig.py and requirements.txt

wget https://raw.githubusercontent.com/cybercommons/cybercommons/master/docs/pages/files/celeryconfig.py

  1. Create SSL directory and copy cyberCommon’s client certificates

mkdir ssl
cp mongodb.pem ssl/
cp key.pem ssl/
cp cert.pem ssl/
cp cacert.pem ssl/

  1. Configure celeryconfig.py to point to client certificates and use corresponding credentials (values in this example between “<” and “>” need to be updated to match your cyberCommon’s configuration. Do not include the “<” and “>” characters.)

broker_url = 'amqp://<username>:<password>@<broker_host>:<broker_port>/<broker_vhost>'
broker_use_ssl = {
    'keyfile': 'ssl/key.pem',
    'certfile': 'ssl/cert.pem',
    'ca_certs': 'ssl/cacert.pem',
    'cert_reqs': ssl.CERT_REQUIRED
}


result_backend = "mongodb://<username>:<password>@<mongo_host>:<mongo_port>/?ssl=true&ssl_ca_certs=ssl/cacert.pem>&ssl_certfile=mongodb.pem>"

mongodb_backend_settings = {
    "database": "<application_short_name>",
    "taskmeta_collection": "tombstone"
}

3.3.2. Configure Tasks

  1. Update requirements.txt to include desired libraries and task handlers.

  2. Update celeryconfig.py to import task handlers that have been included in requirements file.

    imports = ("cybercomq", "name_of_additional_task_handler_library", )

  3. Install requirements

    (virtpy) $ pip install -r requirements.txt

3.3.3. Launch Celery worker

  1. Run in foreground. See Celery Worker Documentation for more information.

    celery worker -Q remote -l INFO -n dev-hostname