Extension `chaostracing`¶


Version	0.4.0
Repository	https://github.com/chaostoolkit-incubator/chaostoolkit-opentracing

This project is an extension for the Chaos Toolkit for OpenTracing and OpenTelemetry.

Here is an example of what it could look like with the Jaeger backend.

OpenTracing

Install¶

This package requires Python 3.6+

To be used from your experiment, this package must be installed in the Python environment where chaostoolkit already lives.

$ pip install -U chaostoolkit-opentracing

Usage¶

Currently, this extension only provides control support to send traces to your provider during the execution of the experiment. It does not yet expose any probes or actions per-se.

NOTE: Please see at the bottom of the page all the supported clients and exporters this control supports.

Declare within the experiment¶

To use this control, you can declare it on a per experiment basis like this:

{
    "configuration": {
        "tracing_provider": "jaeger",
        "tracing_host": "127.0.0.1",
        "tracing_port": 6831,
        "tracing_propagation": "b3"
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

This will automatically create a Jaeger client to emit traces onto the address 127.0.0.1:6831 (over UDP).

Declare within the settings¶

You may also declare the control to be applied to all experiments by declaring the control from within the Chaos Toolkit settings file. In that case, you do not need to set the configuration or the controls at the experiment level and the control will be applied to every experiments you run.

controls:
  opentracing:
    provider:
      type: python
      module: chaostracing.control
      arguments:
        provider: jaeger
        host: 127.0.0.1
        port: 6831
        propagation: b3

Send traces from other extensions¶

You may also access the tracer from other extensions as follows.

For instance, assuming you have an extension that makes a HTTP call you want to trace specifically, you could do this from your extension’s code:

from chaoslib import Configuration, Secrets
import requests
import opentracing

def some_function(configuration: Configuration, secrets: Secrets):
    tracer = opentracing.global_tracer()
    scope = tracer.scope_manager.active
    parent = scope.span

    with tracer.start_span("call-service1", child_of=parent) as span:
        span.set_tag('http.method','GET')
        span.set_tag('http.url', url)
        span.set_tag('span.kind', 'client')
        span.tracer.inject(span, 'http_headers', headers)

        r = requests.get(url, headers=headers)
        span.set_tag('http.status_code', r.status_code)

Because the opentracing exposes a noop tracer when non has been initialized, it should be safe to have that code in your extensions without having to determine if the extension has been enabled in the experiment.

Please note that, Open Tracing scope cannot be shared across threads (while spans can). So, when running this in a background activity, the tracer will not actually be set to the one that was initialized.

Open Tracing Provider Support¶

Jaeger tracer¶

The Jager tracer relies on the OpenTracing protocol which has now be superseded by OpenTelemetry. However, we still provide support for it.

To install the necessary dependencies for the Jaeger tracer, please run:

$ pip install -U jaeger-client~=4.8

Use the following configuration:

{
    "configuration": {
        "tracing_provider": "jaeger",
        "tracing_host": "127.0.0.1",
        "tracing_port": 6831,
        "tracing_propagation": "b3"
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

OpenTelemetry provider¶

To install the baseline dependencies for the OpenTelemetry tracer, please run:

$ pip install -U opentelemetry-api \
    opentelemetry-sdk \
    opentelemetry-opentracing-shim

If you want the b3 propagator, please also install:

$ pip install opentelemetry-propagator-b3

In that case, in all configurations below, add:

"tracing_opentelemetry_baggage_prefix": "b3"

Jaeger thrift exporter¶

If you want to export using the Jaeger thrift protocol, please install:

$ pip install opentelemetry-exporter-jaeger-thrift

Use the following configuration:

{
    "configuration": {
        "tracing_provider": "opentelemetry",
        "tracing_opentelemetry_exporter": "jaeger-thrift",
        "tracing_host": "127.0.0.1",
        "tracing_port": 6831
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

Jaeger grpc exporter¶

If you want to export using the Jaeger grpc protocol, please install:

$ pip install opentelemetry-exporter-jaeger-proto-grpc

Use the following configuration:

{
    "configuration": {
        "tracing_provider": "opentelemetry",
        "tracing_opentelemetry_exporter": "jaeger-grpc",
        "tracing_opentelemetry_collector_endpoint": "localhost:14250",
        "tracing_opentelemetry_collector_endpoint_insecure": true
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

OLTP grpc exporter¶

If you want to export using the OLTP grpc protocol, please install:

$ pip install opentelemetry-exporter-otlp-proto-grpc

Use the following configuration:

{
    "configuration": {
        "tracing_provider": "opentelemetry",
        "tracing_opentelemetry_exporter": "oltp-grpc",
        "tracing_opentelemetry_collector_endpoint": "http://localhost:4317",
        "tracing_opentelemetry_collector_endpoint_insecure": true
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

OLTP HTTP exporter¶

If you want to export using the OLTP HTTP protocol, please install:

$ pip install opentelemetry-exporter-otlp-proto-http

Use the following configuration:

{
    "configuration": {
        "tracing_provider": "opentelemetry",
        "tracing_opentelemetry_exporter": "oltp-http",
        "tracing_opentelemetry_collector_endpoint": "http://localhost:4318",
        "tracing_opentelemetry_collector_endpoint_insecure": true,
        "tracing_opentelemetry_collector_headers": {
            "My-Header": "My-value"
        }
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

Run from containers¶

From Docker¶

Create the following Dockerfile:

FROM chaostoolkit/chaostoolkit:latest

RUN pip install --no-cache-dir -q -U chaostoolkit-opentracing jaeger-client

Obviously adapt the dependencies based on which provider/exporter you want to use.

Then run this image by mounting your experiment file, for instance, something like:

$ docker run -v `pwd`/experiment.json:/home/svc/experiment.json my-image run experiment.json

Make sure you you correctly set the IP address of the traces agent/collector so it can be reached. You can use localhost if you link networks between containers of course too.

Here is an example of a Dockerfile to create an image with all the providers/exporters and their dependencies:

FROM chaostoolkit/chaostoolkit:latest

USER root
RUN apk add --no-cache --virtual build-deps gcc g++ git libffi-dev linux-headers \
        python3-dev musl-dev && \
    apk add libstdc++ && \
    pip install --no-cache-dir -q -U pip setuptools && \
    pip install --prefer-binary --no-cache-dir -q -U chaostoolkit \
    chaostoolkit-opentracing \
    jaeger-client \
    opentelemetry-exporter-jaeger-proto-grpc \
    opentelemetry-api \
    opentelemetry-sdk \
    opentelemetry-opentracing-shim && \
    opentelemetry-propagator-b3 && \
    apk del build-deps
USER 1001

From Kubernetes¶

Assuming you have a container image with the chaostoolkit-opentracing extension installed:

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: ctk-tracing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctk-tracing
  template:
    metadata:
      name: ctk-tracing
      labels:
        app: ctk-tracing
    spec:
      containers:
      - image: my-image
        name: ctk-tracing
        imagePullPolicy: Always
        command:
          - /usr/local/bin/chaos
          - run
          - https://raw.githubusercontent.com/some/place/experiment.json

Obviously you can deliver the experiment as mounted file via volume as well. Again, simply make sure you set the correct address to send the traces to.

Test¶

To run the tests for the project execute the following:

$ pytest

Contribute¶

If you wish to contribute more functions to this package, you are more than welcome to do so. Please, fork this project, make your changes following the usual PEP 8 code style, sprinkling with tests and submit a PR for review.

The Chaos Toolkit projects require all contributors must sign a Developer Certificate of Origin on each commit they would like to merge into the master branch of the repository. Please, make sure you can abide by the rules of the DCO before submitting a PR.

Exported Controls¶

control¶

This module exports controls covering the following phases of the execution of an experiment:

Level	Before	After
Experiment Loading	False	False
Experiment	True	True
Steady-state Hypothesis	True	True
Method	True	True
Rollback	True	True
Activities	True	True

In addition, the controls may define the followings:

Level	Enabled
Validate Control	False
Configure Control	True
Cleanup Control	True

To use this control module, please add the following section to your experiment:

JSONYAML

{
  "controls": [
    {
      "name": "chaostracing",
      "provider": {
        "type": "python",
        "module": "chaostracing.control"
      }
    }
  ]
}

controls:
- name: chaostracing
  provider:
    module: chaostracing.control
    type: python

This block may also be enabled at any other level (steady-state hypothesis or activity) to focus only on that level.

When enabled at the experiment level, by default, all sub-levels are also applied unless you set the automatic properties to false.

Extension chaostracing¶

Install¶

Usage¶

Declare within the experiment¶

Declare within the settings¶

Send traces from other extensions¶

Open Tracing Provider Support¶

Jaeger tracer¶

OpenTelemetry provider¶

Jaeger thrift exporter¶

Jaeger grpc exporter¶

OLTP grpc exporter¶

OLTP HTTP exporter¶

Run from containers¶

From Docker¶

From Kubernetes¶

Test¶

Contribute¶

Exported Controls¶

control¶

Exported Activities¶

control¶

Extension `chaostracing`¶