Managing Jaeger and Elasticsearch ILM
Purpose
By default jaeger stores data in daily indices which may not be the ideal approach for many use cases. If your performance and retention requirements are a little more strict you can delegate the responsibility of rolling-over your indices to Elasticsearch.
The jaeger documentation already describes a use-case with ILM (I encourage you to read it), but it’s kinda funky since it relies on periodical API calls encapsulated inside the jaegertracing/jaeger-es-rollover docker image. The same outcome can be achieved by configuring elasticsearch beforehand, as we will see on this post.
Demo setup
Jaeger uses two indices jaeger-span and jaeger-service, all the operations described below need to be applied to both, but for the sake of simplicity, and to keep the post as concise and clear as possible, I will use jaeger-span as an example (all you need to do is replicate the exact same API requests replacing span with service). I’m running elasticsearch in a docker container locally. You can follow along by running the following docker-compose.
version: "3.7"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
environment:
- xpack.security.enabled=false
- discovery.type=single-node
ports:
- 9200:9200
Create a lifecycle policy
The bellow policy states that a rollover will happen when the current index either gets the max_size of 50 GB or the max_age of 30 days, it also states that indices older that 90 days will be deleted.
curl -X PUT "localhost:9200/_ilm/policy/jaeger-span?pretty" -H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50GB",
"max_age": "30d"
}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}
'
Create an index template
Adapted from here. One should notice that in the lifecycle parameter we are referring our already defined jaeger-span-policy, and our rollover_alias to be jaeger-span-write.
curl -X PUT "localhost:9200/_template/jaeger-span?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["jaeger-span-*"],
"settings":{
"index": {
"number_of_shards":2,
"number_of_replicas":2,
"mapping": {
"nested_fields": {
"limit": 50
}
},
"requests": {
"cache": {
"enable": true
}
},
"lifecycle": {
"name": "jaeger-span-policy",
"rollover_alias": "jaeger-span-write"
}
}
},
"aliases": {
"jaeger-span-read": {}
},
"mappings":{
"dynamic_templates":[
{
"span_tags_map":{
"mapping":{
"type":"keyword",
"ignore_above":256
},
"path_match":"tag.*"
}
},
{
"process_tags_map":{
"mapping":{
"type":"keyword",
"ignore_above":256
},
"path_match":"process.tag.*"
}
}
],
"properties":{
"traceID":{
"type":"keyword",
"ignore_above":256
},
"parentSpanID":{
"type":"keyword",
"ignore_above":256
},
"spanID":{
"type":"keyword",
"ignore_above":256
},
"operationName":{
"type":"keyword",
"ignore_above":256
},
"startTime":{
"type":"long"
},
"startTimeMillis":{
"type":"date",
"format":"epoch_millis"
},
"duration":{
"type":"long"
},
"flags":{
"type":"integer"
},
"logs":{
"type":"nested",
"dynamic":false,
"properties":{
"timestamp":{
"type":"long"
},
"fields":{
"type":"nested",
"dynamic":false,
"properties":{
"key":{
"type":"keyword",
"ignore_above":256
},
"value":{
"type":"keyword",
"ignore_above":256
},
"tagType":{
"type":"keyword",
"ignore_above":256
}
}
}
}
},
"process":{
"properties":{
"serviceName":{
"type":"keyword",
"ignore_above":256
},
"tag":{
"type":"object"
},
"tags":{
"type":"nested",
"dynamic":false,
"properties":{
"key":{
"type":"keyword",
"ignore_above":256
},
"value":{
"type":"keyword",
"ignore_above":256
},
"tagType":{
"type":"keyword",
"ignore_above":256
}
}
}
}
},
"references":{
"type":"nested",
"dynamic":false,
"properties":{
"refType":{
"type":"keyword",
"ignore_above":256
},
"traceID":{
"type":"keyword",
"ignore_above":256
},
"spanID":{
"type":"keyword",
"ignore_above":256
}
}
},
"tag":{
"type":"object"
},
"tags":{
"type":"nested",
"dynamic":false,
"properties":{
"key":{
"type":"keyword",
"ignore_above":256
},
"value":{
"type":"keyword",
"ignore_above":256
},
"tagType":{
"type":"keyword",
"ignore_above":256
}
}
}
}
}
}
'
Creating the first index
We need to create the first index.
curl -X PUT "localhost:9200/jaeger-span-000001" -H 'Content-Type: application/json' -d'
{
"aliases" : {
"jaeger-service-write": {
"is_write_index": "true"
}
},
"settings" : {
"number_of_shards" : 2,
"number_of_replicas" : 2
}
}
'
From now on, our rollover policy will take care of the index rotations. Naming them incrementally (e.g jaeger-span-000002, jaeger-span-000003…).
NOTE: you need to use the flags --es.use-aliases=true, --es.create-index-templates=false.