Background
As part of unifying the developer experience and enabling a more uniform observability stack for one company, I worked on centralizing multiple log sources into a single pane from which the team could set up alerts.
Log centralization can be accomplished in many different ways. In the context of this article, the problem I faced was collecting information from multiple sources into Grafana Loki, which is said to offer a lower maintenance overhead compared to, for example, ElasticSearch (more generally, ELK stack). One troublesome source happened to be an older appliance device that could only be configured to send syslog messages through UDP (with, well, undocumented and custom message format).
Similar articles
Here are a couple of similar articles that got me started on the issue:
- Using Rsyslog and Promtail to relay syslog messages to Loki
- Sending logs from syslog-ng to Grafana Loki
- How I fell in love with logs thanks to Grafana Loki
- Create Rsyslog Service in Kubernetes
Syslog Forwarding Challenges
Sending syslog messages to Loki seems almost straightforward, thanks to relays like Promtail or Vector, designed to listen to syslog messages and feed them to Loki [1] [2]. However, as Grafana's documentation cautions, it's not all smooth sailing:
The recommended deployment is to have a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can handle various specifications and transports that exist (UDP, BSD syslog, …).
Digging deeper into the specifications, some restrictions imposed by the Promtail syslog listener include:
- Listening only functions for TCP (open issue)
- The message format must adhere to IETF Syslog (RFC5424) "standard."
Before delving into any work, I confirmed that similar conditions apply to other agents, such as Grafana Agent or Vector. While they support some use cases, nothing worked out of the box in our context. Issues ranged from lack of UDP support to sporadic UDP support or rejection of messages due to custom formats (for instance, Vector only supports "common variations").
In a straightforward setup, connecting Promtail and Loki might suffice. However, not all external vendors meet the specified conditions (e.g., messages may only be sent via UDP, or the format is bespoke). Alternatively, it may be that the code generating the syslog messages is beyond our control and follows a peculiar specification.
Another infrastructural requirement imposed by my specific environment was container usage, specifically deploying this on Kubernetes. In this article, I'll guide you through a basic setup for deploying rsyslog in a containerized environment, testing it locally with docker-compose, and as a bonus, present a basic Kubernetes setup.
Streamlining with Syslog Forwarding
As per the guidelines laid out in Grafana's aforementioned documentation, the most straightforward approach involves setting up a syslog forwarder. This forwarder should be adept at handling various formats and transports, seamlessly relaying messages to Promtail, which, in turn, dispatches them to Loki. A nifty solution lies in leveraging rsyslog's Alpine Docker image project, conveniently accessible on DockerHub. Despite the container image's vintage, the syslog landscape hasn't undergone significant changes in the past six years. Now, let's delve into the configuration details.
Diagram
The following diagram outlines the solution:
Prerequisites
This article operates under the assumption that the reader possesses a foundational understanding of Docker and Docker Compose, emphasizing the creation of a consistent and replicable environment using these tools to evaluate the rsyslog forwarder.
Please ensure you have the following tools at your disposal:
Setup and configuration
The next sections will walk you through a basic setup using docker compose so that you can follow along and test it in a reproducible environment.
Rsyslog
The configuration underneath is telling rsyslog how to forward its inputs to Promtail:
# rsyslog.conf
module(load="imptcp")
module(load="imudp" TimeRequery="500")
input(type="imptcp" port="10514")
input(type="imudp" port="10514")
module(load="omprog")
module(load="mmutf8fix")
action(type="mmutf8fix" replacementChar="?")
*.* action(type="omfwd" Target="promtail" Port="1514" Protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format")
The first four lines in the configuration detail that our application is set up to listen for both UDP and TCP connections on port 10514. Moving on, the next paragraph specifies that any incoming data should be directed to a target named promtail. This promtail service is identified by its DNS name and expects TCP connections on port 1514. Additionally, it mandates that the forwarded message should adhere to the RSYSLOG_SyslogProtocol23Format template. For an extra layer of robustness, I've included the mmutf8fix module, capable of handling any non-UTF-8 characters in the input. The use of this module is entirely optional.
You can write a simple standalone docker-compose.yaml file to validate the container comes up properly:
# docker-compose.yaml
services:
rsyslog:
image: rsyslog/syslog_appliance_alpine
ports:
- "10514:10514/tcp"
- "10514:10514/udp"
volumes:
- ./rsyslog.conf:/config/rsyslog.conf
- data:/work
environment:
RSYSLOG_CONF: "/config/rsyslog.conf"
volumes:
data:
The usage of RSYSLOG_CONF is explained in the official documentation, the data volume is added because it is a staging work directory for rsyslog that needs to be preserved across restarts. The above configuration can be tested with docker compose up. To validate that the messages can be sent via UDP you can run:
echo '<165>4 An application event log entry...' | nc -v -u localhost 10514
You should see a similar output to the one underneath for the command:
Connection to localhost (127.0.0.1) 10514 port [udp/*] succeeded!
Promtail
The following configuration allows for listening for TCP syslog messages and forwarding them to Loki:
# promtail.yaml
server:
http_listen_port: 9081
grpc_listen_port: 0
positions:
filename: /var/tmp/promtail-syslog-positions.yml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: syslog
syslog:
listen_address: 0.0.0.0:1514
labels:
job: syslog
relabel_configs:
- source_labels:
- __syslog_message_hostname
target_label: hostname
- source_labels:
- __syslog_message_app_name
target_label: app
- source_labels:
- __syslog_message_severity
target_label: level
By extending the docker-compose.yaml file, it is easy to validate that both start up:
@@ -9,6 +9,12 @@ services:
- data:/work
environment:
RSYSLOG_CONF: "/config/rsyslog.conf"
+ promtail:
+ image: grafana/promtail:2.9.4
+ volumes:
+ - ./promtail.yaml:/promtail.yaml
+ command:
+ - -config.file=/promtail.yaml
volumes:
data:
After executing docker compose up again and testing with:
echo '<165>4 2018-10-11T22:14:15.003Z mymach.it e - 1 [ex@32473 iut="3"] An application event log entry...' | nc -v -u localhost 10514
We can see that (after a couple internal retries and finally buffering the message) the log can't be sent to Loki since we have not yet started it (dial tcp: lookup loki on 127.0.0.11:53: server misbehaving):
rsyslog-promtail-1 | level=warn ts=2024-02-22T17:29:09.020479781Z caller=client.go:419 component=client host=loki:3100 msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki:3100/loki/api/v1/push\": dial tcp: lookup loki on 127.0.0.11:53: server misbehaving"
Loki
Spinning up Loki so that Promtail can push logs to it is as simple as:
@@ -15,6 +15,10 @@ services:
- ./promtail.yaml:/promtail.yaml
command:
- -config.file=/promtail.yaml
+ loki:
+ image: grafana/loki:2.9.4
+ ports:
+ - 31000:3100
volumes:
data:
Then once again after restarting the compose application (by killing the current pane and running docker compose up), one final test can be performed:
echo '<165>4 first 2018-10-11T22:14:15.003Z mymach.it e - 1 [ex@32473 iut="3"] An application event log entry...' | nc -v -u localhost 10514
echo '<165>4 second 2018-10-11T22:14:15.003Z mymach.it e - 1 [ex@32473 iut="3"] An application event log entry...' | nc -v -u localhost 10514
Due to the internal buffering the push above is performed twice so that the first message is pushed to Promtail and Loki so that we can validate whether the setup works:
curl http://localhost:31000/loki/api/v1/series
# {"status":"success","data":[{"level":"notice","job":"syslog","hostname":"4","app":"first"}]}
The second message should arrive later when more data is pushed.
Docker compose
To provide a holistic view, here's the final Docker Compose file:
# docker-compose.yaml
services:
rsyslog:
image: rsyslog/syslog_appliance_alpine
ports:
- "10514:10514/tcp"
- "10514:10514/udp"
volumes:
- ./rsyslog.conf:/config/rsyslog.conf
- data:/work
environment:
RSYSLOG_CONF: "/config/rsyslog.conf"
promtail:
image: grafana/promtail:2.9.4
volumes:
- ./promtail.yaml:/promtail.yaml
command:
- -config.file=/promtail.yaml
loki:
image: grafana/loki:2.9.4
ports:
- 31000:3100
volumes:
data:
Summary: Expanding Possibilities
Within this article, I've introduced a fundamental technique to channel UDP syslogs, even with varying formats, to Loki through Promtail. The beauty of this approach lies in its adaptability. You can broaden its scope by tweaking configurations, such as extending rsyslog's setup or adjusting Promtail's forwarding and labeling mechanisms. This flexibility empowers you to tailor the solution to diverse data formats, making it a versatile foundation for your syslog forwarding endeavors.
Bonus: Simple setup in Kubernetes
This configuration can be easily mapped to Kubernetes abstractions to allow for a simple deployment and usage.
Rsyslog
The user needs some kind of an ingress controller or a load balancer to expose the rsyslog forwarder service, and as these are heavily dependent on the context, the following configuration does not take this fully into account.
Configuration
The configuration itself can be kept in a config map for simplicity:
# cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: rsyslog-config
data:
rsyslog.conf: |
module(load="imptcp")
module(load="imudp" TimeRequery="500")
input(type="imptcp" port="10514")
input(type="imudp" port="10514")
module(load="omprog")
module(load="mmutf8fix")
action(type="mmutf8fix" replacementChar="?")
*.* action(type="omfwd" Target="promtail" Port="10514" Protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format")
An alternative approach would be to bake this in the container image itself.
Deployment
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: rsyslog-forwarder
labels:
app: rsyslog-forwarder
spec:
selector:
matchLabels:
app: rsyslog-forwarder
template:
metadata:
labels:
app: rsyslog-forwarder
spec:
containers:
- name: rsyslog-forwarder
image: rsyslog/syslog_appliance_alpine:latest
env:
- name: "RSYSLOG_CONF"
value: "/config/rsyslog.conf"
ports:
- containerPort: 10514
name: syslogudp
protocol: UDP
volumeMounts:
- name: rsyslog-work
mountPath: /work
- name: rsyslog-config
mountPath: /config/rsyslog.conf
readOnly: true
subPath: rsyslog.conf
volumes:
- name: rsyslog-work
persistentVolumeClaim:
claimName: rsyslog-work # TODO: pvc needs to be provided externally
- name: rsyslog-config
configMap:
name: rsyslog-config
Take note of the volume named rsyslog-work. It's hooked up using a persistent volume claim (PVC). This specific storage area plays a crucial role for rsyslog – think of it as the backstage where things get organized.
Now, the nitty-gritty details about which PVC to choose and how to set it up are beyond the scope of this article. Different environments may have various options. If you're in a public cloud, like Google Cloud, making sure this backstage area is available is usually a breeze. They've got something called dynamic PVC provisioning, making your life easier in cloud setups.
Service
No matter how we would like to expose the pods to the external world we would need a service object:
apiVersion: v1
kind: Service
metadata:
labels:
app: rsyslog-forwarder
name: syslog-shipper-rsyslog
spec:
ports:
- name: syslogudp
port: 10514
protocol: UDP
targetPort: syslogudp
selector:
app: rsyslog-forwarder
type: ClusterIP
Depending on the context, theLoadBalancer service type with a static IP might be a better choice, or alternatively, one could expose the deployment for with an ingress object and an ingress controller, such as NGINX Ingress Controller. Tailor your approach based on the specific needs and nuances of your environment.
Promtail
To achieve a comprehensive configuration, leverage the Helm Chart for Promtail. This allows you to seamlessly configure Promtail to listen to syslog and relay messages to Loki, mirroring the setup in the Docker Compose scenario. Below are sample values you can feed to the chart:
# values.yaml
config:
clients:
- url: http://loki:3100/loki/api/v1/push
snippets:
scrapeConfigs: |
- job_name: syslog
syslog:
listen_address: 0.0.0.0:1514
labels:
job: syslog
relabel_configs:
- source_labels:
- __syslog_message_hostname
target_label: hostname
- source_labels:
- __syslog_message_app_name
target_label: app
- source_labels:
- __syslog_message_severity
target_label: level
daemonset:
enabled: false
deployment:
enabled: true
extraPorts:
syslog:
name: tcp-syslog
containerPort: 1514
protocol: TCP
service:
type: ClusterIP
port: 1514
externalIPs: []