Automated observability with Puppet in a zero-trust environment

Haroon Rafique

Manager
Development and Operations
Student Information Systems, ITS

You are here because you are interested in observability and are using Puppet already or thinking about it.

This will be a fairly technical talk. There will be no fancy background graphics, or any memes on display.

I will demonstrate how to build a modern observability stack with Puppet, show a quick demo of the resulting architecture, and then leave some time for questions at the end.

“If you want everything to be familiar, you will never learn anything new, because it can’t be significantly different from what you already know.”

— Rich Hickey

The quote is by Rich Hickey, the author of the programming language Clojure which is a dialect of Lips that runs on the JVM.

I love this quote because it applies directly to how we build infrastructure.

We keep copy-pasting the same scripts and hand-maintaining files because it is what we know. But that familiarity is actively holding us back from scaling. To fix it, we have to recognize where we are getting stuck.

The Toil Trap

• Manual monitoring target updates
• Config drift from reality
• Onboarding friction due to cert. management
• YAML sprawl

Most teams still run observability as a manual process. New servers require manual updates of monitoring targets.

You think the service is running on port 8080, but the reality on the ground has changed.

Certification management is a pain point.

Configuration is scattered across scripts, manifests, and YAML files, making it hard to maintain and audit. This is the bottleneck we want to remove.

Zero-Toil Architecture

• Self-registration
• Automated discovery
• Identity-driven
• Zero-trust

To escape the toil trap, we must make observability and security native properties of the system.

And in this case self-registration and automated discovery go hand in hand.

When we say identity-driven, we mean that the node’s identity and intent are inseparable from the moment it is provisioned.

This allows us to build a zero-trust architecture where every component can verify the authenticity and authorization of every other component without relying on implicit trust or manual configuration.

Let’s look at the five patterns that make this target state possible.

Implementation Blueprint

• Classify nodes automatically during CSR signing
• Separate logic from data using Hiera roles
• Automate service discovery with exported resources
• Secure metrics scraping with Caddy and mTLS
• Standardize monitoring with layered exporter profiles

This slide serves as our roadmap and contract for the rest of the presentation.

Every implementation pattern we cover next will map directly back to these five foundational pillars. I will cover each of these patterns in detail in the following slides.

Components in This Reference Stack

• OpenVox/OpenVox Server for policy and certificates
• OpenVoxDB for exported resource inventory
• VictoriaMetrics for metrics storage
• Vmagent for metrics scraping
• Grafana for visualization

This slide needs 2 clicks for each bullet (for animation).

OpenVox/OpenVox Server is for configuration management.

OpenVoxDB is for storing the inventory of exported resources.

VictoriaMetrics is the time series database. I chose this over Prometheus because it is much more performant, and it is straightforward to scale, even with the single-node version.

vmagent which is part of VictoriaMetrics is for scraping metrics from the nodes.

And Grafana is for visualization.

Pattern 1: CSR Classification

• Immutable identity requested during provisioning via CSR
• Cryptographic proof stored in the certificate extensions
• Tamper-proof classification secure against agent-side overrides
• Automated autosigning (optionally) enabled by secure policy scripts

What is pp_role, and what values should it contain?

• roles
  • monitoring_scraper
  • webserver

Corresponding Hiera data:

roles
├── monitoring_scraper.yaml
└── webserver.yaml

CSR stands for Certificate Signing Request. At the moment of provisioning, the node requests its specific role via a certificate extension in the CSR.

This embeds its identity cryptographically at birth.

And it creates a tamper-proof source of truth that agents cannot override. By keeping node intent attached directly to node identity, we eliminate central merge conflicts in static files and build a pipeline perfectly suited for dynamic, ephemeral fleets.

For demo purposes, I took a shortcut with autosigning, but you can also use a secure policy script to enforce stricter autosigning rules.

pp_role stands for Provisioning Profile Role. Typically, it should contain the roles for the individual VM.

Roles and profiles is not a feature provided by Puppet itself. It is a best-practice design pattern for Puppet that provides an interface between your business logic and reusable Puppet modules.

Pattern 2: Pure Data Roles

• CSR Extension attributes

  /etc/puppetlabs/puppet/csr_attributes.yaml

  ---
  extension_requests:
    pp_role: webserver

• Zero logic inside site manifest or node definitions

  manifests/site.pp

  node default {
    # Lookup classes from Hiera
    lookup('classes', Array[String], 'unique').include
  }

The node’s role is purely data-driven and attached to the certificate. In this case, line 3 specifies that the node’s role is webserver.

The site manifest is just a thin layer that looks up classes based on the role. Line 3 says to search all Hiera levels for a list called classes, merge them into a single list of strings, remove all duplicate class names, and then apply those classes to this node.

• Hiera lookup determines config based on pp_role

  data/roles/webserver.yaml

  ---
  # Webserver role - Apache with monitoring
  # Applied to nodes with pp_role=webserver in csr_attributes.yaml
  classes:
  - profile::apache
  - profile::monitoring::apache_exporter

• Hiera configuration

  excerpt from hiera.yaml

  hierarchy:
  - name: "Per-role data (from CSR attributes)"
    path: "roles/%{trusted.extensions.pp_role}.yaml"

Hiera is a powerful tool for managing configuration data. It allows us to separate data from logic and to create a flexible configuration hierarchy based on the node’s role. Here we are showing the array classes, starting on line 4, that will be applied to this node.

Next up is a small snippet of the hiera configuration that is a single YAML file, and the path shown on line 3 is the node’s trusted role.

Pattern 3: Auto-Discovery

• Exported resources publish state to OpenVoxDB

  @@file { "/etc/vmagent/targets.d/${facts['networking']['fqdn']}_${name}.yaml":
    ensure  => file,
    content => @("EOT"),
      # Managed by Puppet
      - targets:
          - ${facts['networking']['fqdn']}:9090/${name}/metrics
        labels:
          job: ${name}
          instance: ${facts['networking']['fqdn']}
          exporter: ${name}
      | EOT
    tag     => 'vmagent_target',
  }

This is the key automation mechanism of the entire talk. If you remember only one thing from today, make it this pattern.

Instead of manually updating static files, nodes now declare their own state at birth. When a node compiles its catalog, it exports its scrape target and stores it into OpenVoxDB.

Line 1 shows the path for targets.d and line 4 thru line 11 is the contents of the file. Line 12 is the tag that we use to specify the exported targets.

• Dynamic collection gathers all active targets

  # Collect all exported vmagent targets from other nodes
  File <<| tag == 'vmagent_target' |>>

The server scraping the metrics, then queries OpenVoxDB, and dynamically pulls in all active targets. This eliminates the need for human intervention when scaling up new infrastructure.

A new server comes up, it registers itself with OpenVoxDB, and the scraper automatically discovers it as a new target. The discovery can be done on a regular polling interval as well as on-demand. We will see that in the demo.

Pattern 4: Zero-Trust mTLS

• Caddy acts as a sidecar reverse proxy and mTLS server

  excerpt from Caddyfile

  # Authenticate via Puppet CA with mTLS
  tls /etc/caddy/node-cert.pem /etc/caddy/node-key.pem {
    client_auth {
      mode require_and_verify
      trust_pool file /etc/caddy/puppet-ca.pem
    }
  }

• Import metrics exporter configurations

  import /etc/caddy/conf.d/*.caddy

To achieve zero-trust security by default, we must not expose metrics over HTTP in plaintext.

1. Caddy is a lightweight, reverse proxy and TLS server. We can use Caddy as a sidecar proxy.
2. Luckily, we can use the existing Puppet CA to establish mutual TLS between metrics scraper and our nodes.

In this snippet we have configured Caddy to authenticate clients with mutual TLS using the Puppet CA.

Here we are asking Caddy to import all metrics exporter config files from conf.d.

• Proxy the exporter traffic

  excerpt from apache.caddy

  handle_path /apache* {
    reverse_proxy localhost:9117
  }

• Only allow legitimate scrapers

  @authorized_scrapers_snippet {
    expression \
      {tls_client_subject} == "CN=vmagent.local"
  }

• Block unauthorized scrapers

  handle {
   abort
  }

By placing Caddy as a lightweight sidecar, we secure the metrics path with zero changes to the exporter code itself. A lot of times it is quite cumbersome to configure the exporter to add TLS support, let alone mutual TLS. So, we use Caddy as a reverse proxy.

And then we ensure that only authenticated scrapers can access the metrics.

Good luck to anyone else who tries to scrape.

Pattern 5: Layered Observability

• Common profile applies to all nodes

  data/common.yaml

  classes:
  - profile::base

• Baseline metrics applied to every node

  site/profile/manifests/base.pp

  class profile::base {
    include profile::monitoring::node_exporter
  }

• Role-specific exporters are added based on workload

  roles/webserver.yaml

  classes:
  - profile::apache
  - profile::monitoring::apache_exporter

To ensure comprehensive monitoring without creating alert fatigue, we take a layered approach to our exporters.

A base profile applies to every node.

What’s contained in this base profile. Every node automatically receives a single OS-level node exporter.

Then, we use our Hiera roles to dynamically layer on specialized exporters—like PostgreSQL or NGINX—only on the nodes that are actually running those specific workloads.

When our baseline and workload-specific profiles match our Hiera roles, naming and labels stay consistent across every node. This is what finally allows us to use a single, unified query and dashboard model across all of our environments, drastically reducing clutter in the visualization layer.

The Automated Pipeline

Host Lifecycle

• Host boots and starts OpenVox agent

• Agent submits CSR with role intent

• Host applies role-specific profiles

• Exporters publish scrape endpoints

System Response

• CA auto-signs based on secure policy

• Hiera maps data classes to the host

• OpenVoxDB stores the exported assets

• Vmagent polls the target automatically

Security Guardrails For Production

• Strict validation of role-bearing CSRs.
• Role restrictions to prevent privilege escalation.
• Least-privilege paths for authorized scrapers only.
• Continuous auditing of exported resources.

We have to be explicit: complete automation must still be strictly governed. Guardrails like policy-based autosigning scripts prevent rogue agents from spoofing a privileged role or overexposing sensitive metric endpoints. You can have absolute speed and zero-touch workflows without leaving your infrastructure wide open.

Does It Work?

• Initial State: Verify current targets are up.
• The Trigger: Provision a new webserver node.
• The Magic: Watch Puppet execute automatic classification.
• The Result: The new scrape target populates automatically.
• The Payoff: View the live Grafana dashboards.

To prove this isn’t just theoretical vaporware, we are going to do a live run.

Since the rest of the demo takes fairly long, I left the demo in a stage where a few nodes are already onboarded. We will spin up two fresh nodes from scratch and watch it:

self-register

classify itself

and populate directly into Grafana without us touching a single configuration file.

Demo

Operational Impact

• Scales infinitely from isolated labs to massive production fleets.
• Adapts dynamically to volatile node churn and auto-scaling.
• Accelerates visibility by establishing instant scrape targets.
• Minimizes effort by keeping operational overhead flat as the nodes scale.

To tie this back to the operational economics: this architecture completely breaks the linear relationship between the size of your fleet and the size of your monitoring workload. By removing manual interventions, we ensure that adding 1,000 new nodes requires the exact same operator effort as adding a single host, while actively lowering the rate of outages caused by human configuration drift.

Engineering Anti-Patterns

• Mutable scripts installing runtime packages
• Floating tags using latest in production
• Hard-coded credentials in compose files
• Copy-pasted logic repeated across services
• Ungoverned attributes trusted without policy controls

To wrap things up, when you visit the Git repo, you will notice that I took some shortcuts while building out this live demo environment.

You really don’t want to be installing runtime packages in your docker containers. I did that solely to demonstrate the automation. Ideally, you would use already-built images.

In docker-compose, I refer to the latest tag. In production, I would tend to use immutable tags to ensure that I am always running the same version of a service. The third bullet is a no-no.

There is lots of massaging of how services start, stop, and restart. That is all due to the non-availability of systemd in docker containers.

Treating these slides as a transition into future hardening work will ensure your automation doesn’t accidentally introduce security blind spots or fragile dependencies.

Key Takeaways

• Automate observability from the exact second a node boots
• Eliminate bottlenecks by moving classification to CSRs
• Drop manual updates using native exported resources
• Recycle existing security to achieve zero-trust mTLS
• Scale your infrastructure without increasing team toil

To wrap up today’s session, these are the core architectural pillars I hope you take back to your teams.

I invite you all to follow up with me to look at the Git repository, review the Docker compose, Caddy, and Puppet configurations, and discuss how to apply these execution patterns to your own environments.

Bonus Slide: System Architecture

Questions?

Git Repo