Analyze car traffic with Elasticsearch and Kibana

There are currently a lot of construction sites on the my way to work. Therefor i wanted to know when the traffic jams start to occur and whats the best time to get to work.

Google Maps show current traffic on their routing service and “Hey!” there is a API for it: https://developers.google.com/maps/documentation/directions/. You just have to apply for API key there and they give you 2500 queries a day for free.

I started querying the data with curl. Since its a REST API thats really easy:

That will output a lot of JSON data to your screen. And since elasticsearch is schema-less and also uses JSON getting the data into elasticsearch is also really easy. Just pipe the data into another curl command:

That will already works, but to build up time series a timestamp is needed within each document. A small script was needed to reformat the document before pushing it to the elasticsearch index:

Next step was to run this script every 10 minutes in a cronjob:

And to create a simple visualization in kibana based on the routes.legs.duration_in_traffic.value field:

kibana-google-traffic

 

Test-Driven Infrastructure Talk at Docker-Bamberg

On 28th April I held a talk about Test-Driven Infrastructure at the local Docker-Bamberg Meetup.

Agenda:

  • Concepts
    • Puppet
    • Test-Driven Infrastructure
    • Behavior-Driven Development
  • Toolchain
    • A toolchain for TDI
  • Hands on
    • Build a simple webserver for serving a Docker Bamberg weblog
  • Goodies
    • Continuous Integration, Monitoring, Documentation, Checklists, Change Evidence build in!

Slide are available at:

https://markusbenning.de/slides-docker-bamberg/test-driven-infrastructure.html

mtpolicyd 1.21 has been released

Version 1.21 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

What has changed:

  • New feature vhost_by_policy_context
    New option vhost_by_policy_context will if activated tell mtpolicyd to select the VirtualHost based on the policy_context.For example in postfix main.cf use advanced syntax:

    In mtpolicyd.conf:

    The policy_context feature will be available in postfix 3.1 and later.
  • New plugin SMTPVerify
    The SMTPVerify plugin implements address verification at a remote SMTP server with MAIL FROM and RCPT TO commands. It support the following checks:

    • check if the remote SMTP server would accept mail for a address.
      Apply actions or scores if a permanent or temporary error is returnedIf the
    • remote server support the SIZE extension the SIZE will be passed to the remote SMTP server. This way it could be checked if the message exceeds the message size limit or the quota limit of the recipient.
    • Check if the remote SMTP server announces support for STARTTLS
    • Check if there is a TLSA record for the remote SMTP server
    • Check if there is OPENPGPKEY for the recipient

mtpolicyd 1.20 has been released

Version 1.20 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

What has changed:

  • fix SQL connection handling after child fork
    Closing the connection after child fork did not cause a reconnect on all DBI versions. Instead do a reconnect by overwriting the previous connection.
  • improve request logging
    mtpolicyd now logs the plugin that caused the result.The new log format is:

mtpolicyd 1.16 has been released

Version 1.16 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

New Features:

  • Improved SPF support
    The SPF plugins supports now checks on helo and uses postmaster@<heloname> as sender for null-sender mails as defined in RFC. (thx to Scott Kitterman for pointing this out)
  • Support for Spamassassins AWL reputation
    The SaAwlLookup and SaAwlAction plugins can be used to take actions based on the senders reputation stored in Spamassassins AWL database.

Setup an iptables honeypot with fail2ban

The following example shows how to setup an connection honeypot with the fail2ban daemon. It works by logging connection attempts to unused ports with the iptables LOG target and taking ban actions on the source IPs with fail2ban.

The configuration has been tested on a Debian Wheezy box but should also work for other distributions.

On a Debian box, install the fail2ban daemon with:

Create a action definition in /etc/fail2ban/action.d/iptables-honeypot.local:

The start and stop actions will be executed everytime fail2ban starts/stops and will insert the honeypot rules. Adjust the honeyports and honeydev settings for your system. The honeyports line should only list unused ports.

Next create a filter to match the log lines caused by connections to one of the honeyports in /etc/fail2ban/filter.d/iptables-honeypot.local:

How add a honeypot jail like the following to your jail configuration in /etc/fail2ban/jail.local:

Activate the changes by restarting fail2ban:

Be careful to not lock out yourself. Make sure to whitelist your own subnets with the ignoreip setting in [DEFAULT]. You may also want to start with a softer ban action than iptables-allports first.

Use posttls-finger to monitor your DANE configuration in icinga2

First you need to install the posttls-finger command. This command is included in postfix versions >=2.11. On Debian you may just rebuild the packages from unstable for your distribution.

Then download the check_posttls_finger script and make it executable:

Add a command definition to icinga2 by creating /etc/icinga2/conf.d/check_posttls_finger.conf with the following content:

Also add an service definition to /etc/icinga2/conf.d/services.conf:

Now configure the domains your want to monitor in your host definitions. For example to monitor markusbenning.de:

Use delv to monitor your DNSSEC configuration in icinga2

First you need to install delv. delv is a new diagnostic tool like dig, but with improved DNSSEC support (read more). It comes with bind 9.10 and newer. If you’re already using bind >9.10 then it should be already installed. Otherwise you can grab the latest bind tarball, compile it and use the compiled delv binary:

Then download the check_delv nagios plugin script:

Add a command definition to icinga2 by creating /etc/icinga2/conf.d/check_delv.conf with the following content:

Also add an service definition to /etc/icinga2/conf.d/services.conf:

Now configure the domains your want to monitor in your host definitions. For example to monitor markusbenning.de: