/ blog projects miniprojects tagcloud   about contact github flickr where am I?

Had to implement some small but serious infrastructure.

Needed some usual things for internet operations these days like system monitoring, smart logging, graphing and alerting, high availiability, cloud, ids/ips, and mucho security.

Did this many times for random projects but this one was more serious and so I decided to rethink everything with focus on the security infrastructure. Manuals were read and quite a few quick projects were made and there is more to come so this post will be updated with time. I'll quickly go through the novel things I implemented and novel thoughs I have on the subject. This post will be a mess and there is a lot to write,

Quick logistics for nodejs apps

this is boring and already exists but I like the way I do things.

System monitoring?

munin is a collection of perl? scripts from '98 ran by crontab. UGH. nagios is.. yeah, same thing.

went with logstash -> elasticsearch -> kibana for data collection/graphing and riemann for alerting. Didn't figure out active system checks yet (a job for nagios usually. stuff like pinging, checking if websites render correctly, potentially crawling them and such)

all kinds of apps write to logstash. my node apps, log data collectors (lumberjack), local system checks IDS, and such, it works well. Data from everywhere is aggregated and graphed together. I've written something simmilar to this stack as one off apps for debugging complex architecture a bunch of times, was very happy to find out that someone actually already works on such a thing pretty seriously.

didn't like collectd as it seemed like something that hangs out in a munin/nagios oldies crowd. so I wrote:

  • a simple probe that collects data from machines locally and sends it to a central server (in my case logstash) via udp json.



it supports plugins, they are super easy to write. one of the plugins is a meta plugin that runs munin scripts because I'd rather kill myself then write all these data collectors myself.

Geographical internet stuff

csi-internet is there for you if you want to draw some computer stuff on a map. Also good to impress your girlfriend if she got used to cmatrix. As a test of a graphing engine it can ping around, do traceroutes and draw heatmaps. Messy atm as looking at pictures was more fun then looking at good code. work in progress. works on top of d3 and datamaps.

Firewalls

Writing tons of iptables rules is for robots and not humans and so I'd fuck it up eventually. pyromaniac is a thing that renders JSON into iptables commands.

hosts =
    vpn:
        ip: '10.66.1.70'
        publicPorts:
            vpn:
                port: 443

    shell:
        ip: '10.66.1.11'
        ports:
            ssh: 
                port: 22,
                from: 'vpn'
            something:
                port: 666,
                from: 'all'

    git:
        ip: '10.66.1.51'
        ports:
            ssh:
                port: 22
                from: 'all'

    all:
        ip: "10.66.1.2-254"


exports.settings =
    rules:
        forward: [
            { from: 'vpn', to: 'all', comment: 'vpn to everyone TCP' }
            { from: 'vpn', to: 'all', proto: 'udp', comment: 'vpn to everyone UDP' }
        ]

    hosts: hosts

this program is highly specific to my scenario (machines behind a NAT hypervisor with strict control of each connection) I'm showing an aproach here more then sharing my code as something that will work for you.

IDS/IPS

I really like this part, but will write it up later when I have more to show

Mon Mar 16 2015 23:28:16 GMT+0100 (CET), 28 months ago