/ blog projects miniprojects tagcloud   about contact github flickr where am I?

Had to implement some small but serious infrastructure.

Needed some usual things for internet operations these days like system monitoring, smart logging, graphing and alerting, high availiability, cloud, ids/ips, and mucho security.

Did this many times for random projects but this one was more serious and so I decided to rethink everything with focus on the security infrastructure. Manuals were read and quite a few quick projects were made and there is more to come so this post will be updated with time. I'll quickly go through the novel things I implemented and novel thoughs I have on the subject. This post will be a mess and there is a lot to write,

Quick logistics for nodejs apps

this is boring and already exists but I like the way I do things.

System monitoring?

munin is a collection of perl? scripts from '98 ran by crontab. UGH. nagios is.. yeah, same thing.

went with logstash -> elasticsearch -> kibana for data collection/graphing and riemann for alerting. Didn't figure out active system checks yet (a job for nagios usually. stuff like pinging, checking if websites render correctly, potentially crawling them and such)

edit: don't (ab)use elastic! use influxdb stack with graphana,. elastic thing was hard to tweak and used more resources then the whole cluster it's monitoring.

all kinds of apps write to logstash. my node apps, log data collectors (lumberjack), local system checks IDS, and such, it works well. Data from everywhere is aggregated and graphed together. I've written something simmilar to this stack as one off apps for debugging complex architecture a bunch of times, was very happy to find out that someone actually already works on such a thing pretty seriously.

didn't like collectd as it seemed like something that hangs out in a munin/nagios oldies crowd. so I wrote:

  • a simple probe that collects data from machines locally and sends it to a central server (in my case logstash) via udp json.



it supports plugins, they are super easy to write. one of the plugins is a meta plugin that runs munin scripts

Geographical internet stuff

csi-internet is there for you if you want to draw some computer stuff on a map. Also good to impress your girlfriend if she got used to cmatrix. As a test of a graphing engine it can ping around, do traceroutes and draw heatmaps. Messy atm as looking at pictures was more fun then looking at good code. work in progress. works on top of d3 and datamaps.

Firewalls

Writing tons of iptables rules is for robots and not humans and so I'd fuck it up eventually. pyromaniac is a thing that renders JSON into iptables commands.

hosts =
    vpn:
        ip: '10.66.1.70'
        publicPorts:
            vpn:
                port: 443

    shell:
        ip: '10.66.1.11'
        ports:
            ssh: 
                port: 22,
                from: 'vpn'
            something:
                port: 666,
                from: 'all'

    git:
        ip: '10.66.1.51'
        ports:
            ssh:
                port: 22
                from: 'all'

    all:
        ip: "10.66.1.2-254"


exports.settings =
    rules:
        forward: [
            { from: 'vpn', to: 'all', comment: 'vpn to everyone TCP' }
            { from: 'vpn', to: 'all', proto: 'udp', comment: 'vpn to everyone UDP' }
        ]

    hosts: hosts

this program is highly specific to my scenario (machines behind a NAT hypervisor with strict control of each connection) I'm showing an aproach here more then sharing my code as something that will work for you. (a more traditional aproach would be to use ansible for this)

IDS/IPS

I really like this part, but will write it up later when I have more to show

Mon Mar 16 2015 23:28:16 GMT+0100 (CET), 33 months ago

repository

I'm surprised I couldn't find this..

async object or json pattern matching/validation used for validating function/process arguments, api call permissions, messages, etc. supports serialization/deserialization of patterns themselves, which I used to implement a generic db query language on top of this. for now, tests are documentation

extras repository contains cuter things like

  • translation of subset of validator patterns to mongodb queries
  • validated backbone models (model that throws if certain attributes aren't set or if doesn't find appropriate superclass (for mixins))
  • validated backbone model attributes (special accessors that validate the data before its written to a model)
Fri Sep 13 2013 10:35:14 GMT+0200 (CEST), 50 months ago


a bunch of selfish prisoners having sex and mutating. WHOA. a very basic evolutionary algorithm, a last night's play.

Prisoner's dilemma is a scenario in a game theory (check it out if you don't know about it) its interesting to me in part because it explains human moral intuitions through a selfish evolutionary perspective, in other words, why people are generally nice to each other.

In the iterated prisoners dilemma the game is played repeatedly thus each agent has an opportunity to learn about other agents behaviours and potentially punish noncooperation. Cooperation may then arise as an equilibrium outcome.

solutions:

I didn't get tit for tat which I originaly expected, nor did I manage to get any other cooperative equilibrium. cooperation does evolve, but it doesn't last, strange, I expected that cooperative species will raise and stay dominant as I enlarge the number of interactions before a step in the evolution. (its rational to be nice if you need to stick around) need to investigate and think about this some more.

I'll post more info and code later or never. screenshot and someone with a different approach and a spacial dimension.

Tue Sep 10 2013 07:07:06 GMT+0200 (CEST), 51 months ago

After looking at the rorschach test, I wanted more. supirisingly, I couldn't find any rorschach generators online, so, python, tkinter and randomwalk with random size blobs and simmetrical reflection, I thought that some tweaking or more complicated algorithm would be needed but this works great.

click on a canvas for (re)generation.

examples and source.


Tue Sep 10 2013 07:07:06 GMT+0200 (CEST), 51 months ago

I've been hoping to write this for a while, its not a very exciting project so I've been working on this when I'm tired.

its a bit unusual, nodejs-blosxom, ~400 lines of coffescript. this is very much a work in progress, I'd advise you to use it as inspiration more then as a code you run. templates related to my own homepage are included in the repo. might move them laters.

features

  • each post is a markdown file (maybe I should check out ORG mode files?)
  • posts are in a git repo, so they can be written offline and synced with the web later
  • I wanted something like google plus circles, with restricted tags for particular people
  • implicit tagging by placement .md posts in folders
  • explicit tagging/metadata for a particular post by adding a JSON line as a first line in a file
  • tag filtering
  • no javascript on the clientside
  • rss for arbitrary tag combination

immediate todo

  • full set operations on tags
  • think about collapsing large posts when they are initially displayed..
  • also, don't render ALL matching posts, do the next/prev button thing.
  • saving of blog post metadata JSON back to files
  • color private tags differently
  • wiki (some way to link to other posts or tags (just extend the markdown?))

things to figure out

  • something super light for image galleries, I want to rsync images.. might implement this myself. flickr doesn't suck anymore, might hook into flickr?
  • I want to extend the markdown renderer in order to be able to include things like youtube videos or my own images/galleries
  • social network publishing - ifttt.com - I'll just use the rss.

at some point

  • implement tag equalities and implications, for example music is always audio but audio is not always music, audio is always sound though.
  • posts that are related to geographical location + geo search/map display.. I'd use this to mark interesting places, track trip progress and such
Fri Feb 22 2013 05:48:14 GMT+0100 (CET), 57 months ago