Commit 6aa08c08 authored by Jamie Tanna's avatar Jamie Tanna

Remove drafts

As the current drafts are more notes from previous talks I've been to
(and are now wholly behind the times) there's no point keeping them
around - I may as well cut them loose and close any related issues.
parent 79ff9111
Pipeline #41930053 passed with stages
in 7 minutes and 34 seconds
layout: post
categories: fosdem
title: GIFEE - Google Infrastructure for Everyone
tags: containers docker kubernetes
> This article is developed from a talk by [Brandon Philips at FOSDEM 2017][GIFEE-fosdem].
## Why Containers?
Kubernetes, on the other hand, is all about having your distribution running as lightweight as possible. This is in part due to the fact that getting everything to the right version can be difficult. By containerising your application, you can avoid the need for special steps for deployment and configuration management, as it's then all done within a single container. This avoids the issues of trying to get certain versions of libraries playing nicely, on the distribution you're deploying to - for instance, if you use RHEL7.1, and want to be running a very new version of __??__, then you would have to install it via a third-party or from source. This is not ideal, and therefore means that __??__.
On the other hand, the container will contain the pre-packaged environment that is required to run your application, and allows for running multiple __??__. Note that this does, of course, have some drawbacks which I have detailed previously in [Resurrecting Dinosaurs][resurrecting-dinosaurs].
This container can be run on a hardened host, such as a RHEL7.1 image, but the base image could be something like [Alpine Linux][alpine].
Another perk of using containers is knowing that the same image (minus issues like proxies) will work exactly the same way whether it's running in development, continuous integration, on the end environment, and anywhere in between. This increases the confidence the developers can have within their application, and not have to worry even about any differences between their staging and production environments.
## So What's Kubernetes?
Kubernetes is a __??__.
Brandon describes the concept of clustering as a botnet.
### TODO: Replication is Hard
One problem with conventional (pre-config management with on-box agents) server maintenance is that if a given server goes down, you will have to remember exactly what is on the server,
For instance, let's assume that we have three servers:
A - LAMP v1, Client v2
B - LAMP v1, Client v2
C - LAMP v1, Server v3.5
Now, one day, server `B` goes down due to i.e. a power outage.
A - LAMP v1, Client v2
C - LAMP v1, Server v3.5
But we need to run some security upgrades, and so the different servers are bumped to the latest versions.
A - LAMP v1.1, Client v2.0.1
C - LAMP v1.1, Server v3.5.1
But then, when we bring `B` back up online, how is it going to cope?
A - LAMP v1.1, Client v2.1
B ????
C - LAMP v1.1, Server v3.5
`Another example - what if the master node in your identity server goes down?`
Because the fact is that replication is hard, it's a core feature of Kubernetes to run all this for you. Kubernetes has a set of 'control servers' which will manage the Infrastructure for you. These servers will handle what to do if a server dies, and how the 'botnet' will handle it.
- get everything bundled perfectly, will work everywhere
- clustering = botnet
- if a server goes down, you have to remember what's on it, what needs to be on it, if you have or haven't copied latest stuff
- set of control servers that manage things for you - they know when to replicate ie if something dies
- API for control clusters
- has an API server cluster that is the frontend used by CLI
- shouldn't have to care when each server goes down (and nor should the users notice!)
- CoreOS manages who becomes leader (RAFT protocol/algo)
- - show how things break
- can run infrastructure consistently across different providers - cloud or bare metal!
- avoid fragmentation of cloud platforms - run on k8s instead of AWS
- handles things like LB for you
- federation - k8s but instead of machines, it's clusters
- k8s to manage DCs that manage machines
- would mean we can hit multi-cloud so much easier
- but still not there yet (40% current api works so far)
- "who deployed _these_ containers?"
- label them with what they are ie backend, prod, by Manthan
- db deployment is easy - but how do you scale it? Distribute it?
- operators - human knowledge -> software
- operator will manage a lot of things like health of a cluster
- prometheus - can monitor stats
- get nice graphs
- can see LB -> instance
- can go through containers to actual hardware (ie what kernel version is c1ps' first cluster instance's host image running?)
- can do searches to get more details
- next steps - Openid Connect, prepackaged, RBAC
- CoreOS all about securing the internet
- our responsibility to ensure people's data is safe
- "self driving infrastructure"
- don't need to be an expert in patching every component
- ie app, OS, db
- dancing on the edge of secure and insecure
- need to be able to patch and update systems so quickly
[resurrecting-dinosaurs]: {% post_url 2017-02-15-resurrecting-dinosaurs %}
layout: post
title: Analysing Your Projects and Community with
tags: bitergia cauldron opensource community analytics
Parham raised a number of important points about building software, all centred around the fact that software is a human-centric job. Yes, the code we write is performed by a machine, but as the famous saying goes - code is read more times than it's written.
**Imagine murdering them**
> Any fool can write code that a computer can understand. Good programmers write code that humans can understand.
> Martin Fowler
One of Parham's first points was that "no one lists readability or maintainability as a language feature... But it's important". It's definitely interesting seeing that of all the programming languages we have, none of them really praise the readability it provides as a function. And you'll hear __praise__ for functional languages, or those that allow you to turn a large method into a single line of code, but you'll realise that actually it's not something that you really want.
I recently found this when I was debugging a Groovy script that used [Closures][groovy-closures] and its use of trying to fit three closures, a map, and an if statement into three lines made the code very difficult to debug. Being "clever" and participating in [Code Golf][code-golf] is the easiest way to single-handedly enrage future programmers - who may in fact be you!
Parham discussed how languages that don't require explicit error handling can be very costly - for instance, a project he worked on where a `catch` could have saved £3m and a business. The fact that this is enforced in Go gives him a lot more comfort in the code he writes, as it won't be able to blow up as easily.
He went on to discuss how best practices won't always end up with a successful business, but writing suboptimal code often does, due to the time and effort cost in ensuring that your code is always the best it can be. That being said, he wasn't promoting always writing bad code, but instead being pragmatic and conscious of cost of writing "good code now". Additionally, that you need to improve your personal skills so you can more easily talk to Product Owners and persuade them to allow you the time to move deadlines, or work on "tech debt" where needed.
[Matt Brunt fed in on Twitter]( with a quote from [Troy Hunt][troy-hunt]:
> By all means, do your damndest to get it “right” from day one but do so with the expectation that “right” is a temporary state.
This brings home the point of the unattainable "perfect system", and that you'll be constantly chasing your want for
**Imagine praising them**
layout: post
title: Taking Containers from Development to Production
description: <
## docker
need to cut cost, move from VMs to containers
did a search - found docker
running a python + db project:
`docker run -p 5432:5432 postgres`
then running script (in the host)
to actually dev on it, then have to create container - i.e.`docker build && docker run`, as well as having `postgres` working!
becomes a compiled language of sorts - lose the perks of having python (interpreted)
## docker compose!
then it's the case of `docker-compose up`
need to orchestrate things better though
### docker-swarm
can use `docker-compose`, but _not as is_
### kubernetes
experience of 10 years of Google scale
huge community - largest project?
but lots of new concepts
want to use Kube - but don't want to redo `docker-compose`
## Kompose
minikube helps with getting things up and running
smaller application is easy - works well (for a time)
- can't do single jobs
- can't do things like config and secrets
`docker-compose` is built for developers with single-node setups, `Kompose` is not
`restart: no` isn't deploy object, isn't quite the right mapping
layout: post
title: Continuous Delivery for Docker with GitLab Continuous Integration
description: TODO
One of the great things about [GitLab CI][gitlab-ci], that is not available in something like [Travis CI][travis-ci], is that you can provide your own Docker images to be run as part of the CI infrastructure. This, coupled with the `*registry*` that GitLab provides on a per-project basis, means that you can pack all the dependencies required for your application into one built image (`image vs container`).
title: Using the `pry` Gem to debug Chef locally
layout: post
title: Downloading log files over SSH when your log aggregation service doesn't work
description: '**Without making changes to the files themselves**'
tags: break-glass logs ssh blogumentation shell tips
## Context
Let me paint the picture:
- You've had a production incident
- You're on call
- The log aggregation service hasn't worked, so you can't determine what's wrong
- You need to access the logs
- You've been authorized to use the break-glass procedure
- You want to spend as little time on the box, and not change anything
As we're touching a production service, we don't want to make _any_ adverse changes to the instance. This means we want to ensure we don't make any changes like a `chown` or a **??**.
Additionally, as we're touching a production service, we want to spend _as little_ time on the box as possible.
## The How-To
As someone deploying software to some remote server (i.e. Scaleway, DigitalOcean, AWS) you will hopefully have logging capabilities on your application. When something goes wrong, and you don't have log aggregation set up, you'll need to get your logs, and determine what's up.
If you're provisioning your applications with least access privileges, you may have a setup such as:
- flask app
- `flask` user
- `/srv/flask/log/*.log`
When logging into the instance, you will need to SSH as yourself, then log into the other user:
local $ ssh jamie@server
jamie@server $ sudo su - flask
flask@server $
**Note: not if want audit trail?**
Now we're running as the correct user, with the correct access, we should create a tarball of all our logs into a tmp folder:
flask@server $ tar czfv ~/flask-logs-$(date +%Y-%d-%m-%s).tar.gz /srv/flask/log
# ^ seconds since the UNIX Epoch
# ^ i.e. 2017-05-09-1504605248
# ^ verbosely showing me what is in the archive
# ^ to a given file
# ^ a gzip'd archive
# ^ create
Now we have our logs in a separate directory, we can return to our own user account, and give our user ownership of that file, so we can then download it over SSH (given the filename `flask-logs-2017-05-09-1504605248.tar.gz`):
flask@server $ exit
jamie@server $ chown jamie:jamie ~/flask-logs-2017-05-09-1504605248.tar.gz
jamie@server $ exit
local $ scp jamie@server:flask-logs-2017-05-09-1504605248.tar.gz .
# Optionally delete it from the server once it's downloaded locally
local $ ssh jamie@server rm -v flask-logs-2017-05-09-1504605248.tar.gz
Now we have the archive locally, aren't risking anything by being on the box and reading files / making changes, and **???**.
- locate logs
- create tarball in /tmp
- scp to local
- confirm files exist
- delete /tmp/tarball
**Mention how this is the opposite of how it could be done by `chown` etc**
**Mention how this is ONLY in the case you don't have log aggregation set up**
title: Using Netlify Deploy Previews with GitLab Review Apps
As mentioned in `TODO: post_url`, I recently the hosting for my [talks] from GitLab Pages to Netlify.
layout: post
categories: fosdem
title: Managing Infrastructure with OpsTheatre
Managing infrastructure with OpsTheatre
Full self hosted solutions
Mattermost integrated with GitLab ce
Communication is so important - doesn't happen nearly enough
Infrastructure not important
Auto message via mattermost
Not about making the tools themselves - integrate, out-of-the-box
Want combination of publicly acknowledged best practices w open source software glued together using puppet
Quite a bit of overhead - is it worth it?
Not great for non-puppet shops - new stuff to learn and integrate
Works on AWS, but not the "amazon way"
" retrofitting is a bitch"
Not going to be proprietary
Cool stuff!
Uses Vagrant Oscar to provide the actual Vagrant definition, but nicer
Adds updates to hosts file - if all within a group, does fake DNS resolution
layout: post
title: Managing Secrets with Hashicorp Vault
layout: post
title: Security Automated Decryption
What would happen if a DC with encrypted disks went down? Would you have to manually enter it at each machine?
we've currently got to the level of standards i.e. when to encrypt - PCI-DSS
now we're getting the automation (on-off switch)
and then it's defining policy (slider)
trying to move away from having a "key encryption key" for the key which opens the secret
- but this means that someone will know this (for obvious reasons)
so why not use an escrow, which will hold the key
then we need to authenticate to make sure we're who we say, and also authorize
need both the requester and the client to authenticate
need a secure channel
also need a chain of trust to make sure it's legit
oh wait, also backups! for root of trust, as well as the escrow
sounds good? nope, **heartbleed**
- presuming TLS to protect transfer is dangerous; any flaw in TLS means keys are available
- complexity leads to a greater attack surface (all nodes now can be attacked)
- X.509 is hard to get right! (improper sign/trust, etc)
can we use async crypto?
- i.e. Diffie-Hellman
sharing of `c` over the network means that anyone can steal it and decrypt `K` with
also shouldn't be plaintext transferring `K` over the network
recovery with McCallum-Relyea is different
- ephemeral keypair generated for decryption
- `e` and `c` are private
- server only gets "a public key"
- no encryption over the wire
- no state (therefore no (fancy) backups!)
can put keys within secure key hardware i.e. TPM, HSM
if public key is offline, can do the whole thing offline! only need online for decryption
tang and clevis are server and client
decryption is automatic if tang is up and running (i.e. don't need to type anything)
makes it very clean
uses custodia - secrets-as-a-service API
can do this using a block device, too! `-d /dev/sda1`
generates right key i.e. same size as master key
shamir secret share - levels of trust / branches
ie can set threshold - "at least 2 methods from password, yubikey, bluetooth, tang"
`luksmeta` - helper to make it possible to get metadata through LUKS
offloads to the upstream `cryptsetup` instead of doing its own stuff
So -> serverless -> dashboard
-> slack etc
i.e. static site -> serverless /contact endpoints
mostly doing nothing
get data from SO, if new, send notification + add to DB
microservices - i.e. nano service which is a single function / endpoint
- i.e. map a route (via API gateway) to a function
doesn't matter if it's ~1s until written to DB, as it's not realtime
prepare for cold startup
could most likely run on free tier
NoSQL document database (CloudAnt -> CouchDB)
- Cluster of Unreliable Commodity Hardware DB
JSON format
great replication
Apache openwhisk (IBM hosts this)
- own platform
"use big words so people don't realise you're writing a function"
- trigger - event, ie incoming HTTP, tweet matched search term
- rule - what to do when it happens
- action - i.e. function to call, with parameters
- package - collect actions and parameters together
- (grouped so you know what goes with what)
- what if you want parameter(s) for all the functions to use
- sequence - more than one action in a row - i.e. piping/chaining
update === create if not exists
`--web true` i.e. gives it a URL to be used
useful i.e. webhooks, not when behind API gateway
OfflineFirst - client side JS, PouchDB
- then will sync when network appears
- can make changes as required
each function can be owned by a different team
each function can have its own modular, testable codebase
each function can also be a different language
data hygeine - don't make everything hit the DB
- can the function be given the data in params?
Hubot for fancy Slack integration
serverless as the next big thing to remove heavy dependencies and push them to little containers
3-22 mins (if at all) - Earth to Mars light
HTTP request ~44 mins return trip
deploy to `mars-west-1`?
how do you handle a massive latency?
no realtime comms
serverless? nope
radio waves for comms
or interplanetary internet (real word internet - networks)
built for high-latency, unstable networks
multi nodes
- store the data sent to it, then pass it on
- each node retries, rather than whole data load
service workers - proxy between `app -> browser`
remove noisiness
fewer API calls i.e. GraphQL
specify data we want, rather than many calls to get the same response
pouchdb, sync to client
background sync
web RTC - no server, P2P videochat
webtorent for sharing out videos via P2P - no need to have youtube when can be shared around
merkel tree for logs?
interplanetary file system (IPFS? Isn't that Internet Protocol FS)
we're not even building things that work on Earth
soon we need multi-planet
fix it!
can't "quick fix in prod" - eventual consistency
- offline first / sync via PouchDB
- GraphQL to reduce unnecessary API calls
- aside; how will SSL work? extra handshakes
- Node-to-node to get to the
layout: post
title: Why would you want a static site? Isn't Wordpress/(insert blogging platform here) good enough?
description: A discussion about why you would want a static site instead of something like Wordpress.
categories: persuasive
tags: website hosting static cms
**TODO: What's the target audience? Spectat? Techies?**
When deciding how you want to build your personal or professional website, you may flock directly to something like Wordpress, due to the large market share, and because "everyone else is using it". However, it's not always the right answer. You need to ask yourself the following questions:
- How do I want to edit content on my site?
- How do I want to be able to release changes?
- How much time do I want to spend on administration?
- How/do I want to automagically share content?
I'll compare these options for a Content Management System (CMS) Driven Site versus a Static Site.
**TODO: provide definitions?**
## Content Management System Driven Site
### Editing
### Releasing Changes
### Administration
### Content Sharing / Promotion
## Static Site
### Pro
- Deployment fun - {% post_url 2017-07-18-gitlab-review-apps-capistrano %}
- Offline editing
- Use any tool for the job
- Orders of magnitude fewer security issues
### Con
- Time to deploy / time to live
- Less plugins to perform things like auto post to i.e. Twitter
## Database-Driven Site (i.e. CMS)
### Pro
### Con
- Potential sysadmin work
- Bloat
## **Middling**
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment