localghost.io

On devops at a project based shop

2013-06-11T08:57:00+02:00

The term DevOps has many overloaded meanings so let me first state what I think encompasses the term DevOps: everything related to development but not directly to implementing the core product/project's functionality.

More concretely:

Automating the build
Continuous Builds
Continuous Integration
Unit Testing
Integration Tests
Continuous Testing
Automating the deploy procedure
Package the deploy
Continuous Deploy
Centralized error monitoring
Centralized log inspection
Centralized machine monitoring
Custom event monitoring (think statsd)
Machine provisioning
Machine configuration automation
Automating data backups
Automating data synchronization
Automating data-store upgrades

As you can imagine checking every item on this list of quality providing measurements is a massive undertaking. This is usually not scoped in product development let alone in project development. Yet each and everyone of this steps is a huge quality assurance indicator with massive value and if we are to take our trade seriously is something we should all be doing.

I tend to view this check list as shadow work: work that is implicitly implied but never explicitly mentioned.

From what I've seen it's the shops who develop products or maintain a single website tend to do better then project based shops who inherently have shorter development cycles.

A product shop has a continuous life-cycle which rather non-surprisingly makes way for continuous <anything> to creep it's way high on the agenda as they start to form obstacles.

A project shop has a very limited life-cycle, getting something out the door often trumps all and often rightfully so. The rather cruel catch 22 here is that especially these short life-cycle projects benefit the most from the devops checklist. Knowledge of the ins and outs of the moving parts of such a project are lost quickly. All the items on the devops serve as documentation as code and provide a warm welcome back for any developer picking up the project back up after a year or when doing SLA maintenance work.

Limited life-cycle is only one factor hindering project shops checking items off the devops check list. Having multiple projects means that each project comes with its own set of challenges further complicating the matter.

At my previous job at a project shop I helped standardize some of the devops check list and there I had the added benefit that all development happened in .NET on windows machines, the websites were hosted on in house windows servers and development happened rather linearly in SVN. This means that solutions for items on the checklist are reusable and learnings from one project can be transfered to the next. As a developer this meant that introducing new moving parts, say an exotic tech-stack is not something you can easily do. Even in this homogeneous development environment getting to check ALL the items off the list is quite an investment both financially and personally convincing everyone involved of the necessity.

At my current job at a bigger project shop, that prides itself on using whatever tech stack is best for the project and has quite an agile mindset, development happens on Windows, Linux, OSX machines and projects are hosted on the cloud with Google, Amazon and Microsoft or on private dedicated servers behind weird VPN arrangements and project spans range from 6 weeks to three years. On top of that some releases follow a strict sign off procedure by clients others can be released when we deem necessary. Some development is linear and is easily continuously integrated, some follow a heavy branching model.

This means that the knowledge on how to implement each item on the devops check list becomes very fragmented based on environment, language and framework choices, development model and release strategies. For an aspect of our profession already not high on the radar for many developers and managers this quickly becomes too much too handle.

The little dirty secret in our industry that in a lot of cases devops turns into shadow work at best, done in the spare hours of a project.

The big dirty secret is that in a lot of cases this shadow work is simply ignored. This can have many causes

Developers are simply unaware of this work
Developers ignore it because they are time-constraint because this shadow work is not budgeted for.
Developers are aware but managers/project leads do not know how to sell this work to the client.
A client simply refuses to acknowledge the necessity of this shadow work

I think the devops movement is great and more than just the new hip buzzword as it forces a flashlight onto the shadow work and gets it on everybody's agenda.

At @q42 colleague Mark van Straten was given 6 hours a week to improve our devops story and although Mark did an absolutely stellar job getting all the current and past projects to build and deploy automatically we really found that this is not a one man project and 6 hours a week is really not sufficient to take care of ALL the shadow work on the devops checklist.

Having fulfilled a similar role in my previous job Mark and I often debate to How's and Why's of devops and even-though we disagree sometimes on details we share the greater vision of the importance of this shadow work. So we recently called a meeting, <insert meeting bashing comment here> with fellow colleagues who also take an interest and came to the following conclusions:

Devops is NOT a one man operation.
That said a dedicated devop'er might be a very good idea, not every developer has the time to learn puppet/chef and all the other aspects of the ops side of things.
A project technical lead is responsible for getting shadow work on a project's agenda.
This can mean that he/she decides not to something on the checklist for pragmatic reasons.
Everybody in a team is responsible for actually doing devops work.
Devops is not just a back-ender's job there's a lot to automate on the front-end too.
- i.e a JS front-end guy writes scripts and makes sure they can report results in a xunit format so TC knows what it means.
- The CSS guru automates the styleguide generation i.e rendering all the button permutations.
- Writing integration tests in phantomjs
- GruntJS all the things!
- etcetera..

One thing is for sure Devops work is a constant battle but as long it's a battle we are fighting we're on the right track.

Do you work for a project shop and are also fighting the shadow work let me know how you are coping in the comments!

The joys of .NET OSS development

2013-04-03T09:33:00+02:00

Today a quite heated debate opened up on twitter on a number of questions concerning open source development in .NET Doing a full recap of all the points being made would be rather tedious but have a look at Glenn Block's twitter feed most of the question and arguments raised boiled down to:

Does Microsoft do enough to promote OSS projects?
Should Microsoft even step up or should the community do more?
Is there even a OSS .net community?

This also sparked quite an interesting conversation on Jabbr that i sadly just missed out on. I went back to preserve the conversation here if you'd like to read it back. You'll have to scroll down to Wednesday, April 03, 2013, forgive my laziness here.

I've been running an open source project for the past two years with success, I've had many people contribute and submit issues and the project is in use at some great sites out there and I couldn't be happier with the community. The fact remains though that I am still the project's only main maintainer and the only one answering questions on github and stackoverflow.

The only exposure I get is through making sure my documentation is on par and the project looks like its stable and still maintained when people hit the readme page on github. People have tweeted kind words, like this or this, for which I'm super grateful!, that helps create a possitive buzz when someone googles for a .NET elasticsearch client.

To be honest though I think this is too be expected and I am OK with this. Unless you are lucky to have someone on board as a second maintainer, opensource means you have to do all the hard work. I don't write code to please Microsoft so I ask for nothing in return. I write code because I need it, I share it so that others may benefit. If enough people do, which I believe to be the case even in the .NET dev world, you get a community going.

Although I dont expect anything from Microsoft I really don't get anything from Microsoft either. jetBrains donated a free Resharper license, my awesome employer Q42 even donates a day a week of my time to the project. Microsoft does not hand out MVP's for doing well in opensource, no msdn subscription, no reach outs from community managers.

To be fair as a community I do think we too can do better, we don't have a go to website everybody goes to for daily updates on whats going on in the .net world. Finding like minded developers is an exercise in who follows who on twitter.

If we're too thrive as a community doing .NET OSS development it should be despite of Microsoft, not because of it.

Which is also why I'll refrain from bullet-listing everything Microsoft does wrong..

As a closing note I like to think if my project, that serves quite a niche, gets so many involved either through raising issues or sending pull requests I am anything but pessimistic about the community. Even if C# is only the 11th most popular language on github.

Comments from before this blog moved are on this post's GitHub issue.

OSS development, a continuous lesson in humility

2013-03-07T13:41:00+01:00

Yesterday I recieved a bug report on NEST, my .net client for elasticsearch. Now I've received many bug reports in the last 2 years and on several occasions these bug reports made me feel humiliated to the bone.

Yes, I wrote that and yes, that's pretty stupid

Now I am only human, my first reaction when I receive a pretty severe bug report is: Surely this can't be true and Come on! I'm not that stupid! but 90% of the time I investigate and can only conclude: Yes it IS true and Yes I AM that stupid.

Yesterday's bug report, was a prime example of this.

Async requests are not really async
------
Hi,
I noticed that SearchAsync is making synchronous calls in reality.
I think the fault might be with this code (DoAsyncRequest):

if (tcs.Task != null && tcs.Task.Result != null)

As it accesses the Task's Result, we wait for the task to complete...

Fuck me thats pretty severe! Surely this can't be true and Come on! I'm not that stupid! Looking at the code though:

var tcs = new TaskCompletionSource<ConnectionStatus>();
this.Iterate(this._AsyncSteps(request, tcs, data), tcs);
if (tcs.Task != null && tcs.Task.Result != null)
{
    tcs.Task.Result.Request = data;
    tcs.Task.Result.RequestUrl = request.RequestUri.ToString();
    tcs.Task.Result.RequestMethod = request.Method;
}
return tcs.Task;

Damnit! Yes it IS true and Yes I AM that stupid. Fair enough the fix seems easy enough, return a continuation on the task instead.

var tcs = new TaskCompletionSource<ConnectionStatus>();
this.Iterate(this._AsyncSteps(request, tcs, data), tcs);
return tcs.Task.ContinueWith(t =>
{
    cs.Task.Result.Request = data;
    tcs.Task.Result.RequestUrl = request.RequestUri.ToString();
    tcs.Task.Result.RequestMethod = request.Method;
    return tcs.Task.Result;
});

Nest's comes with a demo indexer that tests the troughput of the different IConnection implementations. The code itself looks like this:

var partitionedMessages = msgGenerator.Generate(numMessages).Partition(bufferSize);
Interlocked.Exchange(ref NumSent, 0);
foreach (var messages in partitionedMessages)
{
    var t = client.IndexManyAsync(messages, indexName, bulkParms);
    tasks.Add(t);

    Interlocked.Add(ref NumSent, bufferSize);
    if (NumSent % 10000 == 0)
    {
        Console.WriteLine("Sent {0:0,0} messages to {1}", NumSent, indexName);
    }
}
Task.WaitAll(tasks.ToArray());

msgGenerator generates new Message()'s' without doing ANY IO. So I really should be seeing multiple requests being being processed at the same time.

Yet fiddlers timeline shows me something pretty horrific:

Somethings still blocking damnit.

For reference here's the throughput the test app reports:

Fuck me these are not anywhere near the throughput numbers I reported 4 months ago

HTTP: 12123 docs/s
Thrift: 17575 docs/s

Mother of regressions!

Looking at the "fixed code" again:

var tcs = new TaskCompletionSource<ConnectionStatus>();
this.Iterate(this._AsyncSteps(request, tcs, data), tcs);
return tcs.Task.ContinueWith(t =>
{
    cs.Task.Result.Request = data;
    tcs.Task.Result.RequestUrl = request.RequestUri.ToString();
    tcs.Task.Result.RequestMethod = request.Method;
    return tcs.Task.Result;
});

The TaskCompletionSource still blocks while it coordinates the async steps. Let's try that again:

var tcs = new TaskCompletionSource<ConnectionStatus>();
if (!this._ResourceLock.WaitOne(timeout))
{
    var m = "Could not start the operation before the timeout of " + timeout + "ms completed while waiting for the semaphore";
    tcs.SetResult(new ConnectionStatus(new TimeoutException(m)));
    return tcs.Task;
}
try
{
    return Task.Factory.StartNew(() =>
    {
        this.Iterate(this._AsyncSteps(request, tcs, data), tcs);
        return tcs.Task.Result;
    }, TaskCreationOptions.LongRunning);
}
finally
{
    this._ResourceLock.Release();
}

Here I explicitly start a new Task and on top of that moved the semaphore out of _AsyncSteps() so we delay the creation of Task<T>'s to the very last second. The TaskCreationOptions.LongRunning is also key here to prevent threadpool starvation! The TPL will prefer to really start a new Thread for your task not take one from the threadpool.

Let's see how we are doing now:

Yes! that's more like it!

Nice, we actually improved throughput from 12k 4 months ago to 17k now. Awesome!

This is why developing in the open is so awesome, it keeps you humble and grounded benefiting both your sanity and the quality of the project.

These fixes are pushed to nuget and in Nest 0.9.20.0

Comments from before this blog moved are on this post's GitHub issue.