Mike Nolet of AppNexus started a 2-part webinar on scaling their advertising business and I am glad I followed the first part (and I will definitely be there for the 2nd part!) as some interesting things came out of that. (As a note, for those who are interested, Mike has more details on internet advertising in general on his blog mikeonads.com and you can find more details about these webinars here.) I spoke with Mike after the webinars and he was kind enough to let me publish some of the slides on my blog together of course with my personal opinions on some of them. As I said, you will find the full slides (and not just my screenshots) for free here: https://student.gototraining.com/…2416 or from Mike’s blog.

There is no question that the AppNexus platform is nothing short of complex — at around 14 billion ads transacted a day (Mike will have the precise figures) this is nothing to be sniffed at! But when building something like this, one has to understand from the beginning the challenge ahead. Your project managers will show you the basic equations which tell us that more people in your tech team means more man-hours; and also that more features/complexity means more man-hours too; and just as well more customers mean more man-hours (you have to develop and adapt for customer needs, support the customer and all these technical issues cost you man-hours too!). But what they miss out quite often is that when you bring all these coordinates together, the man-hours spiral out of control! That’s why I was pleased to see that it’s not just me who observes that, as Mike included this into his slides:

So before you go an expand your customer base — have a look at this diagram above. If your team is already working flat out on building features, then you have to plan your market expansion carefully before you cannot deliver anymore — either new features or cannot support on-boarding your clients!

Another interesting aspect which was highlighted in this series, was the infrastructure (have a quick glance at this slide below):

I know the rage nowadays is the cloud — and there are hundreds of business growing in the Amazon, however, one thing is being missed out often when it comes to the cloud (did you know, the cloud can make coffee for you, clean your floors and take the kids to school too — it’s awesome! :D): we are talking here about virtualization! And virtualization means ultimately another layer in between you and the bare metal — and anything that comes in between those 2 will slow you down. Sure, we take the hit nowadays every day for the language we run our code in — unless you’re one of those programmers with a constant cigarette hanging from the corner of your mouth, typing assembler with your eyes closed (because of the cigarette smoke :D), you cannot avoid the runtime layer imposed by the language. In the case of the pre-compiled languages like .NET, JVM languages and so on, the progress made on the JIT’s lower a lot of the impact of this runtime. However, adding another level in the chain will loose you a few extra CPU cycles per operation. Your disk writes and reads won’t be direct for instance — that’s good and bad, but the whole EBS-mounted filesystems require network trips, and while that’s abstracted for you by the PaaS service, it doesn’t mean it’s not there. The CPU’s are virtual too — so your instructions won’t take just N CPU cycles anymore, there will be a couple of more cycles introduced by the virtualization process. Most apps can take that hit — but if you’re planning to operate at gigantic scale, at some point you have to jump out of the cloud (and hopefully land with both feet on the ground!). Not to mention that with the likes of Amazon, the actual bandwidth in and out of your instances depends on the instance size — so for high network throughput you might find out you have to pay for their highest size instance, even though you might not need all the cores thrown in there! I remember back in my Vibrant Media days for similar reasons we have opted for bare metal for similar reasons — even though at the time the Vibrant Media cloud was only around 300+ servers it still mattered. For a while, I guess I considered myself an old-fashion guy thinking about the microseconds — and for finding out that is not the case alone, and there are others like myself, I should probably thank Mike 🙂 There is hope for me after all woohoo 🙂

The last slide I was going to comment on (absolute beauty this and my favourite by far!) is this — please take a long time to look at it, it’s worth it!

Now one of the reasons I like it, is to do with Mike’s personality : it’s good sometimes to see people tell things as they are, and if “shitty support” describes best what you get out of these guys then why not say it? There’s lots of people up in the arms to slam down Microsoft and Windows (a lot of times because it’s been done before and you know you’re not the first or the only one to do it), but when it comes to other companies, the voices seem to dim down. So what if it’s Citrix, or another giant? If they’re shit, they’re shit — nice one, Mike! The second reason I like this slide (there’s many others, but I will stick to just the 2 main ones!) is that it’s good when someone like Mike states that just because “it’s an industry standard” doesn’t mean it’s good enough for you! Can you afford all of your transactions (ads) to go through one of these devices and when it falls over you get a ticket logged in some database and a guy calling you about 4 hours later? If so, wipe out the cross line above, close your eyes and pick randomly any of those names! If however you find out that having one of these things in your network slows things down and as such slows your revenue down (I mean imagine that! an online company making money out of the traffic they “see”! I know, it’s groundbreaking right? :D) then you might want to chuck away the whole PR bullshit about “industry standard” or accept that it might be the standard for a different industry, but not yours! There is no general rule of thumb for some of this stuff — if your online business is truly unique, chances are your requirements (in terms of hardware and software) are as well, so probably a good idea to think a bit before signing on the dotted line for an “industry standard”.

Having learned of Mike’s “passion” for the key/value stores, I can only hope now that the second webinar (which deals with this) will be just as entertaining 🙂 So hopefully you’ll get a few more slides (and more ranting!) from me soon on that!