17.06.13

(Very Very Simple) Bash Script to Delete Old Log Files

Posted in News, Random Thoughts, Tech at 3:25 pm by Liv

So here’s the deal: you have an application running on a Linux box which generates log files. And these files keep mounting up and grow and grow and grow in both number and size and you have start considering cleaning up old logs which you don’t use anymore. I know about logrotate, it’s an awesome tool — assuming that:

• your application knows to close the current log file on a signal (e.g. SIGHUP) — so logrotate can “rotate” it
• or your application is ok with being “hung” for the period of time it takes logrotate to take a snapshot of the file and then truncate the existing one
• and also your application doesn’t have an in-built log rotation mechanism itself

Now, if you happen to have a server application running which is doing a lot of extensive work, you know damn well that you cannot afford to freeze that even if it is for milliseconds, as that would deteriorate the experience to the clients “consuming” your server app. And if it so happens that the amount of logging you do is in the order of a few hundred Mb a day, then you know as well that this won’t take just a couple of milliseconds. Also, if you use something like Java — which is my case and what triggered in the end this blog post, unlikely you’ll be able to hook into SIGHUP or similar; even more, if you already use one of the Log4J’s built-in mechanisms to auto-rotate the log files (be it on size, date etc), then again, you cannot use logrotate unfortunately. However, you will still have each day a new log file being rotated (if you decided to have Log4J rotate on a daily basis) and these files keep mounting up and taking disk space; while you might keep an eye on the log files from the last week, unlikely I’d say in most cases you’d do the same with log files created 3 months ago — so there’s a lot of wasted disk space there!

Read the rest of this entry »

Permalink Comments

Disclaimer

18.04.13

Using Apache Commons CLI to Parse Arguments

Posted in Blogroll, Tech at 2:23 pm by Liv

I’ve talked before on this blog about Apache Commons CLI project and showcased how easy it is to use it to build a rather complex command-line syntax for your Java application. If you find yourself at any point writing an application which needs more than one command line switch, I strongly recommend Commons CLI is the way to go: it takes care of basic “syntax” validation, missing command-line parameters, help screens and so on, and all of this in exchange of pretty much providing a structured set of options describing the parameters expected / accepted by your application.

One of the problems when dealing with command-line arguments is that in a lot of cases the application needs to interpret these not as simple strings but parse them in some different data types and interpret them as numbers, file names, url’s etc. A classic example is specifying the max heap space when starting a Java process:

Java -Xmx512m ...

In the above case, the Java process needs to interpret the argument after -Xmx as a number followed by a “denominator” (e.g. kilobytes, megabytes etc) — as such will employ a process of extracting the digits in a separate string, parsing them, ensure they are a valid number, then separately identify the denominator (“m” in this case), ensure it’s a valid one and find the multiplier (megabytes in this case – i.e. 1024 X 1Kb) and based on these 2 finally compute the final result.

Read the rest of this entry »

Permalink Comments

Disclaimer

08.03.13

What Most Schools Don’t Teach…

Posted in Blogroll, Random Thoughts, Tech at 9:44 pm by Liv

If you work in IT, you have probably by now have seen the famous video on YouTube labelled “What most schools don’t teach” — where Mark Zuckerberg, Bill Gates and a few other prominent figures are talking about how easy it is to program, and how only 10% of the USA schools teach programming. It goes on more to bombard the viewer with some scary figures about how 1,000,000 jobs or more will go unfilled because there are not enough coders — I’m guessing as a motivational message to inspire people (kids?) to start programming.

The thing is, watching the video, I cannot but cringe at the hypocrisy employed in making this video! If the purpose of this message is to increase education efforts (at school level or individually) in the programming sector, then I’d say it probably does it job — in a very lame way, but it sort of does. If however this is intended to be a call to arms for “everyone” to start programming and either change their jobs to a programming job or if they are at the beginning of their career, to set their career path to be in programming, then I’d say the message is a hypocritical one. Even worse, it is setting up the targeted audience for a harsh comedown!

Read the rest of this entry »

Permalink Comments

Disclaimer

28.02.13

S3 Utility on SourceForge.net

Posted in Blogroll, Tech at 1:48 am by Liv

Only a day after I’ve published the S3 utility for buckets with versioning enabled, I took the time finally to put this on SourceForge — woohoo! Still early stages, as I still need to figure out really how to properly release versions into their file system, but as it stands right now at least there is a basic project page, a wiki and issue tracker — not to mention a Subversion repository with the source code.

So let’s start with the basics:

First of all, the project main page is here: https://SourceForge.net/projects/awsversionmgmt/ . This includes links to the download area: https://SourceForge.net/projects/awsversionmgmt/files/ as well the standard SourceForge issue tracking system: https://SourceForge.net/projects/awsversionmgmt/tickets/. I am still working on planning the releases for this, but for now I’ve just added a few issues I have found for the first official version of this (which would be 1.0.0 — naturally!) — if you’ve already tried this tool (unlikely, it’s only been one day since I put up the original post ), feel free to suggest some enhancements using the SourceForge tracking tool.

Also I have created a wiki here: https://SourceForge.net/projects/awsversionmgmt/wiki/. The idea behind it is to make this a dev-oriented wiki — i.e. propose upcoming features and architectures etc. And last but not least, I’ve also managed to deploy the maven-generated site on the project website: http://awsversionmgmt.SourceForge.net.

With that in mind, I think I’ve successfully manage to transfer the project to SourceForge so I’ll maintain it there — though quite likely you will see the occasional blog post here when a new milestone is reached. drop me a line if you want to get involved.

Permalink 1 Comment

Disclaimer

27.02.13

Utility for Version-Enabled AWS S3 Buckets

Posted in Blogroll, Tech at 8:01 pm by Liv

OK, Amazon, I have to tell you something: you dropped the ball a bit on this one! I absolutely love AWS and every day I seem to find something new about them (though, granted, not sure if that says something about the innovation in Amazon or it says something about my ignorance!?). However, having used S3 for a while, I have been waiting for a while for an utility like this that in the end I had to write it myself!

The problem that I have is that I use a few buckets where I have enabled Amazon’s versioning feature on the bucket. This means every time I write a file, S3 stores all the versions of a file, which is so cool because I can keep a full trail of each file changes, together with metadata around it. However, the problem arrives when you delete a file: any S3 file browser you get (including Amazon’s own web based one) shows that the file is no longer there, however, if you proceed to delete all the files in the bucket and then try to delete the bucket, you will get an error message which informs that in fact the bucket is not empty (and as such it cannot be deleted)! If you start digging into it, you will find that AWS API offers a specific set of functions for versioning — and once you employ those you start seeing that your bucket is far from empty and has lots of versions in it; unfortunately those versions are not visible because the file has been deleted, however, unless you delete these versions you cannot delete the bucket (not to mention that they are being stored in S3 in stealth “ninja” mode taking space — and costing you money).

That’s when I started looking for an utility who can deal with this, but up to now, I wasn’t that lucky. And the rule of today’s software tools says that if it’s not there it’s time to write one! Hence this utility — and the post here where I’m offering it for download.

Read the rest of this entry »

Permalink Comments

Disclaimer