Posted on

When a backup is not, and when a restore is a failure

Since writing and speaking a bit more about the “relax! a failure is not an emergency” concept, more and more people approach me with interesting horror stories. I’m scribbling a few backup-related ones here for your enjoyment – and naturally there are important lessons.

Story 1: A place makes backups that get shipped off-site, interstate even. One day a couple of files are lost, and so someone files a request to retrieve said files from the archive. Well, apparently that’s something that should be done as it creates some very stressed responses and a quoted timeline of a few weeks. In the end the issue is resolved through other means and the request stopped – unfortunate, since it would have been very interesting if the requested files would actually ever arrive… clearly a retrieval was not part of the expected process. One also wonders how long a full dataset retrieval would take, or if  it’s even possible!

Story 2: A place has data recovery/failover infrastructure. However, it’s in use as a test environment and consequently an actual event that requires use of this hardware would first need re-imaging of the boxes to get them set up to even receive the data they need to contain. Estimated timeline: days to weeks.

The aboves are extreme, but I think they make the point pretty well. I hear and see lots of cases where there are processes in place for making backups, and off-site transfer. Lovely. However, they do often appear to “forget” the objective of which these processes are merely a part, which is not just shippping out a bucket of bits but potentially using it for failover or recovery!

So you want to test the recovery process, that is the entire trail from the storage all the way to having an actual new functioning copy. And, you want to see how much time that takes.

In Open Query parleance, if you haven’t tested this fully, or the time required is more than the downtime your business can afford, then we say you technically have no backup. Simple. No sense being fuzzy about this, right?

It’s one of the things we help our clients with as part of our service. This is not really a topic that should be regarded as optional.

Posted on
Posted on 3 Comments

What to do with the Falcon engine?

Keep it. Make sure it gets correctly positioned in the coming months.

It appears that with the Oracle acquisition, the reason-to-exist for Falcon is regarded as gone (a non-Oracle-owned InnoDB replacement), previously seen as a strategic imperative – much delayed though.

But look, each engine has unique architectural aspects and thus a niche where it does particularly well. Given that Falcon exists, I’d suggest to not just “ditch it” but have it live as one of the pluggables. What Oracle will do to it is unknown, but Sun/MySQL can make sure of this positioning by making sure in the coming months that Falcon works in 5.1 as a pluggable engine, perhaps also creating a separate bzr project/tree for it on Launchpad.

Then the good work can find its way into the real world, now.

Posted on 3 Comments
Posted on

Open Database Alliance

This alliance is an excellent step, showing the maturity, breadth and depth of expertise for MySQL related services! Of course Open Query is an active early member, with our training and subscription services, and initiatives like the OurDelta builds project.

Kudos to MontyW and PeterZ for driving this further while at the MySQL Conference last month.

Posted on
Posted on

MySQL User Groups on meetup.com – sponsorship – ask Open Query

This is about the ending of the sponsorship of the mysql.meetup.com user groups by Sun/MySQL and their suggested move to Facebook.

If people want to move to Facebook, that’s fine. For those who want to stay but don’t have the local funding, I have an offer for you. Contact Open Query, and we’ll sponsor your group for the coming months. This is not open-ended, I think a more permanent solution is important (moving, sponsorship, whatever) but I want to make the effort for the community to prevent any groups from disappearing now just because of this.

I was the one who originally set up the agreement with meetup.com when they first started charging for meetups (and it turned out to be a very good business model for them!). When that change was announced, quite a few meetup organisers were going to quit. The sponsorship made it easy to stay, although we did lose a few groups still.

I expect the same will happen this time, moving across groups and users to another service will cause some groups to just fold, and others to lose quite a few members. People lead busy lives, and organising or even just showing up at a usergroup involves a little bit of effort. Extra hassle can just be the trigger which ends that involvement.

The mailing list is quite popular on my own group in Brisbane (does Facebook have mailing lists?) and there are plenty of people who are not on Facebook nor care to be. Another aspect is that many related groups (PHP, etc) also are on meetup.com and people will be members of multiple groups. Having people keep track of their groups through multiple sites is awkward, so chances are they’ll drop out of some. An unnecessary waste

It’s just that people dislike having something taken away that was previously theirs, or having to pay for something that was previously free. It’s simple social psychology… people will fight any such instance quite vigorously (or protest by going elsewhere) even if it’s a very small issue. I do appreciate that sentiment, but it can hurt the community.

Posted on
Posted on 1 Comment

Visiting Darwin/NT

From this Tuesday 12 May until Sunday 17 May I’m going to be in Darwin (Northern Territory, Australia), teaching custom MySQL training days for a medical research institute.
DarLUG has been extinct for a while, but perhaps there are some local Linux/OSS people reading this? Please do drop me a line if you’d like to catch up while I’m there!
I’ve only flown through Darwin before, never visited/stayed… and I love the tropics (I lived in Cairns for a while) so this shall be a joyous few days! Oh the tough things we have to do in business. I’m also doing a day trip to Litchfield… Kakadu will have to wait until another time (perhaps with Phoebe) as I’m told it’s best visited over multiple days. It’s extra cool that this trip is happening this year since 2009 marks Charles Darwin‘s 200th birthday.
Posted on 1 Comment
Posted on 1 Comment

Joomla Day Brisbane

After the morning and afternoon tutorials today by Andrew Eddie (Joomla dev lead), tomorrow is Joomla Day – Brisbane Joomla Users Group where I’ll be doing talk as well.

I’ve already noticed that Joomla users are a slightly different crowd. Joomla is a pretty powerful CMS with many modules/extensions, just like Drupal which runs the Open Query web site. I’m not sure the two even compete directly although there might be some overlap. It occurred to me that Joomla might be what I would call an “enabling technology” on the web, just like PHP and MySQL have been since 1995. It has a very easy entry, which of course is both good as well as bad. Again that’s quite similar to the M and the P…. love it or hate it.

I’m kinda agnostic on the subject of CMSses, there’s quite a few out there and I think that most of the main ones have a valid place or niche in the market where they thrive. That makes sense. They shouldn’t each aim to be everything for everybody…. plugin architectures can somewhat guard against that, but there’s more to it than that of course.

Anyway, I think it’s interesting to observe these things, and how different applications have a (slightly) different userbase with different perspectives and objectives.

Posted on 1 Comment
Posted on

Call for Papers: Open Source Developers’ Conference 2009 – Brisbane

This year the fabulous OSDC conference is 25-27 November 2009 and returning to Brisbane (Bardon Centre, Mt.Cootha which is a great venue tucked into a rainforest setting). Stephen Thorne leads the organisation team for this event.

The call for papers is now open, until June 30th. For full details, see OSDC 2009 call for papers.

We’ll definitely be there, particularly since it’s very close to Arjen’s house. And perhaps we’ll get some talks accepted on MySQL and other topics. We’ll definitely be submitting some proposals!

Posted on
Posted on 5 Comments

MySQL docs freedom

As you may or may not know, long long ago (in this universe) I used to be the MySQL documentation team 😉  Yes, a team of one. This was 2001. It was a great and interesting time. The current much extended team is doing a great job with the now much bigger set of docs!

Today, I find myself disagreeing with my former colleagues on one particular aspect, namely its licensing. You see, the documentation has never been released under an open license, it used to be plainly copyright all rights reserved, and later some rights were granted to distribute the docs together with the server.

Statements made earlier by Karen Padir regarding possible opening up of the docs license filled us with hope. Then, Stefan Hinz (the current docs team lead) wrote a blog entry MySQL documentation: no license change. Some of the arguments there we can just plainly disagree on, but fundamentally Sun wants to discourage forks and basically says that if you want to fork the code, you have to write your own docs. Of course they’re entitled to that position, it’s theirs to make. So what’s my problem with this? Of course I’m going to tell, that’s why I started this post.

While the MySQL codebase is GPL and cannot be “taken back” and closed regardless of who owns it. However, the documentation is not protected in this way to guarantee its continued availability to the community.

People have no implicit trust towards big companies (or even smaller corporations), whether it’s the old MySQL AB, Sun Microsystems, Oracle, or another organisation. Their track record is such that at any point strategic decisions can be made that go against everything they were professing the previous week. Which, by the way, I completely appreciate from a business perspective – whether I fundamentally like it or not.

But if you have a business partner, someone you trust, you don’t just shake hands on a critical arrangement, you establish a binding contract so that the terms are laid out clearly, can’t be reinterpreted later, and can’t just be revoked except within the prescribed terms. Still there’s plenty of litigation about contracts, but that’s a whole other matter. Situations change, people responsible change to different people, and companies change owners.

So, the only thing that makes people trust such organisations is a guarantee that has been externalised and thus can’t be revoked unilaterally. The GPL license satisfies that very well for code. Regardless of who owns the code, the fact that it’s GPL means that it can’t be closed up again retrospectively – at least the codebase up to the point where the license changes (if the company owns all the copyright to the code) will always be free.

With the documentation, it’s copyright Sun/MySQL all rights reserved and while certain grants have been made, those restricted liberties are not implicitly irrevocable, i.e. they have not been granted in perpetuity. As it stands now, the current or future owner of that IP could change the license, and hunt down any outstanding copy to enforce the new arrangement. I’m not suggesting they will change anything, but there is no externalised guarantee they won’t.

I believe this is a serious concern for the product as a whole, and hope this concern will be addressed by Sun Microsystems very soon – with action.

Posted on 5 Comments
Posted on 8 Comments

Quiz: Enabling an application for MySQL Replication

A little challenge for you… given an existing app that does not know about separate master/slave connections, and you want to enable working in a replicated infrastructure. Simply redirecting all SELECTs to the slave connection will not work. Why?

Hint: there are at least two reasons, depending on other factors. There may be more.

Comments are set to be moderated so providing answers will not spoil it for others. I’ll leave it run for a bit and then approve all comments.

Posted on 8 Comments
Posted on 5 Comments

Measuring HD latency in ways relevant to MySQL

As I described yesterday, Open Query is doing some tests on SSDs and other devices pretending to be harddisks (SANs, battery-backed RAID controllers, etc). To aid this, I wrote a small tool to test the different kind of I/O operations MySQL would/could do, which is not quite the same as what other general purpose apps would do, and also not what other test tools measure. For instance, it tries Direct I/O as well as fsync() after each write, and also it a range of different I/O block sizes.

In a nutshell, it’s aimed to do what MySQL does, without MySQL! Testing lots of different setups for this particular purpose (even with fantastic tools like MySQL Sandbox) is a complete pest, and changing InnoDB page size requires a recompile. While Percona has tried a larger page size in the past and decided it wasn’t worth it (the default is 16K), I thought it worthwhile to include such a test as the situation may change over time with different devices.

So, this is a little tool for a very specific purpose, and it should not grow beyond that – but do feel free to abuse it for whatever other purpose you reckon fits a similar approach. Oh, and it outputs CSV for easy graphing. To grab the code, go to the hdlatency project on Launchpad. It’s plain C, and GPLv3 licensed.

Posted on 5 Comments