Posted on 5 Comments

Ladies and gentlemen, check your assumptions

I spent some time earlier this week trying to debug a permissions problem in Drupal.

After a lot of head-scratching, it turned out that Drupal assumes that when you run INSERT queries sequentially on a table with an auto_increment integer column, the values that are assigned to this column will also be sequential, ie: 1, 2, 3, …

This might be a valid assumption when you are the only user doing inserts on a single MySQL server, but unfortunately that is not always the situation in which an application runs.

I run MySQL in a dual-master setup, which means that two sequential INSERT statements will never return sequential integers.  The value will always be determined by the  auto_increment_increment and auto_increment_offset settings in the configuration file.

In my case, one master will only assign even numbers, the other only uneven ones.

My patch was accepted, so this problem is now fixed in the Drupal 7 (and hopefully soon in 6 as well) codebase.

The moral of the story is that your application should never make such assumptions about auto_increment columns.  A user may run the application on a completely different architecture, and it may break in interesting and subtle ways.

If you want to use defined integers like Drupal does, make sure you explicitly insert them. Otherwise, you can retrieve the assigned number via the mysql_insert_id() function in PHP or via SELECT LAST_INSERT_ID() in MySQL itself.

Have you checked your code today?

Posted on 5 Comments
Posted on 14 Comments

Good Practice / Bad Practice: Table Aliases

When writing queries, try making a habit out of using short table aliases, no matter how small the query is. Try using an alias that is as short as possible (one letter if possible, two or three when necessary) to avoid clutter in your queries.
Use these aliases in all places where you refer to a field, no matter if it would technically not be necessary. It will make your queries readable, and if you later need to join in other tables, you will not run the risk of ambiguous field names.

Let’s look at an example using the World database. The following two queries both select the 10 biggest cities by population in the continent of Africa, with a population between 1 and 1,5 million.

Good practice:

SELECT
  ci.name AS city_name,
  ci.district,
  co.name AS country_name,
  ci.population
FROM
  Country AS co
  INNER JOIN City AS ci ON (co.code = ci.countrycode)
WHERE
  (co.continent = 'africa') AND
  (ci.population BETWEEN 1000000 AND 1500000)
ORDER BY
  ci.population DESC
LIMIT 10;

bad practice:

select
  City.name,
  District,
  Country.name,
  City.population
from
  Country
  inner join City on City.countrycode = Country.code
where
  continent = 'africa' and
  City.population between 1000000 and 1500000
order by
  City.population desc
limit 10;

Both queries will execute and return the same results, but the first one is much more readable. Issues with respect to the table aliasing in the second query:
1) It is unclear without looking at the table structure whether District and continent are fields of the city or the country table.
2) The full table name needs to be specified for every field that would otherwise be ambiguous. This clutters the query and thus makes it less readable.
3) if I ever need to change the name of the table I am querying, I have to change it in all places instead of just in the from clause. A good example is when I was writing this article: I used lower case c’s for the country and city table names. So after writing the whole statement I had to go back in and change that lowercase c to an uppercase c in close to a dozen places, while in the first query I just needed to change it in one place.

In our good practice / bad practice series, we will provide you with byte/bite sized pieces of advice on what we consider good (or bad) practice in MySQL-land. The topics can be just about anything, so expect random things to come up. Also, the level of advancedness greatly varies. A topic might be a no-brainer for some, a reminder for others and a revelation for a third person. We strive to tender to all of you!

Posted on 14 Comments
Posted on 6 Comments

Your opinion on EC2 and other cloud/hosting options

EC2 is nifty, but it doesn’t appear suitable for all needs, and that’s what this post is about.

For instance, a machine can just “disappear”. You can set things up to automatically start a new instance to replace it, but if you just committed a transaction it’s likely to be lost: MySQL replication is asynchronous, EBS which is slower if you commit your transactions on it, or EBS snapshots which are only periodic (you’d have to add foo on the application end). This adds complexity, and thus the question arises whether EC2 is the best solution for systems where this is a concern.

When pondering this, there are two important factors to consider: a database server needs cores, RAM and reasonably low-latency disk access, and application servers should be near their database server. This means you shouldn’t split app and db servers to different hosting/cloud providers.

We’d like to hear your thoughts on EC2 in this context, as well as options for other hosting providers – and their quirks. Thanks!

Posted on 6 Comments
Posted on

libmemcached packages

Ronald Bradford last week posted about memcached not being multi-threaded on Ubuntu, something he discovered via some small utilities that are bundled with libmemcached, written by Brian Aker.

When I noticed there were no Ubuntu packages for libmemcached (or the CLI tools) I decided to create some.

For your enjoyment: http://ubuntu.cafuego.net/dists/jaunty-cafuego/memcached/ (Source debs are included)

The repository also contains a memcached that has been re-compiled with multithreading enabled.

Posted on
Posted on 3 Comments

How many files does InnoDB have open?

If you use innodb_file_per_table = 1 and innodb_open_files = X (whatever amount is suitable for your server) there’s no way internal to MySQL for finding out how many IBD files InnoDB actually has open. Neither SHOW GLOBAL STATUS LIKE ‘innodb%’ nor SHOW ENGINE INNODB STATUS provide this information.

Many sites do have a growing number of tables, so you’ll want to know when it’s time to up the number (and potentially also open-files-limit). Solution: sudo lsof | grep -c “\.ibd$”

Posted on 3 Comments
Posted on 6 Comments

Update: MySQL tmpdir on tmpfs

Followup on Experiment: MySQL tmpdir on tmpfs, about tmpdir=/dev/shm in my.cnf (it’s not a dynamic variable that can be set at runtime). It’s working well, also confirmed by comments from others that they’ve been using it for a while.

This particular setting is Linux specific. On Solaris, the default /tmp is already on a tmpfs so that’s fine too. Brian reminded me that this tweak is also useful if you’re stuck with a 32-bit OS as you can then utilise some more memory in a practical way.

Extra useful hint from Harrison: if you are using replication, you will also want slave_load_tmpdir=/tmp on your slave (real disk which survives a restart). The issue is that with statement based binary logging, there are many events which create a file for a replicated LOAD DATA INFILE. If you stop your server after some of these events have occurred, but not all, it will break after you restart.

Thanks everyone for their feedback!

Posted on 6 Comments
Posted on 11 Comments

Experiment: MySQL tmpdir on tmpfs

In MySQL, the tmpdir path is mainly used for disk-based sorts (if the sort_buffer_size is not enough) and disk-based temp tables. The latter cannot always be avoided even if you made tmp_table_size and max_heap_table_size quite large, since MEMORY tables don’t support TEXT/BLOB type columns, and also since you just really don’t want to run the risk of exceeding available memory by setting these things too large.

You can see how your server is doing with temporary tables, how many of those become disk tables, and also other created temporary files, by looking at SHOW GLOBAL STATUS LIKE ‘Created_tmp%’;

So why might tmpfs be better? Well, not just any tmpfs, but specifically /dev/shm. Its behaviour (see the kernel docs) may just fit the need, provided you have sufficient “spare” RAM in the machine anyway. Essentially the files will live in kernel cache space, and swap is potentially possible. You can even tweak limits, live. Mind you, this is Linux specific. Any system with glibc 2.2 or higher will have a /dev/shm available.

We’re currently running a comparative test in a production environment with one slave on tmpdir=/dev/shm. It being significantly faster is no real surprise but we just want to make sure that it behaves well over time, in the real world. Naturally it is best to not need disk-based temp tables at all for most queries, but for existing apps you can’t always change problematic design issues “right now”. It’s not a solution, but extra “breathing space” is useful. We’ll report back later with more results and possible insights from this experiment.

2009-06-16: See followup at Update: MySQL tmpdir on tmpfs.

Posted on 11 Comments
Posted on 3 Comments

100% subscription renewal

I’m happy to note (this is internal Open Query happiness but I’m pleased to share) that so far we have a 100% renewal rate for our Proactive Services for MySQL subscriptions. Some of the early clients have grown in the initial period and are have now moved to a higher # of hours (this can also be changed upward during a term), which is of course excellent both for the clients and for us.

I was in eager anticipation of this time since the introduction of the concept late last year, as it is of course the essential proof of whether a subscription service actually works over time. Ideally, you’d want renewal to be a simple straightforward process, with the client having experienced the value of the service. This is relatively straightforward in this case, since it’s not an insurance, emergency or retainer type arrangement – the client actually gets benefits each and every month, so there’s both technical progression as well as ongoing human contact. Seems like a winner!

Along the way we also see a steady influx of new clients. I haven’t been specifically chasing this, as all new concepts take a while to mature, and we also had new people internally. The really cool thing is that our business structure for this service is scalable – I won’t say linearly because at some point the # of internal people involved would require adapting some processes, but it’ll scale a fair way still from where we are now.

Elspeth, our Special Projects Operative, who apart from an ace coder&geek is also organisationally organised, has been a great help with some of the admin aspects of the company. We’re paper-less, but that doesn’t mean there’s no paper. We tend to not produce more, but we do get it from others 😉

Posted on 3 Comments
Posted on

Modular vs Integrated

There’s actually no single “correct” answer! It all depends on

  1. where in a stack the component lives;
  2. the state of the market for that component region;
  3. sometimes even geographic location of the user comes into play.

Yes, for OSS projects modularity is handy in terms of handling contributions, but modularity may not be the best way to deal with a problem in a certain market state and situation!

Research has shown (see, for example, “The Innovator’s Solution” by Clayton Christensen) that the “integrated” region over time actually shifts to a subcomponent of an original integrated component that has since gone modular. An interesting example of this for MySQL its pluggable storage engine interface since version 5.1. MySQL is more modular now, but individual storage engines are tightly integrated for performance reasons, and in some cases they are even proprietary. It’s important to realise that this too is just a phase and not a final state.

But sticking with OSS for this story, distributed version control with (among other things) decent branching/merging make contributions to the core are entirely feasible and even easy. It’s a matter of using a toolchain that supports the most suitable work process, rather than hinder it or force another unsuitable wok process. Good distributed version control systems: bzr (Bazaar), hg (Mercurial), git.

(note: svn/subversion, even with some “distributed” hacks on top of it, does *not* qualify as it’s architecturally unsuitable. The other tools have excellent migration plugins available to get your work process -and your code- out of svn hell. It’s a bit of work but can be a stepwise process to unwarp your work processes and get a sane and more productive development workflow.)

At Open Query we particularly like bzr, because with Launchpad it provides an excellent environment for tracking of bugs, features and ongoing work, merge reviews, and so on. The admin side of Launchpad has some user interface issues, but it is workable.

Sometimes the core of a particular component does need fixing – but the time may not be right for modularity, and it’s important to not get distracted by that geeky desire of “everything modular” as swimming against market forces can be very painful.

Posted on
Posted on 1 Comment

Continuity of power

Last night my residential area lost power for about 2 hours, between 2-4 am. This reminded me of something, and there’s analogies to MySQL infrastructure. Power companies have over recent years invested a lot of money in making the supply more reliable. But, it does fail occasionally still.

From my perspective, the question becomes: is it worth the additional investment for the power companies? Those extra few decimal points in reliability come at a very high cost, and still things can go wrong. So a household (or business) that relies on continuity has to put other measures in place anyway. If the power company has an obligation to deliver to certain standards, it might be more economical for them to provide suitable equipment (UPS, small generator) to these households and business (for free!) and the resulting setup would provide actual continuity rather than merely higher reliability with occasional failures. Everybody wins.

As a general architectural considering, new houses can be designed with low voltage circuits (12-24V) in most areas and 110/240V just in kitchen, laundry. Why? Because most stuff around your house actually runs on low voltage anyway, but uses inefficient heat-generating transformer blocks (power adapters) to get it. Cut out the middleman! It saves money, looks better, is safer (no transformers in ceiling for halogen lighting, etc), and there are fewer points of failure. It’d be fed from central batteries, charged by solar if you wish, and a single transformer (basically like a car battery charger) from mains power. Also think about led lighting for some places and uses, very cheap to run. Apart from all the aforementioned advantages, again it delivers higher uptime since a power failure will then not affect your lights and other stuff running off the low voltage circuit, since its direct power source is a battery – essentially a UPS for most of your house’s electricity needs.

Mind you, I didn’t invent any of this this. It’s been done. All it takes is builders with vision, and/or home owners with initiative. Appying existing technology.

Posted on 1 Comment