Posted on 4 Comments

Tool of the Day: screen

Only the other day I was talking with someone who does a lot of work on the shell command line, but hadn’t used the GNU screen tool, so I’d better scribble a post about it as I regard it as an absolute must-have for any remote work, for multiple reasons.

First of all, what screen does. You start screen inside a terminal session (local or SSH remote), and then you can create additional sessions though Ctrl-A C. The initial screen is number 0, the next one 1, and so on. You can switch between screens with Ctrl-A # where # is the screen number. This way, you can have multiple things going within a single ssh connection, very handy. But that’s not all!

If you get disconnected (it happens 😉 and you reconnect, your screen sessions will still be there, and running too. You can reattach with screen -r. To do a nice disconnect, you can do Ctrl-A D (detach) before closing your ssh connection.

You can also have multiple screen sessions by name, screens within screens (that confuses me for the control keys so I tend not to use that), and an absolute supertrick is that you can actually share a screen session with someone else. That’s sometimes mighty handy with two engineers to look at something, and also for showing things to clients.

The tool itself is absolutely ancient (aka rock solid, in maintenance mode), I did a quick check and I see references as far back as 1987. I remember using it long long ago, might’ve been a XENIX box. I reckon screen’s authors deserve a prize for creating one of the most useful tools ever!

Default Linux installs often don’t have it, but rectifying that is as simple as sudo apt-get install screen or sudo yum install screen. Then, man screen is your friend, but there are also quite a few decent tutorials on the web.

Posted on 4 Comments
Posted on 1 Comment

Tool of the day: inotify

I was actually exploring inotify-tools for something else, but they can also be handy for seeing what goes on below a mysqld process. inotify hooks into the filesystem handlers, and sees which files are accessed. You can then set triggers, or just display a tally over a certain period.

It has been a standard Linux kernel module since 2.6.13 (2005, wow that’s a long time ago already) and can be used through calls or the inotify-tools (commandline). So with the instrumentation already in the kernel, apt-get install inotify-tools is all you need to get started.

 # inotifywatch -v -t 20 -r /var/lib/mysql/* /var/lib/mysql/zabbix/*
Establishing watches...
Setting up watch(es) on /var/lib/mysql/mysql/user.frm
OK, /var/lib/mysql/mysql/user.frm is now being watched.
[...]
Total of 212 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 60 seconds.
total  modify  filename
2371   2371    /var/lib/mysql/relay-log.info
2148   2148    /var/lib/mysql/master.info
1157   1157    /var/lib/mysql/ib_logfile0
24     24      /var/lib/mysql/zabbix/
24     24      /var/lib/mysql/zabbix/history.ibd
8      8       /var/lib/mysql/zabbix/trends_uint.ibd
6      6       /var/lib/mysql/zabbix/items.ibd
5      5       /var/lib/mysql/ibdata1

This is just a limited example from a dev box, but you can see the benefit. You can see which files have been accessed, in what way, and how many times over the specified period. Consequently this provides the most insight if you’re using innodb-file-per-table (or MyISAM) rather than a single InnoDB tablespace. But of course it depends a bit on what you’re looking for.

Posted on 1 Comment
Posted on 8 Comments

Fab tool: watch

A post by Ronald reminded me I wanted to write about watch. The Linux watch command is a fab tool, it allows you to run a program at a set interval, such as every 5 seconds. Handy for keeping an eye on stuff!

Update 2009-06-30: yesterday I wrote “… watch does not re-load and run the program, but goes in and resets the program counter.” (and was un-recommending it for use on production systems for that reason). I was blindly quoting an otherwise trusted source, and it was wholly wrong. Numerous people had a peek at the source code (all the way back to 1997) and found the popen(). So, I stand corrected and my apologies for not doing extra homework in the first place before posting! (I’m leaving the comments as they are)

Posted on 8 Comments
Posted on 3 Comments

Tool of the day: ack – better than grep

I’m decently familiar with grep so I can usually make it do what I want. I frequently need to search for instance MySQL source code for certain pattern strings, and this makes life so much easier. But Akash pointed out ack to me, which has the specific tagline “better than grep” (has the domain even) and I reckon it does live up to that. Win! It’s written in pure Perl, very easy to install (doesn’t even use CPAN if you don’t want).

It recurses into subdirs by default, while ignoring stuff like revision control and binary files. You can search specific types of files through a symbolic name rather than by specifying regexes. And it has colour highlighting, and simply uses the familiar Perl regexes for its pattern matching rather than funky subsets of which there are many distinct ones…

Posted on 3 Comments
Posted on

Modular vs Integrated

There’s actually no single “correct” answer! It all depends on

  1. where in a stack the component lives;
  2. the state of the market for that component region;
  3. sometimes even geographic location of the user comes into play.

Yes, for OSS projects modularity is handy in terms of handling contributions, but modularity may not be the best way to deal with a problem in a certain market state and situation!

Research has shown (see, for example, “The Innovator’s Solution” by Clayton Christensen) that the “integrated” region over time actually shifts to a subcomponent of an original integrated component that has since gone modular. An interesting example of this for MySQL its pluggable storage engine interface since version 5.1. MySQL is more modular now, but individual storage engines are tightly integrated for performance reasons, and in some cases they are even proprietary. It’s important to realise that this too is just a phase and not a final state.

But sticking with OSS for this story, distributed version control with (among other things) decent branching/merging make contributions to the core are entirely feasible and even easy. It’s a matter of using a toolchain that supports the most suitable work process, rather than hinder it or force another unsuitable wok process. Good distributed version control systems: bzr (Bazaar), hg (Mercurial), git.

(note: svn/subversion, even with some “distributed” hacks on top of it, does *not* qualify as it’s architecturally unsuitable. The other tools have excellent migration plugins available to get your work process -and your code- out of svn hell. It’s a bit of work but can be a stepwise process to unwarp your work processes and get a sane and more productive development workflow.)

At Open Query we particularly like bzr, because with Launchpad it provides an excellent environment for tracking of bugs, features and ongoing work, merge reviews, and so on. The admin side of Launchpad has some user interface issues, but it is workable.

Sometimes the core of a particular component does need fixing – but the time may not be right for modularity, and it’s important to not get distracted by that geeky desire of “everything modular” as swimming against market forces can be very painful.

Posted on