Noflushd Patch for 2.6.32+ Kernels

After a recent kernel upgrade (from 2.6.26 to 2.6.34), Noflushd[1] stopped working. Worse, it broke quite badly in that CPU usage spiked as soon as sync's started to happen and in all probability some sort of a continuous loop seems to have kicked in. It appears that Noflushd relies on writing a "0" into the /proc/sys/vm/dirty_writeback_centisecs file which results in the new pdflush (threaded) implementation essentially continuously writing (since the write daemon now sleeps for "0" seconds between writes, and spawns multiple threads, one per partition/mount-point, to do a write). There are a lot of side effects of the pdflush implementation changes and noflushd is one that is quite severely affected.

Noflushd is not the preferred choice for spindowns when you factor in the availability of laptop_mode and the capabilities of hard disks to spindown when there is no activity (as configured via 'hdparm -S ...'). Unfortunately if one uses a Western Digital Scorpio Blue Notebook drive (on Linux), many (most? Or worse, all?) drives seem to have "broken" firmware in that they either spindown extremely aggressively (~8s) or not at all. They blatantly ignore hdparm values and if you factor in the loadcycle ("head parking") that also happens at the same rate of ~8s, there's a very strong likelihood that the drive will die fairly quickly. So, in the unfortunate event you have similar drive behaviour, short of writing a periodic spindown command job (which would not be aware of writes and their benefits on resetting timers) it is best to rely on Noflushd.

After a fair bit of tweaking and debugging, here's a set of changes (see below) that makes Noflushd work as it is supposed to, like with the 2.6.26 (and older) kernels. Essentially these changes also hinge on having the old EXT3 behaviour (journal mode is ordered instead of the now-default writeback) - so make sure that you are using EXT3 with the right behaviour (since this too has changed in the recent kernels).

First a bit of background on the new pdflush implementation. Essentially the /proc/sys/vm/dirty_writeback_centisecs file has a changed meaning (or rather behaviour) when it is set at zero. In older kernels (including 2.6.26), a zero in dirty_writeback_centisecs meant that the background flush daemon was disabled (i.e. not woken up periodically to flush writes to disk). This was crucial in using noflushd correctly to prevent unnecessary spinups of the drives since noflushd used that mechanism to disable the background writes before forcing the hard disk to sleep/standby. In the new kernels, instead of a "0", a "-1" seems to disable the background writes completely (and also results in "correct" behaviour when using noflushd as the harddisk correctly spins down and in general works as before). A fair bit of background on the pdflush changes (not necessarily related to the "0" vs "-1") are here [2,3,4,5].

Disabling the writeback daemon/threads using a "-1" in /proc/sys/vm/dirty_writeback_centisecs (instead of "0" that worked before) requires a few key changes to noflushd (especially when you want it to also work correctly with the older kernels that still write a "0"). One more interesting issue is that if /proc/sys/vm/laptop_mode has a non-zero value in it, the kernel will force a full sync that many seconds after any other write (including a noflushd sync), which will result in a forced disk write and wakeup the disk if it is sleeping. This is a big deviation from the past where a (forced) sync would not force a fresh write later. As a result, to get the disk to spindown correctly (using noflushd), it is essential to disable laptop_mode. Of course all this this is irrelvant if you have a well behaved disk to begin with.

So to support both the new 2.6.32+ kernels and the older kernels, noflushd now respects a "-1" in the dirty_writeback_centisecs entry and uses it accordingly. The patch (below) also has some other useful changes as well. Noflushd will now track how long drives have been spun down as well as how many times it has been spun up/down along with average duration of spin downs. The interval (as specified on the command line) is now based on 10s intervals (so 5 means 50s) instead of the original minute based spec. The new spec allows much finer grained control of the spin down times. The statistics are dumped (via syslog entries) on shutdown as well as when a SIGUSR1 signal is received. Additionally when a SIGHUP is received, the next (default) timeout value is used and that also is logged to syslog making the workings of noflushd much clearer to review and tweak the time settings.

In summary to have noflushd work correctly on the 2.6.32+ kernels, the following changes have to be made to the system configuration:

• Disable laptop mode: /proc/sys/vm/laptop_mode should have "0"
• Disable writeback expiry: /proc/sys/vm/dirty_writeback_centisecs should have "-1" (this is a change from the past when this was "0" to disable writebacks and some other (>0) value to have periodic flushes).
• Use the patched noflushd: Noflushd has now been patched to work correctly with the new kernels ONLY if the write back expiry value is "-1". Otherwise it will default to the old behaviour of writing a "0" in /proc/sys/vm/dirty_writeback_centisecs (which will break in the newer kernels). Caveat emptor. YMMV.
• Disable the automatic hard disk spindowns/tweaks : Use 'hdparm -B 128' (or higher) to disable automatic spindowns of the hard disk. A value of 254 will also disable the auto-head park feature, resulting in a controlled and correct loadcount increase for the hard disk head.
  Note: These hdparm values are specific to the Western Digital Scorpio Blue Series Npotebook IDE drives.
• Other Notes: Noflushd works in conjunction with all the other caveats applicable to delayed writes and laptop mode. See [6] for more information other applicable caveats before relying on noflushd.

Download the patch for noflushd here.

Keywords: linux kernel 2.6.32 2.6.26 writeback expiry dirty disk spindown hdparm pdflush noflushd western digital scorpio blue spindown firmware

URL[1]: http://noflushd.sourceforge.net/
URL[2]: http://lwn.net/Articles/326552/
URL[3]: http://axboe.livejournal.com/1819.html
URL[4]: http://axboe.livejournal.com/2258.html
URL[5]: http://lwn.net/Articles/9521/
URL[6]: http://www.kernel.org/doc/Documentation/laptops/laptop-mode.txt

[/technology] permanent link

Debian's 17th Birthday

Yesterday was Debian's 17th Birthday! Slashdot has a post[1] and OSNews also has a brief post [2] both referencing the same original article[3]. There's a really nice tribute [4] page and comments page [5].

Debian was the first distribution to really elegantly solve the dependency hell problem that plagued the various distributions that existed back in the day. Now most distributions do the "right thing" in that they fetch dependencies automagically, so the package manager is no longer the sole differentiating factor. Debian nevertheless is a really elegant distribution that still strives for stability (at least for the stable distribution). Coupled with the very large platform support and the very large package repository, it is still one of the finest Linux distributions around.

Happy Birthday Debian and here's to looking forward to many many more years of your association.

URL[1]: http://linux.slashdot.org/story/10/08/16/1512223/Happy-17th-Birthday-Debian
URL[2]: http://www.osnews.com/story/23693/Happy_17th_Birthday_Debian
URL[3]: http://digitizor.com/2010/08/16/happy-17th-birthday-debian-and-some-interesting-history/
URL[4]: http://thank.debian.net/thank_debian
URL[5]: http://thank.debian.net/read_thanks

[/misc] permanent link

Quote of the Day for 2010-08-15

Long years ago we made a tryst with destiny, and now the time comes when we
shall redeem our pledge, not wholly or in full measure, but very substantially.
At the stroke of the midnight hour, when the world sleeps, India will awake to
life and freedom. A moment comes, which comes but rarely in history, when we
step out from the old to the new, when an age ends, and when the soul of a
nation, long supressed, finds utterance. It is fitting that at this solemn
moment we take the pledge of dedication to the service of Inida and her people
and to the still larger cause of humanity.
                -- Jawaharlal Nehru, "Tryst with Destiny"
                   Independence Day Speech, 15th August 1947

[/quotes] permanent link

NYTimes on SSL Certificate Issues

The NYT has an article[1] in the Technology section about SSL Certificates and how the profusion of Certificate Authorities (CAs) is a security issue that needs to be addressed. The key issue is that due to the chains of trust (as accepted by browsers like IE and Firefox due to the root certificates that they ship with) entire countries could act as "countries-in-the-middle" and harvest (for lack of a better term) information that the browsers are completely unaware of. The UAE seems to be a case in point (especially with the recent RIM security issues featuring in security headlines across the glove). The article seems to be a continuation on the recent research by the EFF[2] as posted here[3] and here[4].

URL[1]: http://www.nytimes.com/2010/08/14/technology/14encrypt.html
URL[2]: https://www.eff.org/
URL[3]: https://www.eff.org/observatory
URL[4]: https://www.opengear.net/blog/2010/08/08#EFFSSLObservatory-20100808

[/technology] permanent link

Wall Street Journal Covers Internet Privacy and Anonymity

Over the past few days, WSJ has been covering the various facets (and fallacies) of privacy as it relates to the Internet. The articles have been quite interesting and reveal a lot of the behind the scenes goings on by various companies and how they are able to track users and their online excursions. There's one article [5] that also covers privacy as it relates to cell phone usage and location tracking. They also do offer some advise on how to avoid being tracked as well as a reasonable glossary of some terms. All in all a very nice set of reads that are quite revealing.

The series is called "What They Know" [6,7] (very appropriate!).

Nicholas Carr's article "Tracking Is an Assault on Liberty, With Real Dangers" [2], Jessica Vascellaro's "Google Agonizes on Privacy as Ad World Vaults Ahead " [3] and "On the Web's Cutting Edge, Anonymity in Name Only" [4] by Emily Steel and Julia Angwin are particularly nice reads.

On a related note, Bruce Schneier[8] has an interesting blog entry [9] on social networking data (sort of tangential to the WSJ privacy articles that are more browsing/online related vs. the social networking aspects of conscious or intentional exposure). Of course there's also a Slashdot post[9] related to it. All in all there's suddently a spurt of opinions, ideas, thoughts (with their fair share of facts) and it's been a series of interesting reads.

URL[1]: http://online.wsj.com/public/search?article-doc-type={What+They+Know}&HEADER_TEXT=what+they+know
URL[2]: http://online.wsj.com/article/SB10001424052748703748904575411682714389888.html
URL[3]: http://online.wsj.com/article/SB10001424052748703309704575413553851854026.html
URL[4]: http://online.wsj.com/article/SB10001424052748703294904575385532109190198.html
URL[5]: http://online.wsj.com/article/SB10001424052748703467304575383522318244234.html
URL[6]: http://online.wsj.com/public/page/what-they-know-digital-privacy.html
URL[7]: http://blogs.wsj.com/wtk/
URL[8]: http://www.schneier.com/
URL[9]: http://www.schneier.com/blog/archives/2010/08/a_taxonomy_of_s_1.html
URL[10]: http://yro.slashdot.org/story/10/08/10/1352244/Schneiers-Revised-Taxonomy-of-Social-Data

[/technology] permanent link