| "Congress shall make no law... abridging the freedom of speech, or of the press..." |
| "Congress shall make no law... abridging the freedom of speech, or of the press..." |
What follows is based on my observations. My focus is the relative performance of different filesystems, not the raw benchmark numbers of my hardware. For this reason, I have not included any specific model numbers of the hardware.
Part of my "economic stimulus check" went to a 500GB SATA drive. My original intention was to buy two of them, so I could claim, "over a terabyte of disk space!". Alas, I got a little ahead of myself; my system had only one open hard drive bay. With a slightly bruised ego, I returned the unopened second hard drive and began to ponder how to exploit my super-roomy disk space. I quickly settled on one goal: find the fastest journaling filesystem (FS) for my SLAMD64 dual-core computer, with 2G of memory. My testing focused on three main areas: filesystem, disk I/O scheduler, and CPU speed.
I chose ext3, JFS, and XFS for my filesystem options. I specifically excluded ReiserFS from my testing, due to its tendency to bypass many of Linux's internal disk management functions.
So that others may run similar tests on their own systems, I have provided a gzipped tarball containing the scripts and my own test results.
Frankly, the final results stunned me.
My first round of tests was a home-brew hack involving Slackware's package management suite, distributed via threads across several directories, but I found my results too difficult to interpret. I also tried bonnie++, but test after test turned up no clear winner on anything. Part of the problem, I think, is that most of testing in bonnie++ runs in the context of a single thread, leaving each test CPU-bound.
After presenting a little bit of testing results to a discussion board, I learned about irqbalance, which works to distribute the IRQ load evenly across multiple processors, while keeping a particular IRQ response on a single CPU or core as much as possible. On general principles, I installed it immediately.
A little more research reminded me of dbench, the fantastic suite by Andrew Tridgell of Samba fame. His general goal with dbench is to exercise and time all of the basic file operations: create, write, seek, read, rename, lock, unlock, stat, and delete. In addition, the design of dbench specifically includes multi-threading; if a system is designed with multi-processing in mind, dbench will be able to demonstrate its advantage. The results I report here are from dbench, because each run showed a clear winner, by a substantial margin.
As an aside, another favorable point of dbench over bonnie++ is that dbench determines its run length by clock time, rather than number of operations completed. This gives a more deterministic approach to filesystem benchmarking, something I prefer. I'd rather provide a test script that runs for 5 minutes on all systems, than a test that operates on 10,000 files in 30 seconds here, and 20 minutes there.
My first step (while still stuck in bonnie++-land) was to find a set of FS options that provided somewhat good performance. One option that stood out for each was an external journal, on a different controller. By isolating main partition I/O from journal I/O as much as possible, an SMP system can drive both of them at once. This helps all journaling filesystems.
With that in mind, I used /dev/sdb5 for the main partition, with /dev/sda2 for the journal.
The following are the commands and options I used:
ext3: mke2fs -O journal_dev /dev/sda2 #first step
mke2fs -j -J device=/dev/sda2 /dev/sdb5 #second step
JFS: jfs_mkfs -j /dev/sda2 /dev/sdb5
XFS: mkfs.xfs -l logdev=/dev/sda2,lazy-count=1,size=64m -d agcount=8 -f /dev/sdb5
Three things to note:
1. mke2fs looks for /etc/mke2fs.conf, which may contain additional ext2/ext3 options. My system specifies sparse_super, filetype, resize_inode, dir_index, and ext_attr, with a 4K block size and a 128-byte inode size.
2. jfs_mkfs has surprisingly few documented options.
3. mkfs.xfs has many options. The two mandatory options I needed were "logdev=" for the journal (logging) device, and "size=64m" to clarify that only 64 megs (the maximum) of the 2G partition would go to the journal. The other options are "lazy-count=1", which eliminates a serialization point in the superblock, and "agcount=8", meaning 8 allocation groups in the main data partition.
After creating each filesystem, I needed to mount it to /tester. I always passed "-o noatime" to each mount, so that every call to read() would not cause a write to disk.
For ext3, I also specified "data=writeback", which greatly increased overall performance. This option is explained in the Linux kernel documentation as well as the mount(8) man page.
For XFS, I added "logdev=/dev/sda2,logbufs=8,logbsize=262144". One drawback of XFS is that the external journal device must always be specified; it has no data in the superblock to indicate the journal location, other than "internal" or "external". (I will follow up on this in a later article.) I specified 8 logging buffers, to match the number of allocation groups in the filesystem, and gave as much RAM to each log buffer as possible (262,144 bytes).
As with jfs_mkfs, JFS has a dearth of mount options.
In my initial dbench tests, I noticed that my filesystem throughput was mostly stabilized after about 15 seconds of actual test time. Since I was interested more in the short-term throughput typical of program loading and linking (and I tend to be impatient), I shortened the test time to 60 seconds with the following command line:
dbench -t 60 -D /tester $THREAD_COUNT
For a mild system load, I used 5 threads; for a heavy load, I used 20 threads. These are admittedly arbitrary figures, but they did expose well-threaded or poorly-threaded design on my dual-core system.
I created a script to do two primary tasks:
1. Warm the cache with a preliminary run of dbench.
2. Run dbench with each of the four I/O elevator algorithms (noop, deadline, anticipatory, and CFQ) as well as the slowest and fastest CPU speed available on my system. The output of these 8 iterations went to /tmp/$FILESYSTEM-$SPEED-$ELEVATOR.txt.
I also hacked up a little script for preliminary analysis of these output files. It found the best and second-best performers for each dbench operation, followed by an ascending list of overall throughput. I found that the elevator can be more important than CPU speed for a filesystem's performance. The deadline elevator generally did best for all three filesystems in my tests, although the impact of elevator choice varied.
Since not everyone has an SMP system, I also ran the tests after booting with "nosmp".
I was amazed by the numbers. For all but one uniform subset of tests, XFS was the clear winner. I even tried synchronous mounts and encrypted volumes as little "what if?" exercises, and XFS still came out on top. The parameter that cost XFS a total victory was the CFQ elevator on a slower system; ext3 won most of those cases.
One thing that shines about XFS is its highly-threaded design. For every permutation of elevator and CPU speed, XFS scaled upward from 5 threads to 20, although the degree of scaling with "noop" is probably within statistical noise. However, the CFQ elevator had a very large difference between a slow and fast CPU. In fact, (XFS + CFQ + fast CPU speed) on my system clashed so badly that the 5-thread test was the worst one for XFS.
JFS had a surprising disappointment: it never scaled upward. In fact, every single test had better performance with 5 threads than with 20. CPU speed and I/O elevator did not matter. All the 5-thread tests had throughput somewhere between 100-160 MB/s, and all the 20-thread tests came in somewhere between 40-45 MB/s. Once JFS reaches saturation, it performs no better than it would with a synchronous mount (-o sync).
What about ext3? Well, it showed some strange behavior on my system. With a fast CPU, it also showed the characteristic of performing (slightly) better under light load than under heavy load. However, with a slow CPU, all elevator differences disappeared into statistical noise, falling between 160-170 MB/s.
Grouping the results by CPU speed, threads, and I/O elevator, I found that XFS was best on SMP in all but two permutations. For the noop, deadline, and anticipatory elevators, all results come up with XFS best, then ext3, then JFS. With the CFQ elevator and 5 threads, ext3 won, then XFS, then JFS.
With a single CPU and 20 threads, the story was the same: XFS, then ext3, then JFS. However, with the lighter load of 5 threads, there was no uniform winner. ext3 topped the list with the CFQ scheduler and a faster CPU; otherwise, XFS was the winner.
So which was fastest? On my SMP system, XFS with the deadline elevator topped the list, giving over 400 MB/s throughput. I switched my /home and /usr directories to use XFS with external journals, set the deadline elevator on the kernel command line, and suddenly OpenOffice.org went from 6.5 seconds launch time to 3.5 seconds. I am not the only one to notice that XFS performance improves with the deadline elevator.
Another drawback of XFS's external journal is that it is currently impossible to specify the journal location on the kernel command line. The option "rootflags=logdev=/dev/XXXX" is not handled properly, due to shortcomings in the kernel's handling of the first root volume mount. I circumvented this with a hacked initrd, which is another article in itself.
Ten years ago this month (April 1, 1998, to be precise), I got hit with the Chernobyl virus. It wiped out my hard drive's partition table and boot blocks, leaving me with the computing equivalent of a cement block on my desk.
I had been experimenting with Slackware Linux, even getting to the point of having a dual-boot system via AUTOEXEC.BAT and the LOADLIN.EXE Linux loader. But when the partition table got zapped, I made a draconian decision:
NO MORE MICROSOFT WINDOWS FOR ME!
I had seen enough of Linux to be impressed, from the pure 32-bit environment to the system's stability in the face of program naughtiness. Microsoft couldn't make any such claim in Windows 95. I knew that, with such credentials and effort behind Linux, it wasn't going away anytime soon.
The next day, I re-installed Slackware Linux, but this time on my entire hard drive. No Microsoft Windows. None! No safety net, no fancy GUI to hide all the registry options, no programmer in Redmond telling me what I wasn't allowed to do. I had "cast off the bow-lines" and bravely sailed into new waters.
The first few days were tough. I had to re-configure my X server (this was before XFree86 supported monitor detection) and get my Internet dial-up to work again. Then I fired up the Lynx browser and re-downloaded the latest Netscape browser for Linux. Being in the countryside, the best dial-up I could get was 28.8Kbps, and that was on a good day. It usually connected at 26.4Kbps. Either way, the Netscape download still took well over 2 hours.
Since that day, I've seen Linux get better, with:
Two years later, I had done so much with Linux (even ran a web server through dial-up, just because I could), that Mom asked me to build her a Linux system for Christmas. I actually tried to talk her out of it, but when she pointed out the support would be easier with fewer crashes and BSOD's than Windows, well, she convinced me. I put some old parts together, put Mandrake Linux 6 (now Mandriva) on it, and taught her as much Linux as I could during Christmas vacation. She's gone through a few hardware upgrades, and switched to Fedora 8, but she's still using Linux.
Two weeks ago today, I set Dad up with a Linux system. Mom had turned off his laptop to clean it, not knowing that the shutdown would be the laptop's last. So I donated my IBM Thinkpad 660X to the cause, setting it up to boot automatically into XFCE and launch Gnome's Aisleriot solitaire program. It's the only thing he does on the laptop, but for the man who made sure I had food and clothes growing up, I'm glad to help provide his entertainment. And I'm happy to use Linux to do it.
My brother is the only one not using Linux, because his cell phone model is explicitly not supported for anything except Windows and MacOS. Still, he tries to be careful about his web surfing, so he uses the Opera web browser and the Privoxy web firewall.
Actually, it's "Lisp hacker," thanks to funky English phonetics. And I'm not really a hacker in Lisp, but more of a script kiddie (def. 2).
My parents recently got two replacement computers, Mom on her desktop and Dad for his Solitaire machine. Both are running Linux with the GNOME desktop. However, Mom's system is Fedora 8, while Dad's is Slackware 12.0. They both like to play the Klondike version of Solitaire, but the rules weren't to their liking. Specifically, they wanted one-card deals and no limits on re-deals.
The Solitaire rules for AisleRiot are stored as Scheme files in the directory $SHARE/share/gnome-games/aisleriot/games/klondike.scm. Well, Scheme is a dialect of Lisp, and I have a couple books on Lisp...
Their games now work as desired, but the fixes were different between the two machines. Having a modicum of exposure to Lisp made it a lot easier.
Why should patent protections not be available for computer software?
At its core (pun intended), computer memory is just a bunch of transistor-switches. Imagine needing permission from a patent holder, just to turn on all the lights in the building, or to turn lights on and off to make the letters "L O V E" on a skyscraper. Really, it's accomplished with just switches. There may be a patent on the switches themselves, but you are allowed to configure them for any effect you desire.
So it should be with software. The hardware probably has a thousand patents covering its various inventions, but what code and data you load into it (setting the transistor-switches) should be up to you, as well as any resulting effects. If those "resulting effects" includes playing a DVD, then so be it, but the term can be applied to just about any computer program.
(Again, thanks to "the Batman" for spurring this line of thought.)
Slashdot's RSS feed has a couple headlines that are just ripe for a little Microsoft-bashing:
Microsoft Says Vista Has the Fewest Flaws
(I suppose 100,000 bugs instead of 200,000 bugs is a "good thing" in some circles.)
Bill Gates Calls for a 'Kinder Capitalism'
(Kinder to himself.)
Bah. Too easy.
Update 2008-01-26: A response at C|Net to the latter.
Being the geek that I am, I like to challenge myself with new ideas. A few weeks ago, I decided that my latest challenge should be to create a Christmas slideshow. I could put up a few strings of flashy lights, maybe even hang a wreath, but why should I be like my neighbors? I have the power of Linux and Free Software at my fingertips; I should show it to them, right?
My requirements for the project were simple:
The first challenge was to decide on the hardware setup. I have two primary systems: my powerful desktop, my much weaker system for experiments. My desktop system is nowhere near a window, and I didn't feel like moving my entire computer desk. I'm not sure the weaker system is capable of running a slideshow (yes, it really is that weak). I don't have a VGA extension cable, but I do have lots of Cat5 for Ethernet. Ah, the distributed X protocol to the rescue! My desktop runs the slideshow, and the experimental system runs the X server. Already having a serial console on the experimental system is an additional benefit, because I can keep a text login for emergency rescues.
The second challenge was to find properly-licensed images suitable for viewing from the street. I leave this as an "exercise for the readers," who might want pictures for Hanukkah, Sol Invictus, Kwanzaa, or even simply "Happy Birthday." My only word of advice: try not to be too abstract or conceptual. Remember, this really is a show for the "man on the street"; as my geeky retired mother said, Nativity scenes should not look like pickles.
Finally, it was time to make it all work together. To meet requirement #1, I had to make sure the script was suitable for running from /etc/inittab. Bash scripting is my language of choice, but security isn't really part of the language (hence requirement #2). I had initially decided to use OpenOffice.org Impress, but it proved to have difficulty with long-term stability, so I settled on using the GLslideshow module from the XScreensaver collection. The GLslideshow setup also uses less memory.
I wanted to make the setup reasonably portable, so I decided to keep the configuration in environment variables (as many as I could think of). These are the local user for the X server; the remote user and hostname for the GLslideshow client; and a local, writable directory for state management.
The X forwarding capability of OpenSSH makes it the preferred channel for connecting the X server and the GLslideshow client. My desktop system connects directly to the Internet via dial-out, so a mis-routed packet needs to be shielded from snooping. The OpenSSH project provides the ssh-keygen program for automating SSH connections, without requiring a password. On the experimental system running the X server, I logged in as myself and ran ssh-keygen, giving no password. The generated key allows unattended scripts to use SSH sessions.
On the desktop system, I created a new user called "slideshow", with only a home directory, but none of the skeleton files. On my Slackware system, this meant deleting the standard skeleton files after the fact, leaving only .ssh:
useradd -g users slideshow
cd /home/slideshow
rm -rf * .[a-z]* # Be very careful NOT to use ".*"!
mkdir .ssh
Then I copied the .ssh/id_rsa.pub file from the experimental system to the desktop, for password-less SSH connections:
cd .ssh
scp me@remote:.ssh/id_rsa.pub authorized_keys
chmod 600 authorized_keys
chown slideshow:users authorized_keys
I confirmed this was correct, by opening an SSH session from the remote system to the local system, without requiring a password.
The GLslideshow program needed to know where to find the images on my desktop system. In a terminal window, I typed:
xhost +localhost # A different user needs this for X access
su - slideshow
[password]
xscreensaver-demo
Then I clicked on the Advanced tab, selected "Choose Random Image" in the Image Manipulation block, and entered "/home/slideshow/wallpapers" as the directory containing the images. Closing the window saved the parameters.
Next, I created a script on the experimental system, to check the network connection and start the slideshow, if the time is between 5:00PM and 7:00AM. Okay, the script is actually the result of approximately 17 iterations. A few notes:
With this script in /etc/rc.d, I made the following adjustments to /etc/inittab:
With that, I logged in as root and typed "telinit 4" to see if it would work. Voila! Every 20 seconds, a different image appeared.
Bearing in mind the networked nature of the script, I tested its recovery capability by rebooting my desktop. Sure enough, it restarted just fine. In fact, the slideshow was running on the experimental system before my X login screen appeared on my desktop.
To make it fully automatic, I changed the default runlevel in the experimental system's /etc/inittab to 4:
id:4:initdefault:
This is Slackware's default X login runlevel; some (most?) Linux distros use runlevel 5 instead. A reboot of the experimental system made sure the slideshow script would survive a reboot.
Once everything was working together correctly, I put the monitor in the window, so that my neighbors and people walking and driving by could see it.
My experiment has been a personal success, bringing together my skills in Bash scripting, documentation chasing, and the old "scratch an itch" feeling.
Using just free software and a simple bit of shell scripting, I have created a holiday slideshow for my bedroom window. I have proposed an article on this project to Linux.com, but no response yet. When I find a publisher, I'll also post the article here.
Update 2007-12-22: It's over here.
Recently, someone posted a YouTube video showing what happens when the super-user issues the dreaded "rm -rf /" command on a Ubuntu system. I was describing it to my mother, who has lived with my geekiness for 35 years:
Me: "Do you know what happens when you delete all the files on a Linux system?"
My Geeky Retired Mom: "You don't get a C: prompt, do you?"
Haw haw haw! Good one, Mom!
The Swiss government has demonstrated the same appalling incompetence regarding voting security as the United States: Swiss votes to use 'unbreakable' code
A new "unbreakable" encryption method will be keep votes safe for citizens in the Swiss canton (state) of Geneva in the country's upcoming national elections, officials said Thursday.
The city-state will use quantum technology to encrypt election results as they are sent to the capital on Oct. 21...
What good will a very secure network be, when the physical system itself isn't awarded the same protections? How are the votes protected before transmission, while they are stored on the machine?
Maybe they have taken measures to keep the systems physically secure, but the quantum-encryption panacaea won't get them any security that a very large key in OpenSSL can't get them. By the time the key is cracked, another election cycle will be rolling around.
The first major OS war was between VMS and Unix. VMS fans berated the absence of certain "features" in Unix, such as record-level locking, while Unix enthusiasts considered VMS an over-wrought monstrosity. (I've seen the manual set for VMS, and the idea is not unfounded.)
Over time, VMS became relegated to legacy systems and high-end corporate clusters, and AT&T sold off the Unix rights. The NeXT system, developed by erstwhile Apple CEO Steve Jobs, tried to get Unix into the offices and dorm rooms of individuals.
As the market space for VMS became saturated, developers left DEC for Microsoft, providing strong influence for the Windows NT operating system. (See the 100ns timestamp resolution and the alternate file stream operations? Those came from VMS.)
When Steve Jobs returned to Apple, and brought NeXT with him, it was only a matter of time until the Macintosh ran Unix. Mac OS X shows the profound influence of Unix in its low-level command structure, and the influence of NeXT in the user interface. (See the object system with class names starting with NS- ? That's NextStep in there.)
When Windows XP brought the NT kernel to the desktop, and Macintosh brought Unix to the desktop, it became just another stage in an old competition: VMS vs. Unix.
On a current Slashdot discussion appears the brazen comment:
Why do "security experts" like these folks always suggest using nmap to determine what services you are running? Have these folks never heard of netstat?
Sigh. A little knowledge is dangerous, but even a little knowledge is totally lacking in this comment.
It all comes down to trust. In the case of a compromised system, trust is a dangerous thing. Many worms and trojans, in order to hide themselves, will replace the common administration commands (like ps, top, and netstat) with special versions which hide the compromising process. Running ps aux or netstat -utlnp may reveal nothing; only delving into the /proc directory is guaranteed to show what programs are actually running:
cd /proc
for i in */exe ; do
echo $i $(readlink $i)
done
Why use readlink in a loop, instead of simply running ls -l /proc/*/exe? Because ls is one of the most commonly used commands in Unix, it is the primary target for substitution.
If you want to know the plainest network status of a system, your best option is to run NMap from another system. This will maximize the chance of catching the worm or trojan in action. Since NMap does its thing through the network, rather than on the device under test, it allows the administrator to run a relatively more secure OS (such as Ubuntu Linux or OpenBSD) directly on the network with the affected device.
Once the ports in question are identified, the next best step is to capture the network traffic for analysis, using either Wireshark (formerly Ethereal) or the slightly less useful tcpdump. During the capture, it will be a good idea to look on the Web for information regarding the TCP or UDP port(s) and the running worm/trojan.
If you're relying on netstat to tell you what's running on a possibly compromised system, good luck. You aren't yet thinking like the bad guys, and luck is all you've got.
After four years of courtroom maneuvering, and too many attempts by The SCO Group to sneak in "evidence" and "motions" after stated deadlines, they have finally lost the keystone of their lawsuit against Novell:
[T]he court concludes that Novell is the owner of the UNIX and UnixWare Copyrights.
The case isn't done; the lawyers will assure that. But TSG had no leg to stand on, and the court finally affirmed it.
An implicit loser in the case is Microsoft, as the primary funding source for the lawsuit.
It's difficult to come up with two more deserving parties.
Granted, I'm no friend of Microsoft, but once in a while they hit one out of the ballpark. Their Disagreeably Facetious Type Glossary is an example of them "getting one right."
Oh, and for the record: I use a Microsoft Natural Elite Keyboard. It boosts my typing speed by about 20 WPM. I have one for home, and I keep an extra one (with my name on it) for whatever IT job I may have.
The Free Software Foundation's new General Public License version 3 is out, and some people are already sniping at it.
The root question of the various software licenses is that of liberty. The BSD license stipulates only that the user gets neither warranty nor endorsement from the University of California. Other than that, anything goes.
The focus of the GPL, on the other hand, is focused on permission to use the program, including modifying it to make it more fit for a particular circumstance.
The BSD license grants liberty to the developer; the GPL grants liberty to the user.
To enforce this, the GPLv2 stipulates in section 4 that the freedom to distribute GPL-covered programs is revoked, if the distributor attempts to deny a user the ability to modify and use the code.
Some distributors have gotten around this by using the device itself, in some evil combination of DRM and TCP, to render itself essentially "comatose" if the user attempts to tinker with it improperly (and the distributor gets to define "improperly"). The TiVo is the example usually cited.
The GPLv3 was designed to address this concern, by adding software patents and DRM to the list of "forbidden user restrictions." Thus, anyone using a GPLv3-covered program can use it anywhere, any time, for any purpose (including watching an episode of House, MD that you recorded three years ago). No BSD license makes this guarantee.
Jem Matzan is foolish for discounting the GPLv3 out-of-hand. It merely broadens and clarifies the intent of the General Public License, an action reserved to the Free Software Foundation in every version of the GPL. And it doesn't indicate "GNU's decline," but rather the maturation and affirmation of a user's freedom to use devices he owns, as he wants.
Addendum 2007-07-05: An excellent article by Tim B. Lee (no, not Sir Tim Berners-Lee) from the Cato Institute, defending the freedom of Free Software.
A core Java source package is now released as open source: Java 2 Mobile Edition (J2ME), now called "phoneME."
Sun Microsystems is releasing the Java sources under the GPL, marking a significant departure from their use of the Sun CDDL. As geeky Christmas presents go, it would be difficult to surpass this one.
One down, two to go: I am going back to Slackware Linux. The laptop is running it now. I'll install the server system next, and then use it as a temporary holding ground for files from my desktop system while I convert it over. (Update 4:54pm EDT: I'm skipping the server system for now. I need a floppy drive for network-based installs, and it has none.) I'll also be able to use the server system itself for hosting the installation files via NFS.
Using Gentoo Linux was fun; it showed that building an entire system's software from source is possible. It also showed just how much CPU power and Internet speed are needed to do this.
Fedora Linux may be popular, but its managers are not focused enough on security before release.
What's more, the Slackware installer can get you a totally usable system, with KDE, from one 650MB CD. That's right, one CD. If you need international texts for KDE, the packages are available on a second CD, or you can download individual language packs from a Slackware mirror.
"After a Veterans' Affairs employee improperly brought the material home." This phrase summarizes the #1 threat to personal data online: human recklessness. Not ignorance, as with any number of CGI and SQL injection exploits. Just simple, stupid recklessness with other people's information.
Veterans' Administration Secretary Jim Nicholson exposed his ignorance of the severity of the breach with this statement:
I want to emphasize there was no medical records of any veteran and no financial information of any veteran that's been compromised.
This is simply not true. According to the AP article in front of me, the information compromised was for veterans who were discharged after 1975. The military has used Social Security numbers for soldier identification since the 1960's. It is a crucial ingredient in any attempt to steal a person's entire identity, for bank loans and credit cards, clean driving records, even forged documents for illegal aliens. As long as the Internal Revenue Service requires the SSN for its non-corporate tax forms, the SSN is financial information.
I am also waiting to find out how the "midlevel data analyst" was able to obtain unregulated access to production information. During project development, database administrators are supposed to supply bogus data. In this case, the developer should be using names like "Corporal Mickey Mouse" and "Sergeant Woody Woodpecker" with invalid SSN's. Any access to production data should be tightly controlled; the DBA clearly shares some of the blame for allowing unfettered access to so much data during project development. As this incident shows, all the security in the world does no good, when incompetents are guarding the gold mine.
However, the bulk of the blame, and any subsequent punishment, belongs to the "analyst" who so obviously has no clue about data flow. This idiot should never again enjoy our liberties, the trust of whose defenders was so carelessly betrayed.
As I type this, my system needs over 700 megabytes of updates. I can't download them all at once, so I am applying them in smaller chunks of packages and their dependencies. I didn't let it slip that bad. I only installed Fedora Core 5. Well, tried to install.
Fedora Core 5 has possibly the buggiest Linux installation I have ever used. I believe the problems stem from the crucial media verification. I verified the recorded CD's on two different computers, and all reported "PASSED." However, the actual installation process reported something much different. On my laptop, it reported an error unpacking the "xsri" package on the first CD. I rebooted and re-verified the disc, and it reported "PASSED." So I tried the installation again. The installer then reported an error on a different package, after successfully installing the "xsri" package.
What the defrag is going on here?
I tried everything I could think of: disabling DMA, using 16-bit I/O, using the text-mode installer (memory crunch from the GUI?), and nothing seemed to do the trick. Time after time, a package failed. A couple times, the installer itself crashed, giving me a backtrace on the screen. In that case, my only option was to exit and reboot. I gave up the laptop installation.
I was met with similar frustrations on my desktop. The only approach that worked was to use my USB CD-R drive as the source. Surprisingly, using this drive eliminated the errors. (I say "surprisingly" because this drive has never worked as a recorder for me.) This successful approach exposed another problem: Damaged packages from earlier attempts (such as "xsri") no longer appeared as "installed," so their packages were skipped. The net result of all this is an incomplete installation.
I have a usable Fedora Core 5 desktop system now. However, the frustration of the installation is giving me serious pause. I am unable to do a complete update in one step, partly because of my slow internet connection, partly because of a missing dependency for "bind-config."
The laptop is a different, not better, story. I decided to go for broke, telling the installer to start from scratch with a "Typical Desktop" setup. It wiped the partition table, re-partitioned and re-formatted, and then crashed on the second CD, first with a "damaged package" error, then a Python execution error in the Anaconda installer.
The laptop is now running Knoppix.
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. (Tony Hoare)
In an earlier article, I praised delayed library binding as a way to relieve memory pressure. Now, I will show how accessing data sooner, rather than later, can make a program run faster. The example I use is the famous Conway's Game of Life.
The traditional rules are fairly simple. The game is played on a large two-dimensional grid of "cells," which are either "alive" or "dead." The cells' status for the next generation is determined using a set of rules, usually the following:
A "neighbor" is defined as any cell one unit away on either or both axes. For example:
| X | X | |
| X | X | |
| X |
The center cell above has four living neighbors, and will die of "overcrowding" in the next generation. Corner cells have three neighbors, edge cells have five neighbors, and all remaining cells have eight neighbors.
Altering either the rules or the starting conditions creates a different "environment" for this simulated life. This formed the basis of cellular automata theory.
The implementation traditionally follows a simple two-stage approach:
This is sufficient for teaching programming techniques, but it doesn't take into account modern CPU design. The net result of this approach is that each cell's in-memory data is fetched at least twice, possibly more, into the data cache (Dcache): once for neighbor counting, again for determining life or death in the next generation. Data fetched at the beginning of the first stage will be written back to memory and ejected from cache before the scan finishes, in order to make room for more data. When the second stage begins, cell data must be fetched again
Every fetch into Dcache can slow down program execution. Conway's Game of Life, on a large grid, runs amazingly slow. On my Linux-based 2.0 GHz AMD Athlon, with 768 megs of memory, a 1,000x1,000 grid will use no swap, but still requires 81.54 CPU seconds to run 1,000 generations.
One way to reduce Dcache fetches is to act on the data as soon as all dependencies are satisfied. In order to expose the data dependencies, we can combine both stages into a single scan through the cell array. As we scan, we do two things:
The "last neighbor" is determined by the row and column scanning order. In a left-to-right, bottom-to-top approach, a cell's last neighbor is above and to the right. That is, after checking a cell at (X,Y), we can decide if the cell at (X-1,Y-1) will be alive or dead on the next generation. The effect on execution speed is tremendous. For large areas of dead cells, only one neighbor must be fetched into Dcache. If the system's Dcache will hold three rows of cell data, then writeback and ejection of each cell occurs only once per iteration. The increased efficiency becomes even more obvious when cell data must be fetched from swap space on disk.
This new approach creates very real "edge cases" and "corner cases." In order to simplify the scanning, I added a one-cell buffer around the whole array, so that all active cells would have the full complement of neighbor cells. This means, for an N by N grid, the buffer is in rows 0 and N+1, and in columns 0 and N+1. Iteration starts at (1,1) and ends at (N,N). These buffer cells will never become alive. For row 1 and column 1, the second part of the analysis involves only resetting the neighbor count to 0. Column N can be handled at the end of each row's analysis. A separate loop is necessary to guarantee that row N will be properly analyzed. The pseudo-code looks something like this:
For each row x, from 1 to N:
For each cell in column y, from 1 to N:
If cell(x,y) is alive,
Add 1 to all its neighbors' "living neighbor" counts.
If x > 1 and y > 1,
Determine if cell(x-1, y-1) should be alive or dead.
Set "living neighbor" count for cell(x-1, y-1) to 0.
Repeat last 2 steps in last cell of the row:
Determine if cell(x-1, N) should be alive or dead.
Set "living neighbor" count for cell(x-1, N) to 0.
Again, repeat last 2 steps in the last row:
For each cell in column y, from 1 to N:
Determine if cell (N,y-1) should be alive or dead.
Set "living neighbor" count for cells (N, y-1) and (N+1, y-1) to 0.
Last cell is always border, ergo empty:
Set "living neighbor" count for cell(N+1,N+1) to 0.
This looks like more work, and for the programmer, it is. The performance benefits far outweigh the cost of the work; on my system, the memory fetches are reduced by 50%, and CPU time is reduced even more. The same simulation that took 81.54 seconds, is now reduced to 33.65 seconds!
"Premature optimization" may be evil, but serious design consideration to the flow of data through a program's execution will show great benefit later on.
The incredible Sawfish (né Sawmill) window manager is back, after almost three years of neglect. John Harper has incorporated some of the fixes sent his way, so that librep and rep-gtk work better with newer versions of Gnome.
Note: The "official" SourceForge releases of librep, rep-gtk, and Sawfish are not updated. The current code is in the Gnome CVS. Here are the steps I did for a successful build on Fedora Core 4:
$ mkdir sawfish-build ; cd sawfish-build
$ cvs -d:pserver:anonymous@anoncvs.gnome.org:/cvs/gnome co login
$ cvs -z3 -d:pserver:anonymous@anoncvs.gnome.org:/cvs/gnome \
co librep
$ cvs -z3 -d:pserver:anonymous@anoncvs.gnome.org:/cvs/gnome \
co rep-gtk
$ cvs -z3 -d:pserver:anonymous@anoncvs.gnome.org:/cvs/gnome \
co sawfish
And then, in each of the new directories:
$ ./autogen.sh --prefix=/usr
$ make all check
$ su -
# cd <back to the build dir>
# make install && ldconfig
It is not a perfect build process. All three packages complained loudly about mixing signed and unsigned integer types. Still, I got no show-stopping errors, so I started using it. It has crashed only once, running a bit of rep-gtk test code from Bugzilla. I'd say that's "good enough" for my purposes.
I knew I'd want to keep all my Sawfish themes and configuration files. I've been stuck with Metacity for the past two years, and it feels pretty good to right-click a window title bar and send it to the bottom of the window stack.
I used to think the outrageous news about stupid government mostly came from California. Over the past few years, I've noticed that more and more of it seems to be coming from Florida.
The latest juicy bit (pun intended) comes from Broward County, home of Fort Lauderdale. The unexpurgated public records before 2002 are available for general perusal. That includes anything already in the records, including "Social Security numbers, driver's license information and bank account details."
Great, so Castro can go fishing for the people he wants dead. And he can do it from the comfort of his own mansion!
Get a load of this official "excuse" for such recklessness, from the director of the Records Division, Sue Baldwin:
Recorders have no statutory authority to automatically remove Social Security, bank account and driver's license numbers.
That's beside the point. Those records have no business being online for viewing by every petty tyrant and identity thief from all over the world.
This is, without a doubt, one of the biggest failures in recent memory of a government body to serve the public interest, surpassed (so far) only by the sanctioned murder of Terri Schindler. Someone needs to file a lawsuit, soon, before Broward citizens get into massive financial and legal trouble.
In this case, the "light" is the EMF of the Internet.
Guillermo Fariñas has been on a hunger strike for 57 days and counting, to highlight the outrageous suppression of free communication in Cuba. In the Reporters Without Borders Worldwide Press Freedom Index 2005, Cuba ranked #161 out of 167 nations, for precisely the reasons Fariñas wants us to know.
Are you shocked? Here is another shocker for you:
The Google search for "guillermo farinas" has 7 of the top 10 links going to blogs. The identical Yahoo search has 9 of the top 10 links going to blogs.
None of the top 10 links on either search will take you to a major news outlet. Not CNN.com. Not MSNBC.com. Not the New York Times. Not FoxNews.com. (That's surprising.) Even the special "News" searches don't show any attention from the media powerhouses, whose livelihoods depend on the very freedoms Guillermo Fariñas seeks for himself and his fellow Cubans.
This is the legacy of Walter Cronkite, Tom Brokaw, Dan Rather, and Peter Jennings: Once again, the "accredited journalists" can't be bothered. They're too busy trying to tear down George W. Bush.
Shame on them.
In the meantime, the blogs are getting the real story out. Muchas gracias, Sr. Babalu.
¡Abajo Fidel!
I have used my Motorola V262 with Alltel service to get online with my Fedora Core 4 laptop. The setup is very simple, with administrative (root) privileges.
The V262 data cable connects to the USB port, so the device file will be visible as /dev/ttyACM0, handled by the "cdc_acm" kernel module. A Fedora Core 4 system configured with "udev" will take care of this automatically when you plug in the data cable. FC4 provides the necessary tools in GNOME 2.10 to complete the configuration.
Under the Desktop main menu, select "System Settings," then "Network." If you are not logged in as root, you will need the root password to run this tool.
In the Network dialog, click the "Hardware" tab, then click the "New" tool button. Select "Modem" as the hardware type, then click "OK." Use the following settings for the modem:
These are the device settings. To configure the ISP settings, click on the "Devices" tab, then click the "New" tool button. Select the "Modem connection" device type, then click "Forward. I used the following settings to connect to the internet using my Alltel service:
After clicking "Forward," choose "Automatically obtain IP address settings" and check "Automatically obtain DNS information from provider." After clicking "Forward," you should see a confirmation page. If you are satisfied that the displayed settings are correct, click "Apply" to create the connection.
Once the connection is available, you can activate it using the Network Configuration window, by selecting the new connection and clicking the "Activate" tool button. If all goes well, the device's status should change to "Active." If not, check the system log for the reason behind the failure.
When you have a working connection, you can adjust other properties by selecting the device, then clicking the "Edit" tool button. I have set the following:
However, I did not see that any of the Compression options changed the options passed to "pppd" as indicated in the "ps -ax" output. I hand-edited the /etc/sysconfig/network-scripts/ifup-ppp to add the "deflate 10" and "bsdcomp 10" options to the pppd command.
If you set the "Allow all users to enable..." option, then a non-root user can run the /sbin/ifup and /sbin/ifdown scripts, to connect and disconnect. I have a simple script in my home directory, which I can run via a single click in my Gkrellm2 monitor:
#!/bin/sh
ps ax | grep pppd.*alltel | grep -q -v grep
if [ $? -eq 0 ] ; then
/sbin/ifdown alltel
else
/sbin/ifup alltel
fi
exit $?
The only complaint I have about the Alltel internet access is that the TCP SYN/SYNACK phase tends to be slow. Individual TCP connections may take a second or two to initiate. Once established, the 17-20 Kbyte bandwidth is excellent. Web pages with a lot of auxiliary content (images and applets/Flash) can have their overall download speed hurt by the SYN/SYNACK problem; the Tweak Network extension for Firefox can help with this, by reducing the number of TCP connections required per web page.
(Update 2006-03-03: Further analysis shows that the delay is between the HTTP request and response. I suspect some mandatory proxying of port 80 transfers.)
(Update 2008-01-22: The same drivers and configuration will work with the Alltel Hue, aka the Samsung R500.)
If you like your computer animations to include social commentary, check out Cubic Tragedy. At 66 megs for the WMV file (that's a "short"?), you'll need a fast Internet connection to get it before summer arrives.
H/T: Overcaffeinated.
Many computer programs are built using one of two function-binding techniques:
I would like to propose a simple third way: call-time binding, in which a code library is not loaded until it is needed. The immediate benefit is keeping the RSS reasonably low for a process. The trade-off is that some support code is needed, which can actually increase the RSS for a complicated program.
I have included a rudimentary proof-of-concept code sample, to demonstrate an actual implementation.
The philosophy is simple: Don't load code until it's needed. Using dlopen() and dlsym() judiciously means that unnecessary code is never loaded.
First, start with some real functions. Here is add2() in real2.c:
#include <stdio.h>
int add2(int a) {
printf("Now in real add2(a)\n");
printf("Passed a = %d\n", a);
printf("Now exiting real fn(a), returning %d\n", a+2);
return a+2;
}
Next, add a function pointer in fn.h:
extern int (*add2)(int);
#define add2(a) (*add2)(a)
The add2_stub() function in stub.c is linked at build-time, but it will be called only once:
#include <dlfcn.h>
#include <stdio.h>
#include "fn.h"
/* __attribute__((constructor)) means it runs when this library is loaded,
at launch time */
void prep(void) __attribute__((constructor));
int (*add2)(int a);
int add2_stub(int a) {
void *h;
h = dlopen("./real2.so", RTLD_LAZY);
if (!h) {
fprintf(stderr, "%s\n", dlerror());
exit(1);
}
/* do the lookup */
dlerror(); /* clear existing errors */
add2 = (int (*)(int))dlsym(h, "add2");
/* return the real result */
return (*add2)(a);
}
/* the following must appear after all stub functions */
void prep(void) {
add2 = add2_stub;
}
The stub library will be loaded at launch time. Marking the prep() function means it will be called when the library is loaded, causing the (*add2) pointer to point to add2_stub. Remember, in fn.h, the code "add2(x)" translates to "(*add2)(x)", which at launch time will call add2_stub(x).
However, upon calling add2_stub(x), the library with the "real" add2() is loaded, and (*add2) is set to point to the actual, desired code. Later calls to (*add2) go directly to the loaded library's add2() immediately.
The main executable includes the simple function call:
add2(5)
which, the first time, calls the stub function, which loads the library, re-points the pointer, and then invokes the real add2(5). The second time and later executing the above function call results in a simple call to the real add2(5).
Finally, the Makefile to tie it all together:
main: main.o stub.o real.so real2.so
gcc -g -o main -ldl main.o stub.o
clean:
rm -f main.o stub.o real.so real2.so main
main.o: main.c
gcc -g -o main.o -c main.c
stub.o: stub.c
gcc -g -o stub.o -c stub.c
real.so: real.c
gcc -g -o real.so -shared real.c
real2.so: real.c
gcc -g -o real2.so -shared real2.c
The leading whitespace is a tab character, not spaces. Two notes:
The code sample contains a slightly more complicated demonstration, involving two separate functions in two different libraries. User input causes the two libraries to be loaded in different order.
Simple experiments on Linux show that the libraries are mapped using shared pages, even for different processes loading them in different order. Just as with run-time binding, a library mapped to multiple processes won't result in significantly higher memory pressure on the system.
The two advantages should be obvious. If no running program needs to do complex number math, why load the code to do it? Conversely, if a program does need code for complex math, there is no need to load an entire monster library just to get that. And, when the last program needing it exits, the code will be unmapped, reducing the memory pressure.
No, not for the reasons you might think.
The New York Times is collecting information about its web readers. If the NSA was wrong to use cookies on its website, then the Times' tracking borders on felonious.
The Times wants this much information, just to be allowed to read their articles:
This is all required. For all they know, you could be lying through your keyboard, but they still require so much information.
Once you get past the registration, the Times and its affiliates will track your activities with the following:
As a security measure, most web browsers can pop up alert boxes when a website sets a cookie. I do not know of any browsers that provide a similar measure for web bugs.
For all the ruckus the Times raised about the NSA and communication surveillance, it seems "odd" that they do so much of their own tracking.
I just noticed one of the coolest log messages ever:
Clock: inserting leap second 23:59:60 UTC
I didn't have to do anything special to make that happen. It just... did.
This patent application shows Amazon hasn't really started playing nice. They're trying to patent the Slashdot algorithm, which has been in development for over 7 years.
I've pulled my reading list from the main page, since it uses data from Amazon. I won't send them traffic. Who's with me?
Q: How many reboots does it take to get a working Sun Cluster?
A: We might be able to tell you in a couple months.
I've found an OS that I hate even more than Microsoft Window$.
I Hate Solaris.
It has a slow installer. It runs slowly, even after it's installed. Its package management documentation doesn't apply to software on the installation CD's. System resources can't be re-configured without a reboot. Criminy, even Microsoft figured out that people are tired of rebooting!
I hope the OS group at Sun Microsystems got a clue before they designed Solaris 10. Although, judging by their propaganda site, they haven't. They seem to think that a new logo will make the OS better.
Please, somebody put them out of my misery.
(Cross-posted from Slashdot: "Why OpenOffice.org? Open Document Formats.")
Boss wanted me to create a PostScript version of our corporate logo, so it could be scaled as needed.
Source: a poorly rendered GIF.
Equipment: one Linux machine, with OpenOffice.org installed.
I found the matching font, got the dots lined up, converted it to a traced object, found the right "burnt sienna" color... but that pukey-green was nowhere in any color selector I could find.
After hunting for nearly a half hour, for an edit box that would let me enter an arbitrary hex triplet, I just saved the file and quit OOo. Then I unzipped the document, opened the style sheet in NEdit, and changed the hex triplets by hand. Save, exit, re-zip, and open it in OOo to see if the changes were correct. Voila!
I never, never ever would have been able to do that in a Microsoft product. I will grant that Microsoft may have made the hex triplet entry somewhat more obvious, but that doesn't mean I would have been able to find it any more easily. They absolutely control how the user accesses the document. OOo lets you access it any way you want.
Matt Drudge has just given an entire segment of his radio program to Internet security, with special mentions to the Firefox web browser.
Wow. I use Firefox at home, and enjoy the thin, fast, and secure browsing. No pop-ups, nothing hidden, and incredibly customizable.
Thanks, Drudge. We open-source advocates couldn't buy the publicity you just gave us.
I already have 15 pounds of swag, and I've been here only two hours. My feet are killing me, and I haven't had this much fun since BayCon. I've taken only one picture... but who needs photos when you have swag?
I did get to meet the inventor of Project Looking Glass. A very humble fellow, with the kind of humility that's good to see in project leadership.
Cell phone is turned off. I'm glad for that.
Warrants as events update.
I just upgraded to Mozilla 1.6. They really cleaned up the Gecko page renderer, including the table renderer on the calendar. I'm impressed, to say the least. Way to go, guys!
I finally got OProfile working. It profiles the system, using either a timer-based interrupt or the performance counters in the CPU itself. Cool!
Note to Pentium-class and Alpha users: If you want to use the CPU's performance counters, you need to configure your kernel to use the APIC (under "Processor type and features"). IA64 users can use performance counters under the 2.4 kernel only.