Jared Robinson’s Blog

I upgraded one of my machines to Fedora 10 last month, and for me, this release lacks the “wow” appeal that other releases have had. A minor annoyance is that the keyboard repeat delay is broken for me and so far, there is no fix other than disabling keyboard repeat. On the plus side, Fedora 10 includes OpenOffice.org 3 and other new features.

The other day, a co-worker handed me an OpenSolaris 11 Live CD. I booted it, expecting to be underwhelmed like I was with the Solaris 10 JavaDesktop. I was pleasantly surprised, however. Sun’s “Nimbus” GNOME theme knocks the socks off of the boring Fedora window manager themes. The experience felt like I was running Linux. It was responsive, supported my newer hardware, and the system was built with GNU utilities on the command line so I get my favourite options to ‘ls’, ‘grep’, etc. It supported my NVidia card out-of-the box, and had Compiz eye-candy as an option. The only thing I missed (in my superficial test) was the familiar ‘yum’ and ‘rpm’ for package management. I suspect that if I used it from day to day, I’d find other things I miss. Does OpenSolaris support encrypted file systems? Does it have as much optional software as I can get with Fedora Extras?

I’ll keep my eye on OpenSolaris a little more closely in the future.

Preserving the availability of digital artifacts is a goal worthy of pursuit. First, I’ve got thousands of digital family photos, and I don’t want to lose them to hard drive failure or lock them up so that they’re hard to get to. Second, I’ve got my email stored on my computer for the past several years. The recent email is what’s most valuable to me, but every once in a while, I need to search through email archives to find things, like a license key for Date Book 5. Third, it took weeks to install software and configure our laptop. I don’t want to have to repeat that work if the hard drive happens to stop working — especially if a project I’m working on needs to be done soon.

There are three main types of backup that are important to me: Immediate backup, full backup and long-term archival.

Immediate Backup

What I’m currently working on with the computer is usually more important than what I was doing on the computer a few weeks ago. The auto-save and even the “undo” feature of most word processing programs can help me when one of my children touches the keyboard and accidentally deletes most of the text. Auto-save and undo won’t help if my laptop is stolen or the hard drive fails. That’s why I use mozy.com for automated, off-site backups of my Windows laptop. It’s well worth $5.00 per month for this service, and it’s easy to pay for: skip eating out for lunch once per month.

For backup to happen regularly, automation is key — especially for immediate backup. I would make full backups more frequently if it were an automated process. I use a monthly repeating reminder so I remember to backup the things that aren’t automated.

Full Backup

Admittedly, hard drives don’t fail often, and laptops that usually stay at home aren’t often stolen (at least, not in my neighborhood). But when it does happen, it’s a pain to reinstall the myriad of applications we use on a semi-regular basis. This is why a periodic, full backup is valuable. Doing a full backup with optical media takes too much time. External hard drives are much faster, have more capacity, and are inexpensive. They plug in using standard connectors such as USB, FireWire or eSATA. I store my external USB hard drive in a fireproof box.

Long Term Archival

I want the best of my digital memories (e.g. photos) to be preserved for decades or centuries. A CDROM may be readable in ten or twenty years, but not fifty or a hundred. There probably won’t be hardware to read it in fifty years. Will computers in fifty years recognize JPEG format? No idea!

To preserve digital artifacts for that long requires refreshing it periodically into newer formats and storage media. It’s a good idea to use open, standardized formats rather than proprietary formats. For photos, this means to use JPEG and PNG in preference to Photoshop format.

Rather than refresh constantly, there’s the option of printing photos and documents. It’s going to be easier to view a physical photo or a printed document in a hundred years than to unlock the secrets of an old hard drive.

Trust, but verify

I tend to trust my backup solutions, but it’s necessary to verify that they’re working. My brother’s computer periodically downloads my digital photos. I trusted that this was, at least in part, a good off-site backup. I learned recently, however, that his computer deletes old photos when space gets low, which is often.

Resources

Preserving Your Digital Memories: What you can do

A few backup solutions: Mozy, Carbonite, SyncBackSE, and JungleDisk.

Interesting projects to backup using P2P protocols (including featurs such as encryption and fault tolerance): Tahoe with a writeup from LWN and Flud.

Reading the linux kernel documentation on KGDB wasn’t enough for me to be able to use the newly built-in KGDB kernel debugger with version 2.6.26 or 2.6.27. The breakthrough for me was reading part of Jason Wessel’s guide.

I have two machines:

• developer - where I run gdb
• target - where the kernel is being debugged, running in VMware

Configure VMware on the developer machine

• Power down the guest (target)
• Edit the VM guest settings
• Add a serial port
  • Use named pipe /tmp/com_1 (it’s really a UNIX domain socket)
  • Configure it to “Yield CPU on poll” (under Advanced)
• Install ’socat’, if not already installed

Configure and Compile the kernel on the developer or the target machine

• Get kernel 2.6.26 or newer
• make menuconfig # or make gconfig
• Under Kernel Hacking:
  • enable KGDB
  • enable the Magic SysRq key
  • enable “Compile the kernel with debug info”
• Build kernel: make

Configure target

• Enable Magic SysRq key on target:
  • Edit /etc/sysctl.conf and set kernel.sysrq = 1
  • or run sysctl -w kernel.sysrq=1 # this doesn’t survive a reboot
• Install developer kernel
  • On the developer machine:
    rsync -av --exclude .git ./ root@target.host.name:/mnt/work/linux-2.6.26
  • On the target, a RedHat based system:
    make install
make modules_install
• Edit /boot/grub/grub.conf and set timeout=15
• Boot into the newly installed kernel

Start debugging

• On target:
  echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
• On developer:
  socat -d -d /tmp/com_1 PTY: # notice what pty is allocated — /dev/pts/1 in my case
  gdb vmlinux
set remotebaud 115200
target remote /dev/pts/1
• On target, do one of the following:
  • echo "g" > /proc/sysrq-trigger
  • Type ALT-SysRq-G
• Ready, get set, go! Go back to developer machine and use gdb to set breakpoints, continue, etc.

I set up debugging because I wanted to understand the behavior of the kernel when loading a module. It turns out that loading of the module failed because sitting in a debugger delayed the execution, causing a timeout in module load by the time I stepped through the code. Use of printk turned out to work better.

How to enable Intel hardware virtualization on an HP xw4600:

• Boot into the hardware BIOS setup
• Got to Security -> System Security
• Enable both types of virtualization (VTx and VTd)
• Save settings, and power-cycle the machine.

I’m running Linux, Fedora 9, and using KVM, so I run the following:

modprobe kvm-intel  

Loading that module will fail if hardware virtualization isn’t enabled.

I’m not a Django programmer, but for those who are, this may be useful. YouTube has videos of the inagural DjangoCon conference: http://www.youtube.com/results?search_query=djangocon&search=tag

For over a year, I’ve endured a development machine that would lock up under heavy disk I/O. Yesterday, I apparently complained loudly enough that I was given a new machine to replace it. I didn’t want to reinstall Fedora 9, so I transferred my old hard drive to the new machine, as the primary drive. To get it to boot and function properly, here’s what I did:

• Booted with the Fedora 9 install CD into “rescue mode”

• Ran the following commands once I had a shell:

  mount –bind /dev /mnt/sysimage/dev
  mount –bind /sys /mnt/sysimage/sys
  chroot /mnt/sysimage
  mv /boot/initrd-2.6.25…i686.img /boot/initrd-2.6.25…i686.img.orig
  mkinitrd /boot/initrd-2.6.25…i686.img 2.6.25…i686

• Then I ran ‘grub’, and typed the following:

  root (hd0,0)
  setup (hd0)
  quit

• Ejected the install CD, and rebooted. Once booted, I noticed that my network cards weren’t set up quite right. My new network card was listed as “eth2″ in system-config-network, and I didn’t actually have cards for the listed “eth0″ and “eth1″ interfaces anymore. I didn’t know what file to change to get my new card listed as “eth0″, so I ran the following command to find out what files I might need to edit:

  find /etc -type f -print0 | xargs -0 grep “eth[01]“

That command listed the following files, among others:

• /etc/udev/rules.d/70-persistent-net.rules
• /etc/vmware/locations

I edited /etc/udev/rules.d/70-persistent-net.rules and ripped out the assignments for my old NIC interfaces, and set the new one to be “eth0″, then rebooted and used system-config-network to set up my network.

When I ran my VMware guest, VMware Server gave me an error message about not being able to use bridged mode for the selected interface. With my old computer, VMware had used eth1 for bridged networking, and I didn’t have an “eth1″ interface anymore. I edited /etc/vmware/locations and changed “eth1″ to “eth0″, and restarted vmware. This time, bridged mode worked correctly.

Perhaps this is a bit old, but it’s the first time I’ve seen it, and I thought it was interesting enough to share.

http://www.webappsec.org/projects/statistics/

• more than 7% of analyzed sites can be compromised automatically
• Detailed manual and automated assessment using white and black box methods shows that probability to detect high severity vulnerability reaches 96.85%.
• The most prevalent vulnerabilities are Cross-Site Scripting, Information Leakage, SQL Injection and Predictable Resource Location

The Pragmatic Bookshelf is releasing a book on using Git for version control.

Steven Walter released a new command-line front-end for git called yap. It’s not only supposed to make it easier to work with git, but also with subversion repositories. It’s available from http://repo.or.cz/w/yap.git

I’ve often wondered why people seem to prefer either MySQL or PostgreSQL. For the most part, I think it comes down to the following:

• Familiarity.
• Friends (a.k.a. support system) being more familiar with one over the other.
• Ease of getting started. Most web hosting providers support MySQL out-of-the box.
• Name recognition.
• Ease of support.

Here are some resources that could be useful for learning the pros and cons of each database:

• What MySQL can learn from PostgreSQL
• What can PostgreSQL learn from MySQL and the accompanying presentation
• MySQL quirks and limitations
• Why PostgreSQL?

Have you ever wondered what forms of communication are the most and the least effective for software engineers? See Scott Ambler’s “Models of Communication” diagram in his essay. Face-to-face is most effective, and paper is the least effective, with email, telephone and video conferencing falling in-between the two ends of the spectrum.

Have you considered the merits and applicability of RESTful web apps? Here are a few notes I’ve made.

There was quite a discussion about RPC, REST, and message queuing — they are not the same thing. Each one is needed in a different scenario. All are used in building distributed systems.

Wikipedia’s explanation of REST is quite informative, especially their examples of RPC versus REST.

The poster “soabloke” says RPC “Promotes tightly coupled systems which are difficult to scale and maintain. Other abstractions have been more successful in building distributed systems. One such abstraction is message queueing where systems communicate with each other by passing messages through a distributed queue. REST is another completely different abstraction based around the concept of a ‘Resource’. Message queuing can be used to simulate RPC-type calls (request/reply) and REST might commonly use a request/reply protocol (HTTP) but they are fundamentally different from RPC as most people conceive it. ”

The REST FAQ says, “Most applications that self-identify as using “RPC” do not conform to the REST. In particular, most use a single URL to represent the end-point (dispatch point) instead of using a multitude of URLs representing every interesting data object. Then they hide their data objects behind method calls and parameters, making them unavailable to applications built of the Web. REST-based services give addresses to every useful data object and use the resources themselves as the targets for method calls (typically using HTTP methods)… REST is incompatible with ‘end-point’ RPC. Either you address data objects (REST) or you don’t.”

RPC: Remote Procedure Call assumes that people agree on what kinds of procedures they would like to do. RPC is about algorithms, code, etc. that operate on data, rather than about the data itself. Usually fast. Usually binary encoded. Okay for software designed and consumed by a single vendor.

REST: All data is addressed using URLs, and is encoded using a standard MIME type. Data that is made up of other data would simply have URLs pointing to the other data. Assumes that people won’t agree on what they want to do with data, so they let people get the data, and act on it independently, without agreeing on procedures.

Google has made videos available for it’s two-day I/O conference for developers. They cover things such as Python, the Android platform, Google Apps, etc.

I’ve previously mentioned that I bought a Kodak all-in-one printer. When it prints, it prints beautifully. Ours didn’t print well for months, and I finally got around to calling their tech support, which was excellent. They sent me a new print head, which resolved my problem. Unfortunately, I had to stay on the line with tech support for over an hour, and I spilled permanent ink on my pants (it doesn’t wash out like they said it would).

Apparently, Kodak printers have a fundamental flaw in the design of the print head, and I’ll need a new print head about once per year — see the comments on Kodak’s blog. Here’s one of them:

Posted By: ThriftyTechie (7/30/2008)

Before I tell you about my problem, I must say that the Kodak AiO printer phone support is excellent. I’ve had several experiences with your phone support (unfortunately) and every person has been helpful. Kudos. The bad news. It looks like that you have a couple of fundamental engineering flaws in your printer. 1. The non-disposable print head is just not durable enough. 2. The ink cartridges are too small. More frequent ink swaps are a) annoying for the consumer and b) can not possibly good for dependability, durability of the machine. My 5100 printhead failed completely after about 14 months after several months of sub par smeared and greyed-out printing. Lots of ink cartridges wasted on calibrating and testing. I was high on the product’s claims (i.e., save money on ink!), but this product certainly did not live up to the hype.

My time is valuable, so rather than spend over an hour on the line with tech support, I’d rather buy a different printer.

The American Fork City sewage and composting plant is not far from the office where I work, and when the wind blows in this direction, we can smell the human output of an entire city.

It’s not usually a problem, and when it is, we don’t smell it from inside the office. Today is a overpowering exception, and it makes my stomach churn. It’s never been this bad before.

I thought it’d be interesting to know what times of day people were most likely to send me email. My email is stored in mbox format (I used Thunderbird and mutt for email), so I wrote a perl one-liner to analyze it for me.

The first one-liner prints a histogram, in 80 columns, of activity per-hour of the day. The second prints it in a form suitable for import into a spreadsheet

Histogram:

perl -nle '$sum[$1]++ if m/^Date: .* (\d\d):\d\d:\d\d/; END {foreach (@sum) { $max = $_ if $_ > $max }; $div = $max/80; foreach (@sum) { print $i++ . " " . ("#" x ($_ / $div)) . " ($_)";}}' /path/to/Inbox

 0 #################################### (115)
 1 ########################## (84)
 2 ################### (62)
 3 ################ (54)
 4 ############ (40)
 5 ######### (31)
 6 ####### (23)
 7 ######################## (79)
 8 ####################################### (126)
 9 ############################################### (152)
10 ######################################### (133)
11 ###################################### (124)
12 ############################################################### (202)
13 ############################################################## (200)
14 ############################################################ (192)
15 #################################################################### (218)
16 ######################################################################## (229)
17 ################################################################ (206)
18 ################################################## (160)
19 ############################### (101)
20 ##################################### (118)
21 ######################################## (129)
22 ######################################################### (183)
23 ######################################## (129)

Tabular data:

perl -nle '$sum[$1] += 1 if m/^Date: \w{3}, \d+ \w{3} \d{4} (\d\d):\d\d:\d\d/; END {foreach (@sum) { print $i++ . "\t" . $_;} }' /path/to/Inbox

While I was at it, I wanted to know what the most common timezone offsets were. Again, I wrote two separate one-liners. One prints a histogram, and the other doesn’t.

Histogram:

perl -nle '$tz{$1} += 1 if m/^Date: .*([+-]\d{4})/; END {foreach (values %tz) {$max = $_ if $_ > $max }; $div = $max/80; foreach (sort(keys %tz)) { print "$_ " . ("#" x ($tz{$_}/$div)) . " ($tz{$_})"; }}' /path/to/Inbox

Non-histogram:

perl -nle '$tz{$1} += 1 if m/^Date: .*([+-]\d{4})/; END {foreach (sort(keys %tz)) { print "$_ $tz{$_}"; }}' /path/to/Inbox

I subscribe to various email lists, and each has different characteristics. I was surprised to find that my family email box usage pattern was fairly spread out around the clock, except that it drops off significantly during dinner and during the wee hours of the morning. Evening hours are the most active.

I’ve taken the timezone one-liner and modified it to tell me the most common months of the year, or the most common days of the week for email to be sent. For all my email boxes, analyzed over the last few years, email is most active on weekdays, and drops off on weekends.

Mon ############################################################### (5630)
Tue ##################################################################### (6129)
Wed ######################################################################## (6372)
Thu ##################################################################### (6155)
Fri ############################################################ (5329)
Sat ############################## (2675)
Sun ########################## (2368)

I tried translating those one-liners into Ruby, but it wasn’t as compact, and doing it as a one-liner in Java just isn’t going to happen.