A typical weekend

10/01/11 | by admin [mail] | Categories: Hardware, Software, Networking, Mac, OSX

So, this weekend I did the usual housework, eating, sleeping, watching TV etc, that people the world over do of a weekend. The other thing I did was install Debian on a mini-ITX system, and Mac OS X Leopard on a PowerMac G4 Cube.

The Cube is a little upgraded, having an 80GB hard drive, 1.5GB RAM (a Friday night upgrade after ordering 2x 512MB sticks from OWC), Airport, nVidia 6200, and a Sonnet 1.2GHz G4. Since the CPU is upgraded, Leopard will install without issues (it has a 867MHz G4 minimum requirement, although trivial to get around). I did the install using Netboot from a virtualized 10.5 server (another weekend project). I created the boot image from a 10.5.4 ISO image, which was mounted to the VM (Fusion) using the virtual optical drive device.

Boot over the network from the image, and then upgrade the installed 10.4.11 Server (this Cube was set up briefly as a server). There were, needless to say, a couple of hitches. I could not remember the original password for the admin account (I had to reset it last time, as well), so I needed to reset the password using the install disk, which did not work. So, retry to make sure I had clicked all the right buttons etc, and still no go, so a boot to single user mode to edit the local directory using dscl.

Dscl showed that the user accounts had not been imported. The home folders were there, but no account entries in the local node. So, reset the root password, reboot, login and then re-created the account. dsenableroot is a nice trick,

The other, small, issue was that the Airport card could not connect to my draft N WiFi (airport express). Not a big deal since 10/100 Ethernet is working, and I can always plugin in an older basestation (WAP) if needed.

Next trick is to get an SSD drive to saturate that slow 66mbyte ATA bus :)


Mac Myths

09/01/11 | by admin [mail] | Categories: Hardware, Software, Mac, OSX

Working in the Mac world, I sometimes forget how little non-Mac people know about Macs. Even people in IT are surprised to know that Mac OS X Server even exists.

So, there are a few interesting factoids I'd like to share.

Mac OS X is Unix.

To qualify that a little, Mac OS X 10.5 and 10.6 running on Intel Macs is qualified as Unix by the Open Group, the only BSD variant qualified as UNIX03.

This means that Mac OS X is probably the most widely used UNIX in the world. The current Mac userbase is between 30 and 94 million (and the vast majority of these are running 10.5 or higher on Intel Macs). This leads to some interesting conclusions. AFP is probably the most widely used UNIX network file system. HFS+ is probably the most widely used UNIX filesystem.

Mac vs Linux as desktop *Nix.

Estimates of the Linux user-base put it at about 30 million. Mac OS X undoubtedly outnumbers Linux on the desktop, but Linux is still ahead in the server market and probably ahead in the embedded-device/appliance market (despite the success of IOS on iPods, iPads and iPhones).

Macs have Unix tools

There are about 12000 Linux/Unix ports available via Fink. Mac Ports has about 5400. There is some overlap between the two. This compares with about 24000 in Debian.

Mac OS X comes with a large array of standard UNIX tools and FOSS. Mac OS X (the standard version, not the server), comes with bash, tcsh, ksh, ssh, telnet, vnc, ftp, Apache, PHP, Python, BIND, Perl, Ruby, Rails, Kerberos, Postfix, Dovecot, AMaViS, RADIUS, Samba, NFS, X11, gcc, curl, rsync, dtrace, OpenLDAP….

…and CUPS, which is now owned and maintained by Apple, and more.

Macs run Linux.

Linux was first ported to the Mac back in 1997. Debian for PPC Macs was released in 2000. Ports also exist for 68k Macs, and Nubus PPC macs (the first Power Macintosh x100 series). Linux for the PowerPC was first released at the 2.2.x version of the kernel. Current Macs will happily dual (or multi) boot Linux, or run Linux virtualised. It's also possible to run Linux on a Mac, run Mac OS X virtualised under Linux.

Macs have malware.

Malware exists on the Mac. The largest class by numbers is still Office macro viruses. There is a growing number of Trojans (distributed mainly in warez), and web browser (java/flash etc) based malware. The Mac is probably also vulnerable to other application based exploits, like the famous PDF exploit.

Macs also have security software.

Mac OS X has a firewall. Free anti virus is available in the form of clamav, and the frontend ClamXAV. And has been available for many years. Mac OS X Server ships with clamav antivirus since at least 10.3 (don't ask me about 10.2 server, it's too old!).

Macs have multi-button mice

It is a surprisingly common view, that Macs only have single button mice. Multi-button mice have been available for Mac users for 20 odd years. Native support for 2+ buttons and scroll wheels has always been a feature of Mac OS X. I've always used 2 button + scroll wheel mice with my Macs, except for my PowerBook G3 (since it had a built in trackpad). All currently shipping Macs feature multi-'button' and scrolling support, and have done for more than 5 years.

Mac OS X is open

This is an interesting situation. Mac hardware is closed to an extent, and the OS is only designed and licensed to run on Apple hardware. But Mac OS X is Unix, and Unix is a fairly open platform in any flavour. You can install whatever you like, compile what you like, modify and customise a hell of a lot. Nearly all preference/configuration files are in text, or easily convertible to text, and XML formatted. Some of the OS is opensource, although covered by licenses of varying restriction.

The basic BSD framework underlying the OS is called Darwin. The sourcecode is open to read and compile, and to some extent to modify/distribute. Different parts are covered by different licences. Most of the Apple developed components are covered by Apple Public Source License. Apple were releasing ready to install binaries up until Darwin 8 (10.4), but stopped due to limited interest. You can get ready run installs of the Darwin based GNU-Darwin.

Apple developed launchd to replace init/xinit. Basically, launchd is a daemon which starts other daemonised services and agents, both system and user. Originally, it was released under Apple Public Source License, and there was some interest from people such as Ubuntu, however the licence was determined to be too restrictive. Apple has since changed the licence to the Apache License v2 to encourage broader adoption. And, as mentioned, there is CUPS.

Mac OS X and SOE

An interesting feature of Mac OS X, and this comes out of having well defined hardware it runs on, is that you can take a hard drive out of one Mac, and put it in another and it will work (if the computer you are swapping it to meets the minimum system requirements for the OS). There can be some licensing issues with third party software, but by and large it's plug and play. What this means is that half of your SOE is done for you. The OS and drivers is consistent across platforms (different drivers will be used depending on the hardware, but all the drivers are installed and available).

It makes migration a breeze. On top of this, Users accounts are clearly defined. All the users individual settings are in their home folder, a single directory tree, which can be copied between computers and then associated with a log on account. This means you can do cool stuff like keep user accounts on external media, or on a server, and move them between computers.

So you can adopt a fairly radically different approach to SOE builds. All you need to manage is the third party software, which you can do in a variety of ways. You can develop work flows which build an image for deployment, or update a live machine with the software and configuration it needs. You don't need adopt the more conventional approach of identical hardware, single images with OS and sofware which need to be imaged to the hard drive prior to deployment, and centralised, network based, accounts (and all the required hardware to back it up).


Mac OS X 10.4 Intel Install disk

08/08/10 | by admin [mail] | Categories: Software, Mac, OSX

I lost the install disks to my 2nd Gen MacBook Pro. I have Mac Mini 10.4.10 install disks, though. So, I decided to use those.

There are several ways to do this. You can install it onto the Mac Mini (or external drive attached to Mac Mini) and then clone it to the MacBook Pro. You can put the MacBook Pro into target disk mode and plug it into the Mac Mini and run the install on the Mac Mini. Or you can modify the grey install disks to work with the MacBook Pro (or any other supported Intel).

After a bit of searching, I found this.

The file you need to edit on the grey install disks is this:


This is an XML formatted piece of Javascript which contains a function which checks an array to see if the Mac you are using is a supported machine.

The array is defined towards the end of the file and uses the Model Identifier. In the case of my install disks, the array held one value: 'MacMini2,1'

You need to add to this array (or you can replace the single existing value), the Model Identifier of the Mac you wish to use the grey install disks with.

The way I edited the file, since it is on a DVD-ROM, was to use Disk Utility to restore 'Mac OS X Install Disk 1' to a partition on an external USB drive. This has the advantage of allowing me to use the USB drive to install the OS to the MacBook Pro (and quickly). But you could also make a read/write disk image file from the DVD and edit the file on that, then burn the image to a dual layer DVD (or even use Netboot).

I opened the file using Text Editor, but you could use any text editor you like, or an XML editor, or vi or pico or something.

The text we are looking for is something like:

var hwbeSupportedMachines = ['MacMini1,2'];

What we need to do is edit that line so that it reads like:

var hwbeSupportedMachines = ['MacBookPro2,1','Macmini2,1'];

The easy way to get the Model Identifier (the MacBookPro2,1) bit is to open up terminal and issue:

$ sysctl hw.model

The elements of the array need to be comma separated, and you can add as many as you like. The complete list of Intel Macs supporting 10.4 is:

var hwbeSupportedMachines = ['iMac4,1','iMac4,2','iMac5,1','iMac5,2','iMac6,1','iMac7,1','iMac8,1','MacBook2,1','MacBook1,1','MacBook3,1','MacBookPro1,1','MacBookPro1,2','MacBookPro2,2','MacBookPro2,1','MacBookPro3,1','Macmini1,1','Macmini2,1','MacPro1,1','MacPro2,1'];

If you change the array so it reads as above, you can install on any Intel Mac supporting 10.4. Since I used a 10.4.10 disk as my base, there shouldn't be any issues with installing on a Mac that shipped with 10.4.10 or earlier. If you used an earlier version of 10.4, like 10.4.6 you may run into issues trying it with a Mac which shipped with a later version of 10.4 (eg 10.4.8 or 10.4.10). As far as I know, no Macs require 10.4.11 as a minimum.

I got this list of Model Identifiers, and the minimum OS X version support from everymac.com.

I don't speak javascript, but I suspect that the function which reads and evaluates the array could be modified. It looks like it just gets the Model Identifier of the Mac (as modelProp) and compares it against the elements in the array. If it finds it in the array, it returns true. So probably you could just get rid of the bit that does the comparison and just make it always return true.

Of course, it is handy to keep something in the script which will explicitly stop you from installing on unsupported models.

That's it.


When Open Directory goes bad

23/02/10 | by admin [mail] | Categories: Software, Networking, Mac

Open Directory is one of the harder components of Mac OS X server to recover.

I recently had two opportunities to fix bad OD services on two Macs running 10.6 server.

The first server was in a bad way. The OD was not accessible from Workgroup manager, and was not running in Server Admin. Users could not authenticate against the OD.

The usual poking around showed that LDAP was dead. Google lead me to:
sudo db_recover -h /var/db/openldap/openldap-data

Which failed. However, using man db_recover, I came across the option
Perform catastrophic recovery instead of normal recovery.
Which did the trick, nicely.

What caused this failure? Well, a fsck and then permission repair both showed issues. But since it was fixed, I did not spend too much time indulging in 'why'. I am guessing that hiccup caused some file corruption.

The second issue is a more commonplace one. It was a new Snow Leopard server install, and authentication against the OD was taking forever and accessing the OD via WorkGroup Manager was slow and painful.

This was an issue I remember from our early experiences with Leopard Server. Basically, if DNS is not up and running nicely, you get big delays accessing Server Admin, WorkGroup Manager and in authenticating.

This user had set up the server using a FQDN ending in .local. This seemed not to be working nicely. However, they had registered a domain to use with this server (the server will be doing some Web hosting), so a good FQDN was available.

Since there were only 10 users set up, and all the passwords were available, I exported the users and groups using Workgroup Manger and then demoted the server to standalone. I then set up DNS correctly using the registered domain name, and re-promoted the server to OD Master. I then imported the accounts and groups, and reset the passwords. I have found that it is quicker to do this than use the changeip command (and the rest) to change the Kerberos realm, and LDAP, because experience has shown that it is rarely a matter of issuing changeip.


RAID disaster stories

22/02/10 | by admin [mail] | Categories: Hardware, Mac, PC

Over the last few months I've run into a few RAID (mostly RAID-5) horror stories.

Story 1:
MegaRAID breaks killing RAID.

If you have ever had an Xserve G5 with hardware RAID, you are probably familiar with this story. Basically, what happens is that your RAID 5 will die for no apparent reason. Sometimes, a restart and removing one of the drives and replacing it will fix the issue. Sometimes you need to boot from CD and repair the RAID using the terminal commands. Sometimes, you need to re-add it in OpenFirmware.

The problems with this set up are common enough that I routinely remove the cards and set it up as a software mirror.

Story 2:
Xserve RAID disaster 1

The Apple XServe RAID was a nice product, since discontinued. Basically it consisted of a 3u box split into two hardware RAID arrays. The two arrays are independent, but can have software RAID applied to them to create, for example, a mirror. Apple details a number of set ups, including such things as striping the two halves and concatenating the two halves (WTF???). You can probably guess what happens next.

The Xserve RAID was set up with two arrays (RAID level 5). They were then software striped. The RAID was left unmonitored. One side failed (two bad drives on one array). All the data was gone. The only backup was several months old.

The Xserve RAID was set up from scratch, using larger drives and only using one half of the array. A backup system was implemented to create daily backups.

Story 3:
Xserve RAID disaster 2

Similar situation, but in this scenario they simply concatenated the two sides. Then the controller in second half started having 'issues'. This turned out a little better, because most of the data was physically on the first half of the RAID. The downside was that to access the data, the whole concatenated volume needed to be online, but as soon you accessed anything on the second half of the RAID, the concatenated volume went offline and the server needed to be restarted.

Again, there was no current backup. All the data that could be retrieved was taken off the RAID. The Xserve RAID was replaced, and a backup strategy implemented.

Story 4:
PC hardware RAID mirror

This is an older story. This was a cheap linux box with a "RAID" card which basically supported striping and mirroring across 2 volumes. What happened was that the RAID controller itself failed, and wrote garbage to both halves. Some data was retrievable but since this was also the boot drive, the system was down for over a day.

Story 5:
External RAID 5 box

This is another system set up by someone who fails to grasp the basics of RAID. This is another familiar story.
The RAID box used connected to the server via eSATA. It supported a number of useful features - RAID 5, hot spares, mixed drive capacities, web interface, visual+audible alarms, email monitoring etc.
The array was set up as a straight RAID-5. Identical drives were used, all from what turned out to be a bad batch. Two drives failed in close sequence. This company outsourced its IT, and the staff had no training so were unaware of fault until the RAID failed and went off line. They had a backup system in place, but it too was unmonitored, and they were untrained in using it.

They changed IT support companies and pro-actively sought training in how to monitor the RAID and their backup system. A hot spare and email monitoring was set up.

Story 6:
External RAID box.

This was another external RAID box. The company which set it up for them said that they didn't need a separate backup because two drives would need to fail.

One weekend, there was a small fire in office which set off the fire sprinklers. The RAID was destroyed, and with no off site backup they lost their data.

Story 7:
Xserve with hardware RAID

This company shelled out for an Intel Xserve with hardware RAID. The hardware RAID in the intel Xserve replaces the regular SATA back plane. The problem here, again, was the use of identical drives from a bad batch in combination with a hot weekend.

This company leaves their IT equipment running over the weekend, but turns the aircon off. One weekend the temperature hit 43C. They came in on Monday to a server which would not boot. The RAID controller itself failed, and two of the drives were working intermittently. Thankfully, the company had a backup of their vital data.

RAID lessons

A RAID is not a substitute for a backup system, it is just a part of your business continuity strategy. A RAID allows you to continue working while the repair is carried out. A properly implemented RAID reduces the chances of catastrophic failure

Data recovery from a RAID-5 is far more expensive and difficult than data recovery from a mirror or a single drive. This means a good backup system is MORE important if you are running RAID-5.

Don't put all your faith in a RAID controller. Sometimes controllers go bad. Do some research. Be wary of systems that make it hard to change to a non-hardware RAID set up.

Understand the underlying technologies. If you are using hardware RAID-5 it probably uses a proprietary algorithm to write data across the array. If your concern is redundancy, then don't stripe (RAID-0) in software or hardware. Concatenation is rarely useful.

Understand Redundancy. RAID is not a simple technology. There are a number of things that trip up the inexperienced. Use different drives. You are more likely to have two drives fail in quick succession if they are from the same batch. Consider adding a hot spare or using a higher RAID level. The longer it takes to get your RAID back up and running means a greater risk of that second drive failing. Long rebuild times can increase the chances of a second drive failure above acceptable limits.

Educate Users. IT support won't always be around when something goes 'funny'. If users know what to look out for, and they know that a flashing amber light or a beep is not normal, then they can report it. The sooner the problem is noticed, generally the better the outcome is.

Have a backup strategy. An offsite backup makes sense.


<< Previous Page :: Next Page >>

Current Hardware Projects

This is all about my on going fumblings with hardware. Regular entries should provide an indication of the depths of my obsession.

< Previous | Next >

March 2017
Mon Tue Wed Thu Fri Sat Sun
 << <   > >>
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    



Amiga Links

XML Feeds

What is RSS?

powered by b2evolution free blog software