Real NAS

From NAS-Central Buffalo - The Linkstation Wiki
Revision as of 00:34, 11 November 2007 by Ramuk (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Abstract

This howto presents a step by step guide to convert the TeraStation into a real NAS like server. The principal motivation behind this conversion is that to maximize data reliability, the disks containing that data should be off as much as possible. To achieve this goal we will add another hard disk to hold the operating system and possibly some other frequently changing or frequently accessed data. Also, this conversion gives us a safe way to install another Operating System to unlock the usefulness of our TeraStation to its full potential.

Ingredients

  • 1 TeraStation
  • 1 2.5" HD
  • 1 EIDE cable with 3 40 pins connectors
  • 1 40 to 44 pins HD adapter
  • 1 Y power cable

The 2.5" hard disk will be put into the TeraStation as a fifth drive. To do this we also need the EIDE cable with three connectors which will be connected to the motherboard of the TeraStation, the 2,5" hard disk and one of the 3.5" hard disks. It replaces one of the two-connector cables the TeraStation is equipped with. The Y power cable is needed to provide an additional power source for the 2.5" hard disk.

Steps outline

  1. Prepare the soon to be installed OS Hard Disk
  2. Put the OS hard disk into the TeraStation
  3. Modify Buffalo's GNU/Linux distribution
  4. Installing another OS into the OS Hard Disk
  5. Configure and boot the newly installed Operating System
  6. Setup data hard disks

Prepare the soon to be installed OS Hard Disk

In this section we will setup the 2.5in laptop hard disk to have the shape/form the TeraStation expects form a disk that can be booted.

This can be done with the aid of a free EIDE port on a separate computer or directly on the TeraStation.

Using a Separate Computer

If we don't have any big-endian computer on hand and as the mdadm tool isn't endian agnostic, there no nice way to create a RAID partition and transfer files into it that will work on the TeraStation. But that does not present a real problem as we can copy the RAID partition as is (in RAW mode).

First we connect one of the TeraStation's 250GB data disks to our computer and run:

# dd if=/dev/250gb_disk of=original-sectors-0-65.raw bs=8225280 count=66

This gives us an image with the original partition table, the RAID1 partition the Terastation boots from and the swap partition. This image we will transfer to the new OS disk, as well as keep in case we need to recover the TeraStation to it's original state.

Then we disconnect the data disk, connect the soon to be OS disk and run:

# dd if=original-sectors-0-65.raw of=/dev/os_disk bs=8225280 count=66

And we are done!

Also, in case we have not yet setup SSH nor gained root access on the TeraStation, this is a good time to do it, as we have the disk we will boot the TeraStation from connected to our computer.

The RAID1 partition can be mounted to be modified with:

# mount -t xfs /dev/os_disk1 /mnt/teraroot

And in that case, to force the TeraStation to boot from our OS disk we need to connect all four 250GB data disks one by one to our computer and execute:

# dd if=/dev/zero of=/dev/250gb_disk count=64

This is to make them look to the TeraStation as if they were factory new disks.

Using the TeraStation

Note 1: This subsection assumes that we have root access to the TeraStation via SSH.
Note 2: This method isn't actually tested (please make corrections and remove this note when/if it works)

In theory the partition scheme should not matter to the TeraStation nor it should interfere with the boot process unless the OS disk has a RAID partition whose super block suggests it to be /dev/md0 ("Preferred Minor : 0"). But as all data contained in the OS disk will be lost anyway, it may be reasonable to delete the partition table using a computer or an USB enclosure connected to the TeraStation before connecting to the TeraStation via EIDE port.

So if the OS disk gets recognized as /dev/sda we can clean it's partition table running:

# dd if=/dev/zero of=/dev/sda count=64

At this point we can connect all the disks to the TeraStation (this is detailed in the next section), boot it and continue setting up the OS disk.

First we enable DMA on hda and hdb to speed up things a little bit:

# hdparm -d1 /dev/hd{a,b}

Then remove /dev/hdb1 from /dev/md0 (the raid1 array the TeraStation boots form):

# mdadm --manage /dev/md0 --fail /dev/hdb1
# mdadm --manage /dev/md0 --remove /dev/hdb1

At this point it may be good to make a backup of a working RAID1 array used to boot the Terastation to another computer (just in case we make some mistake and have to recover). So we run the following command on the computer we want to back up to:

$ ssh root@terastation "dd if=/dev/hdb bs=8225280 count=66" | cat - > original-sectors-0-65.raw

Then we swap the data disks we just removed with the OS hard disk in the RAID1 array and reboot the TeraStation:

# dd if=/dev/hdb of=/dev/hda bs=8225280 count=66
# dd if=/dev/zero of=/dev/hdb count=64
# reboot

The TeraStation should come up with the first partition from the OS disk (/dev/hda1) part of the RAID1 array (/dev/md0).

Then we make /dev/md0 be exactly /dev/hda1:

# mdadm --manage /dev/md0 --fail /dev/hd{c,e,g}1
# mdadm --manage /dev/md0 --remove /dev/hd{c,e,g}1

Delete the partition tables from the remaining data disks and reboot:

# dd if=/dev/zero of=/dev/hdc count=64
# dd if=/dev/zero of=/dev/hde count=64
# dd if=/dev/zero of=/dev/hdg count=64
# reboot

If everything goes as it should, the Terastation should boot from the OS disk without even touching the DATA disks!

Put the OS hard disk into the TeraStation

The TeraStation has four EIDE channels, each capable of managing up to two disks (one Master and one Slave). This gives us a theoretical limit of 8 disks connected directly to the TeraStation, but in practice we should never connect more than one disk from the same array to one EIDE channel. Because a failed disk can render unusable the whole EIDE channel which would cause all the devices connected to that channel to fail as well. This would mean two disks fail from the array and therefore make the entire array fail.

Having this in mind we connect each of the data disks to it's own EIDE channel and the OS disk as second disk to the channel of any (shouldn't matter which one) on the data disks.

In the author's experience it was easier to fit the OS disk into the TeraStation using the following scheme:

  • OS disk: Master, connected to DISK1 port (/dev/hda)
  • Data disk 1: Slave, connected to DISK1 port (/dev/hdb)
  • Data disks {2,3,4}: Master, connected to DISK{2,3,4} ports (/dev/hd{c,e,g}

This way the OS disk can be placed between the cables going from data disks {2,3,4} to ports labeled DISK{2,3,4} and the cable connecting the OS disk with the data disk 1 and the port labeled DISK1.

Note: This placement has the big disadvantage of the OS disk being completely wrapped in cables without getting any ventilation at all, on the other hand, 2.5in laptop disks are supposed to work fine in those conditions. In any case, it may be better to use a 5400RPM disk instead of a faster 7200RPM one. And please, do experiment with other (better ventilated) placements and edit this section if you find one.

Modifying Buffalo's GNU/Linux distribution

The GNU/Linux distribution the TeraStation comes preinstalled with has a very limited set of tools, which are probably not enough to install and configure most operating systems we may want to install on the TeraStation. So in this section we will install the tools we may need to make a successful installation and setup the boot loader we'll use to boot the soon to be installed OS.

Installing some system tools

As in this howto we try to avoid making any assumptions with respect to which OS we will install, we may need to add some other tools apart from the mentioned below. If that's the case, a good place to get the tools are the Debian PPC stable package repositories as they seem to fit perfectly library wise.

scp 
It is always nice to have a safe way to transfer files form and to the TeraStation and scp is just perfect for that. To install it we need to download the ssh stable deb (ssh_3.8.1p1-8.sarge.4_powerpc.deb) to our computer, extract the data.tar.gz:
$ ar xv ssh_3.8.1p1-8.sarge.4_powerpc.deb data.tar.gz
transfer it to the TeraStation (this one is a little bit tricky as we still don't have scp on the TeraStation):
$ cat data.tar.gz | ssh root@teratsation "cat - > ssh-data.tar.gz"
And extract the files we need from this tar archive (from the TeraStation):
# tar -C / -xzpvf ssh-data.tar.gz ./usr/bin/{scp,sftp}
This will extract just scp and sftp into /usr/bin directory.
coreutils 
From the coreutils package we need to instal the chroot and mknod programs, to install them we need to download the appropriate deb file (coreutils_5.2.1-2_powerpc.deb), extract the data.tar.gz file:
$ ar xv ssh_3.8.1p1-8.sarge.4_powerpc.deb data.tar.gz
and transfer it to the TeraStation:
$ scp data.tar.gz root@terastation:coreutils-data.tar.gz
And extract the files we want on the TeraStation:
# tar -C / -xzpvf coreutils-data.tar.gz ./usr/sbin/chroot ./bin/mknod
nfs 
To be able to mount nfs exports on the TeraStation we'll need to put the sunrpc.o module to /lib/modules/2.4.20_mvl31-ppc_linkstation/kernel/net/ and the lockd.o and nfs.o modules to /lib/modules/2.4.20_mvl31-ppc_linkstation/kernel/fs/ on the TeraStation, all available in one tar archive and load them by running:
# modprobe sunrpc
# modprobe lockd
# modprobe nfs
ncurses 
The ncurses library is needed by many useful packages like screen, less and nano, so in case we want to use any of them we'll have to install ncurses. It is provided by the libncursesw5 deb (libncursesw5_5.4-4_powerpc.deb). To install it, first we get the deb and run:
$ ar xv libncursesw5_5.4-4_powerpc.deb data.tar.gz
$ scp data.tar.gz root@terastation:ncurses-data.tar.gz
And then extract on the TeraStation:
# tar -C / -xzpvf ncurses-data.tar.gz './usr/lib/libncursesw.so*'
screen 
The GNU Screen program is a quite nice tool to have when doing any nontrivial system management (like OS installation). It is provided by the screen deb package (screen_4.0.2-4.1_powerpc.deb). To install it we need to extract the ./usr/bin/screen and ./etc/screenrc files.
less and nano 
There are more and vi, but “less is more” and nano is a little bit friendlier than vi and will serve us as an example of installing a text editor (we will need one to configure the boot loader). This packages are provided by the less and nano debs (less_382-1_powerpc.deb and nano_1.2.4-5_powerpc.deb). We will need ./usr/bin/less form less and ./bin/nano and ./etc/nanorc form nano.
LVM 
In case we want our setup to use LVM (Logical Volume Manager) we just will need to install the LVM userland tools, as the kernel module is already provided by the original Buffalo installation. This tools are provided by the lvm10 deb package (lvm10_1.0.8-8_powerpc.deb). We will need to install the ./lib dir form it. (This will install all programs to /lib/lvm-10/, so to execute the foo program we will have to run # /lib/lvm-10/foo)

Tweaking /etc/init.d scripts

This setup works fine with unchanged scripts, but some scripts should be updated to take into account the changes we are making to our TeraStation.

(This subsection will soon be extended.)

Installing the “boot loader”

To boot into the OS we soon are going to install, this howto proposes the use of a kernel module boot loader written by Chih-Chung Chang (jochang) for the Kuro Box. With it we can safely boot other kernels keeping the flash chip untouched.

To install it we need to download the precompiled module or compile it with the TeraStation's 2.4.20 kernel headers. Put in to somewhere (like /root/) on the TeraStation, write an init script and make the TeraStation execute it at boot.

The is an example for the init script:

#
# bootnewos.sh Boots OS From /dev/hda6.
#

. /etc/default/rcS

MD5_RECOVERING=`/sbin/mdadm -D /dev/md5 | grep "State :" | grep "recovering"`

# We let the kernel to recover the data array
if [ "${MD5_RECOVERING}" != "" ] ; then 
	/sbin/hdparm -d1 /dev/hdb
	exit 0
fi

#Gives us time to correct any problem
/bin/sleep 90

if [ -f /.tonewos ] ; then
	/sbin/mdadm /dev/md5 --stop
	/bin/sync
	/bin/sleep 1
	/bin/sync
	/sbin/insmod /root/loader.o kernel=/root/loader/vmlinux.hda6 cmdline="root=/dev/hda6"
fi

To make it execute at boot:

# cp bootnewos.sh /etc/init.d/
# chmod +x /etc/init.d/bootnewos.sh
# ln -s ../init.d/bootnewos.sh /etc/rc.d/rc3.d/Sa0bootnewos.sh

We should not create the /.tonewos file until our new OS is in working conditions, but when we are ready we just need to execute:

# touch /.tonewos

Installing another OS into the OS Hard Disk

At this point we are finally ready to install the operating system we will use on the TeraStation. To do this we first repartition the OS disk according to our needs and then do the actual installation.

Repartitioning

There are some rules imposed by Buffalo's firmware/kernel we should obey while changing the partition table.

/dev/hda1 
There must be a /dev/md0 for the kernel to boot form. So it is reasonable (and so far the only tested way) to leave /dev/hda1 unchanged (same size and type) so we can follow Buffalo's updates for our Buffalo's GNU/Linux installation.
/dev/hda2 
This partition is not strictly required to be swap for the TeraStation to boot and work, but it is highly recommended for the swap partition to be as close as possible to the beginning of the disk (outer sectors of the disk).
/dev/hda3 
This is the place we'll use as our playground, so to facilitate things we can create an extended partition on it.
/dev/hda4 
For some reason this needs to have the exact type/size/offset as in the original data disks for the TeraStation to boot. It is important to notice that it does not matter if this partition falls outside the physical sectors of the disk.

This leaves us with an empty extended partition we can fill with the partitions we want to use.

To change the partition table the mfdisk program is available, but DO NOT RUN it as if it were the normal fdisk program as then it will replace the existing partition table with a funky useless one. Instead run:

# mfdisk -c /dev/hda

To use it in interactive mode (a la fdisk). Or just to be safe we can set an alias for it to mimic fdisk:

# alias fdisk="mfdisk -c"

So just as an example, in my case I decided to partition as follows:

# fdisk -l /dev/hda

Disk /dev/hda: 40.0 GB, 40007761920 bytes
255 heads, 63 sectors/track, 4864 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
376.4 MiB Buffalo / where the kernel from the firmware will boot to:
/dev/hda1               1          48      385528+  fd  Linux raid autodetect
133.3 MiB swap:
/dev/hda2              49          65      136552+  82  Linux swap / Solaris
The rest of the disk, will contain /dev/hdaX with X≥5:
/dev/hda3              66        4864    38547967+   5  Extended
0 MiB Raid partition that falls out of the disk:
/dev/hda4           30378       30515     1108484   fd  Linux raid autodetect
376.4 MiB To test my Buffalo / mods:
/dev/hda5              66         113      385528+  83  Linux
494 MiB for Linux /:
/dev/hda6             114         244     1052226   83  Linux
The rest of /dev/hda3 for LVM (with /home, /tmp, /usr, /var, ...)
/dev/hda7             245        4864    37110118+  8e  Linux LVM

Instaling the Operating System

In theory the TeraStation should be capable of running any operating system with support for TeraStation's primary hardware components and the capability of running headless.

But being a little bit more practical, it should run:

  1. PPC GNU/Linux distributions compiled without Altivec support should run out of the box.
    • Gentoo Linux: works nicely (howto soon to be posted)
    • Debian GNU/Linux: should work without a glitch (so far untested)
    • Linux From Scratch: should work without a glitch, but may be hard to setup if cross compilation is needed (so far untested)
    • foobar Linux: should work without any problems, but some minor tweaks may be required (so far untested)
  2. Linux/uClibc almost works but has some issues with (I think) the dynamic loader.
    • Gentoo Embedded: have problems
    • uClibc buildroot: the exact same problems as with Gentoo Embedded.
  3. *BSD may work as the CPU, EIDE, NIC and USB chips are supported, but may need some (trivial) changes in the “boot loader” we are using to be able to boot non Linux kernels.
    • NetBSD: should be the best bet (so far untested)

At this point we follow the instructions associated with the operating system we decided to install.

Tweaking and booting the newly installed OS

In this section we will do some minor tweaks specific to our the TeraStation setup.

Addressing the “watchdog”

There is another catch (probably) from how Buffalo's kernel is configured that we need to address before we boot into our new OS. And the catch is that even so there is no real watchdog on the TeraStation, the kernel appears to have a watchdogish something connected to /dev/ttyS1 and managed by the mc_ctld program (the program that should manage an UPS connected to the serial port). So this should only be needed if we will be booting a Linux based OS with the stock kernel.

To work around this catch we can copy the mc_ctld program with the files it uses to an empty directory and execute it chrooted on boot.

Files that we need to copy:

/dev
/dev/ttyS1
/lib
/lib/ld.so.1
/lib/ld-2.3.2.so
/lib/libc.so.6
/lib/libc-2.3.2.so
/etc
/etc/mc_ctld.ini
/usr
/usr/sbin
/usr/sbin/mc_ctld

And add something like the following to be executed by the new OS at boot:

/usr/bin/chroot /empty/dir/we/copied/to /usr/sbin/mc_ctld

It may also be necessary execute on boot (to be confirmed):

echo -n "aa8041" > /dev/ttyS1
echo -n "qqqq" > /dev/ttyS1

And on halt/reboot (also to be confirmed):

echo -n "rrrr" > /dev/ttyS1
echo -n "EEEE" > /dev/ttyS1

SSH Server

As the TeraStation by its nature is headless, it is crucial to configure the new OS to start the SSH server at boot.

Getting ready to boot

If we are going to boot into a Linux based OS with the stock kernel we need to hexedit it to make it boot from the device we want it to boot, else we should skip to the next subsection.

To accomplish this we first need to extract the kernel as shown on Firmware Update and then replace the three occurrences of root=/dev/{md0,ram0,hda1} with root=/dev/hdaX, where X is the partition number we want to boot form (6 in our example). And then transfer it to the TeraStation.

It also may be wise to copy the /lib/modules directory for our newly installed Linux.

Booting

First we need to unmount all the unneeded mounts and stop all RAID devices we can (the only active should be /dev/md0):

# umout /unneeded/mount
# mdadm /dev/mdX --stop

(we can see what is mounted with "mount" and the active raid arrays with "cat /proc/mdstat")

And finally we issue the magic command to boot the new OS:

# sync && insmod loader.o kernel=the_kernel cmdline="root=/dev/hdaX"

(the command line will be ignored by the stock kernel, but it is nice to specify it for consistency sake)

And if everything goes well, in a couple of minutes the TeraStation should come up booted into the OS we just installed!

Post first boot configuration

There is just one tiny thing that needs to be done. And the thing is that when we power the TeraStation on it'll first boot into /dev/md0 wich is supposed to consist of four identical partitions mirrored in RAID1, but we changed that supposition several sections ago, so we should update /dev/md0's superblock (again, just for consistency sake). If we booted Linux, we can do it executing:

# mdadm --manage /dev/md0 --stop
# mdadm --create /dev/md0 -l 1 -f -n 1 /dev/hda1

Also at this point we may want to activate the auto boot into new OS feature creating the .tonewos file in Buffalo's /.

Setting up the DATA hard disks

At this point we can setup the data disks in any way we think will suit better our needs. As the TeraStation should care about nothing more that the OS disk when it boots.

But in reality, this may not hold true as the initrd image that the kernel from firmware boots into may try to access our DATA disk, so some esoteric or close to stock configurations may confuse it.

Open questions

  • Why does it needs the 4th partition?
  • What does the mc_ctld program really do?
  • What configuration and patches should be applied to the stock 2.4.20 kernel sources to compile a bootable kernel?
  • What configuration and patches should be applied to the vanilla 2.6.x kernel sources to compile a bootable kernel?