Difference between revisions of "Real NAS"

From NAS-Central Buffalo - The Linkstation Wiki
Jump to: navigation, search
m (Getting ready to boot)
m (Using the Terastation)
Line 85: Line 85:
  # reboot
  # reboot
If everything goes as it should, the Terastation should boot form the OS disk without even touching the DATA disks!
If everything goes as it should, the Terastation should boot from the OS disk without even touching the DATA disks!
== Installing the OS hard disk ==
== Installing the OS hard disk ==

Revision as of 04:56, 28 July 2005


This howto presents a step by step guide to convert the Terastation into a real NAS like server. The principal idea behind this conversion is that in order to maximize data reliability, the disks containing that data should be turned off for as long periods of time as possible. To archive this goal we will add another hard disk to hold the operating system and possibly some other frequently changing or frequently accessed data. Also, this conversion gives us a safe way to install another Operating System to unlock the usefulness of our Terastation to its full potential.


  • 1 Terastation
  • 1 2.5in HD
  • 1 40 to 44 pins HD adapter
  • 1 EIDE cable with 3 40 pins connectors
  • 1 Y power cable

Steps outline

  1. Prepare the OS hard disk
  2. Install the OS hd into the Terastation
  3. Modify Buffalo's GNU/Linux distribution
  4. Install another Operating System into a separate partition
  5. Configure and boot the newly installed Operating System
  6. Setup data hard disks

Prepare the soon to be installed OS Hard Disk

In this section we will setup the 2.5in laptop hard disk to have the shape/form the Terastation expects form a disk that can be booted.

This can be done with the aid of a free EIDE port on a separate computer or directly on the Terastation.

Using a Separate Computer

If we don't have any big-endian computer on hand and as the mdadm tool isn't endian agnostic, there no nice way to create a raid partition and transfer files into it that will work on the Terastation. But that does not present a real problem as we can copy the raid partition as is (in RAW mode).

First we connect one of the Terastation's 250GB data disks to our computer and run:

# dd if=/dev/250gb_disk of=original-sectors-0-65.raw bs=8225280 count=66

This gives us an image with the original partition table, raid1 partition the Terastation boots from and the swap partition, an image that we will transfer to the os disk, as well as, should keep in case we need to recover the Terastation to it's original state.

Then we disconnect the data disk, connect the soon to be os disk and run:

# dd if=original-sectors-0-65.raw of=/dev/os_disk bs=8225280 count=66

And we are done!

Also, in case we have not jet setup ssh nor gained root access on the Terastation, this is a good time to do it, as we have the disk we will boot the Terastation form connected to our computer.

The raid1 partition can be mounted to be modified with:

# mount -t xfs /dev/os_disk1 /mnt/teraroot

And in that case, to force the Terastation to boot from our OS disk we need to connect all four 250GB data disks one by one to our computer and execute:

# dd if=/dev/zero of=/dev/250gb_disk count=64

This is to make them look to the Terastation as if they were factory new disks.

Using the Terastation

Note 1: This subsection assumes that we have root access to the Terastation via ssh.
Note 2: This method isn't actually tested (please make corrections and remove this note when/if it works)

In theory the partition scheme should not matter to the Terastation nor it should interfere with the boot process unless the OS disk has a raid partition whose super block suggests it to be /dev/md0 ("Preferred Minor : 0"). But as all data contained in the os disk will be lost anyway, it may be reasonable to delete the partition table using a computer or an USB enclosure connected to the Terastation before connecting to the Terastation via EIDE port.

So if the os disk gets recognized as /dev/sda we can clean it's partition table running:

# dd if=/dev/zero of=/dev/sda count=64

At this point we can connect all the disks to the Terastation (this is detailed in the next section), boot it and continue setting up the os disk.

First we enable dma on hda and hdb to speed up things a little bit:

# hdparm -d1 /dev/hd{a,b}

Then remove /dev/hdb1 form /dev/md0 (the raid1 array the Terastation boots form):

# mdadm --manage /dev/md0 --fail /dev/hdb1
# mdadm --manage /dev/md0 --remove /dev/hdb1

At this point it may be good to make a backup of a working raid1 array used to boot the Terastation to another computer (just in case we make some mistake and have to recover). So we run the following command on the computer we want to back up to:

$ ssh root@terastation "dd if=/dev/hdb bs=8225280 count=66" | cat - > original-sectors-0-65.raw

Then we swap the data disks we just removed with the os hard disk in the raid1 array and reboot the Terastation:

# dd if=/dev/hdb of=/dev/hda bs=8225280 count=66
# dd if=/dev/zero of=/dev/hdb count=64
# reboot

The Terastation should come up with the first partition form the OS disk (/dev/hda1) part of the raid1 array (/dev/md0).

Then we make /dev/md0 be exactly /dev/hda1:

# mdadm --manage /dev/md0 --fail /dev/hd{c,e,g}1
# mdadm --manage /dev/md0 --remove /dev/hd{c,e,g}1

Delete the partition tables form the remaining data disks and reboot:

# dd if=/dev/zero of=/dev/hdc count=64
# dd if=/dev/zero of=/dev/hde count=64
# dd if=/dev/zero of=/dev/hdg count=64
# reboot

If everything goes as it should, the Terastation should boot from the OS disk without even touching the DATA disks!

Installing the OS hard disk

The Terastation has four EIDE channels, each capable of managing up to two disks (one Master and one Slave). This gives us a theoretical limit of 8 disks connected directly to the Terastation, but in practice we should never connect more than one disk from the same array to one EIDE channel. Because a failed disk can render unusable the whole EIDE channel failing all devices connected to that channel, causing the lost of two disks form the array instead of one.

Having this in mind we connect each of the data disks to it's own EIDE channel and the OS disk as second disk to the channel of any (should be indifferent) on the data disks.

In the author's experience it was easier to fit the OS disk into the Terastation using the following scheme:

  • OS disk: Master, connected to DISK1 port (/dev/hda)
  • Data disk 1: Slave, connected to DISK1 port (/dev/hdb)
  • Data disks {2,3,4}: Mater, connected to DISK{2,3,4} ports (/dev/hd{c,e,g}

This way the os disk can be placed between the cables going from data disks {2,3,4} to ports labeled DISK{2,3,4} and the cable connecting the OS disk with the data disk 1 and the port labeled DISK1.

Note: This placement has the big disadvantage of the OS disk being completely warped in cables without getting any ventilation at all, on the other hand, 2.5in laptop disks are supposed to work fine in that conditions. In any case, it may result better to use a 5400 rpm disk instead of a faster 7200 rpm one. And please, do experiment with other (better ventilated) placements and edit this section if you find one.

Modifying Buffalo's GNU/Linux distribution

The GNU/Linux the Terastation comes preinstalled with has a very limited set of tools, which are probably not enough to install and configure most operating systems we may want to install on the Terastation. So in this section we will install the tools we may need to make a successful installation and setup the boot loader we'll use to boot the soon to be installed OS.

Installing some system tools

As in this howto we try to avoid making any assumptions with respect to witch OS we will install, we may need to add some other tools apart form the mentioned bellow. If that's the case, a good place to get the tools are the Debian PPC stable package repositories as they stem to fit perfectly library wise.

It is always nice to have a safe way to transfer files form and to the Terastation and scp is just perfect for that. To install it we need to download the ssh stable deb (ssh_3.8.1p1-8.sarge.4_powerpc.deb) to our computer, extract the data.tar.gz:
$ ar xv ssh_3.8.1p1-8.sarge.4_powerpc.deb data.tar.gz
transfer it to the Terastation (this one is a little bit tricky as we still don't have scp on the Tretastation):
$ cat data.tar.gz | ssh root@teratsation "cat - > ssh-data.tar.gz"
And extract the files we need form this tar archive (form the Terastation):
# tar -C / -xzpvf ssh-data.tar.gz ./usr/bin/{scp,sftp}
This will extract just scp and sftp into /usr/bin directory.
From the coreutils package we need to instal the chroot and mknod programs, to install them we need to download the appropriate deb file (coreutils_5.2.1-2_powerpc.deb), extract the data.tar.gz file:
$ ar xv ssh_3.8.1p1-8.sarge.4_powerpc.deb data.tar.gz
and transfer it to the Terastation:
$ scp data.tar.gz root@terastation:coreutils-data.tar.gz
And extract the files we want on the Terastation:
# tar -C / -xzpvf coreutils-data.tar.gz ./usr/sbin/chroot ./bin/mknod
To be able to mount nfs exports on the Terastation we'll need to put the sunrpc.o module to /lib/modules/2.4.20_mvl31-ppc_linkstation/kernel/net/ and the lockd.o and nfs.o modules to /lib/modules/2.4.20_mvl31-ppc_linkstation/kernel/fs/ on the Terastation, all available in one tar archive and load them runing:
# modprobe sunrpc
# modprobe lockd
# modprobe nfs
The ncurses library is needed by many useful packages like screen, less and nano, so in case we want to use any of them we'll have to install ncurses. It is provided by the libncursesw5 deb (libncursesw5_5.4-4_powerpc.deb). To install it, first we get the deb and run:
$ ar xv libncursesw5_5.4-4_powerpc.deb data.tar.gz
$ scp data.tar.gz root@terastation:ncurses-data.tar.gz
And then extract on the Terastation:
# tar -C / -xzpvf ncurses-data.tar.gz './usr/lib/libncursesw.so*'
The GNU Screen program is a quite nice tool to have when doing any nontrivial system management (like OS installation). It is provided by the screen deb package (screen_4.0.2-4.1_powerpc.deb). To install it we need to extract the ./usr/bin/screen and ./etc/screenrc files.
less and nano 
There are more and vi, but “less is more” and nano is a little bit friendlier than vi and will serve us as an example of installing a text editor (we will need one to configure the boot loader). This packages are provided by the less and nano debs (less_382-1_powerpc.deb and nano_1.2.4-5_powerpc.deb). We will need ./usr/bin/less form less and ./bin/nano and ./etc/nanorc form nano.
In case we want our setup to use LVM (Logical Volume Manager) we just will need to install the LVM userland tools, as the kernel module is already provided by the original Buffalo installation. This tools are provided by the lvm10 deb package (lvm10_1.0.8-8_powerpc.deb). We will need to install the ./lib dir form it. (This will install all programs to /lib/lvm-10/, so to execute the foo program we will have to run # /lib/lvm-10/foo)

Tweaking /etc/init.d scripts

This setup works fine with unchanged scripts, but some scripts should be updated to take into account the changes we are making to our Terastation.

(This subsection will soon be extended.)

Installing the “boot loader”

To boot into the OS we soon are going to install, this howto proposes the use of a kernel module boot loader written by Chih-Chung Chang (jochang) for the Kuro Box. With it we can safely boot other kernels keeping the flash chip untouched.

To install it we need to download the precompiled module or compile it with the Terastation's 2.4.20 kernel headers. Put in to somewhere (like /root/) on the Terastation, write an init script and make the Terastation execute it at boot.

The is an example for the init script:

# bootnewos.sh Boots OS From /dev/hda6.

. /etc/default/rcS

MD5_RECOVERING=`/sbin/mdadm -D /dev/md5 | grep "State :" | grep "recovering"`

# We let the kernel to recover the data array
if [ "${MD5_RECOVERING}" != "" ] ; then 
	/sbin/hdparm -d1 /dev/hdb
	exit 0

#Gives us time to correct any problem
/bin/sleep 90

if [ -f /.tonewos ] ; then
	/sbin/mdadm /dev/md5 --stop
	/bin/sleep 1
	/sbin/insmod /root/loader.o kernel=/root/loader/vmlinux.hda6 cmdline="root=/dev/hda6"

To make it execute at boot:

# cp bootnewos.sh /etc/init.d/
# chmod +x /etc/init.d/bootnewos.sh
# ln -s ../init.d/bootnewos.sh /etc/rc.d/rc3.d/Sa0bootnewos.sh

We should not create the /.tonewos file until our new OS is in working conditions, but when we are ready we just need to execute:

# touch /.tonewos

Installing another OS into the OS Hard Disk

At this point we are finally ready to install the operating system we will use on the Terastation. To do this we first repartition the OS disk according to our needs and then do the actual installation.


There are some rules imposed by Buffalo's firmware/kernel we should obey while changing the partition table.

There most be a /dev/md0 for the kernel to boot form. So it is reasonable (and so far the only tested way) to leave /dev/hda1 unchanged (same size and type) so we can follow Buffalo's updates for our Buffalo's GNU/Linux installation.
This partition is not strictly required to be swap for the Terastation to boot and work, but it is highly recommended for the swap partition to be as close as possible to the beginning of the disk (outer sectors of the disk).
This is the place we'll use as our playground, so to facilitate things we can create an extended partition on it.
For some reason this needs to have the exact type/size/offset as in the original data disks for the Terastation to boot. It is important to notice that it does not matter if this partition falls outside the physical sectors of the disk.

This leaves us with an empty extended partition we can fill with the partitions we want to use.

To change the partition table the mfdisk program is available, but DO NOT RUN it as if it were the normal fdisk program as then it will replace the existing partition table with a funky useless one. Instead run:

# mfdisk -c /dev/hda

To use it in interactive mode (a la fdisk). Or just to be safe we can set an alias for it to mimic fdisk:

# alias fdisk="mfdisk -c"

So just as an example, in my case I decided to partition as follows:

# fdisk -l /dev/hda

Disk /dev/hda: 40.0 GB, 40007761920 bytes
255 heads, 63 sectors/track, 4864 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
376.4 MiB Buffalo / and where the kernel from firmware will boot to:
/dev/hda1               1          48      385528+  fd  Linux raid autodetect
133.3 MiB swap:
/dev/hda2              49          65      136552+  82  Linux swap / Solaris
The rest of the disk, will contain /dev/hdaX with X≥5:
/dev/hda3              66        4864    38547967+   5  Extended
0 MiB Raid partition that falls out of the disk:
/dev/hda4           30378       30515     1108484   fd  Linux raid autodetect
376.4 MiB To test my Buffalo / mods:
/dev/hda5              66         113      385528+  83  Linux
494 MiB for Linux /:
/dev/hda6             114         244     1052226   83  Linux
The rest of /dev/hda3 for LVM (with /home, /tmp, /usr, /var, ...)
/dev/hda7             245        4864    37110118+  8e  Linux LVM

Instaling the Operating System

In theory the Terastation should be capable of running any operating system with support for Terastation's primary hardware components and the capability of running headless.

But being a little bit more practical, it should run:

  1. PPC GNU/Linux distributions compiled without altivec support should run out of the box.
    • Gentoo Linux: works nicely (howto soon to be posted)
    • Debian GNU/Linux: should work without a glitch (so far untested)
    • Linux Form Scratch: should work without a glitch, but may be hard to setup if cross compilation is needed (so far untested)
    • foobar Linux: should work without any problems, but some minor tweaks may be required (so far untested)
  2. Linux/uClibc almost works but has some issues with (I think) the dynamic loader.
    • Gentoo Embedded: have problems
    • uClibc buildroot: the exact same problems as with Gentoo Embedded.
  3. *BSD may work as the CPU, EIDE, NIC and USB chips are supported, but may need some (trivial) changes in the “boot loader” we are using to be able to boot non Linux kernels.
    • NetBSD: should be the best bet (so far untested)

At this point we follow the instructions associated with the operating system we decided to install.

Tweaking and booting the newly installed OS

In this section we will do some minor tweaks specific our the Terastation setup.

Addressing the “watchdog”

There is another catch (probably) form how Buffalo's kernel is configured that we need to address before we boot into our new OS. And the catch is that even so there is no real watchdog on the Terastation, the kernel appears to have a watchdogish something connected to /dev/ttyS1 and managed by the mc_ctld program (the program that should manage an UPS connected to the serial port). So this should only be needed if we will be booting a Linux based OS with the stock kernel.

To work around this catch we can copy the mc_ctld program with the files it uses to an empty directory and execute it chrooted on boot.

Files that we need to copy:


And add something like the following to be executed by the new os at boot:

/usr/bin/chroot /empty/dir/we/copied/to /usr/sbin/mc_ctld

It may also be necessary execute on boot (to be confirmed):

echo -n "aa8041" > /dev/ttyS1
echo -n "qqqq" > /dev/ttyS1

And on halt/reboot (also to be confirmed):

echo -n "rrrr" > /dev/ttyS1
echo -n "EEEE" > /dev/ttyS1

SSH Server

As the Terastation by its nature is headless, it is crucial to configure the new os to start the SSH server at boot.

Getting ready to boot

If we are going to boot into a Linux based OS with the stock kernel we need to hexedit it to make it boot from the device we want it to boot, else we should skip to the next subsection.

To accomplish this we first need to extract the kernel as shown on Firmware Update and then replace the three occurrences of root=/dev/{md0,ram0,hda1} with root=/dev/hdaX, where X is the partition number we want to boot form (6 in our example). And then transfer it to the Terastation.

It also may be wise to copy the /lib/modules directory for our newly installed Linux.


First we need to unmount all the unneeded mounts and stop all raid devices we can (the only active should be /dev/md0):

# umout /unneeded/mount
# mdadm /dev/mdX --stop

(we can see what is mounted with "mount" and the active raid arrays with "cat /proc/mdstat")

And finally we issue the magic command to boot the new OS:

# sync && insmod loader.o kernel=the_kernel cmdline="root=/dev/hdaX"

(the command line will be ignored by the stock kernel, but it is nice to specify it for consistency's sake)

And if everything goes well, in a couple of minutes the Terastation should come up booted into the os we just installed!

Post first boot configuration

There is just one tiny thing that needs to be done. And the thing is that when we power the Terastation on it'll first boot into /dev/md0 witch is supposed to consist of four identical partitions mirrored in raid1, but we changed that supposition several sections ago, so we should update /dev/md0's superblock (again, just for consistency's sake). If we booted Linux, we can do it executing:

# mdadm --manage /dev/md0 --stop
# mdadm --create /dev/md0 -l 1 -f -n 1 /dev/hda1

Also at this point we may want to activate the auto boot into new os feature creating the .tonewos file in Buffalo's /.

Setting up the DATA hard disks

At this point we can setup the data disks in any way we think will suit better our needs. As the Terastation should care about nothing more that the OS disk when it boots.

But in reality, this may not hold true as the initrd image that the kernel from firmware boots into may try to access our DATA disk, so some esoteric or close to stock configurations may confuse it.

Open questions

  • Why does it needs the 4th partition?
  • What does the mc_ctld program really do?
  • What configuration and patches should be applied to the stock 2.4.20 kernel sources to compile a bootable kernel?
  • What configuration and patches should be applied to the vanilla 2.6.x kernel sources to compile a bootable kernel?