Enabling Rsync on a Linkstation Mini

From NAS-Central Buffalo - The Linkstation Wiki
Revision as of 11:50, 15 February 2009 by Walton (Talk | contribs)

Jump to: navigation, search

Contents

Purpose

This page tells how to set up the LinkStation Mini to do backups using rsync and ssh. The intended audience are readers who know Linux or some other UNIX, at the root level, but do not know about LinkStations or single chip computers. Note that setting up the Mini for ssh and rsync will void the warranty, and that no warranty is implied or liability accepted by the authors of this page.

[The other purpose of this page is to `learn by writing'. I've already got my Mini set up for rsync, but writing this should close the loose ends. -User:Walton]

Do You Really Want Rsync?

I measured the Mini transfer speed as 1.2 megabytes per second doing scp/rsync, which is not exactly blazing, but it did 11 megabytes per second doing ftp of a large file (this is the limit of my old router at 100Mbits/sec so with a faster router the speed may be faster: the Mini did 30 megabytes per second cp'ing from disk to /dev/null). I suspect the reason for the slow scp/rsync is that the CPU is slow at encrypting.

If you just want to transfer single large files, ftp would be better. However, if you want to transfer complex directory structures, there are some reasons to prefer rsync. The closest thing to rsync in the ftp world that I know is the lftp mirror command, over which rsync has the following advantages:

  • Rsync preserves file and directory modification times, whereas ftp cannot set modification times on the remote system and does not preserve timestamps.
  • Rsync preserves symbolic links, whereas ftp cannot create symbolic links on the remote system and does not preserve them.
  • Rsync has a -vnc option that allows you to check whether your backup matches your original bit-for-bit by computing checksums of the files.

Also, when lftp mirror retrieves a large complex directory system stored by rsync, lftp achieved only a 2.4 megabyte per second transfer rate, and not the 11 megabyte per second rate for large files. My guess is that the Mini is very slow at accessing lots of directory structure: I've seen this in other situations. Perhaps because the xfs file system has a large block size and the Mini only has 128MB RAM and can therefore only cache a few blocks.

Lastly you should always consider getting flash memory stick drives for backup before buying any backup disk. You can do rsync to a flash drive very easily. (Hints: you will likely want to `dd if=/dev/zero of=/dev/YOURFLASH' and then `mkfs -type ext3 /dev/YOURFLASH' to get a good file system on a flash stick drive.)

LinkStation Mini Description

The LinkStation Mini is a very small computer running a fairly full Linux system, in so far as backup is concerned. The manufacturer supports FTP and SAMBA servers (and also LPD for printers), but not rsync or ssh, probably because as RedHat discovered, any attempt to support the latter would result in too many calls to techsupport. However, all the software to use ssh and rsync is already installed, and only a few configuration files need to be edited to make these work.

See the previous section for some timing info.

I put a power meter on the Mini and measured 6-7 watts when it was not transferring data and 7-8 watts when it was transferring data; most excellent.

The LinkStation Mini hardware is:

   ARM9 CPU (266 BogoMIPS: 15 times slower than a 2Gz Intel)Datasheet
   2 hard drives, either 250GB or 500GB each (30 Mbyte/sec transfer to /dev/null)
   128 MB RAM
   one ethernet port (can run at 1 Gb/sec)
   controllers for 2 USB ports (only one connector provided - hub might allow 2 ports)
   controller for several serial ports - no external or internal connector provided
   256KB flash memory - contains uBoot bootstrap loader
                      - not used by Linux or during normal operation

(More details at Disassemble the LS MINI.)

As delivered and then reconfigured to use RAID1, a 250GB disk Mini has the following disk partitions:

   Mounted On         Format        RAID Device         Hardware Devices           Size
   
   /boot              ext3          /dev/md0            /dev/sd{a,b}1              1GB
   /                  xfs           /dev/md1            /dev/sd{a,b}2              5GB
   swap area          swap          /dev/md10           /dev/sd{a,b}5              1GB
   /mnt/array1        xfs           /dev/md2            /dev/sd{a,b}6              236GB

In addition, 15MB of RAM is used as a RAM disk for /tmp. The flash memory which is reported to be only 256KB is not used by Linux. The /boot partition has all the firmware and possibly can be used to restore the machine to its delivered state, possibly by using the web Maintainance/Initialization page `Reset LinkStation Configuration to Factory Defaults'.

/boot also contains a save of the `current configuration', which includes files such as /etc/passwd, /etc/shadow, etc in the file /boot/config_save.tgz.

The XFS file system format is a full featured Linux file system originated by SGI that is particularly well suited to file transfer operations.

As delivered the Mini has 2 passworded users: admin, whose password you can change using the web page, and root, whose password is presumably known only to the manufacturer.

ACP

The Mini responds to a protocol called ACP that allows you to send commands to the Mini and get back responses. It appears that the Mini always responds to this protocol, except while it is booting up, so this is the root protocol, as it were.

Fortunately there is a program called acp_commander that permits ACP commands to be sent from your Linux computer's command line. To use it, download

   http://downloads.nas-central.org/TOOLS/ALL_LS_KB_ARM9/ACP_COMMANDER/acp_commander.jar

and try it out.

First, you need to set up your Mini and connected to the ethernet and establish a fixed IP address for it (using your router). Then you open the Mini's web page using the IP address, set the Mini hostname using the Basic subpage, and if you like, tell your own computer about the hostname you have choosen, say by editing your /etc/hosts file or equivalent. I gave my Mini the name linkstation1.walton (walton is my router's domain name) and will use this name hereafter in place of the Mini's IP address.

IMPORTANT: The Mini comes without RAID turned on, so the full 500GB or 1TB is available. If you want to use RAID1 so the two disks back up each other, you should change to RAID1 NOW, before you start changing other Mini configuration. See the Mini Disk Management/RAID Setup subpage and click on the tool icon.

Next you need to turn off your own computer's firewall (temporarily). I tried punching a hole in my firewall but it did not work. If you do not turn off your firewall you will get a message containing:

   A SocketTimeoutException usually indicates bad firewall settings

Now you can execute on your own computer

   java -jar acp_commander.jar -t linkstation1.walton -c 'ls -l'

This will execute in the Mini the Linux `ls -l' command with root as the current user and / as current directory. Just to test.

If this works you can execute

   java -jar acp_commander.jar -t linkstation1.walton -o

at which point your manufacturer's warranty is gonzo, at least a little bit. This zeros the root password so you can log in as root, and also executes `telnetd' which starts the telnet daemon, so you can now telnet into the Mini.

You can now turn your firewall back on. You are done with ACP.

However, if you want to play before you change anything, you can do

   java -jar acp_commander.jar -t linkstation1.walton -s

which allows you to type shell commands at the Mini. However, each command is executed in a separate instance of the shell, and interactive commands will hang the Mini ACP interface (you can recover by using the Mini web page to reboot).

Some more info is in the ACP Commander README;

   http://downloads.buffalo.nas-central.org/TOOLS/ALL_LS_KB_ARM9/ACP_COMMANDER/README

Many thanks to Georg for writting acp_commander.

Configuring the Mini for SSH and RSYNC

At this point you can log in as root using

   telnet linkstation1.walton

(remember, linkstation1.walton is my abbreviation for the Mini's IP address).

Look around. Get comfortable. Decide you really want to go through with this.

Also, use the `passwd' command to set a secure root password. Be careful what you choose: this is a backup computer and you may not want to have your own computer's passwords flying around the ether, so you may want to use a special password for the Mini.

To enable ssh you need to

   (1) create an /etc/init.d/sshd.sh file
   (2) test running sshd
   (3) edit the /etc/init.d/rcS file so the above file is executed when the Mini reboots
   (4) test rebooting
   (5) edit the /etc/daemonwatch.list file so sshd will be restarted if it dies
   (6) test killing sshd
   (7) change permissions on /etc/profile
   (8) create the /var/log/lastlog file

(Note: all this comes from Open Stock Firmware: many thanks to the authors of this page. Also: my Mini firmware is 1.04 which appears to be a later version than the 1.11a firmware mentioned in this reference, which is for an older LinkStation model.)

The sshd.sh file contents are

   #! /bin/sh
   # Start/stop the SSH daemon
   #
   #
   
   test -f /usr/local/sbin/sshd || exit 0
   
   # this is used by deamonwatch
   ACTIVE_FILE=/var/run/active_sshd
   
   case "$1" in
   start)  echo -n "Start services: sshd"
           /usr/local/sbin/sshd -f /etc/sshd_config 2> /dev/null &
           touch $ACTIVE_FILE
           ;;
   stop)   echo -n "Stop services: sshd"
           killall sshd
           rm -f $ACTIVE_FILE
           ;;
   restart)
           $0 stop
           $0 start
           ;;
   *)      echo "Usage: $0 start|stop|restart"
           exit 1
           ;;
   esac
   exit 0

(Note: the firmware on my Mini is 1.04, and supports daemonwatch. Earlier versions of the firmware might not support daemonwatch, in which case see Open Stock Firmware. To test for the presence of daemonwatch execute `which daemonwatch'.)

You can create sshd.sh by using the vi editor. A bit easier is to execute

   cd /etc/init.d
   cat >sshd.sh
   <copy the above file contents here using your mouse>
   <control-D>
   vi sshd.sh
   <remove the 4 spaces beginning each line: YOU MUST DO THIS FOR FIRST LINE>

After creating sshd.sh set its mode by `chmod 755 sshd.sh'.

Now test ssh. Just

   cd /etc/init.d
   ./sshd.sh start

and try to log into your Mini by executing in another window of your computer

   ssh root@linkstation1.walton

and giving your Mini's root password. This should work. You may get an error message concerning /etc/profile which you can ignore. If things do not work, examine the section of Open Stock Firmware that discusses `Config' (or more explicitly, /etc/sshd_config).

You are not done because sshd will not yet restart on reboot (nor will telnet). To fix this you must add `sshd.sh' to the right spot in the /etc/init.d/rcS file. Do

   cd /etc/init.d
   cp -p rcS rcS.bak
   vi rcS
   /step3
   <add ` sshd.sh' to the end of the next `for' line>
   :wq

That is, rcS is executed to start the system during reboot, and when it gets to step3 it will now execute sshd.sh along with everything else it does.

Now try rebooting the Mini using the web page Maintenance/Restart `Restart Now'. Note that rebooting the Mini is a rather slow operation. Once its rebooted log in again using ssh. Telnet should no longer work (you could restart it by executing `telnetd', which if necessary you can do using acp_commander `-c telnetd').

Next you need to tell daemonwatch about sshd.sh so it will restart sshd if that should die. You need to add a line to /etc/daemonwatch.list. Do

   cd /etc
   cp -p daemonwatch.list daemonwatch.list.bak
   vi daemonwatch.list
   <add the line below to the end of the file>
   :wq
   cd /etc/init.d
   ./daemonwatch.sh restart

The line to add is:

   /var/run/sshd.pid             /var/run/active_sshd    /etc/init.d/sshd.sh restart


Now test. Do

   killall sshd

Your ssh connection should die. But after a minute or so you should be able to reconnect. If you cannot you will have to reboot as above to restart sshd.

Lastly a bit of cleanup:

   chmod 755 /etc/profile
   touch /var/log/lastlog
   chmod 644 /var/log/lastlog

This permits /etc/profile to be properly used when any user logs in via ssh, and gets rid of `lastlog does not exist' error messages in /var/log/messages.

Rsync should now work! You can test it by executing on your own computer:

 cd ~
 mkdir buffalo
 cd buffalo
 rsync -av root@linkstation1.walton:/boot/ boot
 <enter your root password>
 file boot/*

We will talk more about the contents of the /boot directory below.

Adding Users

It seems to me best to use the Mini web page to add users, home/shared directories, groups, etc. as then the standard Mini software will know about them and they will be usable via FTP, SAMBA, etc. Doing this works but you have to do a bit extra for rsync.

First, what do the Mini web pages do?

Their configuration information base is kept in ASCII /etc/melco/* files. So when you make changes using the `Shared Folders Setup', `Group Management', or `User Management' pages you are modifying these files.

In addition to modifying the above information base:

  • Users have a line in /etc/passwd and /etc/shadow. However, the home directory of all users (in /etc/passwd) is /home. Also, all users (except the builtin `guest' user) are added to the hdusers group in /etc/group and have the hdusers group number as their login group in /etc/passwd.
  • Groups have a line in /etc/group listing all the users they contain.
  • Shared files are located in /mnt/array1 (this might be different if you are not using RAID1). However, this file has owner root, group root, and protections drwxrwxrwx.

So to add an `rsync user' named `ruser' do the following:

  1. Using the `User Management' page add `ruser' as a user.
  2. Using the `Group Management' page add `ruser' as a group and put the user `ruser' in this group (normally as its only member).
  3. Using the `Shared Folders/Shared Folders Setup' page add `ruser' as a shared folder, enable it for FTP and read/write, enable access restrictions, and give the user `ruser' read/write access. WARNING: do NOT give any group `read only' access if the user `ruser' is in that group (see below).
  4. Edit the `ruser' line in /etc/passwds to replace `/home' by `/mnt/array1/ruser' and to replace the hdusers group number (which is 100 on my Mini) by the `ruser' group's group number (the group number is the second of the two numbers in the line, and the `rusers' group number is in the /etc/group file).
  5. Execute
   cd /mnt/array1
   chown ruser:ruser ruser
   chmod 750 ruser

Given this ssh ruser@linkstation1.walton will land in the home directory /mnt/array1/ruser running a shell with with user ruser and group ruser. FTP (if the service is enabled via the `Shared Folders/Service Setup' page) will land in the same home directory running an ftp process also with user ruser and group ruser. So all files created in the Mini by ruser will have user ruser and group ruser.

The mini FTP seems to have the strange property that if you use `Shared Folders/Shared Folders Setup' to make a directory both read-write to you using your user name and also read-only to you using one of your group names, the latter takes precedence for FTP, and your FTP access will be read-only. Also mysterious is that the FTP server on the Mini seems to be ignoring /etc/proftpd/proftpd.conf in this matter. So you must NOT make your home directory read-only to your group using the Mini web page, if you ever want to use FTP to write in your home directory. This seems to be a property of the vanilla Mini, and is not related to anything done above.

Security Settings

There are some file permissions that should be changed if ssh is really going to be much used with your Mini. These may not be important, but for the record here is some info.

Some useful commands are

   find / -name walton3 -prune -o -type l -o -perm 666 -o -exec ls -ld {} \;
   find / -name walton3 -prune -o -type l -o -perm 777 -o -exec ls -ld {} \;
   find / -name walton3 -prune -o -group root -o -exec ls -dl {} \;

which will find the system files that have group other than root or that have the permissions 666 or 777. Here the `-name walton3 -prune -o' serves to exclude the files in my walton3 directory which would have produced a ton of spurious listings, since I was already using my Mini for backup and there were plenty of non-system files in that directory.

Based on the output I got I did the following:

   chmod 755 /home /mnt /mnt/array1
   chmod 644 /etc/pam.d/sudo/sudo.pam

Time

The best way to keep the clock at the right time is to use the network time servers. On the Mini this can be done via the `Basic' page by enabling the NTP Server. I used the `0.centos.pool.ntp.org' server because my computer runs CentOS 5.2 and uses that server, and I set the system to synchronize once a day.

Shutdown

There is no way to shut the Mini down from a web page (at least with my 1.04 firmware). You can shut it down by turning the power switch on the back to OFF. This causes the Mini to execute an orderly shutdown: nothing happens for a few seconds, and then the lights will blink for quite a few seconds before the Mini goes dark.

You should not set the power switch to `auto' unless you have the appropriate software, as the Mini may power up correctly, and then after a while power down for no obvious reason.

Power Failure

Power fails in my neighborhood several times a year, so I wanted some assurance the Mini could recover from a power failure. So I unplugged it, waited 10 seconds, and plugged it back in. After it finished restarting, I did a `Disk Management/Disk Check' which runs fsck on /mnt/array1, and `Raid Setup/Begin Immediate RAID Scan' (which takes over an hour, during which time you can use the Mini normally), with no errors detected. -:)

How Boot Works

This section currently just has some information tidbits that hopefully will eventually lead to being able to boot off something other than the Mini hard disk.

Using tcpdump and nmap I determined that the Mini never sends ethernet packets or listens for such until the Linux kernel has been booted from disk (with 99% certainty), no matter what.

Hitting the top button during boot does nothing. However, holding it down when turning power on causes the Mini to go into a special state where it is rapidly blinking just its topmost blue LED. If you release and repress the top button, it goes into a different state in which the topmost blue LED and the LinkStation Logo blue LED blink rapidly, and then after a while just the Logo LED blinks rapidly, which is the normal boot state, and the Mini reboots normally. During this time no packets are sent on the ethernet and the Mini does not listen on sockets.

So if the flash boot pays any attention to the world outside the Mini, it must be the USB port.

References:

   U-boot bootloader
   LS Mini: Serial Port Output - Boot-Log
   LS Mini: Serial Port Output - U-Boot


Result of fdisk -l /dev/sda:

   Disk /dev/sda: 250.0 GB, 250059350016 bytes
   255 heads, 63 sectors/track, 30401 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sda1               1         125     1004031   83  Linux
   /dev/sda2             126         748     5004247+  83  Linux
   /dev/sda4             749       30401   238187722+   5  Extended
   /dev/sda5             749         873     1004031   82  Linux swap
   /dev/sda6             874       30295   236332183+  83  Linux

Result of rsyncing /boot to my CentOS Linux computer and doing file * on the contents (file on the Mini is too ignorant):

   boot/conf_save.tgz:                  gzip compressed data, from Unix, max compression
   boot/hddrootfs.buffalo.updated.done: gzip compressed data, from Unix, last modified: Mon Jun  9 00:19:12 2008
   boot/initrd.buffalo:                 PPCBoot image
   boot/lost+found:                     directory
   boot/u-boot.buffalo:                 data
   boot/uImage.buffalo:                 PPCBoot image

To Do

Investigate how to connect to the bootstrap program uBoot (I could not get nc to work the first time I tried), how to load new firmware using Linux (no Window's system available), how to unbrick a Mini with blank disks. Also adding USB flash drives, and possibly using them as removable disks that substitute for / and /boot.