Difference between revisions of "Terastation Recovery"
m (→EM Mode)
m (→Network Will not Start)
|Line 231:||Line 231:|
=Network Will not Start=
=Network Will not Start=
Revision as of 16:20, 29 September 2008
THIS ARTICLE IS STILL UNDER CONSTRUCTION
- 1 Introduction
- 2 Systems Covered
- 3 Booting Process
- 4 Failure Symptoms
- 5 Replacing the Hard Disks
- 6 Telnet Enabled Firmware
- 7 EM Mode
- 8 Using the Firmware Updater
- 9 Flash Error
- 10 Boot Does not complete
- 11 Network Will not Start
- 12 Clearing Down the System Partition
This article is about recovering TeraStation systems that are failing to load correctly to get them back to a working state
Before attempting to do any of the articles described in this article you want to first read the wiki article on Terastation Data Recovery.
It covers several types of failures:
- A single disk fails
- Failures to start the boot process
- The system fails to complete the boot process
- The system appears to boot OK, but the Web GUI and File Shares are inaccessible.
The systems that are currently covered by this article include all the original PPC based Models:
- Original TeraStation
- TeraStation Home Server
- TeraStation Pro v1
The following ARM based systems are not currently covered:
- Terastation Live
- TeraStation Pro v2
To understand the recovery processes that are described here it is very advantageous to have an understanding of how the boot process works.
It runs in various stages:
- A boot loader is loaded from flash
- The boot loader now runs a Linux kernel and a very basic Linux system from disk, with the system files being run from a RAM disk. The disk arrays are started.
- The Linux kernel is switched to running from the System are (partition 1) of the hard disk.
- The system services such as Samba file sharing, the Buffalo Web GUI and (if relevant) File Transfer service, Printer Service, Telnet and the Media server services are started.
Failures can occur at any of these stages and the recovery process to be followed will be dependant on identifying at which stage he boot process failed.
The Original and the Home Server TeraStations report problems by lighting up a Red DIAG light. This light then flashes a number of times to indicate the type of problem detected.
The TeraStation Pro v1 has a LED display instead, and if problems are encountered it displays a Exx type error code.
If the system fails almost immediately it normally means that there is a flash based problem. These are by far the hardest to recover from.
If the system fails after around 30 seconds or more into the Booting process, then this normally means that there is a problem with the System area on the Hard Disks. This type of problem should almost always be recoverable as long as one is very careful about the steps that are carried out.
It is also possible for a single disk to fail. If you are using a RAID5 or a RAID1 array then this situation can be recovered without losing any user data. This is described in the section on replacing a single drive
Replacing the Hard Disks
This section covers using Hard Disks of up to 500Gb which is the maximum size supported by the standard Buffalo software on the PPC based TeraStations. If you want to try and use disks that are larger than this size look at the article on TeraStation and Large Hard Disks.
Replacing a Single disk in a RAID5 array
If a single disk has failed in a RAID5 array, then the system can be recovered without reasonably easily without any data loss. On the Original and Home servermodels a disk failure is indicated by the Disk Status light turning Red. Note that this light is also used to indicate a Disk Full status (more than 90% of disk used) so if all disk Status lights turn Red this is almost certainly the reason. At this point the TeraStation will continue to operate normally, and if you go into the Web GUI you will be told that one or more of the disk arrays have "Low Disk Space". If a disk has failed then you will be told that instead.
The first thing is to decide if the disk really has failed - or whether some sort of data corruption has made the TeraStation think that the drive has failed. Spurious reports of disk failures seem to be quite frequent, and are particularily likely to happen after a power failure. The best way to do this is to attach the drive to a PC and run the manufacturer's disk diagnostic software to check the drive. If this is not possible, then you can simply clear down that drive and see if the disk comes back to life.
Preparing a Drive
There are a few things you need to check before attempting to use a drive:
- If you have a new drive and it is IDE based is to ensure that the jumpers are set correctly. Most new drives come with the jumpers set for the drive to operate in "Slave" mode on the assumption that they are being added to a working PC. The TeraStation requires that the drive be set to operate in Single/Master mode. There should be a digram on the drive showing what the correct jumper setting are for the different modes of working. For SATA drives as used in the TeraStation Pro this is not an issue.
- Removing any existing partitions. Most new drives come without any partitions and would probably work immediately. However some manufacturers try to be "helpful" and provide their drives already partitioned and formatted. If you have a drive in this state (or you are trying to re-use a previously used drive) these partitions need to be removed.
Using a PC to remove partitions
For most users using a PC is probably the easiest way to proceed. This involves:
- Remove the drive from the TeraStation and attaching it to a PC either internally or via an external disk enclosure. For the IDE based models (Original and Home Server) there are instructions on how to open the case to get at the drives at the back of the manual. For the Pro v1 using SATA drives it is much easier as the door at the front can be opened and the drives simply released and slid out.
- Run the Windows Disk Manager software to remove all the partitions on the drive.
- Replace the drive in the TeraStation.
- Switch on the TeraStation.
Using the TeraStation to remove partitions
This option is only possible if you already have telnet enabled firmware installed on your TeraStation.If you have got telnet enabled firmware installed on your TeraStation:
- telnet to the TeraStation, and log in with a user with root privileges (e.g. myroot)
- Run the disk partitioning software using a command of the form
mfdisk //dev/devicewhere device is hda/hdc/hde/hdg for the four drives in IDE based models or sda/sdb/sdc/sdd for those in the Pro model. Make sure you get the right device name as otherwise you will lose all your data.
- Delete all partitions on the disk
- Save the results back to the partition table. This will exit the mfdisk program with an error message about being unable to resync procee the disks.
- Use the"reboot" command to stop/restart the system
Recovering the Array
- The system should now come up without issues, but when you go into the Web GUI it will tell you that the RAID array has been damaged.
- Select the RAID array and you will be taken to the screen where the array is repaired. You should be able to select the drive that failed (all the others should be greyed out) and given an option to repair the array.
- If you select this option then (depending on the firmware release) you will be given a screen in Japanese asking you to enter a displayed number as a confirmation. Enter the number and then select the left-most button to proceed.
- The system will put up a screen about recovering the array. At this point the new drive will be partitioned and formatted, and added to the RAID array. While this is going on (which normally takes a few minutes) all file access to the TeraStation is suspended.
- Once the array has been "fixed" the File Serving services will be resumed, and the the process of Resyncing the array will start. During this stage file accesss is allowed, but performance is degraded. This stage can take a long time with the length of time depending on the size of the array - as an example a 1Tb array takes about 5.5 hours, and other sizes will be pro-rata.
When replacing the drive is is recommended if at all possible that you replace the drive with another of the same make and model. Some people have success with using a different make/model that is of the same size (or larger) so it isn up to you whether you try this.
When you are going to replace the drive, then the replacement should be unformatted wtih nopartitions on it. If it is an IDE model (the Orignal and Home Server models), then make sure that the jumpers are set it make it the Master/Only drive. Many drives come with the jumpers set to Slave on the assumption that they are being added as an additional drive to an existing PC. If the jumpers are incorrectly set to the Slave position then the drive will not be recognised by the TeraStation.
Replacing All the Hard Disks
The most likely reason for replacing all disks is that you are intending to upgrade the capacity of your TeraStation.
If the new disks are of size 500Gb or less then proceed as follows:
- Replace all existing drives with 4 identical drives. If they are IDE drives ensure that the jumpers are set for them to be Master/Only drives. Also make sure that there are no existing partitions on the drives.
- Switch the TeraStation on and it will come up in EM-Mode ready to accept a reflash.
- You will probably need to reflash the system twice - once to partition/format the drives, and the second time to copy the system software to the new hard disks. If during the flashing process you get a timeout message, then quit the Flashing process. Leave the TeraStation alone until all the disk activity lights have stopped, and then start the second flash. When flashing it is a good idea to use the Debug Mode of the firmware updater, and set the option to "clear user-config".
- At the end of this process the system will be operating as if it was a brand new TeraStation at the new disk capacity.
If you want to use disks that are larger than 750Gb then see this articl on TeraStation and Larger Hard Disksyou need to work around the fact that the PPC based TeraStations use a 2.4 Linux Kernel that is limited to 2Tb in a single file system. You can work around this as described below:
- Start by flashing just as described above, but make sure that you use a telnet enabled firmware release. Such releases can be obtained for PPC based TeraStations from Itimpi's Web site.
Telnet Enabled Firmware
TeraStations internally run the Linux operating system. Buffalo hide this from the average user providing the system "packaged" and with a browser based GUI to control and configure iy. Telnet enabled firmware allows users to log in using a telnet client to control and manipulate the system at the Linux command line level.
This allows users to do things like:
- Configure the system at a more detailed level than allowed for by the Buffalo GUI.
- Upgrade components of the underlying software to newer versions with additional feature and/or bug fixes included.
- Install new applications to extend the functionality of the Terastation.
- In the event of any problems being encountered allow for a level of access that gives more chance of recovering user data without loss.
The changes described in this article require the use of a telnet enabled release. Hopefully the instructions provided are detailed enough that users can carry out the steps provided without needing much linux knowledge.
The standard software supplied with buffalo PPC based TeraStations does not provide for telnet access. Telnet enabled releases of firmware corresponding to virtually all Buffalo firmware releases can be found at itimpi's website. These are identical in functionality to the corresponding Buffalo firmware releases - the modification to add telent functionality being trivial. This means they will have exactly the same bugs (if any) as are found in the Buffalo releases.
The itimpi firmware releases are the ones that have been used while preparing this article. Firmware from other sources should work fine as long as it is telnet enabled.
To use telnet you need a telnet client. A rudimentary one is included with Windows which you can invoked by typing a command of the following form into the Windows Run box:
A freeware one called PuTTY is recommended as a much better alternative. In addition to the standard telnet protocol PuTTY also supports the more secure SSH variant (although additional components need installing at the TeraStation end to support this).
In the forums there is a lot of discission about EM mode (also known as Engineer or Emergency Mode). If the TeraStation is in this mode, and its network connection is working, then you will find that it will have "EM" as part of the node name.
The reason why EM mode is considered important is that in this mode the TeraStation has successfully booted the Linux Kernel from Flash, and is running from a RAM disk (also loaded from Flash). In EM mode the additional services such as File Sharing are not available, but the Linux system running at that point is sufficient to allow for updating the flash, and writing System files to disk.
The TeraStation will go into EM mode anytime it detects that the System Partition on the Hard disks does not appear to contain a working Linux system. In some circumstances, however, the Hard Disk appears to contain a working Linux system, but some of the files on the System partition have been damaged in some way. In such cases special actions need to be taken to force EM mode.
Using the Firmware Updater
Just to be safe, it is a good idea to back up your data before attempting this upgrade. If things go well your data will survive fine, but if anything goes wrong there is a small chance that your data will no longer be accessible.
The computer running this firmware update program must be connected to the TeraStation’s network by a wired connection. A wireless connection will not do! Disable any firewall software on the PC before running the firmware update. Unzip the firmware in the file you downloaded into a folder on your PC.
To apply this update you need to run the firmware updater program that is the folder you created when unzipping the downloaded firmware. The name of this program can vary according to what version of firmware you downloaded, but should be the only executable program in the folder. The firmware updater program runs fine on Windows 2000 and Windows XP SP2. For XP SP1 and other versions of Windows, set the program to run in Windows 2000 compatibility mode Do this by right-clicking on the updater (e.g. HD-HTGL FWUpdate.exe) icon, selecting Properties, and choosing the compatibility tab.Put a check next to Run this program in compatibility mode for: and select Windows 2000 from the drop-box if it’s not already there.
Execute the application file (e.g. HD-HTGL FWUpdate.exe).
This will come up with a dialog like the following:
and report some basic information about the TeraStation including its current firmware version. <p> If you have multiple teraStations, then select the appropriate one from the drop-down list. Note that only TeraStation models for which this update is valid will be listed). Click on the Update button (sometimes labelled as Renew Firmware). On Japanese versions of the Updater program this is the leftmost button on the main screen. On the next window it will ask for an Admin password. By default the username is:Admin and the password is: password. Of course, if you’ve changed the Admin password on the TeraStation, use your new one! The firmware update will take 3 to 5 minutes to complete and while it is running a number of dialogs will be displayed informaing you of progress. If you never get the OK button, or you get a failure or error message, you may want to try the steps below:
- On the computer that you are doing the firmware update from, make sure that any software firewalls (such as Windows Firewall,ZoneAlarm, Black Ice, etc.) are disabled.
- Make sure that your computer is connected to the network with a wired Ethernet connection, not a wireless one.
- If the firmware update still fails, try a different setup:
- Connect the TeraStation directly to your computer, with either a regular Ethernet or crossover cable, directly from the TeraStation’s RJ-45 Ethernet port to the RJ-45 port on your computer’s network interface card.
- Change the IP address of your computer to 192.168.11.100 with a subnet mask of 255.255.255.0.
- Reset the TeraStation by holding down the Init button for 30 seconds. Power down the TeraStation, then turn it back on.
- Ping 192.168.11.150 to make sure that your PC is able to communicate with the TeraStation.
- Click on Start, then select Run.
- In the run window enter "cmd" and press OK. This will give you a Command Prompt window.
- Type ping 192.168.11.150 and press the enter key.<p>If your connection is good, you’ll see the words “Reply from” in response to the 4 attempted pings.<p> If the connection is bad, you’ll likely see “request timed out” instead.
If that happens, something is wrong with your Ethernet connection.
Check your Ethernet cable, reset the TeraStation again, and make sure that there are no firewall programs running on the computer (and that includes the Windows XP firewall!).
Restart the TeraStation and try pinging again until you get a successful ping.
Then go to step 5.
- Rerun the firmware update.
- Once the update has completed, reconnect your TeraStation and PC to your network.
- Change your IP address back to “obtain an IP address automatically”. Windows XP users will need to also make sure that DNS is set back to “obtain DNS server address automatically”.
- Run the IP Setup Utility again and setup your IP address to work with your network
- You may want to log back into the TeraStation to make sure that the unit is setup the way you want it. After doing the reset, all files and folders will remain but any user or folder
If you are having problems you might want to run the Flash Updater in Debug Mode
To do this the first thing you need to do is to check the .INI file (normally called NASUPDATER.INI) that is the same folder as the executable program. You can load it into any text editor such as Notepad. Check that it includes the following lines:
[SpecialFlags] Debug = 1
If their is already a [SpecialFlags] section in this file, then merely add the Debug line.
Once you have done this, when you start the program for flashing it will present its initial screen along the lines indicated earlier.
If you now click on the yellow rings at the top left of the screen you are able to select the Debug option which comes up with the following dialog:
which allows you to control elements of the flashing process.
The default options fora standard flash will be pre-selected. Options you might want to consider altering are:
- BOOT: This indicates that the Boot area of the Flash should be renewed
- KERNEL: This indicates that the copy of the Linux kernel in Flash should be updated.
- HDD: This indicates that the main Linux system held on the Hard Disk should be updated.
- Do Not Check Version: Do not carry out any firmware version checks. Normally you are only allowed to update to a higher version than that already installed. Setting this option allows one to downgrade versions.
- Rebuild Partition Table: This Clears down all partitions on the disk, and then rebuilds the standard Partition table and formats these partitions.
- Delete user-Config: This clears down the user config information that is stored in flash while the flashing is in progress, and then restored to the hard disk on completion. Setting this option will return all settings to factory defaults.
When you have the settings as you want them to be, press the OK button to return to the main screen ready to start flashing.
A Flash Error is indicated by:
- TeraStation Pro: Error E04 on LCD Panel
- TeraStation Original or Home Some Sever: 5 blinks every four seconds of red Diag light
The full list of Diag LED codes can be found [url=http://buffalo.nas-central.org/index.php/Hardware_LEDs here].
Flash errors are actually very rare unless they occur just after you have done a firmware upgrade and it turns out that you have a defective flash chip so that the update failed.
In practise it has been found that a good rule-of-thumb is that if the above error occurs very rapidly (i.e. within about 5 seconds) after power on then it probably really is a Flash error. If it takes much longer (greater than 30 seconds) and the system appears to start its normal booting sequence before this error is given then it is actually much more likely that the system files on the hard disk have become corrupted.
To recover from a genuine flash error requires use of the JTAG method of reqriting the flash. This is covered in a number of wiki articles if you want to attempt to do this yourself. There are also a number of forum members who are prepared to attempt JTAG recoveries on behalf of members. If you are interested in availing yourself of such a facility you should post a forum message giving your location so that members who are geographically closest to you can be identifed.
To recover from the situation where the system will not boot look at the section below.
Boot Does not complete
Network Will not Start
This is the situation where the system appears to boot OK, but at the end of the Boot sequence the LINK/ACT light never becomes active to show that the network has been restored. One cause of this can be that the /etc/networks/interfaces file stored in the Configuration Save area has become corrupt. As the system always over-writes the disk copy from the flash copy on disk as part of the boot sequence some special actions need to be taken to recover from this scenario.
To understand this issue, one needs to understand how the TeraStation Loads/Saves key confiuration information.
- Every time the TeraStation shuts down, it takes its internal configuration files, compresses them and writes the compressed archive to an area of its flash.
- When the TeraStation is booting, it reads the flash, uncompresses the config files and overwrites specific configuration files which reside in the system partition of the hard disk - one of these being the /etc/networking/interfacesfile.
The problem is that if the copy of this file held in the flash Configuration Save area ever gets corrupted (sometimes set to 0 length), then as there is no check of the config files on booting the TeraStation can overwrite the disk based version of the file with the version from the Config Save area. In the case of the /etc/networking/interfaces file this is fatal, because the /sbin/ifup command run during the boot process fails to power up the ethernet interface with the result that the TeraStation is not reachable via the network.
The error remains also in a forced EM-mode (removing all disks), as config data is treated the same way in EM mode.
You cannot recover from this error by flashing, as the TeraStation will not be able to communicate over the ethernet interface, and thus the flashing process cannot be performed.
To recover your TeraStation from this error the following steps are necessary:
- You must have a valid system partition on at least one disk. If you had a previosuly working system you almost certainly already ahve this is place. If not you would need to obtain a disk image of a working system.
- Attach the disk to a PC running Linux.
- Partition 1 (the system partition) will probably automount. Use a
dfcommand to see the device name used - it will vary according to your PC configuration.
- Create a copy of /etc/networking/interfaces (e.g. /etc/networking/interfaces2) and configure it corresponding to your needs. A sample file is shown below.
- Create an executable file /etc/init.d/network_workaround containing the following line to copy the good file over the bad file:
cp /etc/networking/interfaces2 /etc/networking/interfaces
- Create a symlink /etc/rc.d/rcS.d/S39network_workaround -> ../init.d/network_workaround to run the above command at the correct time during the boot sequence.
- Zero partition 1 on disks 2, 3 and 4. This is to force it to use the copy from disk 1. This can be done using commands of the form:
dd if=/dev/null of=/dev/xxx1
where xxx was the device name mentioned above
The purpose of the above modifications is to copy a good version of the /etc/network/interfaces file over the one restored from the flash area at the right point in the boot process so that the ethernet interface is started correctly. After this modification your TS should be reachable again. You can then try to fix it via web interface or flash it with a modified firmware and fix it manually.
A sample valid configuration file is:
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 192.168.1.253 netmask 255.255.255.0 broadcast 192.168.1.255 gateway 192.168.1.1
This file is to use a fixed IP address of 192.168.1.253. You can use a different one if it is more convenient.
Clearing Down the System Partition
There are times when clearing down the system partition will force the system to boot from Flash, and then come up in a mode where you can force the files on the System partition to be rewritten.
Making a Backup of the Partition
It is trongly recommended that if at all possible you make a backup of the system partition before clearing it. This means if that for any reason you have misdiagnosed the problem with your TeraStation and you want to restore it you can do so.
To do this:
- Remove Disk 1 from the teraStation and attach it to another system. This can be done either internally or via an external Disk Enclosure attached via USB or Firewire.
- Boot the system into Linux. If your system does not run Linux normally, then you can use one of the "Live" distirbutions that run from CD/DVD rather than the hard disk. This would avoid you ahving to actually install Linux onto this system.
- Make sure that you know how to run commands with 'root' priveleges. Depending on the variant of linux chosen this may mean either logging in as 'root' or running commands via the 'sudoer' facility.
- The system will probably have automounted the TeraStation hard disk. If so you check what it has been mounted as using the
Now unmount it using a command of the form:
umount /dev/devicenamewhere the devicename value will be the one you noticed was used for the TeraStation disk in the output of the
- Make a copy of the TeraStation system partition using a command of the form:
dd if=/dev/devicename of=/filename
where devicename is as described above, and filename is the output file. This file will be a little under 400Mb in size. Make sure that it is secured somewhere that will not be lost when you reboot the linux system (such as USB stick or ectenal USB drive).