Difference between revisions of "Usenet downloader"

From NAS-Central Buffalo - The Linkstation Wiki
Jump to: navigation, search
(New Page)
 
 
(12 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
{{articles|LSPro|Filesharing}}
 
[Sources/Credits Methanoid & Fantasymick on NAS-central.org, Caveman on OpenFSG.com and DrChair at wl500g.info]
 
[Sources/Credits Methanoid & Fantasymick on NAS-central.org, Caveman on OpenFSG.com and DrChair at wl500g.info]
  
This has been tested and works well on a ARM9 Linkstation Live. Should work on others.
 
Initial work:
 
You need access to the box via Freelink or Open Stock firmware or similar
 
You need IPKG installed and to have picked up the ARM9 feed from NSLU2-Linux.org
 
You need to telnet into the box as root.
 
  
Installing is made much easier by a recent update in the ARM9 feed which now includes a Hellanzb package.
+
Note from author (Methanoid):
+
I no longer recommend this method/program. It's been superceded by the more efficient NZBget, a guide for which is in the forums. Both are more efficient than SABnzbd+ which takes too much resource for a Linkstation (400mhz ARM models) to be usable.
ipkg update
+
ipkg install py-hellanzb (or if you want to use python2.5, then ipkg install py25-hellanzb)
+
  
This installs Python, HellaNZB and all the dependencies such as Twisted, RAR, Par2Cmdline and the faster YEnc decoder  
+
 
+
 
But you will need to configure Hellanzb
+
This has been tested and works well on a ARM9 Linkstation Live. Should work on others. We are using HellaNZB which is a NZB-based usenet "leech" client which also PAR checks and un-archives the files you have downloaded.
+
 
cd /opt/local/bin
+
=Prerequisites=
vi hellanzb.conf
+
 
 +
There are 2 options:
 +
 
 +
* [[Open_Stock_Firmware]]/jtymod + [http://forum.nas-central.org/viewtopic.php?f=19&t=4075&p=39162#p39162 bootstrapping to the optware feed]
 +
* [[:Category:FreeLink]]
 +
 
 +
 
 +
 
 +
 
 +
=Stock firmware/jtymod=
 +
==Installation==
 +
 
 +
ipkg update
 +
ipkg install py24-hellanzb (or py25-hellanzbd if you want to use Python 2.5 rathern than 2.4)
 +
 
 +
This installs Python, HellaNZB and all the dependencies such as Twisted, RAR, Par2Cmdline and the faster YEnc decoder
 +
 
 +
== Configuration ==
 +
You can edit the config file now to get it set up for your provider etc
 +
 
 +
cd /opt/etc
 +
vi hellanzb.conf
 
   
 
   
 
You need to change the settings to match those of your usenet provider (username, password, server,number of connections). Remember, the Linkstation is a NAS box not a high spec PC so don't be too greedy on the number of connections as it might actually make it go slower. For example, I can happily max out a 4Mbit line with 4 connections. Try different numbers and see how you get on.
 
You need to change the settings to match those of your usenet provider (username, password, server,number of connections). Remember, the Linkstation is a NAS box not a high spec PC so don't be too greedy on the number of connections as it might actually make it go slower. For example, I can happily max out a 4Mbit line with 4 connections. Try different numbers and see how you get on.
  
# Set both the username and password to 'None' (without the quotes) if your  
+
<pre>
# usenet server does not require authorization  
+
# Set both the username and password to 'None' (without the quotes) if your  
defineServer(id = 'your usenet',  
+
# usenet server does not require authorization  
hosts = [ 'yourusenet:119' ],  
+
defineServer(id = 'your usenet',  
#hosts = [ 'news.changeme.com', 'morenews.changeme.com:8000' ],  
+
hosts = [ 'yourusenet:119' ],  
 +
#hosts = [ 'news.changeme.com', 'morenews.changeme.com:8000' ],  
  
username = 'your username',  
+
username = 'your username',  
password = 'your password',  
+
password = 'your password',  
#username = None, # no auth  
+
#username = None, # no auth  
#password = None,  
+
#password = None,  
  
connections = 4,  
+
connections = 4,  
  
# Important locations  
+
# Important locations  
Hellanzb.PREFIX_DIR = '/mnt/disk1/shared/download'  
+
Hellanzb.PREFIX_DIR = '/mnt/disk1/shared/download'  
 +
</pre>
  
 
A further important change is to fix the permissions (you want to be able to move files from the download directory presumably?) so change
 
A further important change is to fix the permissions (you want to be able to move files from the download directory presumably?) so change
Hellanzb.UMASK = 0022  to
+
<pre> Hellanzb.UMASK = 0022  </pre>
Hellanzb.UMASK = 0000
+
to
 
+
<pre> Hellanzb.UMASK = 0000</pre>
Move the resulting config file to where the actual Hellanzb python script is located
+
 
+
mv hellanzb.conf /opt/local/bin/hellanzb.conf
+
  
 
Now there should be hellanzb.py and your hellanzb.conf there. Now you can fire up hellanzb either from command line or from script.
 
Now there should be hellanzb.py and your hellanzb.conf there. Now you can fire up hellanzb either from command line or from script.
  
cd /opt/local/bin
+
cd /opt/bin
python hellanzb.py -D  
+
python hellanzb.py -D (or python25 hellanzb.py -D if running Python 2.5)
  
 
The -D makes it run in Daemon mode, i.e it runs in the background until stopped.
 
The -D makes it run in Daemon mode, i.e it runs in the background until stopped.
  
{edit] put script reference in here - the script is something like S71hellanzb and to run sh it. ./opt/etc/init.d/S71hellanzb
+
or you can run via script. Unfortunately the script needs some changes before use
 +
vi /opt/etc/init.d/S71hellanzb
 +
 
 +
Change it to read
 +
 
 +
<pre> #!/bin/sh
 +
nice /opt/bin/hellanzb.py -D</pre>
 +
 
 +
Then check/fix the permissions/ownership and run it (first time only, it will autostart with the Linkstation in future!)
 +
 
 +
cd /opt/etc/init.d
 +
chown root /opt/etc/init.d/S71hellanzb
 +
chmod ugo+rx /opt/etc/init.d/S71hellanzb  
 +
sh S71hellanzb
 +
 
 
If everything went well hellanzb is running and will create the necessary sub-dirs in the dir you created  
 
If everything went well hellanzb is running and will create the necessary sub-dirs in the dir you created  
  
/mnt/disk1/shared/download
+
/mnt/disk1/shared/download
/mnt/disk1/shared/download/nzb  
+
/mnt/disk1/shared/download/nzb  
/mnt/disk1/shared/download/nzb/daemon.current  
+
/mnt/disk1/shared/download/nzb/daemon.current  
/mnt/disk1/shared/download/nzb/daemon.queue  
+
/mnt/disk1/shared/download/nzb/daemon.queue  
/mnt/disk1/shared/download/nzb/daemon.postponed  
+
/mnt/disk1/shared/download/nzb/daemon.postponed  
/mnt/disk1/shared/download/nzb/daemon.temp  
+
/mnt/disk1/shared/download/nzb/daemon.temp  
/mnt/disk1/shared/download/nzb/daemon.processing  
+
/mnt/disk1/shared/download/nzb/daemon.processing  
/mnt/disk1/shared/download/nzb/daemon.working  
+
/mnt/disk1/shared/download/nzb/daemon.working  
  
 
Just pop your NZB file in the /mnt/disk1/shared/download/nzb/daemon.queue dir and hellanzb will start downloading and pop  
 
Just pop your NZB file in the /mnt/disk1/shared/download/nzb/daemon.queue dir and hellanzb will start downloading and pop  
Line 69: Line 97:
 
You can monitor (without a web frontend) by typing
 
You can monitor (without a web frontend) by typing
 
python hellanzb.py status
 
python hellanzb.py status
 
Realistically you will probably want a web based frontend. Hellahella doesn't seme to want to run but Zussaweb works fine with Lighttpd. You could use a 2nd instance of Apache or maybe piggyback on the Apache that runs the LS's web interface but it didn't work for me.
 
  
ipkg update
+
==Adding a webfrontend==
ipkg install lighttpd
+
Realistically you will probably want a web based frontend. Hellahella doesn't seem to want to run, Zussaweb looks pretty for monitoring downloads but the upload NZB and Newzbin ID imports don't work for me. I used Lighttpd but you could use a 2nd instance of Apache or maybe piggyback on the Apache that runs the LS's web interface but it didn't work for me. I chose HellaPHP as it worked 100%, required minimal configuration and the author was prepared to amend it to work 100% on my Linkstation setup. He's also adding some more functionality in future like password protection etc and a new look.
ipkg install fcgi-php
+
 
 +
ipkg update
 +
ipkg install lighttpd php-fcgi
 
   
 
   
 
You'll need to configure (of course)
 
You'll need to configure (of course)
 
   
 
   
vi /opt/etc/lighttpd/lighttpd.conf
+
vi /opt/etc/lighttpd/lighttpd.conf
 
   
 
   
 
change
 
change
#                              "mod_fastcgi",
+
<pre> #                              "mod_fastcgi", </pre>
  into
+
into
                                "mod_fastcgi",
+
<pre>                                "mod_fastcgi",</pre>
change
+
change
url.access-deny            = ( "~", ".inc" )
+
<pre>url.access-deny            = ( "~", ".inc" )</pre>
  into
+
into
url.access-deny            = ( "~", ".inc", ".sqlite" )
+
<pre>url.access-deny            = ( "~", ".inc", ".sqlite" )</pre>
 
   
 
   
  change
+
change
 +
<pre>
 
#fastcgi.server            = ( ".php" =>
 
#fastcgi.server            = ( ".php" =>
 
#                              ( "localhost" =>
 
#                              ( "localhost" =>
Line 98: Line 127:
 
#                              )
 
#                              )
 
#                            )
 
#                            )
 
+
</pre>
  into
+
into
 +
<pre>
 
fastcgi.server            = ( ".php" =>
 
fastcgi.server            = ( ".php" =>
 
                               ( "localhost" =>
 
                               ( "localhost" =>
Line 108: Line 138:
 
                               )
 
                               )
 
                             )
 
                             )
 +
</pre>
 
   
 
   
+
Give some thought as to where you want to have your webpages. I like mine accessible to my Windows PCs so that I can edit directly in Frontpage or whatever but in this case I won't change the defaults.. Also you may need to open a port to the webserver if you want to have your HellaPHP available remotely over the internet.
Give some thought as to where you want to have your webpages. I like mine accessible to my Windows PCs so that I can edit directly in Frontpage or whatever but in this case I won't change the defaults.. Also you may need to open a port to the webserver if you want to have your Zussaweb available remotely over the internet.
+
 
   
 
   
 
Now you are ready to start lighttpd by /opt/etc/init.d/S80lighttpd
 
Now you are ready to start lighttpd by /opt/etc/init.d/S80lighttpd
  
Next download and unpack zussaweb:
+
Download and unpack HellaPHP. Change the values of $disk and $download_nzb_path in config.php to match your download and NZB directories
 +
 +
Now you can use the interface at http://linkstation:8081/hella/ (or wherever you've popped the HellaPHP index.php and other files)
  
cd /opt/tmp
+
== Security ==
wget http://download.sourceforge.net/sourceforge/zussaweb/zussaweb03.tar.gz
+
cd /opt/share/www/lighttpd/
+
tar -xzvf /opt/tmp/zussaweb03.tar.gz
+
  
Next you need to change the values of $disk and $download_nzb_path in the file /opt/share/www/lighttpd/zussaweb/settings.php
+
You might have noticed that HellaPHP has no security. Anyone who knows the IP of the machine you have it running on can view your downloads, interfere with yours and start their own. After tinkering with the PHP files to provide security I found a much simpler solution. You can configure the lighttpd webserver software to protect any page or directory you want with a password.
  
Finally we need to fix a bug:
+
Go into /etc/lighttpd/conf-enabled (it might be /opt/etc/lighttpd/conf-enabled in certain configurations). Enable the mod-auth module with
Edit /opt/share/www/lighttpd/zussaweb/functions.php and change:
+
  
function freediskspace() {
+
ln -s ../conf-available/10-auth.conf 10-auth.conf
        return (round (disk_free_space(disk)/1024/1024,2));
+
 
}
+
Now edit the /etc/lighttpd/lighttpd.conf (or /opt/etc/lighttpd/lighttpd.conf). Make sure that you have the module "mod_auth" in the server.modules
+
section and add the lines
to
+
 
+
<pre>
function freediskspace($disk) {
+
auth.debug = 2
        return (round (disk_free_space($disk)/1024/1024,2));
+
auth.backend = "plain"
+
auth.backend.plain.userfile = "/home/lighttpd/.lighttpdpassword"
+
 
Now you can use the interface at http://linkstation:8081/zussaweb/
+
auth.require = ( "/hella/" =>
+
(
Remaining issues needing fixing
+
"method" => "basic",
 +
"realm" => "Password protected area",
 +
"require" => "user=changeuser"
 +
)
 +
)
 +
</pre>
 +
 
 +
Put in whatever user name you want instead of "changeuser".
 +
 
 +
Now create (or edit) the "/home/lighttpd/.lighttpdpassword" file. You can use any file you want as long as it's accessible by the lighttpd process.
 +
 
 +
Add the line
 +
 
 +
<pre>
 +
changeuser:changeme
 +
</pre>
 +
 
 +
to the file (with the user name you used in the lighttpd.conf file and the password you want).
 +
 
 +
Restart the lighttpd
 +
 
 +
<pre>
 +
lighttpd
 +
</pre>
 +
 
 +
and you're done. Your web interface is now password protected.
 +
 
 +
= Freelink =
 +
 
 +
==Installation==
 +
 
 +
==Configuration==
 +
 
 +
==Adding a webfrontend==
  
1) Getting Hellanzb to start up in daemon mode when LS starts.
+
{{Stubs}}
2) Getting the "upload an NZB" and "enter Newzbin ref" options in Zussaweb to work right.
+

Latest revision as of 08:57, 18 August 2009

[Sources/Credits Methanoid & Fantasymick on NAS-central.org, Caveman on OpenFSG.com and DrChair at wl500g.info]


Note from author (Methanoid): I no longer recommend this method/program. It's been superceded by the more efficient NZBget, a guide for which is in the forums. Both are more efficient than SABnzbd+ which takes too much resource for a Linkstation (400mhz ARM models) to be usable.


This has been tested and works well on a ARM9 Linkstation Live. Should work on others. We are using HellaNZB which is a NZB-based usenet "leech" client which also PAR checks and un-archives the files you have downloaded.

Contents

Prerequisites

There are 2 options:



Stock firmware/jtymod

Installation

ipkg update
ipkg install py24-hellanzb (or py25-hellanzbd if you want to use Python 2.5 rathern than 2.4)

This installs Python, HellaNZB and all the dependencies such as Twisted, RAR, Par2Cmdline and the faster YEnc decoder

Configuration

You can edit the config file now to get it set up for your provider etc

cd /opt/etc
vi hellanzb.conf

You need to change the settings to match those of your usenet provider (username, password, server,number of connections). Remember, the Linkstation is a NAS box not a high spec PC so don't be too greedy on the number of connections as it might actually make it go slower. For example, I can happily max out a 4Mbit line with 4 connections. Try different numbers and see how you get on.

 # Set both the username and password to 'None' (without the quotes) if your 
 # usenet server does not require authorization 
 defineServer(id = 'your usenet', 
 hosts = [ 'yourusenet:119' ], 
 #hosts = [ 'news.changeme.com', 'morenews.changeme.com:8000' ], 

 username = 'your username', 
 password = 'your password', 
 #username = None, # no auth 
 #password = None, 

 connections = 4, 

 # Important locations 
 Hellanzb.PREFIX_DIR = '/mnt/disk1/shared/download' 

A further important change is to fix the permissions (you want to be able to move files from the download directory presumably?) so change

 Hellanzb.UMASK = 0022   

to

 Hellanzb.UMASK = 0000

Now there should be hellanzb.py and your hellanzb.conf there. Now you can fire up hellanzb either from command line or from script.

cd /opt/bin
python hellanzb.py -D (or python25 hellanzb.py -D if running Python 2.5)

The -D makes it run in Daemon mode, i.e it runs in the background until stopped.

or you can run via script. Unfortunately the script needs some changes before use

vi /opt/etc/init.d/S71hellanzb

Change it to read

 #!/bin/sh
nice /opt/bin/hellanzb.py -D

Then check/fix the permissions/ownership and run it (first time only, it will autostart with the Linkstation in future!)

cd /opt/etc/init.d
chown root /opt/etc/init.d/S71hellanzb 
chmod ugo+rx /opt/etc/init.d/S71hellanzb 
sh S71hellanzb

If everything went well hellanzb is running and will create the necessary sub-dirs in the dir you created

/mnt/disk1/shared/download
/mnt/disk1/shared/download/nzb 
/mnt/disk1/shared/download/nzb/daemon.current 
/mnt/disk1/shared/download/nzb/daemon.queue 
/mnt/disk1/shared/download/nzb/daemon.postponed 
/mnt/disk1/shared/download/nzb/daemon.temp 
/mnt/disk1/shared/download/nzb/daemon.processing 
/mnt/disk1/shared/download/nzb/daemon.working 

Just pop your NZB file in the /mnt/disk1/shared/download/nzb/daemon.queue dir and hellanzb will start downloading and pop the unarchived resulting files in your destination directory.

You can monitor (without a web frontend) by typing python hellanzb.py status

Adding a webfrontend

Realistically you will probably want a web based frontend. Hellahella doesn't seem to want to run, Zussaweb looks pretty for monitoring downloads but the upload NZB and Newzbin ID imports don't work for me. I used Lighttpd but you could use a 2nd instance of Apache or maybe piggyback on the Apache that runs the LS's web interface but it didn't work for me. I chose HellaPHP as it worked 100%, required minimal configuration and the author was prepared to amend it to work 100% on my Linkstation setup. He's also adding some more functionality in future like password protection etc and a new look.

ipkg update
ipkg install lighttpd php-fcgi

You'll need to configure (of course)

vi /opt/etc/lighttpd/lighttpd.conf

change

 #                               "mod_fastcgi", 

into

                                "mod_fastcgi",

change

url.access-deny             = ( "~", ".inc" )

into

url.access-deny             = ( "~", ".inc", ".sqlite" )

change

#fastcgi.server             = ( ".php" =>
#                               ( "localhost" =>
#                                 (
#                                   "socket" => "/tmp/php-fastcgi.socket",
#                                   "bin-path" => "/usr/local/bin/php"
#                                 )
#                               )
#                             )

into

fastcgi.server             = ( ".php" =>
                               ( "localhost" =>
                                 (
                                   "socket" => "/tmp/php-fastcgi.socket",
                                   "bin-path" => "/opt/bin/php-fcgi"
                                 )
                               )
                            )

Give some thought as to where you want to have your webpages. I like mine accessible to my Windows PCs so that I can edit directly in Frontpage or whatever but in this case I won't change the defaults.. Also you may need to open a port to the webserver if you want to have your HellaPHP available remotely over the internet.

Now you are ready to start lighttpd by /opt/etc/init.d/S80lighttpd

Download and unpack HellaPHP. Change the values of $disk and $download_nzb_path in config.php to match your download and NZB directories

Now you can use the interface at http://linkstation:8081/hella/ (or wherever you've popped the HellaPHP index.php and other files)

Security

You might have noticed that HellaPHP has no security. Anyone who knows the IP of the machine you have it running on can view your downloads, interfere with yours and start their own. After tinkering with the PHP files to provide security I found a much simpler solution. You can configure the lighttpd webserver software to protect any page or directory you want with a password.

Go into /etc/lighttpd/conf-enabled (it might be /opt/etc/lighttpd/conf-enabled in certain configurations). Enable the mod-auth module with

ln -s ../conf-available/10-auth.conf 10-auth.conf

Now edit the /etc/lighttpd/lighttpd.conf (or /opt/etc/lighttpd/lighttpd.conf). Make sure that you have the module "mod_auth" in the server.modules section and add the lines

auth.debug = 2
auth.backend = "plain"
auth.backend.plain.userfile = "/home/lighttpd/.lighttpdpassword"

auth.require = ( "/hella/" =>
(
"method" => "basic",
"realm" => "Password protected area",
"require" => "user=changeuser"
)
)

Put in whatever user name you want instead of "changeuser".

Now create (or edit) the "/home/lighttpd/.lighttpdpassword" file. You can use any file you want as long as it's accessible by the lighttpd process.

Add the line

changeuser:changeme

to the file (with the user name you used in the lighttpd.conf file and the password you want).

Restart the lighttpd

lighttpd

and you're done. Your web interface is now password protected.

Freelink

Installation

Configuration

Adding a webfrontend

Link.png This article is currently a stub. You can help this Wiki by expanding it

. This template will categorize articles that include it into Category:Stubs.