Project

General

Profile

Building an Ubuntu RAID NAS from scratch

I needed a NAS to store my music, video and other data on.
But I also wanted to run my DomotiGa home automation software on it, combining the two servers I run now to just one.
Commercial NAS devices looked too expensive (even without disks included), and running Gambas on it, seemed very difficult to acomplish.

So I decided to build one myself, trying to keep in mind that the costs should stay low (relative to commercial NAS devices that is)
And the power usage should be kept to a minimum, same for noise level.

I bought this hardware list:

Point of View Ion 330, Atom 330 board EUR 129,00
2 Kingston ValueRAM 4GB (2x2) DDR2 S0Dimm EUR 71,00
External Power Adapter 220V -> 12V 102W EUR 59,00
MiniBox PicoPSU 150W EUR 52,00
4 1.5TB Samsung Green HD HD541 disks EUR 356,00
Coolermaster Elite 360 case EUR 38,00
Sitecom XC-043 External SATA adapter EUR 27,00
SATA Backplane 4-HDD SAS/SATA black EUR 99,00
Kingston Inbouwkit 2,5" -> 3,5" EUR 1,19

Total 832,19 EUR

I added this I had laying around:
OCZ SSD disk 64Gb
But you can use a cheap laptop 2,5 drive, or even an CF flash card and adapter

The case is cheap, and much bigger than you would expect for a NAS, I could have bought an Chenbro NAS case, but the 200+ Euros is not worth it.
I prefer a cheaper one with more room, looks are not that important, as it stands under my desk in the attic anyway.

But instead of buying a seperate power supply, case and sata backplane you can buy a Chenbro server case.

When using a Chenbro case:

Point of View Ion 330, Atom 330 board EUR 129,00
2 Kingston ValueRAM 4GB (2x2) DDR2 S0Dimm EUR 71,00
4 1.5TB Samsung Green HD HD541 disks EUR 356,00
Sitecom XC-043 External SATA adapter EUR 27,00
PCIe riser card EUR ??,??
Chenbro ES34069-131 Home Server Case 120W, mITX EUR 216,00

Total 799,00 EUR

Now Build everything together

Some notes... the drive bay fan makes a hell of a noise, I remove it together with the back panel, and placed the case fan (which came with the case) right in front of it,
connected it to SYS1_FAN header, blows ok, with minimum RPM.

I used the CPU cooler fan which came with the MB, connected it to SYS_FANx
Is't ok regarding noise and cooling capacity.

The SATA card is for the SSD and providing one external eSATA connection (for remote backups), the four onboard connections called SATA0-3 are for the raid disks.
The PicoPSU in combination with the external adapter is a very efficient power supply.

Installing OS

Since reinstalling the NAS bootdisk (my SSD) is not something you want to do every week,
I installed the RC of the new Ubuntu 9.10 on it, I have chosen the 32Bit version so 3rdParty software works ok.
Not possible to address more than 4Gb of memory anyway.

Create a USB disk with the help of unetbootin, select the Ubuntu "Daily_Live" distro.

Boot from it, answer all the standard questions.
I decided to use ext4 on the ssd, some people prefer ext2 because of less writes, but a few hours after running ext2
I got strange inode problems already. fsck could fix it, but I don't trust it after that.

After install reboot, remove USB key.

NOTE: if you are not a vi-wizard you can replace it by nano in the commands below.

And i login as root with
$ sudo bash

So I don't have to add sudo to all commands.

Please don't if you are not a experienced Linux admin, add sudo instead.

Tweak OS

Change mount options to save SSD write cycles.

# vi /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sde1 during installation
UUID=c285adce-c64d-4cd9-bcd9-93eefc4636fe /               ext4    nodiratime,noatime,errors=remount-ro 0       1
# swap was on /dev/sde5 during installation
UUID=d8b8b6fe-e466-4f72-bdd5-621dd914b83e none            swap    sw              0       0

tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
none /var/tmp aufs noatime,br:/tmp=rw:/var/tmp=ro 0 0
#none /var/log aufs noatime,br:/tmp=rw:/var/log=ro 0 0
none /var/cache aufs noatime,br:/tmp=rw:/var/cache=ro 0 0

So added nodiratime and noatime to /
and use tmpfs for all temporary stuff (even the logs).
NOTE: The settings above result in a /tmp full of stuff without the directory structure, have to fix that later.
NOTE2: There are issues if you install software with the /var/log in ram, dirs are created there and that structure is lost
after a reboot, so startup of lighttpd failses for example, so leave it commented like it the above list.

Don't let the machine swap.

# sudo vi /etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

sysctl -w vm.swappiness=1               # Strongly discourage swapping
sysctl -w vm.vfs_cache_pressure=50      # Don't shrink the inode cache aggressively
exit 0

I have a strange issue with DNS resolving when I use my ISP modem as DNS server. (in combination with Ubuntu 9.10 only!)

So I use this work around:
Configured my LAN interface with static/manual IP config instead of DHCP, used 127.0.0.1 as DNS server address.

To have a local DNS server I installed Bind.

# aptitude install bind9

/etc/resolve.conf
..

Reboot.

Partition drives

# aptitute install mdadm

Installing this will also install postfix, we need it later.

Partition the 4 drives into "Linux Raid Autodetect" partitions:

# fdisk /dev/sda

n (new)
p (primary)
1 (1st primary)
t (change type)
fd (set type to linux raid autodetect)
w (write changes and exit fdisk)

repeat for others
# fdisk /dev/sdb
# fdisk /dev/sdc
# fdisk /dev/sdd

Create the raid 5 array

# mdadm --create --verbose /dev/md0 --chunk=128 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

mdadm: layout defaults to left-symmetric
mdadm: size set to 1465135872K
mdadm: array /dev/md0 started.

Watch progress
# watch cat /proc/mdstat

Every 2.0s: cat /proc/mdstat                                       Tue Oct 27 21:41:47 2009

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1r4 sdc1r2 sdb1r1 sda1r0
      4395407616 blocks level 5, 128k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  0.5% (8242944/1465135872) finish=232.1min speed=104588K/sec

unused devices: <none>

This will take some time (240 minutes in my case)

Finished it looks like this:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1r3 sdc1r2 sdb1r1 sda1r0
      4395407616 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

Save raid config to file:
# mdadm --detail --scan

Add it's output to end of /etc/mdadm/mdadm.conf

Remove the first 0 from 00:90, it's a bug!

This is my config:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR my@email.address.nl
MAILFROM server-mdadm

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=4 metadata=0.90 UUID=be4f360f:107170e8:ee5caf04:66443b85

Mail

We need to configure our mail server, so mdadm and later other processes can send us mails.

Postfix is already installed, we need to tweak it's config a bit.

# vi /etc/postfix/main.cf

I had to adjust these lines only:
myhostname = your.host.com
mydestination = server.your.host.com, localhost.localdomain, localhost
relayhost = smtp.online.nl

myhostname needs to be an official fqdn, otherwise your ISP's mailserver will protest (smtp.online.nl in my case)

Set up a forward for the root account, so all local mail gets in your external email box.

# vi /root/.forward
<your@email.com>

# /etc/init.d/postfix reload

You can send a test e-mail with:
$ mail your@email.com
Cc:[enter]
Subject: Test
Some text(ctrl-D)

If you don't receive anything check queue with
$ mailq

Or received local mail with
$ mail

Now trigger mdadm to send us a test e-mail:

# mdadm --monitor --scan --test --oneshot /dev/md0

Check if mdadm daemon runs:
$ ps aux|grep mdadm
root 3214  0.0  0.0 2148   624 ?  Ss Oct27 0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog

Install other disk monitor tools

# aptitude install hddtemp smartmontools

# hddtemp /dev/sd[abcd]
/dev/sda: SAMSUNG HD154UI: 29°C
/dev/sdb: SAMSUNG HD154UI: 29°C
/dev/sdc: SAMSUNG HD154UI: 30°C
/dev/sdd: SAMSUNG HD154UI: 27°C

DomotiGa can connect to hddtemps network socket if you want to get these values, and report when tresholds are being reached.

Configure smartmon

# vi /etc/default/smartmontools

Uncomment start_smartd=yes

This is my config

# Defaults for smartmontools initscript (/etc/init.d/smartmontools)
# This is a POSIX shell fragment

# List of devices you want to explicitly enable S.M.A.R.T. for
# Not needed (and not recommended) if the device is monitored by smartd
#enable_smart="/dev/hda /dev/hdb" 

# uncomment to start smartd on system startup
start_smartd=yes

# uncomment to pass additional options to smartd on startup
#smartd_opts="--interval=1800" 

# vi /etc/smartd.conf

Add these lines to bottom:
/dev/sda -d ata -n standby -H -l error -l selftest -s (O/../../6/07|L/../../5/00|S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail
/dev/sdb -d ata -n standby -H -l error -l selftest -s (O/../../6/07|L/../../5/00|S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail
/dev/sdc -d ata -n standby -H -l error -l selftest -s (O/../../6/07|L/../../5/00|S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail
/dev/sdd -d ata -n standby -H -l error -l selftest -s (O/../../6/07|L/../../5/00|S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail

Less intrusive:
/dev/sda -d ata -n standby -H -l error -l selftest -s (S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail
/dev/sdb -d ata -n standby -H -l error -l selftest -s (S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail
/dev/sdc -d ata -n standby -H -l error -l selftest -s (S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail
/dev/sdd -d ata -n standby -H -l error -l selftest -s (S/../../1|2|3|4|5/08) -m root -M test -M exec /usr/local/sbin/smart_error_mail

Which means:
O/../../6/07 -> Offline Immediate test on Saturday at 7:00AM
L/../../5/00 -> Long Selftest on Friday at 12:00PM
S/../../1|2|3|4|5/08 -> Short Selfttest every Weekday at 8:00AM

Create mail script

# vi /usr/local/sbin/smart_error_mail

#!/bin/sh
LOGFILE="/tmp/smartd.log" 
echo "$SMARTD_MESSAGE\n" > "$LOGFILE" 
echo "----------------------SMARTCTL OUTPUT----------------------------------------------"  >> "$LOGFILE" 
smartctl -a /dev/sda  >> "$LOGFILE" 
smartctl -a /dev/sdb  >> "$LOGFILE" 
smartctl -a /dev/sdc  >> "$LOGFILE" 
smartctl -a /dev/sdd  >> "$LOGFILE" 
mail -s "Harddrive status of Ubuntu NAS" your@email.com < $LOGFILE

Make it executable
# chmod a+x /usr/local/sbin/smart_error_mail

Start SMART service:

# /etc/init.d/smartmontools restart

NOTE: Not sure if all works ok, have to check this.

Configure ntpdate to sync clock every day

Create file with this contents:

# vi /etc/cron.daily/ntpdate

#!/bin/sh
/usr/sbin/ntpdate -s -u europe.pool.ntp.org

Make executable:
# chmod 755 /etc/cron.daily/ntpdate

Create filesystem on the RAID array

# mkfs.ext4 /dev/md0

This took a few minutes...

Set reserved space to 0%

# tune2fs -m 0 /dev/md0

tune2fs 1.41.9 (22-Aug-2009)
Setting reserved blocks percentage to 0% (0 blocks)

# mkdir /mnt/nas

# vi /etc/fstab

Add line:
/dev/md0  /mnt/nas  ext4  nodiratime,noatime,errors=remount-ro 0

# mount -a

Create directory structure

# cd /mnt/nas
# mkdir -p home/ron
# cd home
# chown -R ron ron
# chgrp -R ron ron
# chmod 755 ron

# mkdir /mnt/nas/public
# chgrp -R users /mnt/nas/public
# chmod 0777 -R /mnt/nas/public

Install and configure Samba

# aptitude install samba

# vi /etc/sambas/smb.conf

Performance

Without tweaky anything it get this performance

Copy several DVD images from Windows 7 64Bit to NAS public share:
58MByte/sec write

Copy several DVD images from NAS public folder to Windows 7 client:
75-80MByte/sec read

Not bad I think.

# smbpasswd <user>

Install lighttpd

# aptitude install lighttpd

Install PHP
# aptitude install php5-cgi php5-mysql php5-xmlrpc

To enable PHP5 in Lighttpd, we must modify /etc/php5/cgi/php.ini and add the line cgi.fix_pathinfo = 1 right at the end of the file:
# vi /etc/php5/cgi/php.ini

Change similar line to:
cgi.fix_pathinfo = 1

To enable the fastcgi configuration (which is stored in /etc/lighttpd/conf-available/10-fastcgi.conf), run the following command:
# lighttpd-enable-mod fastcgi

This creates a symlink /etc/lighttpd/conf-enabled/10-fastcgi.conf which points to /etc/lighttpd/conf-available/10-fastcgi.conf

Then we reload Lighttpd:

# /etc/init.d/lighttpd force-reload

Install torrentflux-b4rt

# aptitude install unrar cksfv uudeview php5-cli php5-gd vlc-nox uudeview python-psyco python-crypto \
libxml-simple-perl libxml-dom-perl libdbd-mysql-perl libdigest-sha1-perl \
bittorrent bittornado

# cd /var/www
# wget http://download.berlios.de/tf-b4rt/torrentflux-b4rt_1.0-beta2.tar.bz2
# tar -jvxf torrentflux-b4rt_1.0-beta2.tar.bz2 && cd torrentflux-b4rt_1.0-beta2
# cd clients/transmission

We need to build a patched transmission
# wget http://download.m0k.org/transmission/files/transmission-1.06.tar.bz2   
# tar jxvf transmission-1.06.tar.bz2  
# tar jxvf Transmission-1.06_tfCLI-svn3356.tar.bz2  
# cp Transmission-1.06_tfCLI-svn3356/cli/transmissioncli.c transmission-1.06/cli/transmissioncli.c  
# ./configure --prefix=/usr/local --disable-gtk --sysconfdir=/etc
# make
# make install

Copy the Torrentflux-b4rt html directory to a web visible location

# mv /var/www/torrentflux-b4rt_1.0-beta2/html/ /var/www/torrentflux
# chown -R www-data:www-data /var/www/torrentflux

Create database and mysqluser
# cat > create.sql <<EOF
CREATE DATABASE torrentflux;
GRANT ALL ON torrentflux.* TO torrentflux@localhost IDENTIFIED BY 'torrentflux';
EOF

# mysql -u root -p < create.sql

Run the TorrentFlux-b4rt setup.php script by pointing your browser to:
http://yourserverip/torrentflux/setup.php

Follow the directions onscreen to finish the setup. Enter your MySQL username, password, and database name.

# rm /var/www/torrentflux/setup.php

Then, we make it so that the HTTP server can no longer right to the config folder:
# chown -R root:root /var/www/torrentflux/inc/config/

# cd /mnt/nas
# mkdir torrentflux
# chown -R www-data torrentflux

I use this a download path.

TwonkyMedia server

# mkdir ~/install/twonky
# cd ~/install/twonky
# wget http://www.twonkymedia.com/downloads/twonkymedia-i386-glibc-2.2.5-full-5.0.68.sh
# chmod 755 twonkymedia-i386-glibc-2.2.5-full-5.0.68.sh
# ./twonkymedia-i386-glibc-2.2.5-full-5.0.68.sh

Starting server ...
Starting /usr/local/twonkymedia/twonkymedia ... Daemonizing...
Daemonizing...

Installation finished
...

# sudo adduser twonky

On Ubuntu this will create the user "twonky" and the group "twonky"

Installation

We need to fix permissions.

# chown -R twonky:twonky /usr/local/twonkymedia

The installation is now done and you could start the server via:
# su - twonky -c "/usr/local/twonkymedia/twonkymedia.sh start" 

Add multicast route

# route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

Browse to http://&lt;ipofserver&gt;:9000/config
# aptitude install rcconf
# rcconf
Enable Twonky to start at startup

NOTE: Still need to make starting Twonky with twonky user working.

Related Resources

    Updated by: rdnzl, Updated almost 8 years ago
    Access count: 73050 since 2011-08-25

    Attached Files

    Also available in: PDF HTML TXT