Building Scalable ISPs with open-source softwares

ArticleCategory: [Choose a category for your article]

System Administration

AuthorImage:[Here we need a little image form you]

[Photo of the Author]

TranslationInfo:[Author and translation history]

original in en Atif Ghaffar 

AboutTheAuthor:[A small biography about the author]

Atif is a chameleon. He changes his roles, from System Administrator, to programmer, to teacher, to project manager, to whatever is required to get the job done.
Currently he is working as the Internet Development manager at 4unet and is looking for talented people to hire.
More about him can be found at his homepage

Abstract:[Here you write a little summary]

In my last article, I introduced you on using LDAP under Linux. I had many requests from people who were curious to know how LDAP combined with some fine open-source software can make a web mail or web hosting system.

In this Article we will setup a scalable ISP service based on LDAP and Linux. It will touch many of the issues and questions asked. We will then use a tool called ISPMan to manage it.

To avoid the unnecessary length with the textual examples, I have embedded them in textarea widgets. If you really want to print this page, then save it locally and change textarea to pre tags.
Perhaps this perl-liner will do the trick.
perl -pi.bak -e 's!textarea.*?>!pre>!g' filename

Building an ISP and then managing it is tricky, especially if you want to make it mostly automated, highly available and scalable. Running an ISP requires a team of system administrators, dedicated to the running of machines, creating accounts, managing web-sites, trouble shooting, running a help desk etc.
Most of the time, the help desk has little or no control on the setup. Most of the help desks I have come in touch with desperately need help.

ArticleIllustration:[This is the title picture for your article]

ispman

ArticleBody:[The article body]

Why LDAP?

LDAP is an excellent directory. It can manage not only the username/password of users but much more about users and resources. As we go along we will see how LDAP helps to manage a lot of things centrally.

Whats ISPMan?

ISPMan is the software, I wrote, so people from the IT department dont bug me when they need to create a new domain, set up a new webserver or a change a DNS entry. ISPMan is open source software and is available at http://www.ispman.org. I wont try to explain more about ISPMan itself but the concepts behind it. You can then feel free to try it out and enhance it.

What will this ISP provide?

This ISP will provide DNS, mail, webmail, web hosting etc.

The idea is that a customer comes and wants to host domain "exampledomain.com".
This domain is created centrally by just a few clicks and a lot of magic is managed behind the scenes including setting up DNS, setting up the virtual server for mail and a virtual webserver.
The customer gets one ftp username to access his/her web/ftp server. Any number of users can be created within the domain for email access. Users within the domain may or may not get web space.

There is also the issue of providing internet access. This could be a rather straight forward process or it can get really complicated, so we wont be touching it in this article :).

Virtual domains

Every one wants a website and their own domain's email addresses. Email server have been strangely tied to the system accounts for emails. I dont like it.
The problem arises on multiple levels when you want to manage user1@domain1.com and user1@domain2.ch etc.
There is a lot of unneccesary mappings etc to be done. Hopefully future softwares will be written with domain names in mind.
For example, Cyrus, an excellent IMAP server. Cyrus manages mailboxes (not users). It only allows one mailbox to have a name "aghaffar". If I have another customers from domain "linuxrus.com" who wants a username "aghaffar", then I am out of luck. I would have to create a mailbox by another name and map the user to that mailbox. Another trick would be to create a user "aghaffar.linuxrus.com" but Cyrus used "." as the mailbox delimeter, so cant use that one, neither can I use "aghaffar@linuxrus.com".

Also note that not all of us live in US. A lot of examples in some mailing lists suggest user1@domain1 and user1@domain2. They expect all domain names to end in .com. So we have to keep track of "username" "domain" "TLD"(Top Level Domain).

Our design will take all this into account.

So instead of creating a user called "aghaffar", we will create users in this form. "username_domain_tld".
I dont really remember why I used "_" as the delimeter, but "." was not available because of Cyrus, and there were other problems with other delimiters, for example "&" is a bad one both for shells and URLs.

Softwares powering our ISP

This is a list of software that I was able to use nicely together. You may want to use another software if you wish and if it works for you.

Directory Design

Our directory is based on domains. There are domains, domain users, domain services etc.
For example, a user can only exist in a domain (except some system users such as LDAP admin, Cyrus admin etc)
domain branch has information about users of that domain, the DNS data of the domain, the http data of the domain etc etc
Example

Here we defined a branch for domain "developer.ch", this branch has sub branches for users, dnsdata, and httpdata.
In this example, we defined a uid, gid, homeDirectory etc for the domain, cause we want only the user "domain.tld" to access via ftp.
For example, if the customer who owns developer.ch wants to upload files to her directory, then she will login as user "developer.ch" and the appropriate password to login to the ftpserver which will hopefully chroot to homeDirectory.. etc .. etc.. more about that later.

For other users, we dont define uid, gid, etc cause we dont want them to login via ftp.
You can find a complete LDIF example file here (this file may be outdated cause its from a production machine, and I have added features etc in the new design) or use this link if you want to show it to your boss (again a bit outdated)

LDAP Authentification (No system accounts)

Another great thing is that we do not have to create any system accounts in /etc/passwd, /etc/shadow etc or manage NIS etc.
All accounts are manged in LDAP and authentification is done directly from LDAP.
For this trick we take a lot of help from PAM (Pluggable Authentification Module) pam_ldap. PAM allows you to define which module should take care of the authentification, authorization etc.

For example, my /etc/pam.d/imap, /etc/pam.d/pop and /etc/pam.d/proftpd files say

#%PAM-1.0
auth       sufficient   /lib/security/pam_ldap.so
account    sufficient   /lib/security/pam_ldap.so                               
Now all imap/pop3/ftp authentifications are handed to the ldap server. So user aghaffar_developer_ch gets authentificated and gets his mails even though he does not exists on the system's passwd or nis records.

Managing Dns via LDAP

DNS does not at the moment have an LDAP back end, and perhaps its not a good idea to have an LDAP backend for DNS except for Dynamica DNS which is used in conjunction with DHCP. Any way, we use LDAP to store information about DNS and then extract it to create DNS zone files. This allows us to manage everything centrally and from a secure machine, while not making any changes to DNS.

The DNS entries in the LDAP look like this
dn: ou=dnsdata, domain=4unet.net, o=ispman
domain: 4unet.net
ou: dnsdata
objectclass: top
objectclass: domainrelatedobject
objectclass: posixAccount
uid: 4unet.net
uidNumber: 2000
gidNumber: 1000
homeDirectory: /home/4unet.net
userPassword: {crypt}XXffGGHH
loginShell: /bin/true
The dnsdata branch definition. It also defined a posixAccount(a user) with the same name as the domain name. This user is sort of webmaster, who can log in via ftp and upload files to the designated areas etc.
dn: cn=soarecords, ou=dnsdata, domain=4unet.net, o=ispman
cn: soarecords
primary: ns1.4unet.net
ou: dnsdata
retry: 1800
rootmail: dnsmaster.4unet.net
domain: 4unet.net
minimum: 432000
objectclass: top
objectclass: domainRelatedObject
expire: 1209600
refresh: 21600
This defines the SOA records for the DNS for 4unet.net.
A script is responsible to extract the values and format them into a DNS SOA record to be included in the zone file
dn: cn=nsrecords, ou=dnsdata, domain=4unet.net, o=ispman
domain: 4unet.net
cn: nsrecords
ou: dnsdata
objectclass: top
objectclass: domainRelatedObject
record: @,ns1.4unet.net
record: @,ns2.4unet.net
And these are the NS records.
A script will grab all the record attributes, and split them by character "," and get the target and nameserver to be added to the zonefile
dn: cn=mxrecords, ou=dnsdata, domain=4unet.net, o=ispman
domain: 4unet.net
cn: mxrecords
ou: dnsdata
objectclass: top
objectclass: domainRelatedObject
record: @,10, mx1.4unet.net
record: @,100, mx2.4unet.net
Same as above, but MX records this time.
These records also contains the priority field which is accordingly adjusted in the MX records of the zonefile
dn: cn=arecords, ou=dnsdata, domain=4unet.net, o=ispman
objectclass: top
objectclass: domainRelatedObject
domain: 4unet.net
cn: arecords
ou: dnsdata
record: ns1, 193.247.80.43
record: ns2, 193.247.80.44
record: @,193.247.80.43
record: @,193.247.80.44
These are the A records, simply host, ip address mappings
dn: cn=cnames, ou=dnsdata, domain=4unet.net, o=ispman
objectclass: top
objectclass: domainRelatedObject
domain: 4unet.net
cn: cnames
ou: dnsdata
record: ftp, www
record: mail, www
record: *, www
And the CNAMES or Aliases for the hosts
All these scripts are available with the ISPMan package. Have a look at the screenshot to see how to modify a NS record

Configuring proftpd

You need to have proftpd with the LDAP module installed. If you are going to run ftp with virtual servers etc, and you think it will be quiet often busy then you should run it standalone instead of from inetd.
So comment the ftp lines from your inetd.conf and reload the inetd daemon.
Create a group with the gid 1000 called ftponly. We will give this gid to all domains.

edit your /etc/pam.d/proftpd as shown in LDAP authentification section above.
your /etc/proftpd.conf should look something like this.

To run proftpd, you can type /usr/sbin/proftpd and to kill it, killall /usr/sbin/proftpd

Setting up Cyrus

Compile and install Cyrus SASL and imapd, and the UW-IMAP client's c-sdk. The IMAP, client's sdk may already be installed on your system. Try first rpm -aq | grep imap. If in doubt, compile and install a newer version anyway. create a system user called cyrus and group mail, follow the installation instruction and test that you imap server is working. Once it is working simply set /etc/pam.d/imap as shown above in the LDAP authentification section.

you will need to create a user called cyrus or whatever admin your gave to /etc/imapd.conf in the LDAP directory.
For example if your cyrus admin is called "cyrus", then you may have an entry like the one below in your ldap directory.

dn: uid=cyrus, ou=admins, o=ispman
cn: Cyrus Admin
sn: Cyrus
objectclass: top
objectclass: systemadmins
uid: cyrus
userpassword: XXDDCCYY
ou: admins                                                                      
ISPMan will make these entries for your during setup.

Setting up Postfix

Postfix is very friendly to LDAP. You can make virtual domain and virtual user lookups directly in LDAP so you dont have to specify which domains you should recieve mails for.
For example, if a mail arrives for domain "perl.ch", it should look in all domains and see it there is a domain called "perl.ch", if there is then it should accept the mail instead of returning "Duh MX for perl.ch loops back to myself".

my /etc/postfix/main.cf looks like this

Managing users with ISPMan

ISPMan makes it easy to manage users. Creating a user consists of two steps

  1. creating an entry in the LDAP database
  2. creating a mailbox on the IMAP server
Deleting a user, deleted both of these entries for you.

Scaling to more machines

ISPMan allows you to create user's mailbox to any machine in your mailfarm. For example, you may have mail1, mail2, mail3, mail4 etc, each managing 10,000 users. using the LDAP + Postfix + Cyrus combo, you can deliver mail to any of the internal machines.
For example, mail arrives for bob@developer.ch, Postfix asks ldap server to retun the maildrop for the entry that matches mailacceptinggeneralid=bob@developer.ch, and the LDAP server returns bob@mail5.developer.ch. The mail is then routed to the machine called mail5 in your mailfarm.

Currently I am working along with some cool developers on a IMAP/pop3 proxy that runs on the frontend mail servers on pop3 and imap port and transparently redireects requests to internal machines. So your users will only know one address for example mail.developer.ch or pop.developer.ch to connect to instead of knowing which mailserver their mail resides on.

providing web mail to users

IMP is a great product for providing web mail to users.
You can set an alias of mail.* in apache's httpd.conf to point to the central IMP installation.

For example the following is from my installation

I am working on a slightly modified version of IMP that will handle the following cases.

thus you can allow all your clients to read emails by simply going to http://mail.their.domain

Resources

PAM LDAP Module
The pam_ldap module provides the means for Solaris and Linux workstations to authenticate against LDAP directories, and to change their passwords in the directory.
openLDAP
The OpenLDAP Project is a collaborative effort to develop a robust, commercial-grade, fully featured, and open source LDAP suite of applications and development tools. The project is managed by a worldwide community of volunteers that use the Internet to communicate, plan, and develop the OpenLDAP Suite and its related documentation.
Postfix
The excellent SMTP server
Cyrus SASL library and imapd
The Cyrus Electronic Mail Project is continuing to build a highly scalable enterprise mail system designed for use in a small to large enterprise environments using standards based technologies. The Cyrus technologies will scale from independent use in small departments to a system centrally managed in a large enterprise.
ISPMan (the Exchange killer)
The ISP management software I was plugging in this article
proftpd
A secure ftp server