Freeside:Documentation:MirroringRPMRepo

From Freeside
Revision as of 11:53, 13 April 2009 by Ivan (talk | contribs) (→‎Rate Limiting)

Jump to: navigation, search

Mirroring the Freeside RPM Repository

Introduction

If you installed (or intend to install) Freeside from RPM, you may wish to mirror the Freeside RPM repository locally for a variety of reasons, including faster re-installation after a crash, faster testing when doing periodic disaster recovery tests, baselining the repository if you stage changes to your machines through local mirrors, guaranteeing availability of a repository for disaster recovery, you run modified Freeside RPMs but want to mirror all the prerequisite modules, etc.

The complete repository tree is currently about 3GB in size and growing. We strongly recommend that you do not mirror the whole repository. Mirror only the branch(es) for the distribution(s) you use.

Freeside Repository Structure

The current repository structure is: 

	http://wavetail.420.am/~rsiddall - Base URL containing GPG keys and yum repos.d file
		repo/ - Base of RPM repositories
			centos/
				4/
					freeside-1.7/
						stable/
							i386/
							x86_64/
						prerelease/
							i386/
							x86_64/
						testing/
							i386/
							x86_64/
					freeside-1.9/
						stable/
							i386/
							x86_64/
						prerelease/
							i386/
							x86_64/
						testing/
							i386/
							x86_64/
			sles/
				10/
					freeside-1.7/
						stable/
							i386/
							x86_64/
						prerelease/
							i386/
							x86_64/
						testing/
							i386/
							x86_64/
							self-service/
								i386/
								x86_64/
					freeside-1.9/
						stable/
							i386/
							x86_64/
						prerelease/
							i386/
							x86_64/
						testing/
							i386/
							x86_64/
							self-service/
								i386/
								x86_64/

The "testing" branches contain "daily" CVS builds (build-on-change) and may not actually have a complete set of prerequisite Perl modules.

The "centos" branch should work on RHEL as well as CentOS and compatible operating systems.

We currently build only for i386 and x86_64 architectures.

Due to the way SuSE's libzypp treats RPM repositories, the self-service files are kept in a different branch from the billing server RPMs. This is not necessary on CentOS due to yum's better handling of RPMs.

Proxy Servers

The following assumes you are not accessing the internet through a proxy server, or that you are using a transparent proxy server that requires no special configuration for client programs. If this is not the case, you may need to add command line switches to route internet accesses through your proxy server and to authenticate yourself to the server.

Mirroring via Rsync

Freeside Internet Services does not currently make its RPM repository available for anonymous rsync. We may do so in future and will update these instructions if and when anonymous rsync becomes available.

Using curl Instead of wget

The instructions below use the readily available wget utility to fetch files from the web. Some administrators prefer to use curl. This is perfectly reasonable. To keep the instructions simple, details on using curl have been omitted.

Rate Limiting

We would greatly appreciate it if you would include --limit-rate=100k in the wget command line to limit the load on the Freeside repository server.

Instructions

With the exception of a couple of steps to create a folder and make it available via your web server, the steps for initial setup and refreshing the repository are identical. You could put all these steps in a shell script.

Initial Setup

The Freeside RPM repository is just a web server containing the RPMs and some metadata files. To mirror this, you would download the files into a folder and then make that folder available to your local client machines via a web server.

To set up the mirror initially:

  • Decide which distributions, branches, and architectures you are going to mirror, for example sles/10/freeside-1.7/testing/i386 or centos/4/freeside-1.7.
  • Create a folder somewhere on the machine that will act as your repository server (under /var/www or /srv/www, for example):
mkdir /srv/www/freeside-repo
  • Copy down the repository from the Freeside web server:
cd /srv/www/freeside-repo
wget --mirror --no-parent --no-host-directories --cut-dirs=1 http://wavetail.420.am/~rsiddall/repo/sles/10/freeside-1.7/testing/i386/

The trailing slash is important. It prevents wget from downloading all repositories at the same level as the i386 folder.

  • If using SuSE, copy down the corresponding self-service server files if you want them:
wget --mirror --no-parent --no-host-directories --cut-dirs=1 http://wavetail.420.am/~rsiddall/repo/sles/10/freeside-1.7/testing/self-service/i386/
  • If using yum, copy down the signing keys and the yum repository definition file, and then fix up the URLs in the repository definition file so they point to your web server:
wget --mirror --no-parent --no-directories http://wavetail.420.am/~rsiddall/RPM-GPG-KEY-Freeside
wget --mirror --no-parent --no-directories http://wavetail.420.am/~rsiddall/freeside.repo
perl -pi -e 's|wavetail.420.am/~rsiddall|server.domain.tld/freeside-repo|g' freeside.repo
  • Remove index.html files from the mirror:
find /srv/www/freeside-repo -name 'index.html*' -print -exec /bin/rm {} \;
  • Make the folder available via the web server. For Apache, put a configuration snippet in a file (freeside-repo.conf) in /etc/httpd/conf.d or /etc/apache2/conf.d:
Alias /freeside-repo /srv/www/freeside-repo
<Directory /srv/www/freeside-repo>
Options +Indexes
Allow From All
Order Deny,Allow
</Directory>
  • Restart or reload your web server:
service apache2 reload
  • Test the repository:
    • Try doing an installation or update

Refreshing Your Mirror

To refresh the mirror, just repeat the steps to copy down the files: 

cd /srv/www/freeside-repo
wget --mirror --no-parent --no-host-directories --cut-dirs=1 http://wavetail.420.am/~rsiddall/repo/sles/10/freeside-1.7/testing/i386/
wget --mirror --no-parent --no-host-directories --cut-dirs=1 http://wavetail.420.am/~rsiddall/repo/sles/10/freeside-1.7/testing/self-service/i386/

and if using yum: 

wget --mirror --no-parent --no-directories http://wavetail.420.am/~rsiddall/RPM-GPG-KEY-Freeside
wget --mirror --no-parent --no-directories http://wavetail.420.am/~rsiddall/freeside.repo

(Don't forget to fix up the URLs in the yum repository definition file if it is downloaded.)

Remove index.html files from the mirror: 

find /srv/www/freeside-repo -name 'index.html*' -print -exec /bin/rm {} \;

The RPM signing keys and repository definition file will hardly ever change.

The above commands can be put in a shell script and executed periodically via cron.

Using Your Mirror

Follow the instructions for using the Freeside Internet Services RPM repository (CentOS/RHEL or SuSE), but substitute the URL of your repository.

Initial Installation of Freeside

# Add the folder or web server containing SuSE to your installation sources
yast2 inst_source
# Enable SSL and Perl scripting for the HTTP server
yast2 --install apache2-mod_perl
yast2 http-server modules enable=ssl,perl
# Configure apache for SSL
cd /etc/apache2/vhosts.d
cp -p vhost-ssl.template freeside-ssl.conf
perl -pi -e 's/#ServerName www.example.com/ServerName `hostname`/g;' freeside-ssl.conf # Uncomment and correct ServerName
perl -pi -e 's/#ServerAdmin root\@example.com/you\@domain.tld/g' freeside-ssl.conf # Uncomment and correct ServerAdmin, and any SSL file locations
cd
# Copy server cert (csr, crt, and key) to Apache SSL folders
for ext in csr crt key; do cp -p server.${ext} /etc/apache2/ssl.${ext}; done
# Install zypper
yast2 --install zypper
# Update the system
zypper -n update
# Install PostgreSQL (might need a client if database server is not on same box)
zypper -n install postgresql-server
# Installing Freeside itself
zypper service-add http://server.domain.tld/freeside-repo/repo/sles/10/freeside-1.7/testing/i386
zypper -n install freeside-postgresql freeside freeside-mason perl-Business-OnlinePayment-AuthorizeNet

and then finish the installation as normal, starting PostgreSQL and creating a Freeside user and database; then running freeside-setup, freeside-adduser, and restarting the web server.

Switching Existing Clients to Your Repository

If you've previously installed Freeside (billing server and/or self-service) on a client machine and want to switch it to use your local repository, do the following:

  • For CentOS, etc., replace the existing freeside.repo file:
cd /etc/yum.repos.d
mv freeside.repo freeside.repo.orig
wget http://server.domain.tld/freeside-repo/freeside.repo
  • For SuSE, change the installation sources via zypper or YaST2:
zypper service-list # Get number of source using wavetail.420.am, say '2'
zypper service-delete 2
zypper service-add http://server.domain.tld/freeside-repo/repo/sles/10/freeside-1.7/testing/i386
zypper service-list # Make sure it's correct
Retrieved from "http://secure.freeside.biz/mediawiki/index.php?title=Freeside:Documentation:MirroringRPMRepo&oldid=4560"