This page is about the FAST system, which I developped during my PhD. It allows the scheduler to get informations about the current state of the system as well as the needs of the routine to schedule.
I modified NetSolve, a RPC based client/agent/server system developed at the University of Teneesee to make use of FAST. The result of this integration is described in this article. I also integrated FAST into DIET, a grid enabled Problem Solving Environment project in which my research team is involved trough the ARC INRIA called OURAGAN.
This project is now discontinued.
About FAST
FAST (Fast Agent System Timer) is a tool for dynamic performance forecasting in a Grid environment. FAST is composed of several layers and relies on low level software. First, it uses a network and CPU monitoring software to handle dynamically changing resources, like workload or bandwidth. FAST uses the Network Weather Service (NWS), a distributed system that periodically monitors and dynamically forecasts the performance of various network and computational resources. The dynamic data acquisition module of FAST uses and enhances NWS. FAST also includes routines to model the time and space needs for each triplet (problem ; machine ; parameters set). They are based on benchmarking at installation time on each machine for a representative set of parameters and polynomial data fitting. To store these static data, FAST uses a LDAP tree.
This is Free Software (GPL). For more information, you can check the my publications, or this presentation (in french).
Download: You can download all dependencies in one big tarball. It contains a small makefile, extracting all software in ~/DIET (change the ROOT variable put it somewhere else), configure, and install them. It should work to on linux/{i386,alpha,sparc} and solaris/sparc.
Related software
FAST may leverage several software when found, but none is absolutely mandatory. If compiled without any of those friends, FAST will still be able to give the current load of the machine it runs on. It is really limited, and you may want to install some of the following software to increase FAST's abilities.
What is really needed?
I would say that NWS is almost
mandatory, that LDAP and GSL are welcome, and that BDB is useless for normal
human beings. But of course, it depends on your personal use of the
beast.
GSL: | GNU Scientific Library
is used to interpolate the result of the benchmarks to a polynomial
function during the installation phasis. Project homepage: http://sources.redhat.com/gsl/ Last tested version: gsl-1.2.tar.gz
|
| |
BDB: | Berkley DB is used to
buffer the results of the benchmarks during the installation phases. Project homepage: http://www.sleepycat.com/ Last tested version: db-4.1.25.tar.gz
|
| |
NWS: | Network Weather Service is
used at run time to get information about system availabilities. Project homepage: http://nws.cs.ucsb.edu/ Last tested version: nws-2.11.tar.gz.
|
| |
LDAP: | Lightweight Directory Access
Protocol is used as database at runtime. Project homepage: http://www.openldap.org/ Last tested version: openldap-2.1.21.tar.gz.
|
| |
FAST: | The SDK itself. Last tested version: fast-0.8.7.1.tar.gz (ChangeLog). |
Installing Fast
FAST depends on several other programs and library, and the resulting system is highly distributed. This makes the installation process a bit tedious. This page tries to explain how to do that.
As stated earlier, the system is composed of two subparts. First the bencher, which should be run only once on every machines which will host computational servers, and an interrogation library, which have to be linked in the program wanting to usee FAST forecasts.
This page is organized as follow: we will first install the benching part and its requirements, then install the runtime library, then we will show how to do some benchmarks, and finally, how to install and use a program using FAST.
Installing the bencher and its requirements
The bencher's work is to do some benchmarks of operations on each
computational server, and then get a polynomial function modeling these
results. For that, it depends on two libraries. Since the benchmarking
process can be pretty long, every result is cached in a database using
the Berkley Database
library. Then, once FAST have all the needed results, the GNU Scientific Library is used
to do the polynomial regression.
Once BDB is installed, you can compile FAST to have only the bencher
without the interpolations, but that's not that useful since you won't be
able to use the result. So I won't explain how to compile FAST in such
conditions. Let's install GSL to be able to do the polynomial
regression.
Once GSL is installed, the bencher is fully functional. Let's see how to use it.
Doing some benchs
(see the documentation within the tarball)
Installing the runtime library and its requirements
The runtime library is composed of two main parts.
- The first one is able to get the requirements of a computation on a given machine. This part needs the result of the bencher, which are stored in a LDAP database.
- The second part of the runtime library knows about the actual availability of the system, which are given by the Network Weather Service (NWS).
To install NWS, simply do make, make install. Configuring NWS is a bit more tricky, but you should succeed anyway.
To install LDAP, make, make install in enough. Trick: add
schemacheck offto the slapd.conf so that the ldif files can be imported even if they are syntaxicaly wrong.
Then, you wan to put the result of benchs in LDAP. When running the benchs, you get some ldif files you have to add in the LDAP tree so that FAST can use them. First of all, make sure your server slapd is running before trying to add anything to the tree
Adding the root of the tree
The first file to add is base.ldif. This
file adds the entries constituting the root of the LDAP tree. To add
this file to the tree, type a command like the following:
ldapadd -H ldap://veloce:9009 -D
cn=admin,ou=People,dc=ens-lyon,dc=fr -w toto -x -c -f
base.ldif<br><br>
where:
- -H ldap://veloce:9009 specifies which LDAP server should be contacted. That's a regular URL, so keep the ldap://, which says the protocol used, change "veloce" to the actual name of your server, and change "9009" to the actual port number on which your server ears.
- -D cn=admin,ou=People,dc=ens-lyon,dc=fr specifies who is trying to do the modification (that's the administrator of the LDAP tree, and should be the same as specified in your slapd.conf
- -w toto is the password of the user trying to modify the tree. Should be the same than specified in slapd.conf.
- -x specifies that we want to use the simpliest way of authentication.
- -c ask to not check the syntax of the ldif file too hardly.
- -f base.ldif gives the name of the file to add.
Adding the result of the benchs
Once the root of the tree is added, you can add the files resulting of
the benchmarking on the servers. The command is the same as above for
each one.