Currently, EXFEL has a number of partitions:
- exfel: the partition which should be used by most EXFEL members
- exfel-theory for EXFEL theory with sub-partitions three work packages in EuXFEL:
- upex: the partition for users of the European XFEL
The exfel partitions comprises various hardware platforms (see Maxwell Hardware for details) where different limits and constraints apply. Below are some details for the exfel partition in Maxwell, but it applies to any of the partitions likewise, just replace the partition exfel with your favorite partition.
Support
Please consult with the documentation provided by the European XFEL.
For questions concerning access to any of the EXFEL partitions please contact it-support@xfel.eu.
For any more generic question or problem please send a message to maxwell.service@desy.de.
Have a look at the Maxwell pages, in particular on getting started, storage, running jobs.
Interactive Login Nodes
To login in to exfel part of the maxwell cluster you have different options depending on the hardware required.
- ssh max-display.desy.de: will connect you to one of the display nodes. FastX might your better choice. Please have a look at the Remote Login and the FastX documentation.
- ssh max-exfl.desy.de: will connect you to max-exfl001,14,15 (INTEL based part of the partition). It's a load-balanced alias. Use theses nodes to compile, test INTEL-tailored applications.
- ssh max-wgs: will connect you to the generic login node.
- Unless you use hardware-specific compiler-flags, compilation and job-submission can done from any of the login nodes.
- Please note: max-display.desy.de is directly accessible from outside. All other login nodes can only be reached by first connecting to bastion.desy.de.
Login nodes are always shared resources sometimes used by a large number of concurrent users. Don't run compute or memory intense jobs on the login nodes, use a batch job instead!
The European XFEL Batch resource in Maxwell
As a first step login to one of the login nodes and check which Maxwell resources are available for your account using the my-partitions command:
If it says "yes" for partition "exfel" (or upex etc) you are ready to go. If so you will also see a "yes" at least for partition "all". If not: get in touch with it-support@xfel.eu! Let's assume that you've got the exfel-resource. The exfel-resource also offers great additional opportunities for remote login. It entitles you to use the max-display nodes to connect to the maxwell cluster via your browser or the fastx-client.
Apart from that: If you have an application, which is started by a script called my-application, and doesn't require a GUI, you can simply submit the script as a batch-job:
This works for any application smart enough not to strictly require an X-environment, matlab, comsol, ansys, mathematica, idl and many others can be executed as batch jobs. To make it more convenient you can add the SLURM directives directly into the script:
The email-notification will be sent to <user-id>@mail.desy.de. That should always work, so you don't actually need to specify an email-address. If you do, please make sure it's a valid address. For further examples and instructions please read Running Jobs on Maxwell.
If you think that it's much to complicated to write job-scripts or if you can't afford to invest the time to look into it: we are happy to assist. Please drop a message to maxwell.service@desy.de, we'll try our best.
Running interactive batch jobs
If you absolutely need an interactive environment, X-windows features like a GUI, there are options to do that in the batch environment. For example:
There are a few things to consider:
- Interactive jobs with salloc easily get forgotten, leaving precious resources idle. We do accounting and monitoring!
- Keep the time short: there is hardly a good reason to run an interactive jobs for more than the working hours. Use a batch job instead.
- Terminate allocations as soon as the job is done!
Other Maxwell Resources
Being member of EXFEL and maybe having access to the exfel partition doesn't need to be the end of the story. If you have parallelized applications suitable for the Maxwell cluster you can apply for the Maxwell resource like everyone else on campus. Please send a message to maxwell.service@desy.de briefly explaining your use case. Being also a user of the European XFEL you might also have access to the UPEX partition. You can easily distribute your job over the partitions:
The partition will be selected from exfel OR maxwell OR upex OR all starting with the highest priority partition. So your job will run on the exfel partition if nodes are available, on the maxwell partition if nodes are available and finally on the all partition if none of the other partitions specified have free nodes. Keep in mind that you should however select the partition according to the type of work you are doing. And a job can never combine nodes from different partitions, so check the limits applying to partitions.
To check availability of nodes and characteristics use sinfo (https://slurm.schedmd.com/sinfo.html)
[@max-display001 ~]$ sinfo -p upex -o "%10P %.6D %8c %8L %12l %8m %30f %N" PARTITION NODES CPUS DEFAULTT TIMELIMIT MEMORY AVAIL_FEATURES NODELIST upex 80 80 1:00:00 14-00:00:00 512 INTEL,V4,E5-2698 max-exfl[020-099] upex 2 40 1:00:00 14-00:00:00 256 INTEL,V4,GPU,P100,E5-2640 max-exflg[006-007] upex 4 40 1:00:00 14-00:00:00 256 INTEL,V4,E5-2640 max-exfl[016-019] [@max-display001 ~]$ sinfo -p upex -o "%10P %.6D %10s" PARTITION NODES JOB_SIZE upex 86 1-16