You have submitted jobs but they are not starting? Possible causes could be
- you have been using significantly resources in the last hours/days, so that your priority is reduced compared to other users, that have used less resources and which have now a higher priority
- colleagues in your group have been using signified amounts of your overall group quota or resources, so that you might have a higher user priority but have to wait for your colleagues' jobs to finish
- you have requested large resource chunks, which need to time to be freed. E.g., when you request for you job a slot with 8 CPU cores and 20GB of memory, the Condor batch system has to free and wait for 10 standard jobs (which have 1 CPU core and 2GB memory) to finish on one node.
- your jobs might have been run but failed multiple times, so that the Condor batch systems stopped submitting new jobs so that you can check, why your jobs have failed.
Since the NAF is a shared resource, such a fair distribution management has to be applied.
What can You do
- check the overall load in the BIRD batch system
- check the status of your overall user and group priority
> condor_userpriority
- check the status of your jobs, especially their numbers of starts
> condor_status ...
User Priorities
This is a bit of a simplification but may serve as a starting point when it comes to questions about what you actually get out of the NAF and what others get.
First of all your jobs are depending on the fixed quota of the accountinggroup you are using at submit time, most of the time this will be your primary network group.
As we have the goal to get the pool as busy as possible we allow 'default' jobs to run on any ressource available at submit time, ignoring the actual quota of your accounting group. Hence if you stick to 1-core-3hours-2gb jobs you might be lucky and run thousands of those on other peoples ressources for ex. overnight or of course at any other given time if the ressources are not used by their respective owners.
If you need more cores or time we are not able to broker your jobs on any hardware outside your quota.
Let's assume you are running jobs that depend on your group quota which you share with the other members of your group. In this case the quota will be split amongst the active group members according to their priority. The priority is calculated through previous usage that goes back for 7 days (fairshare) and a standard factor that is usually equal for all members of the group. You can check quotas and usage inside your group using 'condor_userprio':
[root@bird-htc-sched13 ~]# condor_userprio Last Priority Update: 1/9 13:49 Group Config Use Effective Priority Res Total Usage Time Since Requested User Name Quota Surplus Priority Factor In Use (wghted-hrs) Last Usage Resources ---------------------- --------- ------- ------------ --------- ------ ------------ ---------- ---------- BIRD_belle.lite 1548.39 ByQuota 1000.00 0 914658.00 0+00:06 4 userx@desy.de 609.72 1000.00 0 1270.72 0+00:09 usery@desy.de 1902.56 1000.00 1 40142.99 0+00:07 BIRD_hera.bide 580.65 no 1000.00 0 1188112.38 0+00:23 0 userz@desy.de 9560.94 1000.00 1 154993.41 0+00:24 BIRD_hera.lite 1161.29 ByQuota 1000.00 0 1055890.75 0+06:09 0 BIRD_jupyter.lite 240.00 ByQuota 100000.00 0 476.18 0+23:45 0 BIRD_ilc.bide 774.19 no 1000.00 6 1407676.62 <now> 6 user1@desy.de 554.91 1000.00 5 20237.93 <now> user2@desy.de 756.83 1000.00 1 1044.77 <now> BIRD_atlas.lite 5806.45 ByQuota 1000.00 1377 3932596.50 <now> 34801 user3@desy.de 534.79 1000.00 10 3824.85 0+00:12 user4@desy.de 2621.37 1000.00 517 87909.19 <now> uer5@desy.de 76364.93 1000.00 787 8523.70 <now> user6@desy.de 106185.06 1000.00 1 15977.39 <now> user7@desy.de 287283.25 1000.00 72 82995.03 <now> <snip>
The most relevant field is the 'effective priority' and smaller values represent a better ranking for gaining resources and large values denote heavy active users, whose chance for free resources is dwindling.
Each group as a general lite and bide share, where the lite value is used for short running jobs <3h without specific requirements (as these jobs can overflow to under-utilized resources from other groups) - while bide applies for longer running jobs.
HTCondor applies a priority system, so that every user has the change to get resources according to his or her assigned quota - where also the recent resource usage gets convoluted into the calculations
I.e., users/groups, which have recently not been very active in their consumption, have a better chance to get resources for their jobs than heavy users.
At first the general group quota is sorted.
E.g., if the ATLAS group as whole has used many resources during the current time window, than each ATLAS user has a smaller chance to get free resources. Now, when Condor considers to send an ATLAS job to a node, it checks then within the group for the user with the best prio value.
Of course, if you request more resources in the form of CPUs and/or memory for your job than a default 1core/2GB mem job, this job gets weighted accordingly.
Please check this link for further information about user priorities and how they affect your jobs (one hint though: the lower your priority in condor_userprio output the better):
https://htcondor.readthedocs.io/en/v8_9_5/users-manual/priorities-and-preemption.html
Of course there is also a man page on the submit host ...