[Developers] disk space usage reporting

[Erik Schnetter]

A way that is similar in spirit would be per-job quota.  When you  
submit a job, you have to apply for a certain amount of disk space.   
PBS keeps track of that.  If there is not enough space left, you  
cannot submit your job.  If you jobs uses more, it is aborted.

This doesn't have to be implemented with operating system quotas; a  
script could use "du" every hour to check on the jobs' output  
directories.

-erik

On Oct 20, 2006, at 08:38:45, Tom Goodale wrote:

> Yep.  Just having a way of getting estimated disk usage is a good  
> start.
>
> Once we have that, we could, for instance, have an external process  
> get this from all cacti running on a machine which could then raise  
> an alarm if there is likely to be a problem, or even steer storage  
> parameters to try to work around the problem and migrate older data  
> off the system.
>
> Another thing is to know early on that there's just no-way the job  
> is going to be able to write all its data out unless some more  
> space is made.
>
> Cheers,
>
> Tom
>
> On Fri, 20 Oct 2006, David Rideout wrote:
>
>> Indeed, it will be quite difficult in general to predict when the  
>> disk
>> will fill.  But this should not cause us to abandon ideas along these
>> lines.
>>
>> -David
>>
>> On 10/20/06, Steve White <steve.white at aei.mpg.de> wrote:
>>> Frank,
>>>
>>> On 20.10.06, Frank Loeffler wrote:
>>>> Hi,
>>>>
>>>> Steve White wrote:
>>>>> I have in mind:  if Cactus could calculate and sum up how much  
>>>>> data it
>>>>> writes in each iteration (and perhaps compare that to remaining  
>>>>> storage
>>>>> space, and estimate how quickly it will fill it),
>>>>
>>>> While this is a very nice idea, it already fails with two jobs  
>>>> writing
>>>> to the same disc. Each would estimate more time to fillup than one
>>>> really has.
>>>>
>>> Of course, this is a consideration.  There are many pragmatic  
>>> measures one
>>> can take to guess that a problem will soon arise.  For example,  
>>> one could
>>> use the remaining disk space at iteration n, and that at  
>>> iteration n + 1
>>> to make an estimate.
>>>
>>> As I said, the rest is details.
>>>
>>>> Even if it would work, it assumes users who watch to the output  
>>>> of this.
>>>>
>>> They could be notified by e-mail, e.g.  This is another detail.
>>>
>>>
>>> --
>>> Steve White : Programmer
>>> Max-Planck-Institut für Gravitationsphysik      Albert-Einstein- 
>>> Institut
>>> Am Mühlenberg 1, D-14476 Golm, Germany                   
>>> +49-331-567-7625
>>>
>>> _______________________________________________
>>> Developers mailing list
>>> Developers at cactuscode.org
>>> http://www.cactuscode.org/mailman/listinfo/developers
>>>
>>
>> _______________________________________________
>> Developers mailing list
>> Developers at cactuscode.org
>> http://www.cactuscode.org/mailman/listinfo/developers
> _______________________________________________
> Developers mailing list
> Developers at cactuscode.org
> http://www.cactuscode.org/mailman/listinfo/developers


-- 
Erik Schnetter <schnetter at cct.lsu.edu>

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from www.keyserver.net.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://www.cactuscode.org/pipermail/developers/attachments/20061020/7556cf4b/attachment.bin