[Developers] disk space usage reporting

[David Rideout]

In my opinion, a crucial feature to any solution is that it make life
easier for users, not harder.  Let's place as much burden (of managing
finite disk resources) as possible
on Cactus, or on some grid tools at Tom suggests, rather than ourselves.

-David

On 10/20/06, Erik Schnetter <schnetter at cct.lsu.edu> wrote:
> A way that is similar in spirit would be per-job quota.  When you
> submit a job, you have to apply for a certain amount of disk space.
> PBS keeps track of that.  If there is not enough space left, you
> cannot submit your job.  If you jobs uses more, it is aborted.
>
> This doesn't have to be implemented with operating system quotas; a
> script could use "du" every hour to check on the jobs' output
> directories.
>
> -erik
>
> On Oct 20, 2006, at 08:38:45, Tom Goodale wrote:
>
> > Yep.  Just having a way of getting estimated disk usage is a good
> > start.
> >
> > Once we have that, we could, for instance, have an external process
> > get this from all cacti running on a machine which could then raise
> > an alarm if there is likely to be a problem, or even steer storage
> > parameters to try to work around the problem and migrate older data
> > off the system.
> >
> > Another thing is to know early on that there's just no-way the job
> > is going to be able to write all its data out unless some more
> > space is made.
> >
> > Cheers,
> >
> > Tom
> >
> > On Fri, 20 Oct 2006, David Rideout wrote:
> >
> >> Indeed, it will be quite difficult in general to predict when the
> >> disk
> >> will fill.  But this should not cause us to abandon ideas along these
> >> lines.
> >>
> >> -David
> >>
> >> On 10/20/06, Steve White <steve.white at aei.mpg.de> wrote:
> >>> Frank,
> >>>
> >>> On 20.10.06, Frank Loeffler wrote:
> >>>> Hi,
> >>>>
> >>>> Steve White wrote:
> >>>>> I have in mind:  if Cactus could calculate and sum up how much
> >>>>> data it
> >>>>> writes in each iteration (and perhaps compare that to remaining
> >>>>> storage
> >>>>> space, and estimate how quickly it will fill it),
> >>>>
> >>>> While this is a very nice idea, it already fails with two jobs
> >>>> writing
> >>>> to the same disc. Each would estimate more time to fillup than one
> >>>> really has.
> >>>>
> >>> Of course, this is a consideration.  There are many pragmatic
> >>> measures one
> >>> can take to guess that a problem will soon arise.  For example,
> >>> one could
> >>> use the remaining disk space at iteration n, and that at
> >>> iteration n + 1
> >>> to make an estimate.
> >>>
> >>> As I said, the rest is details.
> >>>
> >>>> Even if it would work, it assumes users who watch to the output
> >>>> of this.
> >>>>
> >>> They could be notified by e-mail, e.g.  This is another detail.
> >>>
> >>>
> >>> --
> >>> Steve White : Programmer
> >>> Max-Planck-Institut für Gravitationsphysik      Albert-Einstein-
> >>> Institut
> >>> Am Mühlenberg 1, D-14476 Golm, Germany
> >>> +49-331-567-7625
> >>>
> >>> _______________________________________________
> >>> Developers mailing list
> >>> Developers at cactuscode.org
> >>> http://www.cactuscode.org/mailman/listinfo/developers
> >>>
> >>
> >> _______________________________________________
> >> Developers mailing list
> >> Developers at cactuscode.org
> >> http://www.cactuscode.org/mailman/listinfo/developers
> > _______________________________________________
> > Developers mailing list
> > Developers at cactuscode.org
> > http://www.cactuscode.org/mailman/listinfo/developers
>
>
> --
> Erik Schnetter <schnetter at cct.lsu.edu>
>
> My email is as private as my paper mail.  I therefore support encrypting
> and signing email messages.  Get my PGP key from www.keyserver.net.
>
>
>
>
>
> _______________________________________________
> Developers mailing list
> Developers at cactuscode.org
> http://www.cactuscode.org/mailman/listinfo/developers
>
>
>
>