[INUG-Users] Survey 1 - probe rules function calls
Don Wildman
WILDMAN at uk.ibm.com
Sun Sep 6 16:18:13 EDT 2009
Java vs Perl vs ... has certainly generated some interesting debate worthy
of a separate thread,
It is a tad off topic for my original reason for posting a thread
regarding function calls within the probe rules for a couple of reasons
that may not be apparent to the members who are relatively new to OMNIbus.
Probes may be considered as two distinct sections:-
Firstly, the points that make a probe different to other probes, delivered
as a patch to a base OMNIbus install. These deal with the southbound
protocols and connectivity; log, snmp. corba, telnet, jms etc; and parsing
the various event data formats to a common set of tokens for additional
processing. The probes are written in several languages including java.
Second, the common functionality delivered as a set of libraries with core
OMNIbus. These libraries, written in C, provide the rules, lookup,
ObjectServer connectivity, Encryption, SAF, etc for all probes and are
defined in the generic probe and gateway guide. These common functions
are designed to manipulate the incoming data into the users defined alert
structure within OMNIbus, in particular assigned appropriate values to the
columns that drive the generic triggers within the ObjectServer.
At this time as we look at requirement detail for 2010 onwards I am
seeking to gather INUG views on the functionality that the common
libraries deliver. In other words, what use case scenarios should the
rules function provide as distinct from how should that functionality be
implemented. I have certainly seen some useful input in the stream of
responses to this thread.
We will certainly review the "how" aspects that have emerged in this
thread as we move into implementation, but at the moment are focussed on
describing "what" functionality should the rules syntax support.
Once again my thanks to all who have contributed on both the what and the
how.
Regards
Don Wildman
World Wide Product Manager ** Event Management **
users-bounces at netcoolusers.org wrote on 04/09/2009 23:03:09:
> [image removed]
>
> Re: [INUG-Users] Survey 1 - probe rules function calls
>
> Daniel L. Needles
>
> to:
>
> users
>
> 04/09/2009 23:24
>
> Sent by:
>
> users-bounces at netcoolusers.org
>
> Please respond to users
>
> That was actually my point. Java has a very good garbage collector but
it
> cannot deal with circular references and the alike. PERL is the same but
is
> worse at detection. So all things being the same PERL should be worse
than
> JAVA at memory bleeds but then code isn't the issue.
>
> The code culture is really what determines in these cases the extent of
the
> issue. You can search the PERL world and certainly find leaks, but in
the
> JAVA world the majority of the apps I've worked with have leaks. This
has a
> lot to do with the top down nature of JAVA and how the language is built
to
> encourage using junior programmers to fill in the frameworks. This does
make
> business sense except when you have highly threaded, high transaction,
type
> apps - i.e. probes, pollers, and alike.
>
> (BTW on the JAVA side Thinking in JAVA is an excellent resource.)
>
> -----Original Message-----
> From: users-bounces at netcoolusers.org [
mailto:users-bounces at netcoolusers.org]
> On Behalf Of Greg Hubbard
> Sent: Friday, September 04, 2009 4:50 PM
> To: users at netcoolusers.org
> Subject: Re: [INUG-Users] Survey 1 - probe rules function calls
>
> Apropos of nothing in the title of the thread, I have been reading Daman
> Conway's "Object Oriented Perl" (an excellent book) and there is an
> interesting discussion about how lazy programmers can cause Perl to leak
> memory -- has to do with creating linked data structures and then
deleting
> the closest one (or letting it go out of scope) so the "reference
counting"
> garbage collector never sees the reference counting go to zero so the
memory
> remains allocated. The symbol table may grow in size if you have an
ever
> growing hash. I learned an easy way to make a Perl process crash by
using a
> Perl HTML table formatting module (which formats a table in memory
before
> writing it out) and then creating a very large table. So Perl may have
been
> completely innocent and the programmer should have been traded for
target
> practice Yugo.
>
> On 9/4/09, Jacob Steinberger <trefalgar at realitybytes.net> wrote:
> >
> > I've found memory leaks in Perl:POE ;)
> >
> > Jacob
> >
> > Quoting "Daniel L. Needles" <dneedles at gmail.com>:
> >
> > > I'd have to agree with Dougie here. When it comes to practice I have
> > never
> > > had an issue with memory leaks when it comes to PERL based
applications
> > (and
> > > I do mean apps, not scripts.) PERL does have backward compatibility
> > issues
> > > when using its GUI package TK, but other than that PERL isn't
plagued by
> > the
> > > same issues that follow JAVA around: memory leaks and backward
> > > compatibility.
> > >
> > > -----Original Message-----
> > > From: users-bounces at netcoolusers.org [mailto:
> > users-bounces at netcoolusers.org]
> > > On Behalf Of douglasstvnsn at aol.com
> > > Sent: Friday, September 04, 2009 11:55 AM
> > > To: users at netcoolusers.org
> > > Subject: Re: [INUG-Users] Survey 1 - probe rules function calls
> > >
> > >
> > >
> > >
> > >
> > > Hmmmm... Other than a bug in Perl 5.2, I haven't run into much
memory
> > > leakage / growth problems. I have had scripts that have ran
continuous
> > for
> > > months and were still the same size... Even on HPUX which statically
> > assigns
> > > buffers!
> > >
> > > Now Java...? ARGH!
> > >
> > > Dougie!!!
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Alex Greenbank <agreenbank at uk.ibm.com>
> > > To: users at netcoolusers.org
> > > Sent: Fri, Sep 4, 2009 9:07 am
> > > Subject: Re: [INUG-Users] Survey 1 - probe rules function calls
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Hello,
> > >
> > > That's why I said you'd need to make it persistent, the perl
environment
> > > is instantiated only once and the script compiled only once, but
then
> > > you run into possible memory growth problems of long running
scripts.
> > >
> > > You could have middle ground of it running for a certain period, or
> > > certain number of events before saving it's state and restarting,
but
> > > it's already sounding a bit kludgey.
> > >
> > > Ta,
> > >
> > > -Alex
> > >
> > > users-bounces at netcoolusers.org wrote on 04/09/2009 13:55:43:
> > >> I bet you could embed perl in a probe -- most of the "Hello world"
> > > overhead
> > >> is compilation. I think you would need to be selective about which
> > > portions
> > >> of Perl are exposed in a probe implementation.
> > >>
> > >> GLH
> > >>
> > >>
> > >> On 9/4/09, Alex Greenbank <agreenbank at uk.ibm.com> wrote:
> > >> >
> > >> > Hello,
> > >> >
> > >> > The problem is dealing with runaway processes/scripts. What does
the
> > >> > probe do if the perl script never returns? How does it detect
that it
> > >> > has gone awry?
> > >> >
> > >> > Do you use a timeout? If so, what value? If you want to maintain
200
> > >> > events a second then your script will need to run in less than 5
> msec.
> > >> >
> > >> > If you want any kind of performance you'll have to have a
persistent
> > >> > script and communicate to it through stdin/stdout, or UNIX or TCP
> > >> > socket(s).
> > >> >
> > >> > You can't incur the performance hit of fork()/exec() perl every
time
> > >> > you want to do run it.
> > >> >
> > >> > devtest4# cat h.pl
> > >> > #!/usr/bin/perl
> > >> >
> > >> > use strict;
> > >> >
> > >> > print "Hello world.\n";
> > >> >
> > >> > devtest4# time ./h.pl
> > >> > Hello world.
> > >> >
> > >> > real 0m0.133s
> > >> > user 0m0.010s
> > >> > sys 0m0.040s
> > >> >
> > >> > There's the 5 msec budget blown wide apart already, purely
invoking a
> > >> > "Hello World." bit of perl.
> > >> >
> > >> > 0.133s is 8 "Hello world." invocations per second. 8 events per
> > > second.
> > >> >
> > >> > A persistent script means the possibility of memory growth either
> from
> > >> > your own hand-coded memory leaks, or those that exist in any of
the
> > >> > software you're using within your script. Are you sure Perl-DBI
won't
> > >> > slowly leak?
> > >> >
> > >> > And Perl doesn't make it tremendously easy to write code that
won't
> > >> > grow either. Without explicit memory allocation/freeing calls
you're
> > >> > relying on "use strict", scope and its garbage collection to keep
it
> > >> > all neat and tidy.
> > >> >
> > >> > Don't get me wrong, I like Perl and use it all the time, but for
> > >> > scripts that process data, parse log files or for cgi-bin. All of
> > >> > the memory issues disappear when the process exits after doing
> > >> > its job. I'm not brave enough to write a persistent complex
daemon in
> > >> > Perl.
> > >> >
> > >> > State: if you maintain some form of state, and rely on this to
> process
> > >> > events differently based on this state, are you saving it in case
> your
> > >> > process is restarted (or killed because it exceeds the timeout
> because
> > >> > the database you're talking to is a bit busy). Maintaining and
> writing
> > >> > out state will take another chunk of your 5 msec budget.
> > >> >
> > >> > Yes, perl() would solve all manner of issues, but we don't look
at it
> > >> > from a "what clever stuff can it do for us?" perspective. We have
to
> > >> > look at it from a "how could we implement it and protect against
it
> > >> > breaking what you rely on?"
> > >> >
> > >> > Ideally the rules language should be powerful enough for you to
do
> > >> > everything you want to do to manipulate the event data before
> > > insertion
> > >> > into the ObjectServer in linear predictable time. It's not the
place
> > >> > for accessing external data.
> > >> >
> > >> > Ta,
> > >> >
> > >> > -Alex
> > >> >
> > >> > users-bounces at netcoolusers.org wrote on 03/09/2009 21:52:14:
> > >> > > I agree with a lot that Greg says. Regarding perl... why not
have
> > > the
> > >> > > ability to call perl code from with in the rules files code? I
have
> > >> > > suggested this in the past. Create a new function called...
(get
> > > ready
> > >> > for
> > >> > > this)... "perl()" This function could accept either a block of
> code
> > >> > > directly, or the path to a perl script. All of the elements
that
> > > would
> > >> > > normally be available in the rules file are passed (possibly by
> > >> > reference)
> > >> > > in a hash to the perl code, and are passed back to the rules
file
> > > when
> > >> > the
> > >> > > perl code exits. This would also easily add loops, functions
and
> > > other
> > >> > > capabilities for advanced users, without bloating the base rule
> file
> > >> > > language.
> > >> > >
> > >> > > Rob
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: users-bounces at netcoolusers.org
> > >> > [mailto:users-bounces at netcoolusers.org]
> > >> > > On Behalf Of Greg Hubbard
> > >> > > Sent: Thursday, September 03, 2009 1:32 PM
> > >> > > To: users at netcoolusers.org
> > >> > > Subject: Re: [INUG-Users] Survey 1 - probe rules function calls
> > >> > >
> > >> > > Question 1 -- I guess so.
> > >> > >
> > >> > > Question 2 -- I am biased toward Perl, which is pretty darn
> > > flexible.
> > >> > >
> > >> > > I wrote some "X in Y" code that I used like a function by
> > > including
> > >> > it
> > >> > > where I needed it. It needs some input values, which I store
in
> > >> > specific
> > >> > > elements. It has its own "working storage" using some more
> > > elements. It
> > >> > > returns a single value which is stored in a specific element.
> > > However,
> > >> > it
> > >> > > could have returned multiple elements.
> > >> > >
> > >> > > The reason I did it this way was to encapsulate the logic in
one
> > > file. I
> > >> > > was inspired by the DecodeMOI bits in NcKL (or whatever they
are
> > >> > called).
> > >> > >
> > >> > > So the problem is how to set up the syntax for a function call,
and
> > > to
> > >> > > protect any read only arguments while enabling read/write and
allow
> > >> > multiple
> > >> > > values to be returned (although this might not be a kill shot).
> > > Using
> > >> > > elements (string variables) the way I am requires discipline,
and
> > > anyone
> > >> > > else who edits any file in the hierarchy might clobber my code
by
> > >> > stepping
> > >> > > on one of the internal elements, or vice versa.
> > >> > >
> > >> > > Frankly, I think the probe rules need to be looked over now
that
> > > people
> > >> > are
> > >> > > doing more than just copying elements into output fields. The
lack
> > > of
> > >> > > looping is particularly ugly, and it would be nice to have a
way to
> > > log
> > >> > > messages in a user log that is separate from the standard log
so it
> > > is
> > >> > > easier to review them. On busy probes the log files can turn
over
> > >> > pretty
> > >> > > quick.
> > >> > >
> > >> > > GLH
> > >> > >
> > >> > >
> > >> > > On 9/3/09, Don Wildman <WILDMAN at uk.ibm.com> wrote:
> > >> > > >
> > >> > > > Often discussed and up for review.
> > >> > > >
> > >> > > > Everyone of course knows what a function call is. But I would
bet
> > > that
> > >> >
> > >> > > > if I asked 5 people to design the functionality they would
come
> up
> > >> > > > with 5 solutions with differences of varying subtlety, Indeed
> > > having
> > >> > > > sometime in the distant past programmed in Assembler,
Fortran,
> > > COBOL,
> > >> > > > Pascal, C, and various so called 3G languages but never java,
I
> > > could
> > >> > > > come up with several on my own.
> > >> > > >
> > >> > > > Question 1:-
> > >> > > > Would it be enough if the current use of include files to
repeat
> > > often
> > >> >
> > >> > > > used rule fragments loaded faster and were more memory
efficient?
> > >> > > >
> > >> > > > Question 2:-
> > >> > > > Leaving aside what I see in NcKL, please describe your
scenario
> of
> > > how
> > >> >
> > >> > > > a function call would work in your ideal rule file world.
> (English
> > >> > > > description preferred, not code fragments)
> > >> > > >
> > >> > > > Thanks for your help
> > >> > > > Don Wildman
> > >> > > > Product Manager
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Unless stated otherwise above:
> > >> > > > IBM United Kingdom Limited - Registered in England and Wales
with
> > >> > > > number 741598.
> > >> > > > Registered office: PO Box 41, North Harbour, Portsmouth,
> Hampshire
> > > PO6
> > >> >
> > >> > > > 3AU
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > _______________________________________________
> > >> > > > Sent by the netcoolusers.org "users" mailing list
> > >> > > > Post: users at netcoolusers.org
> > >> > > > Unsubscribe: users-unsubscribe at netcoolusers.org
> > >> > > > Search: http://netcoolusers.org/Search
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Disclaimer: 1) all opinions are my own, 2) I may be completely
> > > wrong,
> > >> > 3) my
> > >> > > advice is worth at least as much as what you are paying for it,
or
> > > your
> > >> > > money cheerfully refunded.
> > >> > > _______________________________________________
> > >> > > Sent by the netcoolusers.org "users" mailing list
> > >> > > Post: users at netcoolusers.org
> > >> > > Unsubscribe: users-unsubscribe at netcoolusers.org
> > >> > > Search: http://netcoolusers.org/Search
> > >> > >
> > >> > >
> > >> > > _______________________________________________
> > >> > > Sent by the netcoolusers.org "users" mailing list
> > >> > > Post: users at netcoolusers.org
> > >> > > Unsubscribe: users-unsubscribe at netcoolusers.org
> > >> > > Search: http://netcoolusers.org/Search
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Unless stated otherwise above:
> > >> > IBM United Kingdom Limited - Registered in England and Wales with
> > > number
> > >> > 741598.
> > >> > Registered office: PO Box 41, North Harbour, Portsmouth,
Hampshire
> PO6
> > > 3AU
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > _______________________________________________
> > >> > Sent by the netcoolusers.org "users" mailing list
> > >> > Post: users at netcoolusers.org
> > >> > Unsubscribe: users-unsubscribe at netcoolusers.org
> > >> > Search: http://netcoolusers.org/Search
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Disclaimer: 1) all opinions are my own, 2) I may be completely
wrong,
> > > 3) my
> > >> advice is worth at least as much as what you are paying for it, or
your
> > >> money cheerfully refunded.
> > >> _______________________________________________
> > >> Sent by the netcoolusers.org "users" mailing list
> > >> Post: users at netcoolusers.org
> > >> Unsubscribe: users-unsubscribe at netcoolusers.org
> > >> Search: http://netcoolusers.org/Search
> > >
> > >
> > >
> > >
> > >
> > >
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
PO6
> > 3AU
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Sent by the netcoolusers.org "users" mailing list
> > > Post: users at netcoolusers.org
> > > Unsubscribe: users-unsubscribe at netcoolusers.org
> > > Search: http://netcoolusers.org/Search
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Sent by the netcoolusers.org "users" mailing list
> > > Post: users at netcoolusers.org
> > > Unsubscribe: users-unsubscribe at netcoolusers.org
> > > Search: http://netcoolusers.org/Search
> > >
> > > _______________________________________________
> > > Sent by the netcoolusers.org "users" mailing list
> > > Post: users at netcoolusers.org
> > > Unsubscribe: users-unsubscribe at netcoolusers.org
> > > Search: http://netcoolusers.org/Search
> > >
> >
> >
> >
> > _______________________________________________
> > Sent by the netcoolusers.org "users" mailing list
> > Post: users at netcoolusers.org
> > Unsubscribe: users-unsubscribe at netcoolusers.org
> > Search: http://netcoolusers.org/Search
> >
>
>
>
> --
> Disclaimer: 1) all opinions are my own, 2) I may be completely wrong,
3) my
> advice is worth at least as much as what you are paying for it, or your
> money cheerfully refunded.
> _______________________________________________
> Sent by the netcoolusers.org "users" mailing list
> Post: users at netcoolusers.org
> Unsubscribe: users-unsubscribe at netcoolusers.org
> Search: http://netcoolusers.org/Search
>
> _______________________________________________
> Sent by the netcoolusers.org "users" mailing list
> Post: users at netcoolusers.org
> Unsubscribe: users-unsubscribe at netcoolusers.org
> Search: http://netcoolusers.org/Search
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
More information about the Users
mailing list