[INUG-Users] Users Digest, Vol 59, Issue 23
Daniel L. Needles
dneedles at gmail.com
Sat Oct 24 13:35:21 EDT 2009
Mark,
Might want to do a count. If TIME_WAIT is the majority, it indicates the TCP
connections are not timing out correctly. Though TCP has a 3 way handshake
to open a connection it is a four way close. So the client side can close
the connection. If the server doesn't see the closure, depending on
parameters it can take a while for the server to close the connection. This
would at least get the denial of service attack by the product under
control. 8-)
On the other end of things - why Netcool is doing a denial of service attack
on itself I have seen this issue with the wrong version of JAVA client as
well with object server bugs on various releases. Often it is a combination
of both. I would investigate the client JAVA version that have caused the
problem in the past and see if there is a correlation. I would also open a
PMR in the off chance that this problem has been identified and/or IBM can
correct it.
Thanks,
Daniel
-----Original Message-----
From: users-bounces at netcoolusers.org [mailto:users-bounces at netcoolusers.org]
On Behalf Of mark.3.thompson at bt.com
Sent: Saturday, October 24, 2009 11:26 AM
To: users at netcoolusers.org
Subject: Re: [INUG-Users] Users Digest, Vol 59, Issue 23
We have done netstat diagnostics and the connections vary from being
"ESTABLISHED" to "TIME_WAIT". I have not looked any deeper into the TCP
side of things because I never suspected I would see evidence of the
problem in the traffic, more the symptom, I will try and catch this on
the next issue that we come across.
Thanks, Mark.
-----Original Message-----
From: users-bounces at netcoolusers.org
[mailto:users-bounces at netcoolusers.org] On Behalf Of
users-request at netcoolusers.org
Sent: 24 October 2009 17:00
To: users at netcoolusers.org
Subject: Users Digest, Vol 59, Issue 23
Send Users mailing list submissions to
users at netcoolusers.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.netcoolusers.org/mailman/listinfo/users
or, via email, send a message with subject or body 'help' to
users-request at netcoolusers.org
You can reach the person managing the list at
users-owner at netcoolusers.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Users digest..."
Today's Topics:
1. Rising Netcool Omnibus Objectserver Connections
(mark.3.thompson at bt.com)
2. Re: Rising Netcool Omnibus Objectserver Connections
(Daniel L. Needles)
----------------------------------------------------------------------
Message: 1
Date: Fri, 23 Oct 2009 23:03:02 +0100
From: <mark.3.thompson at bt.com>
Subject: [INUG-Users] Rising Netcool Omnibus Objectserver Connections
To: <users at netcoolusers.org>
Message-ID:
<90AE9C8202E991478D279E6A3053E623013C760C at E03MVX3-UKDY.domain1.systemhos
t.net>
Content-Type: text/plain; charset="us-ascii"
I have a problem that has been intermittently happening for a while now
and I am starting to think it is either a bug or in-compatibility issue
between the objectserver version & maybe the desktop client. The problem
is to do with the number of objectserver connections (port 4100 as
standard) rising to 500+ with the culprit always being a desktop client,
obviously 500+ connections breaks the max number allowed and legitimate
connections start to fail/stop connecting.
The problem always arises following a genuine outage of the objectserver
primary and the culprit is (thus far) always been an laptop/PC where an
Omnibus client has been connected but not manned from before the
failure, i.e. a person has not at the PC/laptop to see the
disconnect/re-connect warning message when the Objectserver has failed.
When the service is restored this rogue Omnibus client starts connecting
to the objectserver eventually running up 100's of connections, please
note it is not the same laptop/PC, it could be any.
We are running a 3 layer Omnibus model, collection, aggregation and
display layer (6 objectservers), the clients connect to the display
layer but also have a writer connection to the aggregation layer, it is
at the aggregation layer that we see the excessive connections.
Here is the output from the version command at the aggregation layer
objectserver, the clients are at various versions of 7.x:
root at tpomni02 # ./nco_objserv -version
Netcool/OMNIbus Object Server - Version 7.1
Copyright (C) 1994 - 2005, Micromuse Ltd. All rights reserved.
10/23/09 22:59:00: Warning: W-BASE-004-066: Failed to load properties
file /opt/netcool/omnibus/etc/NCOMS.props: No such file or directory
Code Revision: 5.2.76
Library Revisions:
libnetcool: 5.2.76
libncmd: 5.2.76
libnregion: 5.2.62
libnmemstore: 5.2.64
libnsecurity: 5.2.62
libnstore: 5.2.69
libnproc: 5.2.76
libnauto: 5.2.67
libnipc: 5.2.76
libnstk: 5.2.62
libnobjserv: 5.2.67
network::ipv6: 5.2.62
Compilation Date: Fri May 25 17:25:10 BST 2007
Compilation Machine: SunOS 5.8 sparc
Compilation System: sol8-build2.hursley.ibm.com
Code Generation: PRODUCTION
Registered debug facilities:
nco_objserv_profiler_timings[ON], signal[ON], thread[ON],
cmd[ON],
region[ON], nco_objserv_profiler[ON], ipc_s_rpc[ON],
memstore[ON],
nco_objserv[ON], clock[ON], timer[ON], sec_author[ON],
prop_mgr[ON],
cb_mgr[ON], arg_mgr[ON], ipc_s_res[ON], nco_objserv_vvtr[ON],
ipc_s_not[ON], store[ON], auto[ON], proc[ON],
region_mutation[ON],
sec_audit[ON], module[ON], ipc_s_ini[ON], ipc_s_mut[ON],
ipc_s_evt[ON]
Has anyone seen this issue before or can offer any advice on how to
track down this issue further.
Thanks, Mark.
Mark Thompson
BT Innovate & Design|DKV
IT Service Management|SM(IT)
t:+44(0)114 2772709|m:+44(0)771 0055307
e:mark.3.thompson at bt.com
This electronic message contains information from British
Telecommunications plc, which may be privileged or confidential. The
information is intended for use only by the individual(s) or entity
named above. If you are not the intended recipient, be aware that any
disclosure, copying, distribution or use of the contents of this
information is strictly prohibited. If you have received this electronic
message in error, please notify me by telephone or email (to the number
or email address above) immediately.
Activity and use of the British Telecommunications plc e-mail system is
monitored to secure its effective operation and for other lawful
business purposes. Communications using this system will also be
monitored and may be recorded to secure effective operation and for
other lawful business purposes.
British Telecommunications plc. Registered office: 81 Newgate Street
London EC1A 7AJ Registered in England no: 1800000
------------------------------
Message: 2
Date: Sat, 24 Oct 2009 09:16:37 -0500
From: "Daniel L. Needles" <dneedles at gmail.com>
Subject: Re: [INUG-Users] Rising Netcool Omnibus Objectserver
Connections
To: <users at netcoolusers.org>
Message-ID: <5211E1CD1DA74D2C99821B238608A4F3 at dadlaptop>
Content-Type: text/plain; charset="us-ascii"
If you do a netstat, what state are these connection in on the server?
Also
have you been able to snoop the traffic at all to see the TCP
transaction? I
saw problems similar to this and it was caused by a difference in the
way
the TCP stacks handled resets from the client. It was corrected by OS
fixes.
But it has been a couple years since I've seen this.
-----Original Message-----
From: users-bounces at netcoolusers.org
[mailto:users-bounces at netcoolusers.org]
On Behalf Of mark.3.thompson at bt.com
Sent: Friday, October 23, 2009 5:03 PM
To: users at netcoolusers.org
Subject: [INUG-Users] Rising Netcool Omnibus Objectserver Connections
I have a problem that has been intermittently happening for a while now
and I am starting to think it is either a bug or in-compatibility issue
between the objectserver version & maybe the desktop client. The problem
is to do with the number of objectserver connections (port 4100 as
standard) rising to 500+ with the culprit always being a desktop client,
obviously 500+ connections breaks the max number allowed and legitimate
connections start to fail/stop connecting.
The problem always arises following a genuine outage of the objectserver
primary and the culprit is (thus far) always been an laptop/PC where an
Omnibus client has been connected but not manned from before the
failure, i.e. a person has not at the PC/laptop to see the
disconnect/re-connect warning message when the Objectserver has failed.
When the service is restored this rogue Omnibus client starts connecting
to the objectserver eventually running up 100's of connections, please
note it is not the same laptop/PC, it could be any.
We are running a 3 layer Omnibus model, collection, aggregation and
display layer (6 objectservers), the clients connect to the display
layer but also have a writer connection to the aggregation layer, it is
at the aggregation layer that we see the excessive connections.
Here is the output from the version command at the aggregation layer
objectserver, the clients are at various versions of 7.x:
root at tpomni02 # ./nco_objserv -version
Netcool/OMNIbus Object Server - Version 7.1
Copyright (C) 1994 - 2005, Micromuse Ltd. All rights reserved.
10/23/09 22:59:00: Warning: W-BASE-004-066: Failed to load properties
file /opt/netcool/omnibus/etc/NCOMS.props: No such file or directory
Code Revision: 5.2.76
Library Revisions:
libnetcool: 5.2.76
libncmd: 5.2.76
libnregion: 5.2.62
libnmemstore: 5.2.64
libnsecurity: 5.2.62
libnstore: 5.2.69
libnproc: 5.2.76
libnauto: 5.2.67
libnipc: 5.2.76
libnstk: 5.2.62
libnobjserv: 5.2.67
network::ipv6: 5.2.62
Compilation Date: Fri May 25 17:25:10 BST 2007
Compilation Machine: SunOS 5.8 sparc
Compilation System: sol8-build2.hursley.ibm.com
Code Generation: PRODUCTION
Registered debug facilities:
nco_objserv_profiler_timings[ON], signal[ON], thread[ON],
cmd[ON],
region[ON], nco_objserv_profiler[ON], ipc_s_rpc[ON],
memstore[ON],
nco_objserv[ON], clock[ON], timer[ON], sec_author[ON],
prop_mgr[ON],
cb_mgr[ON], arg_mgr[ON], ipc_s_res[ON], nco_objserv_vvtr[ON],
ipc_s_not[ON], store[ON], auto[ON], proc[ON],
region_mutation[ON],
sec_audit[ON], module[ON], ipc_s_ini[ON], ipc_s_mut[ON],
ipc_s_evt[ON]
Has anyone seen this issue before or can offer any advice on how to
track down this issue further.
Thanks, Mark.
Mark Thompson
BT Innovate & Design|DKV
IT Service Management|SM(IT)
t:+44(0)114 2772709|m:+44(0)771 0055307
e:mark.3.thompson at bt.com
This electronic message contains information from British
Telecommunications plc, which may be privileged or confidential. The
information is intended for use only by the individual(s) or entity
named above. If you are not the intended recipient, be aware that any
disclosure, copying, distribution or use of the contents of this
information is strictly prohibited. If you have received this electronic
message in error, please notify me by telephone or email (to the number
or email address above) immediately.
Activity and use of the British Telecommunications plc e-mail system is
monitored to secure its effective operation and for other lawful
business purposes. Communications using this system will also be
monitored and may be recorded to secure effective operation and for
other lawful business purposes.
British Telecommunications plc. Registered office: 81 Newgate Street
London EC1A 7AJ Registered in England no: 1800000
_______________________________________________
Sent by the netcoolusers.org "users" mailing list
Post: users at netcoolusers.org
Unsubscribe: users-unsubscribe at netcoolusers.org
Search: http://netcoolusers.org/Search
------------------------------
_______________________________________________
Sent by the netcoolusers.org "users" mailing list
Post: users at netcoolusers.org
Unsubscribe: users-unsubscribe at netcoolusers.org
Search: http://netcoolusers.org/Search
End of Users Digest, Vol 59, Issue 23
*************************************
_______________________________________________
Sent by the netcoolusers.org "users" mailing list
Post: users at netcoolusers.org
Unsubscribe: users-unsubscribe at netcoolusers.org
Search: http://netcoolusers.org/Search
More information about the Users
mailing list