Amount of Cisco devices for a proxy agent

Nov 30, 2010 at 2:11 PM

Hi,

We're using the 1.1.0.19c version of the xSNMP MP with the Cisco add-on for our SCOM2007R2 setup running on Server 2008R2 with SQL2008SP1.
We have ~250 Cisco devices currently being monitored.

Our proxy agent has registry changes that extends the cache to 160Mb from the original 30Mb as described in http://support.microsoft.com/kb/975057
The agent also has overrides that extends the Health Service Private Bytes and Monitoring Host Private Bytes to 1610612736 as well as Monitoring Host Handle Count set to 10000 (same levels as the Mgmt Server).
The server it resides on is a Dual SixCore AMD Opteron 2.1GHz with 16Gb RAM running Server2008R2.
The problem is that according to the documentation, a dedicated proxy agent should be able to monitor 1500-3000 nodes. We have <250 and are experiencing problems with the agent where the above monitors go red a couple of times a day..

Is this normal / as designed, or is there a problem in our setup?

Note - We have disabled everything in the xSNMP MP and have then enabled the following for ~210 devices:

Monitors:
-
Rules:
Alert on xSNMP Device Reboot
Alert on xSNMP Device Reboot (SNMP Trap)

and the following on top of the above for ~40 additional devices:

Monitors:
xSNMP Cisco Device Chassis Fan Notification
xSNMP Cisco Device Chassis Power Notification
xSNMP Cisco Device Chassis Voltage Notification
xSNMP Cisco Switch Stack Fan Status
xSNMP Cisco Switch Stack Major Alarm Status
xSNMP Cisco Switch Stack Minor Alarm Status
xSNMP Cisco Fan Status
xSNMP Cisco Memory Pool Utilization
xSNMP Cisco Module Status
xSNMP Cisco Power Supply Status
xSNMP Cisco Processor Utilization 

Rules:
Alert on xSNMP Device Reboot
Alert on xSNMP Device Reboot (SNMP Trap)
Collect xSNMP Cisco Device CPU Utilization
Collect xSNMP Cisco Memory Pool Utilization

 

Thanks in advance!

Nov 30, 2010 at 9:31 PM

Hi Jon,

Its 1500-3000 monitored objects, noting that an object includes chassis and network interfaces (and more)!! Its not 1500-3000 devices.

FWIW, I have 275 devices (mostly Cisco/APC and Dell) covered by the xSNMP on one server that is a VM with 2 cores and 4GB and the servers doing several other tasks as well. I have no issues HOWEVER, on my Cisco's I have disabled discovery of network interfaces hence massively reducing my number of objects (I then enable for selective interfaces, mainly WAN and uplink ports).

Cheers,

Aengus

Nov 30, 2010 at 9:34 PM

From the manual;

A dedicated agent proxy should be able to effectively monitor between 1500-3000 monitored
objects (devices, interfaces, chassis objects, volumes, etc). 

Dec 1, 2010 at 2:53 PM

I removed all devices and disabled the Network Interface discovery.
After that I were able to add ~300 devices to a virtual server with just 2 cores and 2GB RAM (= more devices on a server with much lower than our physical server as described above)..
This is an extreme improvement as the same server were only able to host <50 devices before the interface discovery were disabled. When I reached 500 devices I got the same problems as with the physical one though.. So I'll just start up two such virtual servers to split the load.. ;)

AengusM - A huge thanks for pointing me in the correct direction!

To the xSNMP developers - Please include a note on this in the documentation. To me it is a strange problem - high load when there are interfaces discovered that are not monitored, not discovering them solved it.
The entire Cisco xSNMP MP should have a different approach in my opinion - everything disabled by default and enabling the monitors as needed. If just adding the MP on a server with lots of Network Devices the load may kill the server..
The same should be the rule for Network Interfaces in the base xSNMP MP as they seem to be such a performance killer, even if just discovered.
Anyway.. Thanks for a great free product :-)

 

Feb 18, 2011 at 8:02 PM
AengusM wrote:

Hi Jon,

Its 1500-3000 monitored objects, noting that an object includes chassis and network interfaces (and more)!! Its not 1500-3000 devices.

FWIW, I have 275 devices (mostly Cisco/APC and Dell) covered by the xSNMP on one server that is a VM with 2 cores and 4GB and the servers doing several other tasks as well. I have no issues HOWEVER, on my Cisco's I have disabled discovery of network interfaces hence massively reducing my number of objects (I then enable for selective interfaces, mainly WAN and uplink ports).

Cheers,

Aengus


How do you re-enable discovery for specific interfaces? The only way I can see to do it is by type or index. Well, I have uplink ports of ethernet type that are all different indexes. Therefore, I can't really filter by index or type.

I basically want to monitor all serial ports and only ethernet uplink ports. I do not want to discovery all ports as the first time I did this it brought SCOM to its knees. Just like the post above, the interface discovery should be disabled by default.

Feb 20, 2011 at 10:52 PM

In my case it is a bit of manual work...

- Over ride xSNMP Network Interface Discovery and set Enabled to False

- The network guy's get the index of the interfaces they want to monitor (in our case its mostly wan ports) and they create an override using the Index filter, e.g. they will go to the Authoring tab > Filter to xSNMP network Interfaces > Override > Specific xSNMP > BeijingWAN_01 > set enabled to true and enforced and change the Filter approriately, maybe something like (2|23)

- and of course depending on the level of monitoring required we have a filter to include the interfaces in one of the xSNMP network Interface groups..in our case we only want up/down for everything so a broad filter to include all interfaces in "xSNMP Network Interfaces - Status Monitoring Only"

It would be nice to not have to look up the index but thats all we have at the moment, thankfully routers and switches don't change frequently here.

HTH's

Aengus

 

Feb 25, 2011 at 10:23 PM

Apparently only one process is used for the Health Service and that is apparently what runs all the proxied stuff. That means you can add more cores/processors and not help. You have to run more proxies, maybe run one virtual proxy server per processor on VMM or ESX.

Please! Please correct me if I'm wrong.

After reading this thread I understand why my proxy running nearly 2800 interfaces plus web, service, and DB proxies is losing its health service regularly.