For me it was a case of a misconfigured firewall.
Previous week discussions were of a new device going into the network and for it to be managed. Sure not a problem I told them.
Solarwinds can do that - the MIBs are already there for that device. It'll be easy as, just load it in and allow it to be polled from of these 3 NPM servers.
Sunday morning 6:40am, I'm about to take my kids swimming and my phone starts dancing around the kitchen table to their delight.
"Damn - what's broken" was my first though. Sure enough - SAM has started flooding me with an email - one for each DB instance file that has less than 5% storage available.
Call the SQL DBA and get him to give me some more room at the cost of another application. All good, off to swimming lessons we go. Phew.
20min before getting into the office Monday morning and my phone is doing it again. WHAT! WHY!
DBA gets into action and moves some other DBs over to a different drive after I've shut down those applications. Great. (except from the abuse from the DBA for buggering up his Sunday morning)
Checks confirm what was only a 28GB Solarwinds NPM DB Thursday, has now grown into a 109GB mammoth and there's nowhere else to grow!
Jump onto SAM. Check the SQL server - Solarwinds instance and scroll down to the "Top 10 Tables by Size" - what's this? A 10GB Traps table and a 68GB TrapVarbind table too!!!!!
Breath. Ok know we know the impact and the SQL Table at fault, so what's the cause?
Into the Trap Tab we go and see alot of noise from a single device.
Sheesh! Its that new device - they put it in last week, AND CONFIGURED IT TO SEND INFORMATIONAL TRAPS - to all 3 SERVERS!!!
Ok speak with the culprit and clarify what he's done verses what you advised and get him to reconfigure immediately.
[this misconfig resulted in the 3 NPM servers writing the each trap once received into the same Db]
Risk removed, it's time to clean up.
As our policy requires us to maintain the default 30days for traps/events etc I couldn't just purge the whole 2 tables out, so off to engage the DBA with the SQL query of the offenders to be removed.
Hopefully now we'll have a happier DBA - well he should be a bit happier with me as Solarwinds helped find the cause of his weekend pain and I wasn't the root cause
Since I don't have direct DB access, I've created https://thwack.solarwinds.com/ideas/4720 - vote it up if your sitting there laughing to yourself cause you've been there, done that!
[Love to be able to purge out entries in the Database without having to rely on the DBA to do it for you - I can't due to access restrictions between departments etc]
Anyway be careful out there when multiple people are able to impact your environment since your retention period may not purge your entries quick enough if there is a specific policy for these other types of information to be captured.
And monitor your monitor - a single config mishap on a device could blow your hard work into large reporting gaps (and very peeved off people that don't have complete shiny reports)
[PS: Our instance was built on the good old "this is just a POC to see if its any good" concept - you know, on that hardware that can't be upgraded any further cause you've hit the physical limit]
How about you?.....