Long time since I talked about Grid Control, but this one was weird enough to share. Hostnames and installation paths obfuscated as usual.
After a recent reconfiguration exercise for an Oracle 11g Grid Control agent, all targets monitored by the agent were showing "unknown" status, agent metrics were unavailable, and the performance page host target itself was throwing strange (to me, anyway) "No such metric, Switching to last 24 hours view" errors.
A quick look at the emagent.trc file in $ORACLE_HOME/sysman/log showed a lot of errors of the following form, for every monitored target on the system:
2010-12-03 21:50:35,716 Thread-3027233680 ERROR engine: [oracle_emd,server.domain:1830,ProcessInfo] : nmeegd_GetMetricData failed : Missing Properties : [perlBin]
perlBin? That's a new one for me. None of the recent work on the agent configuration had touched the perl installation, and lightweight command-line testing of the perl installed in the ORACLE_HOME didn't reveal any problems. That meant it was time to start digging in $ORACLE_HOME/sysman/config, where there are lots of files with .properties extensions. For once, the most obvious place to look turned out to be the right place.
The culprit? Oh, just one little character in the emd.properties file:
Change /perlBin to perlBin, restart the agent, and all of a sudden everything looks way more sane. Whew. How that wayward / got to be there in the first place is a mystery that can wait for another day. Probably my fault; it often is.