Ilmar Kerm

Oracle, databases, Linux and maybe more

For a while now Oracle is using standardised log structure under Automatic Diagnostic Repository (ADR) – this includes database alert.log, listener log, CRS log, ASM log and much more. All this is very valuable information for monitoring.

I’m a fan of using Grafana (using InfluxDB as a data source) for monitoring, since in Grafana it is very easy to create your own customised dashboards and alerts – so lately I’ve ben creating my own monitoring agents to gather send Oracle monitoring information to InfluxDB.

InfluxDB is fast time series database, perfect for storing monitoring data – each record must have timestamp associated and data retention is also natively built in.

Since our Oracle environment is exploding in size, my goal is also to have monitoring tools/agents that figure out what to do themselves, deployed using Ansible. All without any human involvment.

Here is monitoring agent with following high level features:

  • Searches for all recently modified alert/log.xml files under ADR base
  • Parses newly added log messages for any tags that Oracle itself has added (like component and message level)
  • Parses log messages against custom regular expressions to parse out custom information

Code is here:

https://github.com/ilmarkerm/monitoring/blob/main/oracle/adr-log.py

It is expected to run as SystemD service. I currently left in some Ansible tags, since this is how I distribute the agent to all servers (and to make sure people do no just take the code and run it without reading and understanding it).

Here you can see some small snippets how to return quite a lot of useful attributed about Oracle database directly from CRS. And the code I use to do an automated RAC rolling restart.

If you are doing many HTTPS calls from PL/SQL to many different sites, managing oracle wallet with all the required trusted certificates is quite painful. But Linux distros already come bundled with all the trusted CA certs from Mozilla, would it be nice just to convert it to Oracle wallet format so PL/SQL could use it?

Here is an Ansible role just for this, you could run it regularly to make sure you have the latest Mozilla trusted certs.

https://github.com/ilmarkerm/oracle-wallet-mozilla-ca

I have quite many different databases and would like to log in to them from my linux box using my AD username and password. Creating tnsnames.ora aliases for all of them is too much work, I’d like to just specify hostname and service from command line.

Bash aliases to the rescue, but simple alias is not enough in this case, so just take it as an example how can more complex aliases be created in Bash.

Just copy these functions to $HOME/.bashrc, modify them according to your needs and done 🙂

This creates two “aliases” – sqls for secure database connection and sql for insecure. I also need to use instantclient driver for database connection (Radius authentication).

Usage:

sqls rac-scan.example.com myapplicationservice.prod
sqls rac-scan.example.com myapplicationservice.prod dbauser
SQLS_OPTS="-verbose" sqls rac-scan.example.com myapplicationservice.prod

So your RAC database hangs/has occasional stalls/you want to do an emergency reboot?
Franck Pachot has written a good article what traces to get for troubleshooting or for Oracle support

I had a problem – all RAC instances seem to “stall” occasionally so I wanted to execute Francks script on all instances at the same time when the problem was happening.

Ansible to the rescue.

First I pushed out the following script to all instances, this script actually does the diagnostics dump.

Then the Ansible playbook to execute the script above on all instances at the same time and afterwards download all traces to your local ansible controller host: