{"id":781,"date":"2024-11-19T17:41:26","date_gmt":"2024-11-19T16:41:26","guid":{"rendered":"https:\/\/ilmarkerm.eu\/blog\/?p=781"},"modified":"2024-11-20T07:35:36","modified_gmt":"2024-11-20T06:35:36","slug":"exporting-oracle-rest-data-services-metrics-for-prometheus","status":"publish","type":"post","link":"https:\/\/ilmarkerm.eu\/blog\/2024\/11\/exporting-oracle-rest-data-services-metrics-for-prometheus\/","title":{"rendered":"Exporting Oracle Rest Data Services metrics for Prometheus"},"content":{"rendered":"\n<p>Tested with ORDS 24.3 running using GraalVM 21.<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.oracle.com\/en\/database\/oracle\/oracle-rest-data-services\/24.3\/ordig\/deploying-and-monitoring-oracle-rest-data-services.html#GUID-AC5E2F60-8EF5-4E82-A08B-60450438F48B\">ORDS documentation has a chapter how to push ORDS metrics to OpenTelemetry endpoint<\/a>, which also is supported by Prometheus. But Prometheus traditionally is using the opposite method, that Prometheus itself will regularly connect to monitored service endpoints and scrape all its metrics. Similar JavaAgent method can also be deployed to expose ORDS metrics as a traditional Prometheus pull based endpoint.<\/p>\n\n\n\n<p>For this we need to deploy Prometheus JMX exporter as a Java agent in ORDS, this will expose \/metrics endpoint that Prometheus can scrape.<\/p>\n\n\n\n<p>First, <a href=\"https:\/\/github.com\/prometheus\/jmx_exporter\">JMX exporter downloads and documentation is available here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Download JMX exporter to ORDS host\ncurl -o jmx_prometheus_javaagent.jar \"https:\/\/repo1.maven.org\/maven2\/io\/prometheus\/jmx\/jmx_prometheus_javaagent\/1.0.1\/jmx_prometheus_javaagent-1.0.1.jar\"<\/code><\/pre>\n\n\n\n<p>Create configuration file, config.yaml with the following contents. It will format the JMX data into more usable Prometheus metric names.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">rules:<br>    # Reformatting Oracle UCP metrics to have more usable names and adding data types<br>    - pattern: \"oracle.ucp.admin.UniversalConnectionPoolMBean&lt;name=.+, poolName=\\\\|(.+)\\\\|(.+)\\\\|.+>&lt;>connectionsClosedCount\"<br>      name: oracle_ucp_connectionsClosedCount<br>      type: COUNTER<br>      labels:<br>          poolName: $1_$2<br>    - pattern: \"oracle.ucp.admin.UniversalConnectionPoolMBean&lt;name=.+, poolName=\\\\|(.+)\\\\|(.+)\\\\|.+>&lt;>connectionsCreatedCount\"<br>      name: oracle_ucp_connectionsCreatedCount<br>      type: COUNTER<br>      labels:<br>          poolName: $1_$2<br>    - pattern: \"oracle.ucp.admin.UniversalConnectionPoolMBean&lt;name=.+, poolName=\\\\|(.+)\\\\|(.+)\\\\|.+>&lt;>cumulative(\\\\w+)\"<br>      name: oracle_ucp_cumulative$3<br>      type: COUNTER<br>      labels:<br>          poolName: $1_$2<br>    - pattern: \"oracle.ucp.admin.UniversalConnectionPoolMBean&lt;name=.+, poolName=\\\\|(.+)\\\\|(.+)\\\\|.+>&lt;>(\\\\w+)\"<br>      name: oracle_ucp_$3<br>      type: GAUGE<br>      labels:<br>          poolName: $1_$2<br>    # This pattern below will add all the rest, tons of detailed java internal things<br>    # Comment out if you do not want to see them<br>    - pattern: '.*'<\/pre>\n\n\n\n<p>I will assume, that jmx_prometheus_javaagent.jar and config.yaml are placed under \/home\/ords<\/p>\n\n\n\n<p>Next, change ORDS startup script so it would include the JMX agent. The easiest way is to use environment variable _JAVA_OPTIONS for it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Set Startup Java options\n# 10.10.10.10 is my local server IP where metrics exporter will bind to, default is localhost\n# 21022 is the port JMX exporter will listen to\n# With this ORDS metrics would be exposed as http:\/\/10.10.10.10:21022\/metrics\n\nexport _JAVA_OPTIONS=\"-javaagent:\/home\/ords\/jmx_prometheus_javaagent.jar=10.10.10.10:21022:\/home\/ords\/config.yaml\"\n\n# Start ORDS in standalone mode as usual\nords serve<\/code><\/pre>\n\n\n\n<p>Below is my full ORDS SystemD service file &#8211; \/etc\/systemd\/system\/ords.service<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;Unit]\nDescription=Oracle Rest Data Services\nAfter=syslog.target network.target\n\n&#91;Service]\nType=simple\nUser=ords\nGroup=ords\nRestart=always\nRestartSec=30\nEnvironment=\"_JAVA_OPTIONS=-Xms3G -Xmx3G -javaagent:\/home\/ords\/jmx_prometheus_javaagent.jar=10.10.10.10:21022:\/home\/ords\/config.yaml\"\nEnvironment=\"JAVA_HOME=\/home\/ords\/graalvm\"\n#Environment=\"JAVA_TOOL_OPTIONS=-Djava.util.logging.config.file=\/home\/ords\/logging.conf\"\nExecStart=\/home\/ords\/ords\/bin\/ords --config \/etc\/ords\/config serve --secure --port 8443 --key \/etc\/ords\/server.key --certificate \/etc\/ords\/server.pem\n\n&#91;Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\n<p>After restarting ORDS I can query its metrics endpoint.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl http:\/\/10.10.10.10:21022\/metrics\n\n# You will see many Java and JVM metrics in the output. Example...\njvm_memory_pool_max_bytes{pool=\"Compressed Class Space\"} 1.073741824E9\njvm_memory_pool_max_bytes{pool=\"G1 Eden Space\"} -1.0\njvm_memory_pool_max_bytes{pool=\"G1 Old Gen\"} 3.221225472E9\njvm_memory_pool_max_bytes{pool=\"G1 Survivor Space\"} -1.0\njvm_memory_pool_max_bytes{pool=\"Metaspace\"} -1.0\njvm_memory_pool_used_bytes{pool=\"CodeHeap 'non-nmethods'\"} 1822336.0\njvm_memory_pool_used_bytes{pool=\"CodeHeap 'non-profiled nmethods'\"} 5918080.0\njvm_memory_pool_used_bytes{pool=\"CodeHeap 'profiled nmethods'\"} 2.3397888E7\njvm_memory_pool_used_bytes{pool=\"Compressed Class Space\"} 7328848.0\njvm_memory_pool_used_bytes{pool=\"G1 Eden Space\"} 2.57949696E8\njvm_memory_pool_used_bytes{pool=\"G1 Old Gen\"} 2.280663304E9\njvm_memory_pool_used_bytes{pool=\"G1 Survivor Space\"} 8528.0\njvm_memory_pool_used_bytes{pool=\"Metaspace\"} 6.750048E7\njvm_memory_used_bytes{area=\"heap\"} 2.538621528E9\njvm_memory_used_bytes{area=\"nonheap\"} 1.05967632E8\njvm_threads_deadlocked_monitor 0.0\njvm_threads_peak 62.0\njvm_threads_started_total 62.0\njvm_threads_state{state=\"BLOCKED\"} 0.0\njvm_threads_state{state=\"NEW\"} 0.0\njvm_threads_state{state=\"RUNNABLE\"} 12.0\njvm_threads_state{state=\"TERMINATED\"} 0.0\njvm_threads_state{state=\"TIMED_WAITING\"} 20.0\njvm_threads_state{state=\"UNKNOWN\"} 0.0\njvm_threads_state{state=\"WAITING\"} 15.0\n\n# ORDS database connection pool metrics will be exported like this\n# Just an example... all UCP attributes are exported, for all ORDS connection pools\n\noracle_ucp_abandonedConnectionTimeout{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_abandonedConnectionTimeout{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_abandonedConnectionsCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_abandonedConnectionsCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_availableConnectionsCount{poolName=\"backoffice_lo\"} 10.0\noracle_ucp_availableConnectionsCount{poolName=\"marketing_communications_2_lo\"} 10.0\noracle_ucp_averageBorrowedConnectionsCount{poolName=\"backoffice_lo\"} 1.0\noracle_ucp_averageBorrowedConnectionsCount{poolName=\"marketing_communications_2_lo\"} 1.0\noracle_ucp_averageConnectionWaitTime{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_averageConnectionWaitTime{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_borrowedConnectionsCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_borrowedConnectionsCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_bufferSize{poolName=\"backoffice_lo\"} 1024.0\noracle_ucp_bufferSize{poolName=\"marketing_communications_2_lo\"} 1024.0\noracle_ucp_connectionHarvestMaxCount{poolName=\"backoffice_lo\"} 1.0\noracle_ucp_connectionHarvestMaxCount{poolName=\"marketing_communications_2_lo\"} 1.0\noracle_ucp_connectionHarvestTriggerCount{poolName=\"backoffice_lo\"} 2.147483647E9\noracle_ucp_connectionHarvestTriggerCount{poolName=\"marketing_communications_2_lo\"} 2.147483647E9\noracle_ucp_connectionRepurposeCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_connectionRepurposeCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_connectionValidationTimeout{poolName=\"backoffice_lo\"} 15.0\noracle_ucp_connectionValidationTimeout{poolName=\"marketing_communications_2_lo\"} 15.0\noracle_ucp_connectionWaitTimeout{poolName=\"backoffice_lo\"} 3.0\noracle_ucp_connectionWaitTimeout{poolName=\"marketing_communications_2_lo\"} 3.0\noracle_ucp_connectionsClosedCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_connectionsClosedCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_connectionsCreatedCount{poolName=\"backoffice_lo\"} 10.0\noracle_ucp_connectionsCreatedCount{poolName=\"marketing_communications_2_lo\"} 10.0\noracle_ucp_createConnectionInBorrowThread{poolName=\"backoffice_lo\"} 1.0\noracle_ucp_createConnectionInBorrowThread{poolName=\"marketing_communications_2_lo\"} 1.0\noracle_ucp_cumulativeConnectionBorrowedCount{poolName=\"backoffice_lo\"} 1.0\noracle_ucp_cumulativeConnectionBorrowedCount{poolName=\"marketing_communications_2_lo\"} 1.0\noracle_ucp_cumulativeConnectionReturnedCount{poolName=\"backoffice_lo\"} 1.0\noracle_ucp_cumulativeConnectionReturnedCount{poolName=\"marketing_communications_2_lo\"} 1.0\noracle_ucp_cumulativeConnectionUseTime{poolName=\"backoffice_lo\"} 60.0\noracle_ucp_cumulativeConnectionUseTime{poolName=\"marketing_communications_2_lo\"} 30.0\noracle_ucp_cumulativeConnectionWaitTime{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_cumulativeConnectionWaitTime{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_cumulativeFailedConnectionWaitCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_cumulativeFailedConnectionWaitCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_cumulativeFailedConnectionWaitTime{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_cumulativeFailedConnectionWaitTime{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_cumulativeSuccessfulConnectionWaitCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_cumulativeSuccessfulConnectionWaitCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_cumulativeSuccessfulConnectionWaitTime{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_cumulativeSuccessfulConnectionWaitTime{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_failedAffinityBasedBorrowCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_failedAffinityBasedBorrowCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_failedRCLBBasedBorrowCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_failedRCLBBasedBorrowCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_failoverEnabled{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_failoverEnabled{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_inactiveConnectionTimeout{poolName=\"backoffice_lo\"} 1800.0\noracle_ucp_inactiveConnectionTimeout{poolName=\"marketing_communications_2_lo\"} 1800.0\noracle_ucp_initialPoolSize{poolName=\"backoffice_lo\"} 10.0\noracle_ucp_initialPoolSize{poolName=\"marketing_communications_2_lo\"} 10.0\noracle_ucp_labeledConnectionsCount{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_labeledConnectionsCount{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_loggingEnabled{poolName=\"backoffice_lo\"} 0.0\noracle_ucp_loggingEnabled{poolName=\"marketing_communications_2_lo\"} 0.0\noracle_ucp_maxConnectionReuseCount{poolName=\"backoffice_lo\"} 1000.0<\/code><\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Tested with ORDS 24.3 running using GraalVM 21. ORDS documentation has a chapter how to push ORDS metrics to OpenTelemetry endpoint, which also is supported by Prometheus. But Prometheus traditionally is using the opposite method, that Prometheus itself will regularly connect to monitored service endpoints and scrape all its metrics. Similar JavaAgent method can also [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[58,4,66],"class_list":["post-781","post","type-post","status-publish","format-standard","hentry","category-blog-entry","tag-monitoring","tag-oracle","tag-ords"],"_links":{"self":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts\/781","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/comments?post=781"}],"version-history":[{"count":11,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts\/781\/revisions"}],"predecessor-version":[{"id":794,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts\/781\/revisions\/794"}],"wp:attachment":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/media?parent=781"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/categories?post=781"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/tags?post=781"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}