{"id":328,"date":"2016-07-09T23:41:50","date_gmt":"2016-07-09T21:41:50","guid":{"rendered":"https:\/\/ilmarkerm.eu\/blog\/?p=328"},"modified":"2016-08-09T16:06:02","modified_gmt":"2016-08-09T14:06:02","slug":"experiences-with-moving-to-multitenant-in-12-1-0-2","status":"publish","type":"post","link":"https:\/\/ilmarkerm.eu\/blog\/2016\/07\/experiences-with-moving-to-multitenant-in-12-1-0-2\/","title":{"rendered":"Experiences with moving to multitenant in 12.1.0.2"},"content":{"rendered":"<p>Lately I&#8217;ve been upgrading our 11g Standard Edition databases in test environments  to 12c Enterprise Edition and also plugging them in to a multitenant container database.<br \/>\nIt&#8217;s a new technology for Oracle, but I was still quite surprised about the number of issues I faced when trying to plug in an existing non-CDB database. After resolving all these issues it has been quite painless process since then.<\/p>\n<p>In short, upgrading 11g database to 12c pluggable database involves the following steps:<br \/>\n* Upgrade 11g database to 12c using the normal database upgrade procedures. This step will result in 12c non-CDB database.<br \/>\n* In the target CDB database plug in the new upgraded database as a new pluggable database.<br \/>\n* Run <strong>$ORACLE_HOME\/rdbms\/admin\/noncdb_to_pdb.sql<\/strong> in the new PDB. This step will convert the non-CDB data dictionary to a PDB data dictionary. After this step you can open the newly added PDB.<\/p>\n<p>This post is mostly about the issues I encountered when running the last described step &#8211; executing $ORACLE_HOME\/rdbms\/admin\/noncdb_to_pdb.sql. Hopefully it&#8217;ll be helpful if you hit similar problems.<\/p>\n<p>Version: 12.1.0.2, 2-node RAC<br \/>\nPatches: April 2016 PSU + OJVM PSU<br \/>\nPlatform: Oracle Linux 6 x86-64<\/p>\n<h2>noncdb_to_pdb.sql takes a really long time to execute<\/h2>\n<p>This was the first problem I encountered. After 1,5 hours I killed my session. That was really weird, because executing it should onbly take about 20 minutes according to Oracle documentation. The step script was stuck on was:<\/p>\n<pre><code>-- mark objects in our PDB as common if they exist as common in ROOT\n<\/code><\/pre>\n<p>Looking at the wait events the session was not waiting for a blocker, it was actively executing many parallel sessions.<br \/>\nI found the following blog post that described the same problem and the described solution also helped for me: <a href=\"https:\/\/bdrouvot.wordpress.com\/2014\/10\/17\/watch-out-for-optimizer_adaptive_features-as-it-may-have-a-huge-negative-impact\/\">Link to Bertrand Drouvot blog<\/a><\/p>\n<p>But one addition, instead of modifying the noncdb_to_pdb.sql script, I executed ALTER SESSION before running noncdb_to_pdb.sql.<\/p>\n<pre><code class=\"sql\">SQL&gt; alter session set container=newpdb;\nSQL&gt; alter session set optimizer_adaptive_features=false;\nSQL&gt; $ORACLE_HOME\/rdbms\/admin\/noncdb_to_pdb.sql\n<\/code><\/pre>\n<h2>noncdb_to_pdb.sql hangs at alter pluggable database close<\/h2>\n<p>Next issue I faced &#8211; noncdb_to_pdb.sql just hanged mid-execution and the statement it was executing was<\/p>\n<pre><code class=\"sql\">SQL&gt; alter pluggable database \"&amp;pdbname\" close;\n<\/code><\/pre>\n<p>The session was waiting for <strong>opishd<\/strong>.<\/p>\n<p>Solution: Apply bugfix for <strong>Bug 20172151 &#8211; NONCDB_TO_PDB.SQL SCRIPT HANGS DURING UPGRADE<\/strong>. This will just update noncdb_to_pdb.sql script itself to execute <strong>alter pluggable database &#8220;&amp;pdbname&#8221; close IMMEDIATE instances = all;<\/strong> instead of normal close.<\/p>\n<h2>noncdb_to_pdb.sql fails with ORA-600 [kspgsp2]<\/h2>\n<p>That was a fun one \ud83d\ude42 Not every time, but most executions noncdb_to_pdb.sql failed almost at the end with the following message:<\/p>\n<pre><code class=\"sql\">SQL&gt; alter session set \"_ORACLE_SCRIPT\"=true;\n\nSession altered.\n\nSQL&gt;\nSQL&gt; drop view sys.cdb$tables&amp;pdbid;\ndrop view sys.cdb$tables5\n*\nERROR at line 1:\nORA-00600: internal error code, arguments: [kspgsp2], [0xBF3C9E3F8], [2174],\n[recyclebin], [], [], [], [], [], [], [], []\n<\/code><\/pre>\n<p>Every time at the same drop view statement. Search in Oracle support did not give me anything helpful, there were many ORA-600 [kspgsp2] issues, but nothing matched my case. Finally I noticed that one argumnt was <strong>[recyclebin]<\/strong> and decided to try turning the recyclebin off for the session. It helped.<\/p>\n<h2>Successful non-CDB to PDB conversion<\/h2>\n<p>Getting successful execution of noncdb_to_pdb.sql required me to:<br \/>\n* Apply patch 20172151<br \/>\n* Running noncdb_to_pdb.sql using the following sequence of commands:<\/p>\n<pre><code class=\"sql\">SQL&gt; alter session set container=newpdb;\nSQL&gt; alter session set optimizer_adaptive_features=false;\nSQL&gt; alter session set recyclebin=off;\nSQL&gt; @$ORACLE_HOME\/rdbms\/admin\/noncdb_to_pdb.sql\n<\/code><\/pre>\n<h2>Take care of the services!<\/h2>\n<p>This problem may be our environment specific, but I&#8217;ll describe it anyway.<br \/>\nWe use services a lot, all applications that connect to the database get their own dedicated service. So the applications connect using JDBC connection string that looks something like this:<\/p>\n<pre><code>jdbc:oracle:thin:@\/\/clusername.example.com\/application.scrum.example.com\n<\/code><\/pre>\n<p>Where <strong>application<\/strong> is configured as a service using srvctl and <strong>scrum.example.com<\/strong> is database domain name depending on the environment. The same application in QA environment will have connection string:<\/p>\n<pre><code>jdbc:oracle:thin:@\/\/clusername.example.com\/application.qa.example.com\n<\/code><\/pre>\n<p>We decided to use only one CDB for all environments, but db_domain parameter cannot be different for each PDB. In order to not change the application connection strings I had to create the new services in srvctl using the FULL service name, then Oracle will not append the database domain name to the service name:<\/p>\n<pre><code>srvctl add service -database cdb -preferred cdb1,cdb2 -pdb newpdb -service application.scrum.example.com\nsrvctl start service -database cdb -service application.scrum.example.com\n<\/code><\/pre>\n<p>After adding the services for the first database all of them started just fine and applications connected just fine, but when starting the services for the second environment (qa) I got the following error:<\/p>\n<pre><code>srvctl add service -database cdb -preferred cdb1,cdb2 -pdb newpdbqa -service application.qa.example.com\nsrvctl start service -database cdb -service application.qa.example.com\n\n...\nORA-44311: service application not running\nORA-06512: at \"SYS.DBMS_SYS_ERROR\", line 86\nORA-06512: at \"SYS.DBMS_SERVICE_ERR\", line 40\nORA-06512: at \"SYS.DBMS_SERVICE\", line 421\nORA-06512: at line 1\n...\n<\/code><\/pre>\n<p>But when I tried to add a new service that did not exist previously it started just fine. I started digging into services on CDB level and found that all imported PDB-s also imprted their old short name services to CDB:<\/p>\n<pre><code class=\"sql\">SQL&gt; alter session set container=cdb$root;\n\nSession altered.\n\nSQL&gt; select name from cdb_services order by 1;\n\nNAME\n------------------------------------------------\nSYS$BACKGROUND\nSYS$USERS\n...\napplication\napplication\napplication.scrum.example.com\napplication2\napplication2\napplication2.scrum.example.com\n...\n<\/code><\/pre>\n<p>I just assumed it can be confusing for CDB if different PDB-s have conflicting services running and I manually went into each PDB and removed the old short service names.<\/p>\n<pre><code class=\"sql\">SQL&gt; alter session set container=newpdb;\nSQL&gt; exec dbms_service.delete_service('application');\nSQL&gt; exec dbms_service.delete_service('application2');\nSQL&gt; alter session set container=newpdbqa;\nSQL&gt; exec dbms_service.delete_service('application');\nSQL&gt; exec dbms_service.delete_service('application2');\n<\/code><\/pre>\n<p>After that new services started just fine.<\/p>\n<pre><code class=\"sql\">SQL&gt; alter session set container=cdb$root;\nSQL&gt; select name from cdb_services order by 1;\n\nNAME\n------------------------------------------------\nSYS$BACKGROUND\nSYS$USERS\n...\napplication.scrum.example.com\napplication2.scrum.example.com\napplication.qa.example.com\napplication2.qa.example.com\n...\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Lately I&#8217;ve been upgrading our 11g Standard Edition databases in test environments to 12c Enterprise Edition and also plugging them in to a multitenant container database. It&#8217;s a new technology for Oracle, but I was still quite surprised about the number of issues I faced when trying to plug in an existing non-CDB database. After [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[47,4],"class_list":["post-328","post","type-post","status-publish","format-standard","hentry","category-blog-entry","tag-multitenant","tag-oracle"],"_links":{"self":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts\/328","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/comments?post=328"}],"version-history":[{"count":10,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts\/328\/revisions"}],"predecessor-version":[{"id":339,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/posts\/328\/revisions\/339"}],"wp:attachment":[{"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/media?parent=328"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/categories?post=328"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ilmarkerm.eu\/blog\/wp-json\/wp\/v2\/tags?post=328"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}