Ilmar Kerm

Oracle, databases, Linux and maybe more

Most (all?) of the Oracle ZFS Storage Appliance (ZS3-4 in my case) management functions are also exposed through REST API so it is possible to script and automate the storage management tasks. No proprietary clients or GUI needed, just need to write a simple script that can send some REST commands over HTTPS to the appliance and parse the JSON output.

I had a task of automating test database refresh from production and part of that workflow is also cloning and attaching LUNs from ZFSSA. I did not find any sample code to access ZFSSA online, so here am I publishing my code. It is a Python script that makes a use of a few ZFSSA REST API calls and publishes them as Fabric tasks.

This script exposes four tasks:

  • clone – Creates a new clone from replication
  • attach_stage1 – Attaches all LUNs in a project to a single target server
  • attach_stage2 – Attaches all LUNs in a project to multiple target servers
  • drop – Deletes a project
#!/usr/bin/env fab

from fabric.api import task,hosts,settings,env
from fabric.utils import abort
import requests, json, os
from datetime import date

today =

url = ""

# ZFS project name, where the clone will be stored, it is created during clone operation
zfsproject = "TEST_REFRESH-%s" % today.__format__('%Y%m%d')

# Replication source name
repsource = "prodc1n1"

# Replication package ID
reppackage = "6a7e390e-b159-4d12-d111-a519f0f5168e"

# ZFSSA authentication credentials, it reads username and password from environment variables ZFSUSER and ZFSPASSWORD
zfsauth = (os.getenv('ZFSUSER'), os.getenv('ZFSPASSWORD'))

jsonheader={'Content-Type': 'application/json'}

# This gets the pool name and last sync timestamp
def read_pool():
  global zfspool, lastsync, zfsname
  # Read the replication package properties
  r = requests.get("%s/api/storage/v1/replication/sources/%s/packages/%s" % (url, repsource, reppackage), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code != 200:
    abort("Error getting information about the specified replication package: %s %s" % (r.status_code, r.text))
  j = json.loads(r.text)
  lastsync = j["package"]["last_sync"]
  # Read the project in replication package to get the pool name
  for proj in j["package"]["projects"]:
    r2 = requests.get("%s%s" % (url, proj["href"]), auth=zfsauth, verify=False, headers=jsonheader)
    if r2.status_code != 200:
      abort("Error getting the replication project details: %s %s" % (r2.status_code, r2.text))
    j2 = json.loads(r2.text)
    zfspool = j2["project"]["pool"]
    zfsname = j2["project"]["name"]
    break # Only want the first project

def clone():
  print "Last sync time: %s" % lastsync
  # First check is the target project name already exists
  r = requests.get("%s/api/storage/v1/pools/%s/projects/%s" % (url, zfspool, zfsproject), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code != 404:
    abort("ZFS project %s already exists (or other error): %s" % (zfsproject, r.status_code))
  print "Cloning %s to project %s" % (zfsname, zfsproject)
  payload = { 'target_project': zfsproject }
  r3 = requests.put("%s/api/storage/v1/replication/sources/%s/packages/%s/clone" % (url, repsource, reppackage), auth=zfsauth, verify=False, data=json.dumps(payload), headers=jsonheader)
  if r3.status_code == 202:
    print "Cloned"
    abort("ZFS Clone failed: %s" % r3.text)
  # Snapshot target project to preserve the initial state
  payload = { 'name': 'initial-backup' }
  r ="%s/api/storage/v1/pools/%s/projects/%s/snapshots" % (url, zfspool, zfsproject), auth=zfsauth, verify=False, headers=jsonheader, data=json.dumps(payload))
  if r.status_code == 201:
    print "Initial snapshot created"
    abort("Creating initial snapshot failed")

def attach_luns(initgroups):
  print "Attaching LUNs in project %s" % zfsproject
  r = requests.get("%s/api/storage/v1/pools/%s/projects/%s/luns" % (url, zfspool, zfsproject), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code == 200:
    j = json.loads(r.text)
    payload = { 'status': 'online', 'targetgroup': 'iscsi-all', 'initiatorgroup': initgroups }
    for lun in j["luns"]:
      if "fault" in lun:
        abort("Fault received from ZFSSA: %s" % r.text)
      print lun["name"]
      rx = requests.put("%s/api/storage/v1/pools/%s/projects/%s/luns/%s" % (url, zfspool, zfsproject, lun["name"]), auth=zfsauth, verify=False, headers=jsonheader, data=json.dumps(payload))
      if rx.status_code == 202 or len(initgroups) > 1:  # The "or len(initgroups) > 1" part is to fix ZFSSA api bug, when initgroups is > 1, then it reports internal error, but completes the task
        print "Attached"
        abort("Attaching failed: %s" % rx.text)
    abort("Getting list of LUNs failed: %s" % r.text)

def attach_stage1():
  # Attach all LUNs in a project to a single initiator group "server1-iscsi"

def attach_stage2():
  # Attach all LUNs in a project to a multiple initiator groups (for example to use them as shared storage in a RAC setup)
  attach_luns(["cluster1n01-iscsi", "cluster1n02-iscsi", "cluster1n03-iscsi"])


def drop_initial_snapshot():
  print "Dropping initial snapshot"
  r = requests.delete("%s/api/storage/v1/pools/%s/projects/%s/snapshots/initial-backup" % (url, zfspool, zfsproject), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code == 204:
    print "Initial snapshot dropped"
    abort("Dropping initial snapshot failed")

def drop():
  print "Dropping project %s" % zfsproject
  r = requests.delete("%s/api/storage/v1/pools/%s/projects/%s" % (url, zfspool, zfsproject), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code == 204:
    print "OK"
    abort("Dropping project failed: %s" % r.text)

ZFSSA REST API works fine if each LUN has only a single initiator group, but when assigning LUN to multiple initiator groups (to be used in a RAC setup), I observed the following errors (they must be ZFSSA REST API bugs):

  • When modifying LUN to be part of multiple initiator groups (PUT request to /api/storage/v1/pools/poolname/projects/projectname/luns/lunname), then the API responds with (argument numbers may be different): {“fault”: {“message”: “invalid input argument (assignednumber: invalid literal for int() with base 10: ‘0,0,0’)”, “code”: 400, “name”: “ERR_INVALID_ARG”}}. Nevertheless, ZFS GUI confirms that the operation was completed successfully. This script just ignores the returned error message, when assigning LUN to multiple initiator groups.
  • If any LUN in the project has multiple initiator groups defined, then API call to get the list of LUNs (GET request to /api/storage/v1/pools/poolname/projects/projectname/luns) in a project returns an error immediately: {“luns”:[{“fault”: {“message”: “internal error”, “code”: 500, “name”: “ERR_INTERNAL”}}]}. This script cannot reattach the LUNs in this case, since getting the list of LUNs fails.