Building up & testing a new 9TB SATA RAID10 NFSv4 NAS, part II

Initial DRBD Sync – 9TB in 4.5 days

After much blood, sweat and tears getting xtendx up and running in our new data center with Green.ch this summer, I finally had time to complete a major “high availability” aspect of our new file server platform. Our basic architecture is fairly simple: a couple of application servers backed by the network attached storage.

The network attached storage (NAS) is comprised of a pair of nearly identical self-built (Building up & testing a new 9TB SATA RAID10 NFSv4 NAS, part I) file servers. The next, more complex step was to configure a Class C (Primary + Secondary) DRBD cluster. In an effort to mitigate some risk and spread the workload over the course of the summer, I staged the entire installation:

  1. May: Build NAS 0 (thanks to my wife, Robyn, too!)
  2. Early June: Install NAS 0 into data center, copy over production data
  3. Mid-June: Put NAS 0 into production as a simple ext4 + NSFv4 file server
  4. July: Build NAS 1 (Again, a thanks to my wife)
  5. July + August: Install NAS 1 into data center, configure as a HA NAS: drbd + ext4 + NSFv4.
  6. September: Copy data to NAS 1, take NAS 1 into production
  7. Late-September: configure NAS 0 as secondary node in a HA NAS: drbd + ext4 + NSFv4.
  8. Early-October: Initialize DRBD synchronization.

Kicking off the block-level disk synchronization was a big deal. These servers are a running production system and it is paramount that existing service delivery was not impacted. At first I had left the DRBD sync rate unaltered, which I believe effectively means ‘fast as you can’. This quickly resulted in poor read times for production applications, so I kicked it down dramatically while the sync was in progress with the drbdsetupcommand:

 sudo drbdsetup /dev/drbd1 syncer -r 24M

After fooling around with various rates, I settled on 24M. It’s much lower than the system could theoretically synchronize at, but that is not the goal. This production value is also now configured in /etc/drbd.d/global_common.conf:

common {
    #snip
    syncer {
        # rate after al-extents use-rle cpu-mask verify-alg csums-alg
        rate 24M; 
    }
}

Interestingly, the synchronization rate does not match disk IO as measured by Cacti via SNMP; in fact it is roughly half. (I have no idea why.) As you can see below, it took a solid 4.5 days for the synchronization of a single 9TB device to complete:

Primary DRBD Node Read and Write IO (MBps)

Secondary DRBD Node Read and Write IO (MBps)

A bonus to this work on a production system is that the DRBD sync reserved as a real-world production load test. I now know with a respectable degree of certainty what our NAS is capable of, and when it is approaching stressed.