ZFS poolに障害発生、暫定対応

2017年3月15日FreeBSD,Mac

久々に、zpool statusをみたら、大変な事になっていました。tankのひとつのypoolはミラーリング運用しているのですが、一台だけの運用してになってしっており、ミラーリングできていませんでした。

# zpool status
  pool: rpool
 state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
	still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
	pool will no longer be accessible on software that does not support feature
	flags.
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Feb 22 08:51:43 2014
config:

	NAME         STATE     READ WRITE CKSUM
	rpool        ONLINE       0     0     0
	  gpt/zroot  ONLINE       0     0     0

errors: No known data errors

  pool: upool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
	repaired.
  scan: resilvered 293M in 0h1m with 0 errors on Sat Dec  3 18:48:39 2016
config:

	NAME           STATE     READ WRITE CKSUM
	upool          DEGRADED     0     0     0
	  mirror-0     DEGRADED     0     0     0
	    gpt/disk4  ONLINE       0     0     0  block size: 512B configured, 4096B native
	    gpt/disk2  FAULTED      5   156     0  too many errors
	  mirror-1     ONLINE       0     0     0
	    gpt/disk3  ONLINE       0     0     0
	    gpt/disk5  ONLINE       0     0     0  block size: 512B configured, 4096B native
	logs
	  gpt/zil      ONLINE       0     0     0
	cache
	  gpt/l2arc    ONLINE       0     0     0

errors: No known data errors

  pool: ypool
 state: DEGRADED
status: One or more devices has been removed by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
  scan: resilvered 47.0M in 0h0m with 0 errors on Sat Dec  3 18:46:51 2016
config:

	NAME                      STATE     READ WRITE CKSUM
	ypool                     DEGRADED     0     0     0
	  mirror-0                DEGRADED     0     0     0
	    gpt/disk0             ONLINE       0     0     0
	    15691383435259036286  REMOVED      0     0     0  was /dev/gpt/disk1

errors: No known data errors

げげっ! ディスクを交換するのが正しい対処ですが、取り敢えず、onlineにしてみました。

# zpool online ypool gpt/disk1

そうしたら、無事にディスクを認識し、ミラーリングを始めました。

# zpool status -v ypool
  pool: ypool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Mar 12 21:59:41 2017
        8.05G scanned out of 1.32T at 30.1M/s, 12h42m to go
        8.04G resilvered, 0.60% done
config:

	NAME           STATE     READ WRITE CKSUM
	ypool          ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    gpt/disk0  ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0  (resilvering)

errors: No known data errors

データコピーが終了し、無事に平常運転に戻りました。

# zpool status -v ypool
  pool: ypool
 state: ONLINE
  scan: resilvered 14.3G in 0h8m with 0 errors on Sun Mar 12 22:07:58 2017
config:

	NAME           STATE     READ WRITE CKSUM
	ypool          ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    gpt/disk0  ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0

errors: No known data errors

でも、メインで使っているupoolのdisk2にエラーが残っています。これを何とかしないと。