Can not replace a disk because it says it's part of the pool

[drescherjm]

I had a power outage last night and my main pool on a server is now listed as degraded.

pool: zfs_data_4
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: scrub repaired 0B in 06:13:35 with 0 errors on Sun May 26 06:40:36 2024
config:

    NAME                                   STATE     READ WRITE CKSUM
    zfs_data_4                             DEGRADED     0     0     0
      raidz2-0                             DEGRADED     0     0     0
        13174824313391367059               UNAVAIL      0     0     0  was /dev/sda1
        ata-HGST_HUH721010ALE600_7JH4MZ3C  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ47B3Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_7JH0VTBG  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ386AZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SHXP00Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ4RYLZ  ONLINE       0     0     0
      raidz2-1                             ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ4BD2Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ7T9BZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ329AZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ5079Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ34TNZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ4NDVZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ3UUYZ  ONLINE       0     0     0
    logs
      mirror-2                             ONLINE       0     0     0
        nvme0n1p1                          ONLINE       0     0     0
        nvme1n1p1                          ONLINE       0     0     0
    cache
      nvme0n1p2                            ONLINE       0     0     0
      nvme1n1p2                            ONLINE       0     0     0

I believe I have figured out that the missing disk is /dev/sdd1
[root@fs2 by-serial]# blkid /dev/sdd1 /dev/sdd1: LABEL="zfs_data_4" UUID="16457623129041658883" UUID_SUB="13174824313391367059" TYPE="zfs_member" PARTLABEL="zfs-5673b36e2be4d4da" PARTUUID="83e73662-d104-ad43-9c7d-b1c0b2a02bb2"
However every attempt to add this disk back says its in the active pool.
`
[root@fs2 by-serial]# zpool replace -f zfs_data_4 13174824313391367059 sdd1
invalid vdev specification
the following errors must be manually repaired:
/dev/sdd1 is part of active pool 'zfs_data_4'

[root@fs2 by-serial]# zpool attach zfs_data_4 13174824313391367059 sdd
invalid vdev specification
use '-f' to override the following errors:
/dev/sdd1 is part of active pool 'zfs_data_4'
[root@fs2 by-serial]# zpool attach -f zfs_data_4 13174824313391367059 sdd
invalid vdev specification
the following errors must be manually repaired:
/dev/sdd1 is part of active pool 'zfs_data_4'

[root@fs2 by-serial]# ls -al /dev/disk/by-id | grep ata-HGST_HUH | grep -v part | cat -n
1 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SHXP00Z -> ../../sdh
2 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ329AZ -> ../../sdm
3 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ34TNZ -> ../../sdi
4 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ386AZ -> ../../sdg
5 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ3UUYZ -> ../../sdt
6 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ47B3Z -> ../../sde
7 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ4BD2Z -> ../../sdk
8 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ4NDVZ -> ../../sds
9 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ4RYLZ -> ../../sdj
10 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ5079Z -> ../../sdn
11 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_1SJ7T9BZ -> ../../sdl
12 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_7JH0VTBG -> ../../sdf
13 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_7JH3UJ3C -> ../../sdd
14 lrwxrwxrwx. 1 root root 9 Jun 14 07:38 ata-HGST_HUH721010ALE600_7JH4MZ3C -> ../../sdc
`

[drescherjm]

zfs_data_4:
version: 5000
name: 'zfs_data_4'
state: 0
txg: 833914
pool_guid: 16457623129041658883
errata: 0
compatibility: 'openzfs-2.1-linux'
hostname: 'fs2'
com.delphix:has_per_vdev_zaps
vdev_children: 3
vdev_tree:
type: 'root'
id: 0
guid: 16457623129041658883
create_txg: 4
children[0]:
type: 'raidz'
id: 0
guid: 2039784339945930487
nparity: 2
metaslab_array: 256
metaslab_shift: 34
ashift: 12
asize: 70005713010688
is_log: 0
create_txg: 4
com.delphix:vdev_zap_top: 129
children[0]:
type: 'disk'
id: 0
guid: 13174824313391367059
path: '/dev/sda1'
devid: 'ata-HGST_HUH721010ALE600_7JH3UJ3C-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy1-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot01'
whole_disk: 1
not_present: 1
DTL: 22
create_txg: 4
com.delphix:vdev_zap_leaf: 130
children[1]:
type: 'disk'
id: 1
guid: 6790539378353110680
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_7JH4MZ3C-part1'
devid: 'ata-HGST_HUH721010ALE600_7JH4MZ3C-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy0-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot00'
whole_disk: 1
DTL: 200
create_txg: 4
com.delphix:vdev_zap_leaf: 131
children[2]:
type: 'disk'
id: 2
guid: 5684703596058222722
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ47B3Z-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ47B3Z-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy2-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot02'
whole_disk: 1
DTL: 199
create_txg: 4
com.delphix:vdev_zap_leaf: 132
children[3]:
type: 'disk'
id: 3
guid: 12340788000472239213
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_7JH0VTBG-part1'
devid: 'ata-HGST_HUH721010ALE600_7JH0VTBG-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy3-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot03'
whole_disk: 1
DTL: 198
create_txg: 4
com.delphix:vdev_zap_leaf: 133
children[4]:
type: 'disk'
id: 4
guid: 6883146859437043938
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ386AZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ386AZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy4-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot04'
whole_disk: 1
DTL: 197
create_txg: 4
com.delphix:vdev_zap_leaf: 134
children[5]:
type: 'disk'
id: 5
guid: 14090618046934364423
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SHXP00Z-part1'
devid: 'ata-HGST_HUH721010ALE600_1SHXP00Z-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy5-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot05'
whole_disk: 1
DTL: 196
create_txg: 4
com.delphix:vdev_zap_leaf: 135
children[6]:
type: 'disk'
id: 6
guid: 351640508312637901
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ4RYLZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ4RYLZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy7-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot07'
whole_disk: 1
DTL: 194
create_txg: 4
com.delphix:vdev_zap_leaf: 136
children[1]:
type: 'raidz'
id: 1
guid: 5270299909617887841
nparity: 2
metaslab_array: 770
metaslab_shift: 34
ashift: 12
asize: 70005713010688
is_log: 0
create_txg: 39
com.delphix:vdev_zap_top: 642
children[0]:
type: 'disk'
id: 0
guid: 13316645572900280117
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ4BD2Z-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ4BD2Z-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy8-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot08'
whole_disk: 1
DTL: 208
create_txg: 39
com.delphix:vdev_zap_leaf: 643
children[1]:
type: 'disk'
id: 1
guid: 14667806819913958257
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ7T9BZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ7T9BZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy9-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot09'
whole_disk: 1
DTL: 207
create_txg: 39
com.delphix:vdev_zap_leaf: 644
children[2]:
type: 'disk'
id: 2
guid: 16477520597537057665
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ329AZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ329AZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy10-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot10'
whole_disk: 1
DTL: 206
create_txg: 39
com.delphix:vdev_zap_leaf: 645
children[3]:
type: 'disk'
id: 3
guid: 1614505816682977461
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ5079Z-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ5079Z-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy11-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot11'
whole_disk: 1
DTL: 205
create_txg: 39
com.delphix:vdev_zap_leaf: 646
children[4]:
type: 'disk'
id: 4
guid: 4334627491417169684
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ34TNZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ34TNZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy6-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot06'
whole_disk: 1
DTL: 204
create_txg: 39
com.delphix:vdev_zap_leaf: 647
children[5]:
type: 'disk'
id: 5
guid: 11907400153241021123
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ4NDVZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ4NDVZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy32-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot16'
whole_disk: 1
DTL: 202
create_txg: 39
com.delphix:vdev_zap_leaf: 648
children[6]:
type: 'disk'
id: 6
guid: 2203726962061804941
path: '/dev/disk/by-id/ata-HGST_HUH721010ALE600_1SJ3UUYZ-part1'
devid: 'ata-HGST_HUH721010ALE600_1SJ3UUYZ-part1'
phys_path: 'pci-0000:4b:00.0-sas-exp0x50030480217c6abf-phy33-lun-0'
vdev_enc_sysfs_path: '/sys/class/enclosure/0:0:24:0/Slot17'
whole_disk: 1
DTL: 201
create_txg: 39
com.delphix:vdev_zap_leaf: 649
children[2]:
type: 'mirror'
id: 2
guid: 15187751301081486645
metaslab_array: 783
metaslab_shift: 29
ashift: 12
asize: 34355019776
is_log: 1
create_txg: 84
com.delphix:vdev_zap_top: 776
children[0]:
type: 'disk'
id: 0
guid: 10736577599945821169
path: '/dev/nvme0n1p1'
vdev_enc_sysfs_path: '/sys/bus/pci/slots/102'
whole_disk: 0
DTL: 193
create_txg: 84
com.delphix:vdev_zap_leaf: 777
children[1]:
type: 'disk'
id: 1
guid: 12923533618372533758
path: '/dev/nvme1n1p1'
vdev_enc_sysfs_path: '/sys/bus/pci/slots/106'
whole_disk: 0
DTL: 192
create_txg: 84
com.delphix:vdev_zap_leaf: 778
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data

[drescherjm]

I ended up taking the drive out of the system putting it in a different system and running zpool labelclear -f on it. Then putting it back in and the replace has started:

`
pool: zfs_data_4
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jun 14 09:46:07 2024
253G / 40.7T scanned at 2.01G/s, 0B / 40.6T issued
0B resilvered, 0.00% done, no estimated completion time
config:

    NAME                                   STATE     READ WRITE CKSUM
    zfs_data_4                             DEGRADED     0     0     0
      raidz2-0                             DEGRADED     0     0     0
        replacing-0                        DEGRADED     0     0     0
          13174824313391367059             REMOVED      0     0     0  was /dev/sda1
          sdd1                             ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_7JH4MZ3C  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ47B3Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_7JH0VTBG  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ386AZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SHXP00Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ4RYLZ  ONLINE       0     0     0
      raidz2-1                             ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ4BD2Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ7T9BZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ329AZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ5079Z  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ34TNZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ4NDVZ  ONLINE       0     0     0
        ata-HGST_HUH721010ALE600_1SJ3UUYZ  ONLINE       0     0     0
    logs
      mirror-2                             ONLINE       0     0     0
        nvme0n1p1                          ONLINE       0     0     0
        nvme1n1p1                          ONLINE       0     0     0
    cache
      nvme0n1p2                            ONLINE       0     0     0
      nvme1n1p2                            ONLINE       0     0     0

errors: No known data errors
`