2 つの大きな mdadm アレイがあり、それらをボリューム グループに結合しています。最近、2 番目のアレイに新しいドライブを追加していたときに、停電が発生しました。通常、停電後にアレイを再起動したり、再形成を再開したりする際に大きな問題は発生しませんが、今回は大きな問題が発生しています。
システムの詳細: CentOS 6.8 x64
Linux myserver 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
最初に起動すると、アレイは組み立てられますが、劣化しているため起動しません。次の方法では起動できません:
mdadm -R /dev/md126
次にそれを停止します。
mdadm -S /dev/md126
普通に組み立てると起動しません:
mdadm --assemble --scan
mdadm: device 26 in /dev/md/128 has wrong state in superblock, but /dev/sdn seems ok
mdadm: /dev/md/128 assembled from 12 drives and 2 spares - not enough to start the array while not clean - consider --force.
mdadm: No arrays found in config file or automatically
だから私は次のように組み立てます:
mdadm --assemble --scan --force
mdadm: clearing FAULTY flag for device 1 in /dev/md/128 for /dev/sdn
mdadm: Marking array /dev/md/128 as 'clean'
mdadm: /dev/md/128 has been started with 12 drives (out of 13) and 2 spares.
この時点で、私のセッションはハングしています。しかし、デバイスに ssh で再度接続すると、他のセッションがハングしたままでもコマンドを実行できます。
/proc/mdstat を cat する
md128 : active raid6 sdf[0] sdl[9](S) sdn[14](S) sdo[6] sdm[7] sdc[10] sdb[11] sdd[12] sde[13] sdk[5] sdj[4] sdi[3] sdh[2] sdg[1]
23441323008 blocks super 1.2 level 6, 512k chunk, algorithm 2 [13/12] [UUUUUUUUUUUU_]
[>....................] reshape = 0.0% (128496/3906887168) finish=133658963.6min speed=0K/sec
bitmap: 1/30 pages [4KB], 65536KB chunk
md127 : active (auto-read-only) raid6 sdaa[2] sdab[13] sdac[0] sdad[15] sdy[4] sdx[5] sdz[14] sdw[6] sdr[9] sds[8] sdq[10] sdp[16] sdu[11] sdv[7] sdt[12]
50789533184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
unused devices: <none>
再形成は、そのポイントを超えることは決してありません。常にそこで止まっています。
dmesgに次のエラーが表示されます
created bitmap (30 pages) for device md128
md128: bitmap initialized from disk: read 2 pages, set 1 of 59615 bits
md128: detected capacity change from 0 to 24003914760192
md: reshape of RAID array md128
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
md128:
md: using 128k window, over a total of 3906887168k.
unknown partition table
INFO: task mdadm:2357 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000002 0 2357 2179 0x00000004
ffff880821773688 0000000000000082 0000000000000000 0000000000000086
ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001 0 2428 2 0x00000000
ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
[<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
[<ffffffff81431ae5>] md_thread+0x115/0x150
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff814319d0>] ? md_thread+0x0/0x150
[<ffffffff810a640e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff810a6370>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task blkid:2430 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
blkid D 0000000000000002 0 2430 1 0x00000000
ffff8808211d36f8 0000000000000082 0000000000000000 0000000000000082
ffff8808211d3688 ffffffff810633a3 0000013d9831af63 ffff88081f96d440
ffff88081dc65ed0 0000000100104045 ffff88081e615ad8 ffff8808211d3fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff8142af00>] ? try_module_get+0x30/0xb0
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff81340b4b>] ? mix_pool_bytes_extract+0x16b/0x180
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81151319>] ? zone_statistics+0x99/0xc0
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811cfc8d>] submit_bh+0x11d/0x1f0
[<ffffffff811d2a9c>] block_read_full_page+0x27c/0x3a0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811513ee>] ? __inc_zone_page_state+0x2e/0x30
[<ffffffff81145480>] ? __lru_cache_add+0x40/0x90
[<ffffffff811d7498>] blkdev_readpage+0x18/0x20
[<ffffffff811441aa>] __do_page_cache_readahead+0x20a/0x210
[<ffffffff81144251>] force_page_cache_readahead+0x71/0xa0
[<ffffffff81144773>] page_cache_sync_readahead+0x43/0x50
[<ffffffff81130128>] generic_file_aio_read+0x558/0x700
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8115f3d0>] ? unmap_region+0x110/0x130
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2357 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000002 0 2357 2179 0x00000004
ffff880821773688 0000000000000082 0000000000000000 0000000000000086
ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001 0 2428 2 0x00000000
ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
[<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
[<ffffffff81431ae5>] md_thread+0x115/0x150
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff814319d0>] ? md_thread+0x0/0x150
[<ffffffff810a640e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff810a6370>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task blkid:2430 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
blkid D 0000000000000002 0 2430 1 0x00000000
ffff8808211d36f8 0000000000000082 0000000000000000 0000000000000082
ffff8808211d3688 ffffffff810633a3 0000013d9831af63 ffff88081f96d440
ffff88081dc65ed0 0000000100104045 ffff88081e615ad8 ffff8808211d3fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff8142af00>] ? try_module_get+0x30/0xb0
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff81340b4b>] ? mix_pool_bytes_extract+0x16b/0x180
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81151319>] ? zone_statistics+0x99/0xc0
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811cfc8d>] submit_bh+0x11d/0x1f0
[<ffffffff811d2a9c>] block_read_full_page+0x27c/0x3a0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811513ee>] ? __inc_zone_page_state+0x2e/0x30
[<ffffffff81145480>] ? __lru_cache_add+0x40/0x90
[<ffffffff811d7498>] blkdev_readpage+0x18/0x20
[<ffffffff811441aa>] __do_page_cache_readahead+0x20a/0x210
[<ffffffff81144251>] force_page_cache_readahead+0x71/0xa0
[<ffffffff81144773>] page_cache_sync_readahead+0x43/0x50
[<ffffffff81130128>] generic_file_aio_read+0x558/0x700
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8115f3d0>] ? unmap_region+0x110/0x130
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2481 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000003 0 2481 2459 0x00000004
ffff8808226ef688 0000000000000086 0000000000000000 0000000000000086
ffff8808226ef618 ffffffff810633a3 0000014fb75098d5 ffff88081f96d440
ffff88081dc65ed0 0000000100117098 ffff88081e7f7068 ffff8808226effd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff8117ec7c>] ? transfer_objects+0x5c/0x80
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8117f66e>] ? cache_alloc_refill+0x9e/0x240
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff81151319>] ? zone_statistics+0x99/0xc0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task pvscan:2518 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvscan D 0000000000000003 0 2518 2505 0x00000000
ffff8808205e3688 0000000000000086 0000000000000000 0000000000000086
ffff8808205e3618 ffffffff810633a3 000001550b5d4884 ffff88081f96d440
ffff88081dc65ed0 000000010011ca13 ffff88081ddf5ad8 ffff8808205e3fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff811d7ef0>] ? blkdev_get+0x10/0x20
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff81397a3f>] ? scsi_device_put+0x2f/0x40
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task mdadm:2357 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mdadm D 0000000000000002 0 2357 2179 0x00000004
ffff880821773688 0000000000000082 0000000000000000 0000000000000086
ffff880821773618 ffffffff810633a3 0000013e9de36948 ffff88081f96d440
ffff88081dc65ed0 000000010010516a ffff88081d713068 ffff880821773fd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff81130d00>] ? mempool_alloc+0x20/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff810a6bce>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02455ef>] make_request+0x19f/0xcb0 [raid456]
[<ffffffff812ab7de>] ? __sg_alloc_table+0x7e/0x130
[<ffffffff81056cc5>] ? gup_pte_range+0xe5/0x130
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8143066f>] md_make_request+0xdf/0x240
[<ffffffff8127ddb0>] generic_make_request+0x240/0x5a0
[<ffffffff811d941c>] ? do_direct_IO+0x57c/0xfa0
[<ffffffff811d60cb>] ? bio_alloc_bioset+0x5b/0xf0
[<ffffffff8127e180>] submit_bio+0x70/0x120
[<ffffffff811daabd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811db127>] __blockdev_direct_IO+0x77/0xe0
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff811d73a7>] blkdev_direct_IO+0x57/0x60
[<ffffffff811d6320>] ? blkdev_get_block+0x0/0x20
[<ffffffff8113028b>] generic_file_aio_read+0x6bb/0x700
[<ffffffff8143877f>] ? md_ioctl+0x31f/0x1ac0
[<ffffffff811d7f00>] ? blkdev_open+0x0/0xc0
[<ffffffff81196d47>] ? __dentry_open+0x257/0x380
[<ffffffff811d6891>] blkdev_aio_read+0x51/0x80
[<ffffffff81199a6a>] do_sync_read+0xfa/0x140
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811d66bc>] ? block_ioctl+0x3c/0x40
[<ffffffff811af522>] ? vfs_ioctl+0x22/0xa0
[<ffffffff811af6c4>] ? do_vfs_ioctl+0x84/0x580
[<ffffffff8123aa06>] ? security_file_permission+0x16/0x20
[<ffffffff8119a365>] vfs_read+0xb5/0x1a0
[<ffffffff8119b116>] ? fget_light_pos+0x16/0x50
[<ffffffff8119a6b1>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task md128_reshape:2428 blocked for more than 120 seconds.
Not tainted 2.6.32-642.4.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
md128_reshape D 0000000000000001 0 2428 2 0x00000000
ffff88081fdcfa60 0000000000000046 0000000000000000 0000000000000086
ffff88081fdcf9f0 ffffffff810633a3 0000013d983220e0 ffff88081f96d440
ffff88081dc65ed0 0000000100103f9f ffff88081ea1e5f8 ffff88081fdcffd8
Call Trace:
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa0240805>] get_active_stripe+0x2d5/0x880 [raid456]
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa02447c0>] reshape_request+0x440/0xa20 [raid456]
[<ffffffff810633a3>] ? __wake_up+0x53/0x70
[<ffffffffa02450b2>] sync_request+0x312/0x3a0 [raid456]
[<ffffffff81430ff7>] md_do_sync+0x6c7/0xd60
[<ffffffff81431ae5>] md_thread+0x115/0x150
[<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff814319d0>] ? md_thread+0x0/0x150
[<ffffffff810a640e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff810a6370>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
現在、pvscan などのコマンドや、md アレイを制御するその他のコマンドがハングしています。基本的に、アレイを制御できる状態に戻すには、再起動するしかありません。再起動しても、最後にはハングし、その時点でハードリセットする必要があります。
これを解決する方法について何かアイデアはありますか? Ubuntu Live CD から起動しようとしましたが、アレイを組み立てようとしたときに全体がフリーズしてしまいました。
答え1
最終的に、次のコマンドを実行してこの配列を回復することができました。
mdadm --create /dev/md126 --level=6 --raid-devices=14 --name=gigantor:128 /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdo /dev/sdm missing missing /dev/sdc /dev/sdb /dev/sdd /dev/sde --assume-clean
次に、2 つのデバイスを再度追加します。
mdadm --add /dev/md126 /dev/sdn
mdadm --add /dev/md126 /dev/sdl
その後、LVM 情報も破損していたため、それを回復する必要がありました。
pvcreate --uuid "kvEA4X-vobg-2Ipz-ITF1-ZhtW-Ewej-6liKVx" --restorefile /etc/lvm/backup/vg_data /dev/md126
vgcfgrestore vg_data
lvchange -ay /dev/vg_data/lvm0
次に、LVM の xfs_check を実行しようとしましたが、修復を実行してログを破棄する必要があると表示されました。
xfs_repair -L /dev/mapper/vg_data-lvm0
これを修復した後、LVM をマウントできるようになり、データはそのまま残りました。
LVM を修復中です。
Personalities : [raid6] [raid5] [raid4]
md126 : active raid6 sdm[14] sdl[7] sdk[15] sdn[6] sdj[5] sdi[4] sdg[2] sde[0] sdf[1] sdh[3] sdc[12] sda[11] sdd[13] sdb[10]
46882646016 blocks super 1.2 level 6, 512k chunk, algorithm 2 [14/12] [UUUUUUUU__UUUU]
[==========>..........] recovery = 53.6% (2097061864/3906887168) finish=2448.0min speed=12321K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
md127 : active raid6 sdac[15] sdaa[13] sdz[2] sdab[0] sdq[9] sdy[14] sdx[4] sdr[8] sdp[10] sdo[16] sdv[6] sdu[7] sdw[5] sds[12] sdt[11]
50789533184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
unused devices: <none>
痛いですね…。