From: bgamari@gmail.com (Ben Gamari)
Subject: [ltt-dev] [RFC PATCH] block: Fix bio merge induced high I/O latency
Date: Tue, 20 Jan 2009 17:23:08 -0500 [thread overview]
Message-ID: <720e76b80901201423p6b82aa77kdfbf3eddb8592bd5@mail.gmail.com> (raw)
In-Reply-To: <720e76b80901201222m72ae2e98l972c81ef5886a12e@mail.gmail.com>
The kernel build finally finished. Unfortunately, it crashes quickly
after booting with moderate disk IO, bringing down the entire machine.
For this reason, I haven't been able to complete a fio benchmark.
Jens, what do you think about this backtrace?
- Ben
BUG: unable to handle kernel paging request at 0000000008
IP: [<ffffffff811c4b2d>] cfq_remove_request+0xb0/0x1da
PGD b2902067 PUD b292e067 PMD 0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0t
CPU 0
Modules linked in: aes_x86_64 aes_generic i915 drm i2c_algo_bit rfcomm bridge s]
Pid: 3903, comm: evolution Not tainted 2.6.29-rc2ben #16
RIP: 0010:[<ffffffff811c4b2d>] [<ffffffff811c4b2d>] cfq_remove_request+0xb0/0xa
RSP: 0018:ffff8800bb853758 EFLAGS: 00010006
RAX: 0000000000200200 RBX: ffff8800b28f3420 RCX: 0000000009deabeb
RDX: 0000000000100100 RSI: ffff8800b010afd0 RDI: ffff8800b010afd0
RBP: ffff8800bb853788 R08: ffff88011fc08250 R09: 000000000cf8b20b
R10: 0000000009e15923 R11: ffff8800b28f3420 R12: ffff8800b010afd0
R13: ffff8800b010afd0 R14: ffff88011d4e8000 R15: ffff88011fc08220
FS: 00007f4b1ef407e0(0000) GS:ffffffff817e7000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 00000000b284b000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process evolution (pid: 3903, threadinfo ffff8800bb852000, task ffff8800da0c2de)
Stack:
ffffffff811ccc19 ffff88011fc08220 ffff8800b010afd0 ffff88011d572000
ffff88011d4e8000 ffff88011d572000 ffff8800bb8537b8 ffffffff811c4ca8
ffff88011fc08220 ffff88011d572000 ffff8800b010afd0 ffff88011fc08250
Call Trace:
[<ffffffff811ccc19>] ? rb_insert_color+0xbd/0xe6
[<ffffffff811c4ca8>] cfq_dispatch_insert+0x51/0x72
[<ffffffff811c4d0d>] cfq_add_rq_rb+0x44/0xcf
[<ffffffff811c5519>] cfq_insert_request+0x34d/0x3d1
[<ffffffff811b6d81>] elv_insert+0x1a9/0x250
[<ffffffff811b6ec3>] __elv_add_request+0x9b/0xa4
[<ffffffff811b9769>] __make_request+0x3c4/0x446
[<ffffffff811b7f53>] generic_make_request+0x2bf/0x309
[<ffffffff811b8068>] submit_bio+0xcb/0xd4
[<ffffffff810f170b>] submit_bh+0x115/0x138
[<ffffffff810f31f7>] ll_rw_block+0xa5/0xf4
[<ffffffff810f3886>] __block_prepare_write+0x277/0x306
[<ffffffff8112c759>] ? ext3_get_block+0x0/0x101
[<ffffffff810f3a7e>] block_write_begin+0x8b/0xdd
[<ffffffff8112bd66>] ext3_write_begin+0xee/0x1c0
[<ffffffff8112c759>] ? ext3_get_block+0x0/0x101
[<ffffffff8109f3be>] generic_file_buffered_write+0x12e/0x2e4
[<ffffffff8109f973>] __generic_file_aio_write_nolock+0x263/0x297
[<ffffffff810e4470>] ? touch_atime+0xdf/0x101
[<ffffffff8109feaa>] ? generic_file_aio_read+0x503/0x59c
[<ffffffff810a01ed>] generic_file_aio_write+0x6c/0xc8
[<ffffffff81128c72>] ext3_file_write+0x23/0xa5
[<ffffffff810d2d77>] do_sync_write+0xec/0x132
[<ffffffff8105da1c>] ? autoremove_wake_function+0x0/0x3d
[<ffffffff8119c880>] ? selinux_file_permission+0x40/0xcb
[<ffffffff8119c902>] ? selinux_file_permission+0xc2/0xcb
[<ffffffff81194cc4>] ? security_file_permission+0x16/0x18
[<ffffffff810d3693>] vfs_write+0xb0/0x10a
[<ffffffff810d37bb>] sys_write+0x4c/0x74
[<ffffffff810114aa>] system_call_fastpath+0x16/0x1b
Code: 48 85 c0 74 0c 4c 39 e0 48 8d b0 60 ff ff ff 75 02 31 f6 48 8b 7d d0 48 8
RIP [<ffffffff811c4b2d>] cfq_remove_request+0xb0/0x1da
RSP <ffff8800bb853758>
CR2: 0000000000100108
---[ end trace 6c5ef63f7957c4cf ]---
On Tue, Jan 20, 2009 at 3:22 PM, Ben Gamari <bgamari at gmail.com> wrote:
> On Tue, Jan 20, 2009 at 2:37 AM, Jens Axboe <jens.axboe at oracle.com> wrote:
>> On Mon, Jan 19 2009, Mathieu Desnoyers wrote:
>>> * Jens Axboe (jens.axboe at oracle.com) wrote:
>>> Yes, ideally I should re-run those directly on the disk partitions.
>>
>> At least for comparison.
>>
>
> I just completed my own set of benchmarks using the fio job file
> Mathieu provided. This was on a 2.5 inch 7200 RPM SATA partition
> formatted as ext3. As you can see, I tested all of the available
> schedulers with both queuing enabled and disabled. I'll test the Jens'
> patch soon. Would a blktrace of the fio run help? Let me know if
> there's any other benchmarking or profiling that could be done.
> Thanks,
>
> - Ben
>
>
> mint maxt
> ==========================================================
> queue_depth=31:
> anticipatory 35 msec 11036 msec
> cfq 37 msec 3350 msec
> deadline 36 msec 18144 msec
> noop 39 msec 41512 msec
>
> ==========================================================
> queue_depth=1:
> anticipatory 45 msec 9561 msec
> cfq 28 msec 3974 msec
> deadline 47 msec 16802 msec
> noop 35 msec 38173 msec
>
next prev parent reply other threads:[~2009-01-20 22:23 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-01-17 0:44 [ltt-dev] [Regression] High latency when doing large I/O Mathieu Desnoyers
2009-01-17 16:26 ` [ltt-dev] [RFC PATCH] block: Fix bio merge induced high I/O latency Mathieu Desnoyers
2009-01-17 16:50 ` Leon Woestenberg
2009-01-17 17:15 ` Mathieu Desnoyers
2009-01-17 19:04 ` Jens Axboe
2009-01-18 21:12 ` Mathieu Desnoyers
2009-01-18 21:27 ` Mathieu Desnoyers
2009-01-19 18:26 ` Jens Axboe
2009-01-20 2:10 ` Mathieu Desnoyers
2009-01-20 7:37 ` Jens Axboe
2009-01-20 12:28 ` Jens Axboe
2009-01-20 14:22 ` Mathieu Desnoyers
2009-01-20 14:24 ` Jens Axboe
2009-01-20 15:42 ` Mathieu Desnoyers
2009-01-20 23:06 ` Mathieu Desnoyers
2009-01-20 23:27 ` Mathieu Desnoyers
2009-01-21 0:25 ` Mathieu Desnoyers
2009-01-21 4:38 ` Ben Gamari
2009-01-21 4:54 ` Mathieu Desnoyers
2009-01-21 6:17 ` Ben Gamari
2009-01-22 22:59 ` Mathieu Desnoyers
2009-01-23 3:21 ` KOSAKI Motohiro
2009-01-23 4:03 ` Mathieu Desnoyers
2009-02-10 3:36 ` [ltt-dev] [PATCH] mm fix page writeback accounting to fix oom condition under heavy I/O Mathieu Desnoyers
2009-02-10 3:55 ` Nick Piggin
2009-02-10 5:23 ` Linus Torvalds
2009-02-10 5:56 ` Nick Piggin
2009-02-10 6:12 ` Mathieu Desnoyers
2009-02-02 2:08 ` [ltt-dev] [RFC PATCH] block: Fix bio merge induced high I/O latency Mathieu Desnoyers
2009-02-02 11:26 ` Jens Axboe
2009-02-03 0:46 ` Mathieu Desnoyers
2009-01-20 13:45 ` Mathieu Desnoyers
2009-01-20 20:22 ` Ben Gamari
2009-01-20 22:23 ` Ben Gamari [this message]
2009-01-20 23:05 ` Mathieu Desnoyers
2009-01-22 2:35 ` Ben Gamari
2009-01-19 15:45 ` Nikanth K
2009-01-19 18:23 ` Jens Axboe
2009-01-17 20:03 ` Ben Gamari
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=720e76b80901201423p6b82aa77kdfbf3eddb8592bd5@mail.gmail.com \
--to=bgamari@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox