| Message ID | 20190620003244.261595-1-ndesaulniers@google.com |
|---|---|
| State | New |
| Headers | show |
| Series | arm64: defconfig: update and enable CONFIG_RANDOMIZE_BASE | expand |
Hi Nick, On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote: > Generated via: > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig > <enable CONFIG_RANDOMIZE_BASE aka KASLR> > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig > $ mv defconfig arch/arm64/configs/defconfig Hmm, I'm in two minds about whether we want this on by default. On the plus side, it gets us extra testing coverage, although the /vast/ majority of firmware implementations I run into either don't pass a seed or don't provide a working EFI_RNG. Perhaps that's just a chicken-and-egg problem which can be solved if we shout loud enough when we fail to randomize; we'll also eventually be in a better position when CPUs start implementing the v8.5 RNG instructions (but don't hold your breath unless you have an unusually high lung capacity). On the flip side, I worry that it could make debugging more difficult, but I don't know whether that's a genuine concern or not. I'm assuming you've debugged your fair share of crashes from KASLR-enabled kernels; how bad is it? (I'm thinking of the case where somebody mails you part of a panic log and a .config). Irrespective of the above, I know Catalin was running into issues with his automated tests where the kernel would die silently during early boot with some seeds. That's a bit rubbish if it's still the case -- Catalin? Finally, I know that (K)ASLR can be a bit controversial amongst security folks, with some seeing it as purely a smoke-and-mirrors game with no tangible benefits other than making us feel better about ourselves. Is it still the case that it can be trivially bypassed, or do you see it actually preventing some attacks in production? Sorry for the barrage of questions, but I think enabling this one by default is quite a significant thing to do and probably deserves a bit of scrutiny beforehand. Cheers, Will
On Thu, 20 Jun 2019 at 09:47, Will Deacon <will.deacon@arm.com> wrote: > > Hi Nick, > > On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote: > > Generated via: > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig > > <enable CONFIG_RANDOMIZE_BASE aka KASLR> > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig > > $ mv defconfig arch/arm64/configs/defconfig > > Hmm, I'm in two minds about whether we want this on by default. On the plus > side, it gets us extra testing coverage, although the /vast/ majority of > firmware implementations I run into either don't pass a seed or don't > provide a working EFI_RNG. Perhaps that's just a chicken-and-egg problem > which can be solved if we shout loud enough when we fail to randomize; we'll > also eventually be in a better position when CPUs start implementing the > v8.5 RNG instructions (but don't hold your breath unless you have an > unusually high lung capacity). > For testing coverage purposes, exercising the relocation machinery etc even on no-kaslr boots would be beneficial imo. (The kernel is relocated once for non-kaslr boots and twice for kaslr boots on kaslr capable kernels) > On the flip side, I worry that it could make debugging more difficult, but I > don't know whether that's a genuine concern or not. I'm assuming you've > debugged your fair share of crashes from KASLR-enabled kernels; how bad is > it? (I'm thinking of the case where somebody mails you part of a panic log > and a .config). > When you are debugging using GDB, it can get a bit tedious, since you have to pass the offset when you load the symbols. However, in that case, you can just pass 'nokaslr' unless you are debugging something that is affected by the randomization. For reading backtraces etc, nothing really changes, since we get symbol+nnn/mmm entries (and the full panic log prints the KASLR offset as well, in case it matters) > Irrespective of the above, I know Catalin was running into issues with his > automated tests where the kernel would die silently during early boot with > some seeds. That's a bit rubbish if it's still the case -- Catalin? > Yes, it would be good if we could fix that. In fact, I would argue that having this change in would have increased the likelihood that someone else would have spotted it and fixed it :-) In fact, given how many Android phones are running this code: Nick, can you check if there are any KASLR related kernel fixes that haven't been upstreamed? > Finally, I know that (K)ASLR can be a bit controversial amongst security > folks, with some seeing it as purely a smoke-and-mirrors game with no > tangible benefits other than making us feel better about ourselves. Is it > still the case that it can be trivially bypassed, or do you see it actually > preventing some attacks in production? > > Sorry for the barrage of questions, but I think enabling this one by default > is quite a significant thing to do and probably deserves a bit of scrutiny > beforehand. > I think it is mostly controversial among non-security folks, who think that every mitigation by itself should be bullet proof. Security folks tend to think more about how each layer reduces the attack surface, hopefully resulting in a secure system when all layers are enabled. So KASLR is known to be broken unless you enable KPTI as well, so that is something we could take into account. I.e., mitigations that don't reduce the attack surface at all are just pointless complexity, which should obviously be avoided. Another thing to note is that the runtime cost of KASLR is ~zero, with the exception of the module PLTs. However, the latter could do with some additional coverage as well, so in summary, I think enabling this is a good thing. Otherwise, we could disable full module randomization so that the module PLT code doesn't get used in practice. Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote: > > On Thu, 20 Jun 2019 at 09:47, Will Deacon <will.deacon@arm.com> wrote: > > > On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote: > > > Generated via: > > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig > > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig > > > <enable CONFIG_RANDOMIZE_BASE aka KASLR> > > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig > > > $ mv defconfig arch/arm64/configs/defconfig FWIW, it looks like CONFIG_RANDOMIZE_BASE is set in x86 defconfigs. > > > > Hmm, I'm in two minds about whether we want this on by default. On the plus > > side, it gets us extra testing coverage, although the /vast/ majority of > > firmware implementations I run into either don't pass a seed or don't > > provide a working EFI_RNG. Perhaps that's just a chicken-and-egg problem > > which can be solved if we shout loud enough when we fail to randomize; we'll > > also eventually be in a better position when CPUs start implementing the > > v8.5 RNG instructions (but don't hold your breath unless you have an > > unusually high lung capacity). > > > > For testing coverage purposes, exercising the relocation machinery etc > even on no-kaslr boots would be beneficial imo. (The kernel is > relocated once for non-kaslr boots and twice for kaslr boots on kaslr > capable kernels) Reminds me of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fd9dde6abcb9bfe6c6bee48834e157999f113971 (but maybe unrelated?). > > > On the flip side, I worry that it could make debugging more difficult, but I > > don't know whether that's a genuine concern or not. I'm assuming you've > > debugged your fair share of crashes from KASLR-enabled kernels; how bad is > > it? (I'm thinking of the case where somebody mails you part of a panic log > > and a .config). I don't recall specific cases where KASLR made debugging difficult. I went and spoke to our stability team that debugs crash reports from the field. My understanding is that we capture full ramdumps. They have a lot of custom tooling for debugging, but they did not recall ever having to disable KASLR to debug further. We've had KASLR enabled since I think the 2016 Pixel 1, so I assume their tooling accounts for the seed/offset. I think if a full ramdump of the kernel image is loaded into GDB with the matching kernel image it "just works" but could be mistaken. For external developers, "nokaslr" boot time param is pretty standard. > In fact, given how many Android phones are running this code: Nick, > can you check if there are any KASLR related kernel fixes that haven't > been upstreamed? I spoke with the android common kernel team that's trying to burn down their out of tree patches. I triple checked a doc they had where they had audited every last patch, looking for for KASLR and CONFIG_RANDOMIZE_BASE. I also triple checked our internal bug tracker for burning down the out of tree patches. Finally I'm scanning each branch of our android-common trees via `git log --all --grep <KASLR|CONFIG_RANDOMIZE_BASE>`. I haven't found anything yet, and the team doesn't expect any out of tree patches related to that feature. Sorry for not responding sooner, but I'm still going through our 4.4, 4.9, 4.14, and 4.19 branches. I know I put some backports into an unreleased device's kernel to support LLD (backports of some of your patches). All of those patches are now upstream in mainline, and LLD has its bugs fixed. In fact, I think the backports I did of your patches that broke CONFIG_RANDOMIZE_BASE were actually to support LLD (not Clang) (sorry again for that). I'm currently upgrading AOSP's prebuilt version of Clang. Once that exists, I'll clean up that device's kernel. If you know of any android vendor relying on out of tree hacks to KASLR, and they're getting them from android common kernels, please shoot me a quick email. Every out of tree patch is a burden IMO. Also, for internal code reviews of backports, I've been much better/more aggressive about the backports being sent to stable (when they aren't large features) and then pulling them in from there (sorry, Sami). > > > Finally, I know that (K)ASLR can be a bit controversial amongst security > > folks, with some seeing it as purely a smoke-and-mirrors game with no > > tangible benefits other than making us feel better about ourselves. Is it > > still the case that it can be trivially bypassed, or do you see it actually > > preventing some attacks in production? > > > > Sorry for the barrage of questions, but I think enabling this one by default > > is quite a significant thing to do and probably deserves a bit of scrutiny > > beforehand. > > > > I think it is mostly controversial among non-security folks, who think > that every mitigation by itself should be bullet proof. Security folks > tend to think more about how each layer reduces the attack surface, > hopefully resulting in a secure system when all layers are enabled. + Kees, Sami, Jeff It's a relatively low cost part of our defense in depth strategy. Maybe (Kees, Sami, Jeff) have more thoughts? > > So KASLR is known to be broken unless you enable KPTI as well, so that > is something we could take into account. I.e., mitigations that don't > reduce the attack surface at all are just pointless complexity, which > should obviously be avoided. (Note to Sami + Jeff if they had KPTI on their radar) > > Another thing to note is that the runtime cost of KASLR is ~zero, with > the exception of the module PLTs. However, the latter could do with > some additional coverage as well, so in summary, I think enabling this > is a good thing. Otherwise, we could disable full module randomization > so that the module PLT code doesn't get used in practice. > > Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Olof mentioned on IRC that I should resend without the other defconfig changes. Do others have thoughts on that? -- Thanks, ~Nick Desaulniers
On Fri, Jun 21, 2019 at 1:28 PM Nick Desaulniers <ndesaulniers@google.com> wrote: > > On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel > <ard.biesheuvel@linaro.org> wrote: > > I think it is mostly controversial among non-security folks, who think > > that every mitigation by itself should be bullet proof. Security folks > > tend to think more about how each layer reduces the attack surface, > > hopefully resulting in a secure system when all layers are enabled. > > + Kees, Sami, Jeff > It's a relatively low cost part of our defense in depth strategy. > Maybe (Kees, Sami, Jeff) have more thoughts? Right -- the thought is that it provides more benefit than complication. It is hardly a perfect defense, but it does provide building blocks to more interesting situations. For example, once execute-only memory is more common, KASLR + XOM means there is a not insignificant defense against automated ROP. And KASLR is a general precursor to fine-grained KASLR (i.e. randomizing on function). > > So KASLR is known to be broken unless you enable KPTI as well, so that > > is something we could take into account. I.e., mitigations that don't > > reduce the attack surface at all are just pointless complexity, which > > should obviously be avoided. > > (Note to Sami + Jeff if they had KPTI on their radar) I prefer that KPTI always stay enabled. :) > > Another thing to note is that the runtime cost of KASLR is ~zero, with > > the exception of the module PLTs. However, the latter could do with > > some additional coverage as well, so in summary, I think enabling this > > is a good thing. Otherwise, we could disable full module randomization > > so that the module PLT code doesn't get used in practice. > > > > Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Reviewed-by: Kees Cook <keescook@chromium.org> -- Kees Cook
On Thu, Jun 20, 2019 at 08:46:58AM +0100, Will Deacon wrote: > On Wed, Jun 19, 2019 at 05:32:42PM -0700, Nick Desaulniers wrote: > > Generated via: > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig > > <enable CONFIG_RANDOMIZE_BASE aka KASLR> > > $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig > > $ mv defconfig arch/arm64/configs/defconfig [...] > Irrespective of the above, I know Catalin was running into issues with his > automated tests where the kernel would die silently during early boot with > some seeds. That's a bit rubbish if it's still the case -- Catalin? I stopped seeing these failures about 1-2 months ago, not sure exactly what changed. I'm ok to enable KASLR in defconfig. -- Catalin
Hi Nick, Kees, Ard, Thanks for the responses. On Fri, Jun 21, 2019 at 01:27:45PM -0700, Nick Desaulniers wrote: > On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel > <ard.biesheuvel@linaro.org> wrote: > > On Thu, 20 Jun 2019 at 09:47, Will Deacon <will.deacon@arm.com> wrote: > > > On the flip side, I worry that it could make debugging more difficult, but I > > > don't know whether that's a genuine concern or not. I'm assuming you've > > > debugged your fair share of crashes from KASLR-enabled kernels; how bad is > > > it? (I'm thinking of the case where somebody mails you part of a panic log > > > and a .config). > > I don't recall specific cases where KASLR made debugging difficult. I > went and spoke to our stability team that debugs crash reports from > the field. My understanding is that we capture full ramdumps. They > have a lot of custom tooling for debugging, but they did not recall > ever having to disable KASLR to debug further. We've had KASLR > enabled since I think the 2016 Pixel 1, so I assume their tooling > accounts for the seed/offset. > > I think if a full ramdump of the kernel image is loaded into GDB with > the matching kernel image it "just works" but could be mistaken. For > external developers, "nokaslr" boot time param is pretty standard. > > > In fact, given how many Android phones are running this code: Nick, > > can you check if there are any KASLR related kernel fixes that haven't > > been upstreamed? > > I spoke with the android common kernel team that's trying to burn down > their out of tree patches. I triple checked a doc they had where they > had audited every last patch, looking for for KASLR and > CONFIG_RANDOMIZE_BASE. I also triple checked our internal bug tracker > for burning down the out of tree patches. Finally I'm scanning each > branch of our android-common trees via `git log --all --grep > <KASLR|CONFIG_RANDOMIZE_BASE>`. I haven't found anything yet, and the > team doesn't expect any out of tree patches related to that feature. > Sorry for not responding sooner, but I'm still going through our 4.4, > 4.9, 4.14, and 4.19 branches. Thanks for having a look. It could be that we've fixed the issue Catalin was running into in the past -- he was going to see if the problem persists with mainline, since it was frequent enough that it was causing us to ignore the results from our testing infrastructure when RANDOMIZE_BASE=y. > > So KASLR is known to be broken unless you enable KPTI as well, so that > > is something we could take into account. I.e., mitigations that don't > > reduce the attack surface at all are just pointless complexity, which > > should obviously be avoided. > > (Note to Sami + Jeff if they had KPTI on their radar) I mean, we could have RANDOMIZE_BASE select UNMAP_KERNEL_AT_EL0 if you like? The latter is already default y and hidden behind EXPERT. > > Another thing to note is that the runtime cost of KASLR is ~zero, with > > the exception of the module PLTs. However, the latter could do with > > some additional coverage as well, so in summary, I think enabling this > > is a good thing. Otherwise, we could disable full module randomization > > so that the module PLT code doesn't get used in practice. > > > > Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> > > Olof mentioned on IRC that I should resend without the other defconfig > changes. Do others have thoughts on that? That's not a bad idea. If you do that, feel free to add my Ack to the one adding RANDOMIZE_BASE=y: Acked-by: Will Deacon <will@kernel.org> Will
On Mon, 24 Jun 2019 at 11:57, Will Deacon <will@kernel.org> wrote: > > Hi Nick, Kees, Ard, > > Thanks for the responses. > > On Fri, Jun 21, 2019 at 01:27:45PM -0700, Nick Desaulniers wrote: > > On Thu, Jun 20, 2019 at 1:17 AM Ard Biesheuvel > > <ard.biesheuvel@linaro.org> wrote: > > > On Thu, 20 Jun 2019 at 09:47, Will Deacon <will.deacon@arm.com> wrote: > > > > On the flip side, I worry that it could make debugging more difficult, but I > > > > don't know whether that's a genuine concern or not. I'm assuming you've > > > > debugged your fair share of crashes from KASLR-enabled kernels; how bad is > > > > it? (I'm thinking of the case where somebody mails you part of a panic log > > > > and a .config). > > > > I don't recall specific cases where KASLR made debugging difficult. I > > went and spoke to our stability team that debugs crash reports from > > the field. My understanding is that we capture full ramdumps. They > > have a lot of custom tooling for debugging, but they did not recall > > ever having to disable KASLR to debug further. We've had KASLR > > enabled since I think the 2016 Pixel 1, so I assume their tooling > > accounts for the seed/offset. > > > > I think if a full ramdump of the kernel image is loaded into GDB with > > the matching kernel image it "just works" but could be mistaken. For > > external developers, "nokaslr" boot time param is pretty standard. > > > > > In fact, given how many Android phones are running this code: Nick, > > > can you check if there are any KASLR related kernel fixes that haven't > > > been upstreamed? > > > > I spoke with the android common kernel team that's trying to burn down > > their out of tree patches. I triple checked a doc they had where they > > had audited every last patch, looking for for KASLR and > > CONFIG_RANDOMIZE_BASE. I also triple checked our internal bug tracker > > for burning down the out of tree patches. Finally I'm scanning each > > branch of our android-common trees via `git log --all --grep > > <KASLR|CONFIG_RANDOMIZE_BASE>`. I haven't found anything yet, and the > > team doesn't expect any out of tree patches related to that feature. > > Sorry for not responding sooner, but I'm still going through our 4.4, > > 4.9, 4.14, and 4.19 branches. > > Thanks for having a look. It could be that we've fixed the issue Catalin was > running into in the past -- he was going to see if the problem persists with > mainline, since it was frequent enough that it was causing us to ignore the > results from our testing infrastructure when RANDOMIZE_BASE=y. > I had no idea this was the case. I can look into it if we are still seeing failures. > > > So KASLR is known to be broken unless you enable KPTI as well, so that > > > is something we could take into account. I.e., mitigations that don't > > > reduce the attack surface at all are just pointless complexity, which > > > should obviously be avoided. > > > > (Note to Sami + Jeff if they had KPTI on their radar) > > I mean, we could have RANDOMIZE_BASE select UNMAP_KERNEL_AT_EL0 if you like? > The latter is already default y and hidden behind EXPERT. > IIRC, when KASLR is enabled (and we have a seed), we override the runtime decision to out out of KPTI, and so even uarchs that don't require the Meltdown mitigations it provides will still be using it. So I'd be fine with just adding a note to the UNMAP_KERNEL_AT_EL0 Kconfig help text that even non-affected uarchs have a use for it if KASLR is enabled, but given that it is already behind EXPERT, I don't think more hand holding is necessary. > > > Another thing to note is that the runtime cost of KASLR is ~zero, with > > > the exception of the module PLTs. However, the latter could do with > > > some additional coverage as well, so in summary, I think enabling this > > > is a good thing. Otherwise, we could disable full module randomization > > > so that the module PLT code doesn't get used in practice. > > > > > > Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> > > > > Olof mentioned on IRC that I should resend without the other defconfig > > changes. Do others have thoughts on that? > > That's not a bad idea. If you do that, feel free to add my Ack to the one > adding RANDOMIZE_BASE=y: > > Acked-by: Will Deacon <will@kernel.org> > > Will
On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote: > On Mon, 24 Jun 2019 at 11:57, Will Deacon <will@kernel.org> wrote: > > Thanks for having a look. It could be that we've fixed the issue Catalin was > > running into in the past -- he was going to see if the problem persists with > > mainline, since it was frequent enough that it was causing us to ignore the > > results from our testing infrastructure when RANDOMIZE_BASE=y. > > I had no idea this was the case. I can look into it if we are still > seeing failures. I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in a guest on TX2. It takes a few tries to trigger just with kaslr, enabling lots of other DEBUG_* options makes the failures more deterministic. I can't really say it's kaslr's fault here, only that I used to consistently get it in this configuration. For some reason, I can no longer reproduce it on arm64 for-next/core (or maybe it just takes more tries and my script doesn't catch this). The fault is in the ip_tables module, the __this_cpu_read in xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled sequence in my build: 0000000000000188 <ipt_do_table>: ... 258: d538d080 mrs x0, tpidr_el1 25c: aa1303f9 mov x25, x19 260: b8606b34 ldr w20, [x25, x0] # modprobe iptable_filter [ 45.618896] Unable to handle kernel paging request at virtual address ffffeda0ffbe1388 [ 45.620545] Mem abort info: [ 45.621035] ESR = 0x96000005 [ 45.621559] Exception class = DABT (current EL), IL = 32 bits [ 45.622527] SET = 0, FnV = 0 [ 45.623032] EA = 0, S1PTW = 0 [ 45.623684] Data abort info: [ 45.624192] ISV = 0, ISS = 0x00000005 [ 45.624849] CM = 0, WnR = 0 [ 45.625361] swapper pgtable: 4k pages, 48-bit VAs, pgdp = 0000000026cb760e [ 45.626537] [ffffeda0ffbe1388] pgd=00000000bfffa003, pud=0000000000000000 [ 45.627864] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 45.628844] Modules linked in: iptable_filter cfg80211 rfkill 8021q garp crct10dif_ce stp mrp llc ip_tables x_tables ipv6 [ 45.630700] Process kworker/u8:5 (pid: 173, stack limit = 0x0000000047fc7e17) [ 45.632056] CPU: 3 PID: 173 Comm: kworker/u8:5 Not tainted 5.2.0-rc1 #1 [ 45.633377] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 45.634752] Workqueue: rpciod rpc_async_schedule [ 45.635684] pstate: 80400005 (Nzcv daif +PAN -UAO) [ 45.636674] pc : ipt_do_table+0xd8/0x4b8 [ip_tables] [ 45.637686] lr : ipt_do_table+0xc4/0x4b8 [ip_tables] [ 45.638685] sp : ffff000010aa35a0 [ 45.639358] x29: ffff000010aa35a0 x28: ffffeda1f7eb0ce8 [ 45.640424] x27: ffffeda1e69f1380 x26: ffff5d2c2314a288 [ 45.641493] x25: ffff5d2c0cfbf388 x24: ffffeda1fb5a9000 [ 45.642556] x23: ffff5d2c2314a220 x22: 0000000000000003 [ 45.643636] x21: ffffeda1f79c15d8 x20: ffffeda1e69f1780 [ 45.644727] x19: ffff5d2c0cfbf388 x18: 0000000000000000 [ 45.645814] x17: 0000000000000000 x16: ffff5d2d0bc16d78 [ 45.646908] x15: 0000000000000000 x14: 0900000017ea6502 [ 45.647994] x13: 014a023b4751727f x12: 0000000000002238 [ 45.649060] x11: ffff5d2d0e0de000 x10: 0000000000004000 [ 45.650117] x9 : 000000000000000e x8 : ffffeda1fb5a9000 [ 45.651178] x7 : ffffeda1f7eb0c00 x6 : 00000000000000e8 [ 45.652263] x5 : 0000000000000080 x4 : 00009074f2c22000 [ 45.653286] x3 : 0000000000000200 x2 : ffffeda1fc41c440 [ 45.654314] x1 : ffff5d2c2314a000 x0 : 00009074f2c22000 [ 45.655382] Call trace: [ 45.655896] ipt_do_table+0xd8/0x4b8 [ip_tables] [ 45.656825] iptable_filter_hook+0x1c/0x28 [iptable_filter] [ 45.657938] nf_hook_slow+0x44/0xe8 [ 45.658647] __ip_local_out+0x150/0x250 [ 45.659421] ip_local_out+0x20/0x80 [ 45.660142] __ip_queue_xmit+0x1b0/0x540 [ 45.660937] ip_queue_xmit+0x10/0x18 [ 45.661661] __tcp_transmit_skb+0x50c/0xad8 [ 45.662494] tcp_write_xmit+0x6cc/0x1010 [ 45.663294] __tcp_push_pending_frames+0x38/0xc0 [ 45.664236] tcp_push+0x150/0x170 [ 45.664912] tcp_sendmsg_locked+0xaa0/0xc40 [ 45.665754] tcp_sendmsg+0x34/0x58 [ 45.666442] inet_sendmsg+0x48/0x210 [ 45.667175] sock_sendmsg+0x18/0x30 [ 45.667889] xs_sendpages+0xfc/0x2d8 [ 45.668610] xs_tcp_send_request+0x9c/0x1b0 [ 45.669454] xprt_transmit+0x100/0x5b8 [ 45.670210] call_transmit+0x8c/0xa0 [ 45.670938] __rpc_execute+0xbc/0x580 [ 45.671687] rpc_async_schedule+0x28/0x48 [ 45.672493] process_one_work+0x244/0x680 [ 45.673306] worker_thread+0x40/0x3f0 [ 45.674053] kthread+0x128/0x130 [ 45.674710] ret_from_fork+0x10/0x18 [ 45.675436] Code: b0f4f3d3 910e2273 d538d080 aa1303f9 (b8606b34) [ 45.676665] ---[ end trace 9d1f75fe6e41c0d7 ]--- [ 45.677590] Kernel panic - not syncing: Fatal exception in interrupt [ 45.678851] SMP: stopping secondary CPUs [ 45.679670] Kernel Offset: 0x5d2cfb600000 from 0xffff000010000000 [ 45.680882] PHYS_OFFSET: 0xffff925ec0000000 [ 45.681727] CPU features: 0x002,23800438 [ 45.682518] Memory Limit: none [ 45.683143] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- -- Catalin
On Tue, 25 Jun 2019 at 17:39, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote: > > On Mon, 24 Jun 2019 at 11:57, Will Deacon <will@kernel.org> wrote: > > > Thanks for having a look. It could be that we've fixed the issue Catalin was > > > running into in the past -- he was going to see if the problem persists with > > > mainline, since it was frequent enough that it was causing us to ignore the > > > results from our testing infrastructure when RANDOMIZE_BASE=y. > > > > I had no idea this was the case. I can look into it if we are still > > seeing failures. > > I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in > a guest on TX2. It takes a few tries to trigger just with kaslr, > enabling lots of other DEBUG_* options makes the failures more > deterministic. I can't really say it's kaslr's fault here, only that I > used to consistently get it in this configuration. For some reason, I > can no longer reproduce it on arm64 for-next/core (or maybe it just > takes more tries and my script doesn't catch this). > > The fault is in the ip_tables module, the __this_cpu_read in > xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled > sequence in my build: > > 0000000000000188 <ipt_do_table>: > ... > 258: d538d080 mrs x0, tpidr_el1 > 25c: aa1303f9 mov x25, x19 > 260: b8606b34 ldr w20, [x25, x0] > This was fixed recently by arm64/kernel: kaslr: reduce module randomization range to 2 GB (and arm64/module: deal with ambiguity in PRELxx relocation ranges to some extent)
On Tue, Jun 25, 2019 at 05:42:49PM +0200, Ard Biesheuvel wrote: > On Tue, 25 Jun 2019 at 17:39, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote: > > > On Mon, 24 Jun 2019 at 11:57, Will Deacon <will@kernel.org> wrote: > > > > Thanks for having a look. It could be that we've fixed the issue Catalin was > > > > running into in the past -- he was going to see if the problem persists with > > > > mainline, since it was frequent enough that it was causing us to ignore the > > > > results from our testing infrastructure when RANDOMIZE_BASE=y. > > > > > > I had no idea this was the case. I can look into it if we are still > > > seeing failures. > > > > I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in > > a guest on TX2. It takes a few tries to trigger just with kaslr, > > enabling lots of other DEBUG_* options makes the failures more > > deterministic. I can't really say it's kaslr's fault here, only that I > > used to consistently get it in this configuration. For some reason, I > > can no longer reproduce it on arm64 for-next/core (or maybe it just > > takes more tries and my script doesn't catch this). > > > > The fault is in the ip_tables module, the __this_cpu_read in > > xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled > > sequence in my build: > > > > 0000000000000188 <ipt_do_table>: > > ... > > 258: d538d080 mrs x0, tpidr_el1 > > 25c: aa1303f9 mov x25, x19 > > 260: b8606b34 ldr w20, [x25, x0] > > This was fixed recently by > > arm64/kernel: kaslr: reduce module randomization range to 2 GB > > (and arm64/module: deal with ambiguity in PRELxx relocation ranges to > some extent) Thanks. This explains it. And another weird case that triggers only with 64K pages, KASan and KASLR combination (guest on TX2). My test script modprobes all the modules it finds installed (including some test kernel modules like lock torture). At some point during modprobing, vmalloc trips over the WARN_ON(!pte_none(*pte)) in vmap_pte_range(): # modprobe <some random module> [ 341.631037] WARNING: CPU: 3 PID: 737 at mm/vmalloc.c:153 vmap_page_range_noflush+0x300/0x410 [ 341.632433] Modules linked in: pwm_fan altera_freeze_bridge fpga_region fpga_bridge reset_meson_audio_arb cros_ec_baro cros_ec_sensors rockchip_saradc cros_ec_light_prox industrialio_triggered_buffer kfifo_buf cros_ec_sensors_core spi_meson_spifc spi_meson_spicc spi_bcm2835 spi_ [ 341.632533] pwm_sun4i pwm_rcar aes_neon_bs chacha_neon chacha_generic sha3_generic aes_neon_blk sm3_generic sha512_arm64 test_udelay backtracetest locktorture torture br_netfilter bridge ip6table_mangle ip6table_nat nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tab [ 341.661768] CPU: 3 PID: 737 Comm: modprobe Tainted: G W 5.2.0-rc3-00015-gd1b79ba6a070 #82 [ 341.663644] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 341.665065] pstate: 20400005 (nzCv daif +PAN -UAO) [ 341.666027] pc : vmap_page_range_noflush+0x300/0x410 [ 341.667015] lr : map_vm_area+0x8c/0xd8 [ 341.667806] sp : ffffb618f4877690 [ 341.668488] x29: ffffb618f4877690 x28: ffff200026cc0008 [ 341.669552] x27: dfff200000000000 x26: ffff0400023f0000 [ 341.670611] x25: ffff200026bea818 x24: ffffb6193fd411f8 [ 341.671718] x23: ffff040002400000 x22: ffffb6193fd50000 [ 341.672769] x21: ffff040002400000 x20: ffff040004d7d503 [ 341.673836] x19: 0000000000000000 x18: ffffb618f11d65b0 [ 341.674903] x17: 0000000000002036 x16: ffff200024320da8 [ 341.675979] x15: 0000000000001000 x14: 00000000000001ba [ 341.677043] x13: 0000000000054f18 x12: ffff0ffdb0c7a6b7 [ 341.678093] x11: 1fffeffdb0c7a6b6 x10: 1ffff6c327fa823f [ 341.679197] x9 : ffff0400223f0000 x8 : 000036193fd50000 [ 341.680246] x7 : ffff200026bea000 x6 : 0000000000000001 [ 341.681362] x5 : 00000000f2f2f200 x4 : ffff200027c236c8 [ 341.682459] x3 : ffff0800023f0000 x2 : ffff7fed863d3580 [ 341.683517] x1 : 000036193fd411f8 x0 : 00e000005f880f93 [ 341.684582] Call trace: [ 341.685089] vmap_page_range_noflush+0x300/0x410 [ 341.686050] map_vm_area+0x8c/0xd8 [ 341.686742] __vmalloc_node_range+0x4e4/0x718 [ 341.687608] kasan_module_alloc+0xb4/0x168 [ 341.688440] module_alloc+0xc0/0x240 [ 341.689171] load_module+0x18ac/0x65c0 [ 341.689970] __se_sys_finit_module+0x138/0x148 [ 341.690849] __arm64_sys_finit_module+0x68/0x98 [ 341.691765] el0_svc_handler+0x14c/0x360 [ 341.692539] el0_svc+0x8/0xc [ 341.693167] ---[ end trace 1e09f33962be79c8 ]--- From this point, the kernel gets into some weird state with a few other page faults at check_memory_region+0x64/0x1c8. Eventually, during poweroff, I get the splat below. It seems that even though vzalloc() allocated, the memory isn't there (which is not surprising given the above warning. [ 1016.168708] Unable to handle kernel paging request at virtual address ffff0400021c200a [ 1016.170104] Mem abort info: [ 1016.170598] ESR = 0x96000007 [ 1016.172011] Exception class = DABT (current EL), IL = 32 bits [ 1016.173079] SET = 0, FnV = 0 [ 1016.173614] EA = 0, S1PTW = 0 [ 1016.174177] Data abort info: [ 1016.174688] ISV = 0, ISS = 0x00000007 [ 1016.175513] CM = 0, WnR = 0 [ 1016.176064] swapper pgtable: 64k pages, 48-bit VAs, pgdp=000000005dec0000 [ 1016.177251] [ffff0400021c200a] pgd=00000000bfd50003, pud=00000000bfd50003, pmd=00000000bfd40003, pte=0000000000000000 [ 1016.179155] Internal error: Oops: 96000007 [#4] PREEMPT SMP [ 1016.180122] Modules linked in: ghash_generic lcd pwm_fan altera_freeze_bridge fpga_region fpga_bridge reset_meson_audio_arb cros_ec_baro cros_ec_sensors rockchip_saradc cros_ec_light_prox industrialio_triggered_buffer kfifo_buf cros_ec_sensors_core spi_meson_spifc spi_meson_spic [ 1016.180226] pwm_bcm2835 pwm_tegra pwm_sun4i pwm_rcar aes_neon_bs chacha_neon chacha_generic sha3_generic aes_neon_blk sm3_generic sha512_arm64 test_udelay backtracetest locktorture torture br_netfilter bridge ip6table_mangle ip6table_nat nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 i [ 1016.212762] CPU: 1 PID: 1 Comm: systemd Tainted: G D W 5.2.0-rc3-00015-gd1b79ba6a070 #82 [ 1016.214653] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 1016.216081] pstate: 20400005 (nzCv daif +PAN -UAO) [ 1016.217152] pc : n_tty_open+0x54/0x148 [ 1016.217931] lr : n_tty_open+0x18/0x148 [ 1016.218781] sp : ffffb618e0707660 [ 1016.219505] x29: ffffb618e0707660 x28: ffffb618e0fe7b00 [ 1016.220662] x27: 0000000000000000 x26: 0000000000020101 [ 1016.221758] x25: 1ffff6c31f6dbd68 x24: 0000000000000000 [ 1016.222866] x23: ffffb618f3069058 x22: 0000000000000000 [ 1016.223987] x21: 000000010002bb56 x20: ffff200010e10000 [ 1016.225097] x19: ffffb618f3069000 x18: 0000000000000000 [ 1016.226200] x17: 0000000000000000 x16: 0000000000000000 [ 1016.227296] x15: 0000000000000000 x14: 0000000000000000 [ 1016.228364] x13: 1ffff6c31c0e0f93 x12: ffff0ffdb0c7f1df [ 1016.229479] x11: 1fffeffdb0c7f1de x10: 1ffff6c31c04d0e1 [ 1016.230594] x9 : ffff200030e10000 x8 : 000036193fff0000 [ 1016.231700] x7 : ffff200026bea000 x6 : 0000000000000008 [ 1016.232812] x5 : 00000000f2f2f200 x4 : ffff200027c236c8 [ 1016.233930] x3 : ffff240010e10000 x2 : 1fffe400021c200a [ 1016.235043] x1 : dfff200000000000 x0 : ffff200010e10050 [ 1016.236149] Call trace: [ 1016.236693] n_tty_open+0x54/0x148 [ 1016.237409] tty_ldisc_open.isra.1+0x94/0x128 [ 1016.238334] tty_ldisc_setup+0x40/0xd0 [ 1016.239117] tty_init_dev+0x180/0x3a0 [ 1016.239906] tty_open+0x438/0x780 [ 1016.240627] chrdev_open+0x194/0x408 [ 1016.241385] do_dentry_open+0x2ec/0xbf0 [ 1016.242189] vfs_open+0x8c/0xc0 [ 1016.242862] path_openat+0x694/0x3310 [ 1016.243641] do_filp_open+0x13c/0x200 [ 1016.244402] do_sys_open+0x1dc/0x320 [ 1016.245181] __arm64_sys_openat+0x88/0xc8 [ 1016.246031] el0_svc_handler+0x14c/0x360 [ 1016.246867] el0_svc+0x8/0xc [ 1016.247472] Code: d2c40001 f2fbffe1 d343fc02 f944c2b5 (38e16841) [ 1016.248794] ---[ end trace 1e09f33962be79ce ]--- [ 1016.366872] printk: systemd: 46 output lines suppressed due to ratelimiting [ 1016.369964] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 1016.371385] SMP: stopping secondary CPUs [ 1016.372226] Kernel Offset: 0x141d0000 from 0xffff200010000000 [ 1016.373378] PHYS_OFFSET: 0xffffc9e780000000 [ 1016.374242] CPU features: 0x0002,23800438 [ 1016.375049] Memory Limit: none [ 1016.375702] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- -- Catalin
On Tue, 25 Jun 2019 at 18:03, Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Tue, Jun 25, 2019 at 05:42:49PM +0200, Ard Biesheuvel wrote: > > On Tue, 25 Jun 2019 at 17:39, Catalin Marinas <catalin.marinas@arm.com> wrote: > > > On Mon, Jun 24, 2019 at 12:06:18PM +0200, Ard Biesheuvel wrote: > > > > On Mon, 24 Jun 2019 at 11:57, Will Deacon <will@kernel.org> wrote: > > > > > Thanks for having a look. It could be that we've fixed the issue Catalin was > > > > > running into in the past -- he was going to see if the problem persists with > > > > > mainline, since it was frequent enough that it was causing us to ignore the > > > > > results from our testing infrastructure when RANDOMIZE_BASE=y. > > > > > > > > I had no idea this was the case. I can look into it if we are still > > > > seeing failures. > > > > > > I've seen the panic below with 5.2-rc1, defconfig + RANDOMIZE_BASE=y in > > > a guest on TX2. It takes a few tries to trigger just with kaslr, > > > enabling lots of other DEBUG_* options makes the failures more > > > deterministic. I can't really say it's kaslr's fault here, only that I > > > used to consistently get it in this configuration. For some reason, I > > > can no longer reproduce it on arm64 for-next/core (or maybe it just > > > takes more tries and my script doesn't catch this). > > > > > > The fault is in the ip_tables module, the __this_cpu_read in > > > xt_write_recseq_begin() inlined in ipt_do_table(). The disassembled > > > sequence in my build: > > > > > > 0000000000000188 <ipt_do_table>: > > > ... > > > 258: d538d080 mrs x0, tpidr_el1 > > > 25c: aa1303f9 mov x25, x19 > > > 260: b8606b34 ldr w20, [x25, x0] > > > > This was fixed recently by > > > > arm64/kernel: kaslr: reduce module randomization range to 2 GB > > > > (and arm64/module: deal with ambiguity in PRELxx relocation ranges to > > some extent) > > Thanks. This explains it. > > And another weird case that triggers only with 64K pages, KASan and > KASLR combination (guest on TX2). My test script modprobes all the > modules it finds installed (including some test kernel modules like lock > torture). At some point during modprobing, vmalloc trips over the > WARN_ON(!pte_none(*pte)) in vmap_pte_range(): > When KASAN and KASLR are both enabled, modules are allocated in the dedicated module window, since the vmalloc space is already shadowed by KASAN zero pages and so the modules must be kept out of it. since they have their own real shadow pages that are allocated on demand. Looking at the backtrace, it seems like the failure may be due to the shadow space clashing, probably because the top of the module region exceeds MODULES_END. Does the below help at all? (patch soup courtesy of gmail, apologies) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index f713e2fc4d75..7e94e1f948c9 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -32,6 +32,7 @@ void *module_alloc(unsigned long size) { + u64 module_alloc_end = module_alloc_base + MODULES_VSIZE; gfp_t gfp_mask = GFP_KERNEL; void *p; @@ -39,9 +40,11 @@ void *module_alloc(unsigned long size) if (IS_ENABLED(CONFIG_ARM64_MODULE_PLTS)) gfp_mask |= __GFP_NOWARN; + if (IS_ENABLED(CONFIG_KASAN)) + module_alloc_end = MODULES_END; + p = __vmalloc_node_range(size, MODULE_ALIGN, module_alloc_base, - module_alloc_base + MODULES_VSIZE, - gfp_mask, PAGE_KERNEL_EXEC, 0, + module_alloc_end, gfp_mask, PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, __builtin_return_address(0)); if (!p && IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) &&
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 4d583514258c..54d35e847836 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -68,6 +68,7 @@ CONFIG_KEXEC=y CONFIG_CRASH_DUMP=y CONFIG_XEN=y CONFIG_COMPAT=y +CONFIG_RANDOMIZE_BASE=y CONFIG_HIBERNATION=y CONFIG_WQ_POWER_EFFICIENT_DEFAULT=y CONFIG_ARM_CPUIDLE=y @@ -86,7 +87,6 @@ CONFIG_ARM_TEGRA186_CPUFREQ=y CONFIG_ARM_SCPI_PROTOCOL=y CONFIG_RASPBERRYPI_FIRMWARE=y CONFIG_INTEL_STRATIX10_SERVICE=y -CONFIG_TI_SCI_PROTOCOL=y CONFIG_EFI_CAPSULE_LOADER=y CONFIG_IMX_SCU=y CONFIG_IMX_SCU_PD=y @@ -191,7 +191,6 @@ CONFIG_PCIE_QCOM=y CONFIG_PCIE_ARMADA_8K=y CONFIG_PCIE_KIRIN=y CONFIG_PCIE_HISI_STB=y -CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_HISILICON_LPC=y @@ -365,8 +364,8 @@ CONFIG_SPI_PL022=y CONFIG_SPI_ROCKCHIP=y CONFIG_SPI_QUP=y CONFIG_SPI_S3C64XX=y -CONFIG_SPI_SPIDEV=m CONFIG_SPI_SUN6I=y +CONFIG_SPI_SPIDEV=m CONFIG_SPMI=y CONFIG_PINCTRL_SINGLE=y CONFIG_PINCTRL_MAX77620=y @@ -658,7 +657,6 @@ CONFIG_ARM_MHU=y CONFIG_IMX_MBOX=y CONFIG_PLATFORM_MHU=y CONFIG_BCM2835_MBOX=y -CONFIG_TI_MESSAGE_MANAGER=y CONFIG_QCOM_APCS_IPC=y CONFIG_ROCKCHIP_IOMMU=y CONFIG_TEGRA_IOMMU_SMMU=y @@ -696,9 +694,7 @@ CONFIG_ARCH_TEGRA_210_SOC=y CONFIG_ARCH_TEGRA_186_SOC=y CONFIG_ARCH_TEGRA_194_SOC=y CONFIG_ARCH_K3_AM6_SOC=y -CONFIG_SOC_TI=y CONFIG_TI_SCI_PM_DOMAINS=y -CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y CONFIG_EXTCON_USB_GPIO=y CONFIG_EXTCON_USBC_CROS_EC=y CONFIG_MEMORY=y
Generated via: $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig <enable CONFIG_RANDOMIZE_BASE aka KASLR> $ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make savedefconfig $ mv defconfig arch/arm64/configs/defconfig Removes explicit enablement of: CONFIG_TI_SCI_PROTOCOL CONFIG_TI_MESSAGE_MANAGER CONFIG_SOC_TI CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND Signed-off-by: Nick Desaulniers <ndesaulniers@google.com> --- arch/arm64/configs/defconfig | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) -- 2.22.0.410.gd8fdbe21b5-goog