| Message ID | 1400163385-19863-3-git-send-email-julien.grall@linaro.org |
|---|---|
| State | Superseded, archived |
| Headers | show |
On 15/05/14 15:16, Julien Grall wrote: > Grant mapping can be used for DMA request. The dev_bus_addr returned by the > hypercall is the MFN (not the IPA). Currently Linux is using this address (via > swiotlb) to program the DMA. > When the device is protected by IOMMU the request will fail. We have to > add 1:1 mapping in the domain p2m to allow DMA request working. > > This is valid because DOM0 has its memory mapped 1:1 and therefore we know > that RAM and devices cannot clash. I though a bit more about this patch. It doesn't handle the case where 2 grant use the same MFN, which is valid. In this case, we should remove the 1:1 mapping only when the last grant is removed. I'm not sure how to handle this case. Regards,
On Sat, 2014-05-17 at 19:12 +0100, Julien Grall wrote: > > On 15/05/14 15:16, Julien Grall wrote: > > Grant mapping can be used for DMA request. The dev_bus_addr returned by the > > hypercall is the MFN (not the IPA). Currently Linux is using this address (via > > swiotlb) to program the DMA. > > When the device is protected by IOMMU the request will fail. We have to > > add 1:1 mapping in the domain p2m to allow DMA request working. > > > > This is valid because DOM0 has its memory mapped 1:1 and therefore we know > > that RAM and devices cannot clash. > > I though a bit more about this patch. It doesn't handle the case where 2 > grant use the same MFN, which is valid. > > In this case, we should remove the 1:1 mapping only when the last grant > is removed. > > I'm not sure how to handle this case. Urk, yes indeed. This is all hypervisor side stuff I think, right? I wonder if the grant table maptrack stuff is helpful here, i.e. could it contain a reference count (if it doesn't already)? Some care and attention would be needed to make it safe, but it might be doable. Ian.
On 05/19/2014 10:45 AM, Ian Campbell wrote: > On Sat, 2014-05-17 at 19:12 +0100, Julien Grall wrote: >> >> On 15/05/14 15:16, Julien Grall wrote: >>> Grant mapping can be used for DMA request. The dev_bus_addr returned by the >>> hypercall is the MFN (not the IPA). Currently Linux is using this address (via >>> swiotlb) to program the DMA. >>> When the device is protected by IOMMU the request will fail. We have to >>> add 1:1 mapping in the domain p2m to allow DMA request working. >>> >>> This is valid because DOM0 has its memory mapped 1:1 and therefore we know >>> that RAM and devices cannot clash. >> >> I though a bit more about this patch. It doesn't handle the case where 2 >> grant use the same MFN, which is valid. >> >> In this case, we should remove the 1:1 mapping only when the last grant >> is removed. >> >> I'm not sure how to handle this case. > > Urk, yes indeed. > > This is all hypervisor side stuff I think, right? There is 2 issues (one in hypervisor side, the other in DOM0). The issue I'm talking here is in the hypervisor side. > I wonder if the grant table maptrack stuff is helpful here, i.e. could > it contain a reference count (if it doesn't already)? Some care and > attention would be needed to make it safe, but it might be doable. It seems that mapcount (common/grant_table.c), do the job for us. I will give a look on it. The other issue (from DOM0 side) comes from the swiotlb implementation for ARM (see arch/arm/xen/p2m.c). We are unable to handle multiple PFN linked to a same MFN. This case will be silently ignore and still work (???). This is happening with Freebsd on ARM when portsnap is executed for the first time. I'm not sure why. Regards,
On 19/05/14 16:20, Julien Grall wrote: > On 05/19/2014 10:45 AM, Ian Campbell wrote: >> On Sat, 2014-05-17 at 19:12 +0100, Julien Grall wrote: >>> >>> On 15/05/14 15:16, Julien Grall wrote: >>>> Grant mapping can be used for DMA request. The dev_bus_addr returned by the >>>> hypercall is the MFN (not the IPA). Currently Linux is using this address (via >>>> swiotlb) to program the DMA. >>>> When the device is protected by IOMMU the request will fail. We have to >>>> add 1:1 mapping in the domain p2m to allow DMA request working. >>>> >>>> This is valid because DOM0 has its memory mapped 1:1 and therefore we know >>>> that RAM and devices cannot clash. >>> >>> I though a bit more about this patch. It doesn't handle the case where 2 >>> grant use the same MFN, which is valid. >>> >>> In this case, we should remove the 1:1 mapping only when the last grant >>> is removed. >>> >>> I'm not sure how to handle this case. >> >> Urk, yes indeed. >> >> This is all hypervisor side stuff I think, right? > > There is 2 issues (one in hypervisor side, the other in DOM0). > > The issue I'm talking here is in the hypervisor side. > >> I wonder if the grant table maptrack stuff is helpful here, i.e. could >> it contain a reference count (if it doesn't already)? Some care and >> attention would be needed to make it safe, but it might be doable. > > It seems that mapcount (common/grant_table.c), do the job for us. I will > give a look on it. > > The other issue (from DOM0 side) comes from the swiotlb implementation > for ARM (see arch/arm/xen/p2m.c). We are unable to handle multiple PFN > linked to a same MFN. > > This case will be silently ignore and still work (???). This is > happening with Freebsd on ARM when portsnap is executed for the first > time. I'm not sure why. Not sure I'm following what's going on here, are you saying that FreeBSD blkfront uses the same grant ref on two (or more) requests at the same time? Roger.
On Tue, 2014-05-20 at 09:30 +0200, Roger Pau Monné wrote: > Not sure I'm following what's going on here, are you saying that FreeBSD > blkfront uses the same grant ref on two (or more) requests at the same time? My understanding was that there were two grant refs to the same mfn, which could happen if two threads were doing (direct?) IO to buffers which happened to lie on the same page, or with aio etc. Ian.
On 05/20/2014 09:51 AM, Ian Campbell wrote: > On Tue, 2014-05-20 at 09:30 +0200, Roger Pau Monné wrote: >> Not sure I'm following what's going on here, are you saying that FreeBSD >> blkfront uses the same grant ref on two (or more) requests at the same time? > > My understanding was that there were two grant refs to the same mfn, > which could happen if two threads were doing (direct?) IO to buffers > which happened to lie on the same page, or with aio etc. Sorry I wasn't clear on the use-case. I'm trying to run portsnap on a FreeBSD ARM guest. The first time I load the software, the guest is sending (in xbd_queue_cb) at the same time multiple grant with the same IPA (guest address). This will result to the same MFN from DOM0/Xen POV. I did a bit of debug, and this come from the same thread. I have virtual buffer where each page of this buffer pointed to the same physical address. FreeBSD is asking to write this buffer to the disk but from what I see it mostly contains garbage (full of 0xc2). I can't find where this comes and wondering if you have any thoughts. I was wondering if this things happen on FreeBSD x86. Regards,
On Tue, 2014-05-20 at 12:19 +0100, Julien Grall wrote: > FreeBSD is asking to write this buffer to the disk but from what I see > it mostly contains garbage (full of 0xc2). > > I can't find where this comes and wondering if you have any thoughts. 0xc2 is the value which Xen uses to scrub pages when debug=y (see xen/common/page_alloc.c:scrub_one_page). Ian.
On 05/20/2014 12:25 PM, Ian Campbell wrote: > On Tue, 2014-05-20 at 12:19 +0100, Julien Grall wrote: >> FreeBSD is asking to write this buffer to the disk but from what I see >> it mostly contains garbage (full of 0xc2). >> >> I can't find where this comes and wondering if you have any thoughts. > > 0xc2 is the value which Xen uses to scrub pages when debug=y (see > xen/common/page_alloc.c:scrub_one_page). I though about it, but I find strange that the guest is trying to write non-modified page to the disk. I will try to debug a bit more on it.
diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index eac228c..4ce3962 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -1199,6 +1199,7 @@ int create_grant_host_mapping(unsigned long addr, unsigned long frame, { int rc; p2m_type_t t = p2m_grant_map_rw; + struct domain *d = current->domain; if ( cache_flags || (flags & ~GNTMAP_readonly) != GNTMAP_host_map ) return GNTST_general_error; @@ -1206,13 +1207,32 @@ int create_grant_host_mapping(unsigned long addr, unsigned long frame, if ( flags & GNTMAP_readonly ) t = p2m_grant_map_ro; - rc = guest_physmap_add_entry(current->domain, addr >> PAGE_SHIFT, - frame, 0, t); + rc = guest_physmap_add_entry(d, addr >> PAGE_SHIFT, frame, 0, t); if ( rc ) - return GNTST_general_error; - else - return GNTST_okay; + goto gerror; + + /* Grant mapping can be used for DMA request. The dev_bus_addr returned by + * the hypercall is the MFN (not the IPA). For device protected by + * an IOMMU, Xen needs to add a 1:1 mapping in the domain p2m to + * allow DMA request working. + * This is only valid when the domain is directed mapped + */ + if ( is_domain_direct_mapped(d) && need_iommu(d) ) + { + rc = guest_physmap_add_entry(d, frame, frame, 0, t); + if ( rc ) + goto unmap; + } + + return GNTST_okay; + +unmap: + guest_physmap_remove_page(d, addr >> PAGE_SHIFT, frame, 0); + +gerror: + return GNTST_general_error; + } int replace_grant_host_mapping(unsigned long addr, unsigned long mfn, @@ -1226,6 +1246,9 @@ int replace_grant_host_mapping(unsigned long addr, unsigned long mfn, guest_physmap_remove_page(d, gfn, mfn, 0); + if ( is_domain_direct_mapped(d) && need_iommu(d) ) + guest_physmap_remove_page(d, mfn, mfn, 0); + return GNTST_okay; }