| Message ID | 20230613102905.2808371-3-usama.anjum@collabora.com |
|---|---|
| State | New |
| Headers | show |
| Series | Implement IOCTL to get and optionally clear info about PTEs | expand |
I'll send next revision now. On 6/14/23 11:00 PM, Michał Mirosław wrote: > (A quick reply to answer open questions in case they help the next version.) > > On Wed, 14 Jun 2023 at 19:10, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: >> On 6/14/23 8:14 PM, Michał Mirosław wrote: >>> On Wed, 14 Jun 2023 at 15:46, Muhammad Usama Anjum >>> <usama.anjum@collabora.com> wrote: >>>> >>>> On 6/14/23 3:36 AM, Michał Mirosław wrote: >>>>> On Tue, 13 Jun 2023 at 12:29, Muhammad Usama Anjum >>>>> <usama.anjum@collabora.com> wrote: > [...] >>>>>> + if (cur_buf->bitmap == bitmap && >>>>>> + cur_buf->start + cur_buf->len * PAGE_SIZE == addr) { >>>>>> + cur_buf->len += n_pages; >>>>>> + p->found_pages += n_pages; >>>>>> + } else { >>>>>> + if (cur_buf->len && p->vec_buf_index >= p->vec_buf_len) >>>>>> + return -ENOMEM; >>>>> >>>>> Shouldn't this be -ENOSPC? -ENOMEM usually signifies that the kernel >>>>> ran out of memory when allocating, not that there is no space in a >>>>> user-provided buffer. >>>> There are 3 kinds of return values here: >>>> * PM_SCAN_FOUND_MAX_PAGES (1) ---> max_pages have been found. Abort the >>>> page walk from next entry >>>> * 0 ---> continue the page walk >>>> * -ENOMEM --> Abort the page walk from current entry, user buffer is full >>>> which is not error, but only a stop signal. This -ENOMEM is just >>>> differentiater from (1). This -ENOMEM is for internal use and isn't >>>> returned to user. >>> >>> But why ENOSPC is not good here? I was used before, I think. >> -ENOSPC is being returned in form of true error from >> pagemap_scan_hugetlb_entry(). So I'd to remove -ENOSPC from here as it >> wasn't true error here, it was only a way to abort the walk immediately. >> I'm liking the following erturn code from here now: >> >> #define PM_SCAN_BUFFER_FULL (-256) > > I guess this will be reworked anyway, but I'd prefer this didn't need > custom errors etc. If we agree to decoupling the selection and GET > output, it could be: > > bool is_interesting_page(p, flags); // this one does the > required/anyof/excluded match > size_t output_range(p, start, len, flags); // this one fills the > output vector and returns how many pages were fit > > In this setup, `is_interesting_page() && (n_out = output_range()) < > n_pages` means this is the final range, no more will fit. And if > `n_out == 0` then no pages fit and no WP is needed (no other special > cases). Right now, pagemap_scan_output() performs the work of both of these two functions. The part can be broken into is_interesting_pages() and we can leave the remaining part as it is. Saying that n_out < n_pages tells us the buffer is full covers one case. But there is case of maximum pages have been found and walk needs to be aborted. I'll just add is_interesting_page() in next version. > >>>>> For flags name: PM_REQUIRE_WRITE_ACCESS? >>>>> Or Is it intended to be checked only if doing WP (as the current name >>>>> suggests) and so it would be redundant as WP currently requires >>>>> `p->required_mask = PAGE_IS_WRITTEN`? >>>> This is intended to indicate that if userfaultfd is needed. If >>>> PAGE_IS_WRITTEN is mentioned in any of mask, we need to check if >>>> userfaultfd has been initialized for this memory. I'll rename to >>>> PM_SCAN_REQUIRE_UFFD. >>> >>> Why do we need that check? Wouldn't `is_written = false` work for vmas >>> not registered via uffd? >> UFFD_FEATURE_WP_ASYNC and UNPOPULATED needs to be set on the memory region >> for it to report correct written values on the memory region. Without UFFD >> WP ASYNC and UNPOUPULATED defined on the memory, we consider UFFD_WP state >> undefined. If user hasn't initialized memory with UFFD, he has no right to >> set is_written = false. > > How about calculating `is_written = is_uffd_registered() && > is_uffd_wp()`? This would enable a user to apply GET+WP for the whole > address space of a process regardless of whether all of it is > registered. I wouldn't want to check if uffd is registered again and again. This is why we are doing it only once every walk in pagemap_scan_test_walk(). > >>> While here, I wonder if we really need to fail the call if there are >>> unknown bits in those masks set: if this bit set is expanded with >>> another category flags, a newer userspace run on older kernel would >>> get EINVAL even if the "treat unknown as 0" be what it requires. >>> There is no simple way in the API to discover what bits the kernel >>> supports. We could allow a no-op (no WP nor GET) call to help with >>> that and then rejecting unknown bits would make sense. >> I've not seen any examples of this. But I've seen examples of returning >> error if kernel doesn't support a feature. Each new feature comes with a >> kernel version, greater than this version support this feature. If user is >> trying to use advanced feature which isn't present in a kernel, we should >> return error and not proceed to confuse the user/kernel. In fact if we look >> at userfaultfd_api(), we return error immediately if feature has some bit >> set which kernel doesn't support. > > I think we should have a way of detecting the supported flags if we > don't want a forward compatibility policy for flags here. Maybe it > would be enough to allow all the no-op combinations for this purpose? Again I don't think UFFD is doing anything like this. > >>>>> [...] >>>>>> --- a/include/uapi/linux/fs.h >>>>>> +++ b/include/uapi/linux/fs.h >>>>>> +/* >>>>>> + * struct page_region - Page region with bitmap flags >>>>>> + * @start: Start of the region >>>>>> + * @len: Length of the region in pages >>>>>> + * bitmap: Bits sets for the region >>>>> >>>>> '@' is missing for the third field. BTW, maybe we can call it >>>>> something like `flags` or `category` (something that hints at the >>>>> meaning of the value instead of its data representation). >>>> The deification of this struct says, "with bitmap flags". Bitmap was a >>>> different name. I'll update it to flags. >>> >>> From the implementation and our discussions I guess the >>> `bitmap`/`flags` field is holding a set of matching categories: a bit >>> value 1 = pages are in this category, value 0 = pages are not in this >>> category. >>> >>>>>> +/* >>>>>> + * struct pm_scan_arg - Pagemap ioctl argument >>>>>> + * @size: Size of the structure >>>>>> + * @flags: Flags for the IOCTL >>>>>> + * @start: Starting address of the region >>>>>> + * @len: Length of the region (All the pages in this length are included) >>>>> >>>>> Maybe `scan_start`, `scan_len` - so that there is a better distinction >>>>> from the structure's `size` field? >>>> As start and len already communicate the meaning. We are making things more >>>> verbose. >>> >>> We are describing (in the name) only that it is a range, but not of >>> what or what purpose. That information is only in the docstring, but >>> it is harder to get by someone just reading the code. >> Agreed. But I'm using same names, start and len which mincore (a historic >> syscall) is using. I've followed mincore here. > > mincore() doesn't take parameters as a struct, but as three positional > arguments (whose names don't matter nor appear at call point) - I > wouldn't take it as a precedent for structure field naming. > >>>>>> + * @vec: Address of page_region struct array for output >>>>>> + * @vec_len: Length of the page_region struct array >>>>>> + * @max_pages: Optional max return pages >>>>>> + * @required_mask: Required mask - All of these bits have to be set in the PTE >>>>>> + * @anyof_mask: Any mask - Any of these bits are set in the PTE >>>>>> + * @excluded_mask: Exclude mask - None of these bits are set in the PTE >>>>>> + * @return_mask: Bits that are to be reported in page_region >>>>>> + */ >>>>> >>>>> I skipped most of the page walk implementation as maybe the comments >>>>> above could make it simpler. Reading this patch and the documentation >>>>> I still feel confused about how the filtering/limiting parameters >>>> I'm really sad to hear this. I've been working on making this series from >>>> so many revisions. I was hopping that it would make complete sense to >>>> reviewers and later to users. >>>> >>>> What do you think is missing which is restricting these patches getting >>>> accepted to upstream? >>>> >>>>> should affect GET, WP and WP+GET. Should they limit the pages walked >>>>> (and WP-ed)? Or only the GET's output? How about GET+WP case? >>>> The address range needs to be walked until max pages pages are found, user >>>> buffer is full or whole range is walked. If the page will be added to user >>>> buffer or not depends on the selection criteria (*masks). There is no >>>> difference in case of walk for GET, WP and GET+WP. Only that WP doesn't >>>> take any user buffer and just WPs the whole region. >>> >>> Ok, then this intent (if I understand correctly) does not entirely >>> match the implementation. Let's split up the conditions: >>> >>> 1. The address range needs to be walked until max pages pages are found >>> >>> current implementation: the address range is walked until max pages >>> matching masks (incl. return_mask) are reported by GET (or until end >>> of range if GET is not requested). >>> Maybe we need to describe what "found" means here? >> Found means all the pages which are found to be fulfilling the masks and we >> have added it to the user buffer. I can add the comment on top of >> pagemap_scan_private struct? But I don't think that it is difficult to >> understand the meaning of found_pages and also we compare it with max_pages >> which makes things very easy to understand. > > After fixing `return_mask` and the selection/action split I think > "pages found" might work - as now the count will be exactly what pages > match the required/anyof/excluded criteria. > >>> 2. user buffer is full >>> Matches implementation except in GET+WP edge cases. >> I'm not sure which edge case you are referring to? Probably for hugetlb >> error return case? > > Yes, that one. > > Best Regards > Michał Mirosław
On Thu, 15 Jun 2023 at 16:52, Michał Mirosław <emmir@google.com> wrote: > On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: > > I'll send next revision now. > > On 6/14/23 11:00 PM, Michał Mirosław wrote: > > > (A quick reply to answer open questions in case they help the next version.) > > > > > > On Wed, 14 Jun 2023 at 19:10, Muhammad Usama Anjum > > > <usama.anjum@collabora.com> wrote: > > >> On 6/14/23 8:14 PM, Michał Mirosław wrote: > > >>> On Wed, 14 Jun 2023 at 15:46, Muhammad Usama Anjum > > >>> <usama.anjum@collabora.com> wrote: > > >>>> > > >>>> On 6/14/23 3:36 AM, Michał Mirosław wrote: > > >>>>> On Tue, 13 Jun 2023 at 12:29, Muhammad Usama Anjum > > >>>>> <usama.anjum@collabora.com> wrote: > > >>>>> For flags name: PM_REQUIRE_WRITE_ACCESS? > > >>>>> Or Is it intended to be checked only if doing WP (as the current name > > >>>>> suggests) and so it would be redundant as WP currently requires > > >>>>> `p->required_mask = PAGE_IS_WRITTEN`? > > >>>> This is intended to indicate that if userfaultfd is needed. If > > >>>> PAGE_IS_WRITTEN is mentioned in any of mask, we need to check if > > >>>> userfaultfd has been initialized for this memory. I'll rename to > > >>>> PM_SCAN_REQUIRE_UFFD. > > >>> > > >>> Why do we need that check? Wouldn't `is_written = false` work for vmas > > >>> not registered via uffd? > > >> UFFD_FEATURE_WP_ASYNC and UNPOPULATED needs to be set on the memory region > > >> for it to report correct written values on the memory region. Without UFFD > > >> WP ASYNC and UNPOUPULATED defined on the memory, we consider UFFD_WP state > > >> undefined. If user hasn't initialized memory with UFFD, he has no right to > > >> set is_written = false. > > > > > > How about calculating `is_written = is_uffd_registered() && > > > is_uffd_wp()`? This would enable a user to apply GET+WP for the whole > > > address space of a process regardless of whether all of it is > > > registered. > > I wouldn't want to check if uffd is registered again and again. This is why > > we are doing it only once every walk in pagemap_scan_test_walk(). > > There is no need to do the checks repeatedly. If I understand the code > correctly, uffd registration is per-vma, so it can be communicated > from test_walk to entry/hole callbacks via a field in > pagemap_scan_private. Actually... this could be exposed as a page category for the filter (e.g. PAGE_USES_UFFD_WP) and then you could just make the ioctl() to work for your usecase without tracking the ranges at the userspace side. Best Regards Michał Mirosław
Please review the v19. I hope to get your reviewed by tag soon. On 6/15/23 7:58 PM, Michał Mirosław wrote: > On Thu, 15 Jun 2023 at 16:52, Michał Mirosław <emmir@google.com> wrote: >> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum >> <usama.anjum@collabora.com> wrote: >>> I'll send next revision now. >>> On 6/14/23 11:00 PM, Michał Mirosław wrote: >>>> (A quick reply to answer open questions in case they help the next version.) >>>> >>>> On Wed, 14 Jun 2023 at 19:10, Muhammad Usama Anjum >>>> <usama.anjum@collabora.com> wrote: >>>>> On 6/14/23 8:14 PM, Michał Mirosław wrote: >>>>>> On Wed, 14 Jun 2023 at 15:46, Muhammad Usama Anjum >>>>>> <usama.anjum@collabora.com> wrote: >>>>>>> >>>>>>> On 6/14/23 3:36 AM, Michał Mirosław wrote: >>>>>>>> On Tue, 13 Jun 2023 at 12:29, Muhammad Usama Anjum >>>>>>>> <usama.anjum@collabora.com> wrote: >>>>>>>> For flags name: PM_REQUIRE_WRITE_ACCESS? >>>>>>>> Or Is it intended to be checked only if doing WP (as the current name >>>>>>>> suggests) and so it would be redundant as WP currently requires >>>>>>>> `p->required_mask = PAGE_IS_WRITTEN`? >>>>>>> This is intended to indicate that if userfaultfd is needed. If >>>>>>> PAGE_IS_WRITTEN is mentioned in any of mask, we need to check if >>>>>>> userfaultfd has been initialized for this memory. I'll rename to >>>>>>> PM_SCAN_REQUIRE_UFFD. >>>>>> >>>>>> Why do we need that check? Wouldn't `is_written = false` work for vmas >>>>>> not registered via uffd? >>>>> UFFD_FEATURE_WP_ASYNC and UNPOPULATED needs to be set on the memory region >>>>> for it to report correct written values on the memory region. Without UFFD >>>>> WP ASYNC and UNPOUPULATED defined on the memory, we consider UFFD_WP state >>>>> undefined. If user hasn't initialized memory with UFFD, he has no right to >>>>> set is_written = false. >>>> >>>> How about calculating `is_written = is_uffd_registered() && >>>> is_uffd_wp()`? This would enable a user to apply GET+WP for the whole >>>> address space of a process regardless of whether all of it is >>>> registered. >>> I wouldn't want to check if uffd is registered again and again. This is why >>> we are doing it only once every walk in pagemap_scan_test_walk(). >> >> There is no need to do the checks repeatedly. If I understand the code >> correctly, uffd registration is per-vma, so it can be communicated >> from test_walk to entry/hole callbacks via a field in >> pagemap_scan_private. > > Actually... this could be exposed as a page category for the filter > (e.g. PAGE_USES_UFFD_WP) and then you could just make the ioctl() to > work for your usecase without tracking the ranges at the userspace > side. I'm not sure about page category. ASAIK the current check isn't bad when we already mention in documentation that memory must be registered with UFFD WP before using write feature of the IOCTL. Just like mincore mentions in documentation that user buffer will be filled with values based on the length of the region. Kernel doesn't care if user had provided smaller buffer and kernel overwrites because of user's own issue. I want to follow the same path. If user doesn't read documentation and follow it, he should be punished with the error. > > Best Regards > Michał Mirosław
On Thu, 15 Jun 2023 at 17:16, Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > > Please review the v19. I hope to get your reviewed by tag soon. > > On 6/15/23 7:58 PM, Michał Mirosław wrote: > > On Thu, 15 Jun 2023 at 16:52, Michał Mirosław <emmir@google.com> wrote: > >> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum > >> <usama.anjum@collabora.com> wrote: > >>> I'll send next revision now. > >>> On 6/14/23 11:00 PM, Michał Mirosław wrote: > >>>> (A quick reply to answer open questions in case they help the next version.) > >>>> > >>>> On Wed, 14 Jun 2023 at 19:10, Muhammad Usama Anjum > >>>> <usama.anjum@collabora.com> wrote: > >>>>> On 6/14/23 8:14 PM, Michał Mirosław wrote: > >>>>>> On Wed, 14 Jun 2023 at 15:46, Muhammad Usama Anjum > >>>>>> <usama.anjum@collabora.com> wrote: > >>>>>>> > >>>>>>> On 6/14/23 3:36 AM, Michał Mirosław wrote: > >>>>>>>> On Tue, 13 Jun 2023 at 12:29, Muhammad Usama Anjum > >>>>>>>> <usama.anjum@collabora.com> wrote: > >>>>>>>> For flags name: PM_REQUIRE_WRITE_ACCESS? > >>>>>>>> Or Is it intended to be checked only if doing WP (as the current name > >>>>>>>> suggests) and so it would be redundant as WP currently requires > >>>>>>>> `p->required_mask = PAGE_IS_WRITTEN`? > >>>>>>> This is intended to indicate that if userfaultfd is needed. If > >>>>>>> PAGE_IS_WRITTEN is mentioned in any of mask, we need to check if > >>>>>>> userfaultfd has been initialized for this memory. I'll rename to > >>>>>>> PM_SCAN_REQUIRE_UFFD. > >>>>>> > >>>>>> Why do we need that check? Wouldn't `is_written = false` work for vmas > >>>>>> not registered via uffd? > >>>>> UFFD_FEATURE_WP_ASYNC and UNPOPULATED needs to be set on the memory region > >>>>> for it to report correct written values on the memory region. Without UFFD > >>>>> WP ASYNC and UNPOUPULATED defined on the memory, we consider UFFD_WP state > >>>>> undefined. If user hasn't initialized memory with UFFD, he has no right to > >>>>> set is_written = false. > >>>> > >>>> How about calculating `is_written = is_uffd_registered() && > >>>> is_uffd_wp()`? This would enable a user to apply GET+WP for the whole > >>>> address space of a process regardless of whether all of it is > >>>> registered. > >>> I wouldn't want to check if uffd is registered again and again. This is why > >>> we are doing it only once every walk in pagemap_scan_test_walk(). > >> > >> There is no need to do the checks repeatedly. If I understand the code > >> correctly, uffd registration is per-vma, so it can be communicated > >> from test_walk to entry/hole callbacks via a field in > >> pagemap_scan_private. > > > > Actually... this could be exposed as a page category for the filter > > (e.g. PAGE_USES_UFFD_WP) and then you could just make the ioctl() to > > work for your usecase without tracking the ranges at the userspace > > side. > I'm not sure about page category. ASAIK the current check isn't bad when we > already mention in documentation that memory must be registered with UFFD > WP before using write feature of the IOCTL. You could relax the (documentation) rule to be "WP works only on ranges registeder via UFFD for ASYNC_WP". That way you allow people, who don't read documentation to shoot their foot, but don't block people that know what they are doing from exploiting the nice feature that they don't need to track all the WP-registered ranges calling the ioctl() for each one and instead can just call it once for the whole address space. Best Regards Michał Mirosław
On 6/16/23 1:00 AM, Michał Mirosław wrote: > On Thu, 15 Jun 2023 at 17:16, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: >> >> Please review the v19. I hope to get your reviewed by tag soon. >> >> On 6/15/23 7:58 PM, Michał Mirosław wrote: >>> On Thu, 15 Jun 2023 at 16:52, Michał Mirosław <emmir@google.com> wrote: >>>> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum >>>> <usama.anjum@collabora.com> wrote: >>>>> I'll send next revision now. >>>>> On 6/14/23 11:00 PM, Michał Mirosław wrote: >>>>>> (A quick reply to answer open questions in case they help the next version.) >>>>>> >>>>>> On Wed, 14 Jun 2023 at 19:10, Muhammad Usama Anjum >>>>>> <usama.anjum@collabora.com> wrote: >>>>>>> On 6/14/23 8:14 PM, Michał Mirosław wrote: >>>>>>>> On Wed, 14 Jun 2023 at 15:46, Muhammad Usama Anjum >>>>>>>> <usama.anjum@collabora.com> wrote: >>>>>>>>> >>>>>>>>> On 6/14/23 3:36 AM, Michał Mirosław wrote: >>>>>>>>>> On Tue, 13 Jun 2023 at 12:29, Muhammad Usama Anjum >>>>>>>>>> <usama.anjum@collabora.com> wrote: >>>>>>>>>> For flags name: PM_REQUIRE_WRITE_ACCESS? >>>>>>>>>> Or Is it intended to be checked only if doing WP (as the current name >>>>>>>>>> suggests) and so it would be redundant as WP currently requires >>>>>>>>>> `p->required_mask = PAGE_IS_WRITTEN`? >>>>>>>>> This is intended to indicate that if userfaultfd is needed. If >>>>>>>>> PAGE_IS_WRITTEN is mentioned in any of mask, we need to check if >>>>>>>>> userfaultfd has been initialized for this memory. I'll rename to >>>>>>>>> PM_SCAN_REQUIRE_UFFD. >>>>>>>> >>>>>>>> Why do we need that check? Wouldn't `is_written = false` work for vmas >>>>>>>> not registered via uffd? >>>>>>> UFFD_FEATURE_WP_ASYNC and UNPOPULATED needs to be set on the memory region >>>>>>> for it to report correct written values on the memory region. Without UFFD >>>>>>> WP ASYNC and UNPOUPULATED defined on the memory, we consider UFFD_WP state >>>>>>> undefined. If user hasn't initialized memory with UFFD, he has no right to >>>>>>> set is_written = false. >>>>>> >>>>>> How about calculating `is_written = is_uffd_registered() && >>>>>> is_uffd_wp()`? This would enable a user to apply GET+WP for the whole >>>>>> address space of a process regardless of whether all of it is >>>>>> registered. >>>>> I wouldn't want to check if uffd is registered again and again. This is why >>>>> we are doing it only once every walk in pagemap_scan_test_walk(). >>>> >>>> There is no need to do the checks repeatedly. If I understand the code >>>> correctly, uffd registration is per-vma, so it can be communicated >>>> from test_walk to entry/hole callbacks via a field in >>>> pagemap_scan_private. >>> >>> Actually... this could be exposed as a page category for the filter >>> (e.g. PAGE_USES_UFFD_WP) and then you could just make the ioctl() to >>> work for your usecase without tracking the ranges at the userspace >>> side. >> I'm not sure about page category. ASAIK the current check isn't bad when we >> already mention in documentation that memory must be registered with UFFD >> WP before using write feature of the IOCTL. > > You could relax the (documentation) rule to be "WP works only on > ranges registeder via UFFD for ASYNC_WP". That way you allow people, > who don't read documentation to shoot their foot, They'll shoot their foot and have no idea why they are getting false results. Isn't it better that they get error and they go read the documentation and then register with UFFD first? I think, returning error is way better than not returning anything. > but don't block > people that know what they are doing from exploiting the nice feature > that they don't need to track all the WP-registered ranges calling the > ioctl() for each one and instead can just call it once for the whole > address space. > > Best Regards > Michał Mirosław
On 6/16/23 1:07 AM, Michał Mirosław wrote: > On Thu, 15 Jun 2023 at 17:11, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: >> On 6/15/23 7:52 PM, Michał Mirosław wrote: >>> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum >>> <usama.anjum@collabora.com> wrote: >>>> I'll send next revision now. >>>> On 6/14/23 11:00 PM, Michał Mirosław wrote: >>>>> (A quick reply to answer open questions in case they help the next version.) >>>>> >>>>> On Wed, 14 Jun 2023 at 19:10, Muhammad Usama Anjum >>>>> <usama.anjum@collabora.com> wrote: >>>>>> On 6/14/23 8:14 PM, Michał Mirosław wrote: >>>>>>> On Wed, 14 Jun 2023 at 15:46, Muhammad Usama Anjum >>>>>>> <usama.anjum@collabora.com> wrote: >>>>>>>> >>>>>>>> On 6/14/23 3:36 AM, Michał Mirosław wrote: >>>>>>>>> On Tue, 13 Jun 2023 at 12:29, Muhammad Usama Anjum >>>>>>>>> <usama.anjum@collabora.com> wrote: >>>>> [...] >>>>>>>>>> + if (cur_buf->bitmap == bitmap && >>>>>>>>>> + cur_buf->start + cur_buf->len * PAGE_SIZE == addr) { >>>>>>>>>> + cur_buf->len += n_pages; >>>>>>>>>> + p->found_pages += n_pages; >>>>>>>>>> + } else { >>>>>>>>>> + if (cur_buf->len && p->vec_buf_index >= p->vec_buf_len) >>>>>>>>>> + return -ENOMEM; >>>>>>>>> >>>>>>>>> Shouldn't this be -ENOSPC? -ENOMEM usually signifies that the kernel >>>>>>>>> ran out of memory when allocating, not that there is no space in a >>>>>>>>> user-provided buffer. >>>>>>>> There are 3 kinds of return values here: >>>>>>>> * PM_SCAN_FOUND_MAX_PAGES (1) ---> max_pages have been found. Abort the >>>>>>>> page walk from next entry >>>>>>>> * 0 ---> continue the page walk >>>>>>>> * -ENOMEM --> Abort the page walk from current entry, user buffer is full >>>>>>>> which is not error, but only a stop signal. This -ENOMEM is just >>>>>>>> differentiater from (1). This -ENOMEM is for internal use and isn't >>>>>>>> returned to user. >>>>>>> >>>>>>> But why ENOSPC is not good here? I was used before, I think. >>>>>> -ENOSPC is being returned in form of true error from >>>>>> pagemap_scan_hugetlb_entry(). So I'd to remove -ENOSPC from here as it >>>>>> wasn't true error here, it was only a way to abort the walk immediately. >>>>>> I'm liking the following erturn code from here now: >>>>>> >>>>>> #define PM_SCAN_BUFFER_FULL (-256) >>>>> >>>>> I guess this will be reworked anyway, but I'd prefer this didn't need >>>>> custom errors etc. If we agree to decoupling the selection and GET >>>>> output, it could be: >>>>> >>>>> bool is_interesting_page(p, flags); // this one does the >>>>> required/anyof/excluded match >>>>> size_t output_range(p, start, len, flags); // this one fills the >>>>> output vector and returns how many pages were fit >>>>> >>>>> In this setup, `is_interesting_page() && (n_out = output_range()) < >>>>> n_pages` means this is the final range, no more will fit. And if >>>>> `n_out == 0` then no pages fit and no WP is needed (no other special >>>>> cases). >>>> Right now, pagemap_scan_output() performs the work of both of these two >>>> functions. The part can be broken into is_interesting_pages() and we can >>>> leave the remaining part as it is. >>>> >>>> Saying that n_out < n_pages tells us the buffer is full covers one case. >>>> But there is case of maximum pages have been found and walk needs to be >>>> aborted. >>> >>> This case is exactly what `n_out < n_pages` will cover (if scan_output >>> uses max_pages properly to limit n_out). >>> Isn't it that when the buffer is full we want to abort the scan always >>> (with WP if `n_out > 0`)? >> Wouldn't it be duplication of condition if buffer is full inside >> pagemap_scan_output() and just outside it. Inside pagemap_scan_output() we >> check if we have space before putting data inside it. I'm using this same >> condition to indicate that buffer is full. > > I'm not sure what do you mean? The buffer-full conditions would be > checked in ..scan_output() and communicated to the caller by returning > N less than `n_pages` passed in. This is exactly how e.g. read() > works: if you get less than requested you've hit the end of the file. > If the file happens to have size that is equal to the provided buffer > length, the next read() will return 0. Right now we have: pagemap_scan_output(): if (p->vec_buf_index >= p->vec_buf_len) return PM_SCAN_BUFFER_FULL; if (p->found_pages == p->max_pages) return PM_SCAN_FOUND_MAX_PAGES; pagemap_scan_pmd_entry(): ret = pagemap_scan_output(bitmap, p, start, n_pages); if (ret >= 0) // success make_UFFD_WP and flush else buffer_error You are asking me to do: pagemap_scan_output(): if (p->vec_buf_index >= p->vec_buf_len) return 0; if (p->found_pages == p->max_pages) return PM_SCAN_FOUND_MAX_PAGES; pagemap_scan_pmd_entry(): ret = pagemap_scan_output(bitmap, p, start, n_pages); if (ret > 0) // success make_UFFD_WP and flush else if (ret == 0) // buffer full return PM_SCAN_BUFFER_FULL; else //other errors buffer_error So you are asking me to go from consie code to write more lines of code. I would write more lines without any issue if it improves readability and logical sense. But I don't see here any benefit. > >>>>>>> While here, I wonder if we really need to fail the call if there are >>>>>>> unknown bits in those masks set: if this bit set is expanded with >>>>>>> another category flags, a newer userspace run on older kernel would >>>>>>> get EINVAL even if the "treat unknown as 0" be what it requires. >>>>>>> There is no simple way in the API to discover what bits the kernel >>>>>>> supports. We could allow a no-op (no WP nor GET) call to help with >>>>>>> that and then rejecting unknown bits would make sense. >>>>>> I've not seen any examples of this. But I've seen examples of returning >>>>>> error if kernel doesn't support a feature. Each new feature comes with a >>>>>> kernel version, greater than this version support this feature. If user is >>>>>> trying to use advanced feature which isn't present in a kernel, we should >>>>>> return error and not proceed to confuse the user/kernel. In fact if we look >>>>>> at userfaultfd_api(), we return error immediately if feature has some bit >>>>>> set which kernel doesn't support. >>>>> >>>>> I think we should have a way of detecting the supported flags if we >>>>> don't want a forward compatibility policy for flags here. Maybe it >>>>> would be enough to allow all the no-op combinations for this purpose? >>>> Again I don't think UFFD is doing anything like this. >>> >>> If it's cheap and easy to provide a user with a way to detect the >>> supported features - why not do it? >> I'm sorry. But it would bring up something new and iterations will be >> needed to just play around. I like the UFFD way. > > Let's then first agree on what would have to be changed. I guess we > could leverage that `scan_len = 0` doesn't make much sense otherwise > and let it be used to check the other fields for support. We are making things more and more complex. I don't like multi-plexing variables. Can you give examples where multi-plexing has been done on variables inside linux kernel? Muti-plexing means user gives input and takes output from same variable. It makes variable double meaning. > > Best Regards > Michał Mirosław
On Fri, 16 Jun 2023 at 08:57, Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > > On 6/16/23 1:07 AM, Michał Mirosław wrote: > > On Thu, 15 Jun 2023 at 17:11, Muhammad Usama Anjum > > <usama.anjum@collabora.com> wrote: > >> On 6/15/23 7:52 PM, Michał Mirosław wrote: > >>> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum > >>> <usama.anjum@collabora.com> wrote: > >>>> I'll send next revision now. > >>>> On 6/14/23 11:00 PM, Michał Mirosław wrote: > >>>>> (A quick reply to answer open questions in case they help the next version.) [...] > >>>>> I guess this will be reworked anyway, but I'd prefer this didn't need > >>>>> custom errors etc. If we agree to decoupling the selection and GET > >>>>> output, it could be: > >>>>> > >>>>> bool is_interesting_page(p, flags); // this one does the > >>>>> required/anyof/excluded match > >>>>> size_t output_range(p, start, len, flags); // this one fills the > >>>>> output vector and returns how many pages were fit > >>>>> > >>>>> In this setup, `is_interesting_page() && (n_out = output_range()) < > >>>>> n_pages` means this is the final range, no more will fit. And if > >>>>> `n_out == 0` then no pages fit and no WP is needed (no other special > >>>>> cases). > >>>> Right now, pagemap_scan_output() performs the work of both of these two > >>>> functions. The part can be broken into is_interesting_pages() and we can > >>>> leave the remaining part as it is. > >>>> > >>>> Saying that n_out < n_pages tells us the buffer is full covers one case. > >>>> But there is case of maximum pages have been found and walk needs to be > >>>> aborted. > >>> > >>> This case is exactly what `n_out < n_pages` will cover (if scan_output > >>> uses max_pages properly to limit n_out). > >>> Isn't it that when the buffer is full we want to abort the scan always > >>> (with WP if `n_out > 0`)? > >> Wouldn't it be duplication of condition if buffer is full inside > >> pagemap_scan_output() and just outside it. Inside pagemap_scan_output() we > >> check if we have space before putting data inside it. I'm using this same > >> condition to indicate that buffer is full. > > > > I'm not sure what do you mean? The buffer-full conditions would be > > checked in ..scan_output() and communicated to the caller by returning > > N less than `n_pages` passed in. This is exactly how e.g. read() > > works: if you get less than requested you've hit the end of the file. > > If the file happens to have size that is equal to the provided buffer > > length, the next read() will return 0. > Right now we have: > > pagemap_scan_output(): > if (p->vec_buf_index >= p->vec_buf_len) > return PM_SCAN_BUFFER_FULL; > if (p->found_pages == p->max_pages) > return PM_SCAN_FOUND_MAX_PAGES; Why do you need to differentiate between those cases? > pagemap_scan_pmd_entry(): > ret = pagemap_scan_output(bitmap, p, start, n_pages); > if (ret >= 0) // success > make_UFFD_WP and flush > else > buffer_error > > You are asking me to do: > > pagemap_scan_output(): > if (p->vec_buf_index >= p->vec_buf_len) > return 0; > if (p->found_pages == p->max_pages) > return PM_SCAN_FOUND_MAX_PAGES; This should be instead: n_pages = min(p->max_pags - p_found_pages, n_pages) ... return n_pages; > pagemap_scan_pmd_entry(): > ret = pagemap_scan_output(bitmap, p, start, n_pages); > if (ret > 0) // success > make_UFFD_WP and flush > else if (ret == 0) // buffer full > return PM_SCAN_BUFFER_FULL; > else //other errors > buffer_error And this would be: if (ret > 0 && WP) WP + flush if (ret < n_pages) return -ENOSPC; > So you are asking me to go from consie code to write more lines of code. I > would write more lines without any issue if it improves readability and > logical sense. But I don't see here any benefit. Please see the clarifications above. Best Regards Michał Mirosław
On 6/19/23 1:16 PM, Michał Mirosław wrote: > On Fri, 16 Jun 2023 at 08:57, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: >> >> On 6/16/23 1:07 AM, Michał Mirosław wrote: >>> On Thu, 15 Jun 2023 at 17:11, Muhammad Usama Anjum >>> <usama.anjum@collabora.com> wrote: >>>> On 6/15/23 7:52 PM, Michał Mirosław wrote: >>>>> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum >>>>> <usama.anjum@collabora.com> wrote: >>>>>> I'll send next revision now. >>>>>> On 6/14/23 11:00 PM, Michał Mirosław wrote: >>>>>>> (A quick reply to answer open questions in case they help the next version.) > [...] >>>>>>> I guess this will be reworked anyway, but I'd prefer this didn't need >>>>>>> custom errors etc. If we agree to decoupling the selection and GET >>>>>>> output, it could be: >>>>>>> >>>>>>> bool is_interesting_page(p, flags); // this one does the >>>>>>> required/anyof/excluded match >>>>>>> size_t output_range(p, start, len, flags); // this one fills the >>>>>>> output vector and returns how many pages were fit >>>>>>> >>>>>>> In this setup, `is_interesting_page() && (n_out = output_range()) < >>>>>>> n_pages` means this is the final range, no more will fit. And if >>>>>>> `n_out == 0` then no pages fit and no WP is needed (no other special >>>>>>> cases). >>>>>> Right now, pagemap_scan_output() performs the work of both of these two >>>>>> functions. The part can be broken into is_interesting_pages() and we can >>>>>> leave the remaining part as it is. >>>>>> >>>>>> Saying that n_out < n_pages tells us the buffer is full covers one case. >>>>>> But there is case of maximum pages have been found and walk needs to be >>>>>> aborted. >>>>> >>>>> This case is exactly what `n_out < n_pages` will cover (if scan_output >>>>> uses max_pages properly to limit n_out). >>>>> Isn't it that when the buffer is full we want to abort the scan always >>>>> (with WP if `n_out > 0`)? >>>> Wouldn't it be duplication of condition if buffer is full inside >>>> pagemap_scan_output() and just outside it. Inside pagemap_scan_output() we >>>> check if we have space before putting data inside it. I'm using this same >>>> condition to indicate that buffer is full. >>> >>> I'm not sure what do you mean? The buffer-full conditions would be >>> checked in ..scan_output() and communicated to the caller by returning >>> N less than `n_pages` passed in. This is exactly how e.g. read() >>> works: if you get less than requested you've hit the end of the file. >>> If the file happens to have size that is equal to the provided buffer >>> length, the next read() will return 0. >> Right now we have: >> >> pagemap_scan_output(): >> if (p->vec_buf_index >= p->vec_buf_len) >> return PM_SCAN_BUFFER_FULL; >> if (p->found_pages == p->max_pages) >> return PM_SCAN_FOUND_MAX_PAGES; > > Why do you need to differentiate between those cases? > >> pagemap_scan_pmd_entry(): >> ret = pagemap_scan_output(bitmap, p, start, n_pages); >> if (ret >= 0) // success >> make_UFFD_WP and flush >> else >> buffer_error >> >> You are asking me to do: >> >> pagemap_scan_output(): >> if (p->vec_buf_index >= p->vec_buf_len) >> return 0; > >> if (p->found_pages == p->max_pages) >> return PM_SCAN_FOUND_MAX_PAGES; > > This should be instead: > > n_pages = min(p->max_pags - p_found_pages, n_pages) > ... > return n_pages; You are missing the optimization here that we check for full buffer every time adding to user buffer. This was added to remove extra iteration of page walk if buffer is full already. The way you are suggesting will remove it. So you are returning remaining pages to be found now. This doesn't seem right. If max_pages is 520, found_pages is 0 and n_pages is 512 before calling pagemap_scan_output(). found_pages would become 512 after adding 512 pages to output buffer. But n_pages would return 8 instead of 512. You were saying we should return the number of pages added to the output buffer. > >> pagemap_scan_pmd_entry(): >> ret = pagemap_scan_output(bitmap, p, start, n_pages); >> if (ret > 0) // success >> make_UFFD_WP and flush >> else if (ret == 0) // buffer full >> return PM_SCAN_BUFFER_FULL; >> else //other errors >> buffer_error > > And this would be: > > if (ret > 0 && WP) > WP + flush > > if (ret < n_pages) > return -ENOSPC; > >> So you are asking me to go from consie code to write more lines of code. I >> would write more lines without any issue if it improves readability and >> logical sense. But I don't see here any benefit. > > Please see the clarifications above. > > Best Regards > Michał Mirosław
On Tue, 20 Jun 2023 at 13:16, Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > On 6/19/23 1:16 PM, Michał Mirosław wrote: > > On Fri, 16 Jun 2023 at 08:57, Muhammad Usama Anjum > > <usama.anjum@collabora.com> wrote: > >> > >> On 6/16/23 1:07 AM, Michał Mirosław wrote: > >>> On Thu, 15 Jun 2023 at 17:11, Muhammad Usama Anjum > >>> <usama.anjum@collabora.com> wrote: > >>>> On 6/15/23 7:52 PM, Michał Mirosław wrote: > >>>>> On Thu, 15 Jun 2023 at 15:58, Muhammad Usama Anjum > >>>>> <usama.anjum@collabora.com> wrote: > >>>>>> I'll send next revision now. > >>>>>> On 6/14/23 11:00 PM, Michał Mirosław wrote: > >>>>>>> (A quick reply to answer open questions in case they help the next version.) > > [...] > >>>>>>> I guess this will be reworked anyway, but I'd prefer this didn't need > >>>>>>> custom errors etc. If we agree to decoupling the selection and GET > >>>>>>> output, it could be: > >>>>>>> > >>>>>>> bool is_interesting_page(p, flags); // this one does the > >>>>>>> required/anyof/excluded match > >>>>>>> size_t output_range(p, start, len, flags); // this one fills the > >>>>>>> output vector and returns how many pages were fit > >>>>>>> > >>>>>>> In this setup, `is_interesting_page() && (n_out = output_range()) < > >>>>>>> n_pages` means this is the final range, no more will fit. And if > >>>>>>> `n_out == 0` then no pages fit and no WP is needed (no other special > >>>>>>> cases). > >>>>>> Right now, pagemap_scan_output() performs the work of both of these two > >>>>>> functions. The part can be broken into is_interesting_pages() and we can > >>>>>> leave the remaining part as it is. > >>>>>> > >>>>>> Saying that n_out < n_pages tells us the buffer is full covers one case. > >>>>>> But there is case of maximum pages have been found and walk needs to be > >>>>>> aborted. > >>>>> > >>>>> This case is exactly what `n_out < n_pages` will cover (if scan_output > >>>>> uses max_pages properly to limit n_out). > >>>>> Isn't it that when the buffer is full we want to abort the scan always > >>>>> (with WP if `n_out > 0`)? > >>>> Wouldn't it be duplication of condition if buffer is full inside > >>>> pagemap_scan_output() and just outside it. Inside pagemap_scan_output() we > >>>> check if we have space before putting data inside it. I'm using this same > >>>> condition to indicate that buffer is full. > >>> > >>> I'm not sure what do you mean? The buffer-full conditions would be > >>> checked in ..scan_output() and communicated to the caller by returning > >>> N less than `n_pages` passed in. This is exactly how e.g. read() > >>> works: if you get less than requested you've hit the end of the file. > >>> If the file happens to have size that is equal to the provided buffer > >>> length, the next read() will return 0. > >> Right now we have: > >> > >> pagemap_scan_output(): > >> if (p->vec_buf_index >= p->vec_buf_len) > >> return PM_SCAN_BUFFER_FULL; > >> if (p->found_pages == p->max_pages) > >> return PM_SCAN_FOUND_MAX_PAGES; > > > > Why do you need to differentiate between those cases? > > > >> pagemap_scan_pmd_entry(): > >> ret = pagemap_scan_output(bitmap, p, start, n_pages); > >> if (ret >= 0) // success > >> make_UFFD_WP and flush > >> else > >> buffer_error > >> > >> You are asking me to do: > >> > >> pagemap_scan_output(): > >> if (p->vec_buf_index >= p->vec_buf_len) > >> return 0; > > > >> if (p->found_pages == p->max_pages) > >> return PM_SCAN_FOUND_MAX_PAGES; > > > > This should be instead: > > > > n_pages = min(p->max_pags - p_found_pages, n_pages) > > ... > > return n_pages; > You are missing the optimization here that we check for full buffer every > time adding to user buffer. This was added to remove extra iteration of > page walk if buffer is full already. The way you are suggesting will remove it. > > So you are returning remaining pages to be found now. This doesn't seem > right. If max_pages is 520, found_pages is 0 and n_pages is 512 before > calling pagemap_scan_output(). found_pages would become 512 after adding > 512 pages to output buffer. But n_pages would return 8 instead of 512. You > were saying we should return the number of pages added to the output buffer. Ok, if we want this optimization, then i'd rework it so that we have: bool pagemap_scan_output(..., int *n_pages) { limit n_pages; ... return have_more_room_in_output; } The compiler should remove the pointer and memory storage for `n_pages` when inlining the function. Best Regards Michał Mirosław
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 507cd4e59d07..0d49e14e97aa 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -19,6 +19,7 @@ #include <linux/shmem_fs.h> #include <linux/uaccess.h> #include <linux/pkeys.h> +#include <linux/minmax.h> #include <asm/elf.h> #include <asm/tlb.h> @@ -1765,11 +1766,523 @@ static int pagemap_release(struct inode *inode, struct file *file) return 0; } +#define PM_SCAN_OP_WRITE BIT(63) +#define PM_SCAN_FOUND_MAX_PAGES (1) +#define PM_SCAN_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \ + PAGE_IS_PRESENT | PAGE_IS_SWAPPED) +#define PM_SCAN_OPS (PM_SCAN_OP_GET | PM_SCAN_OP_WP) +#define IS_PM_SCAN_GET(flags) (flags & PM_SCAN_OP_GET) +#define IS_PM_SCAN_WP(flags) (flags & PM_SCAN_OP_WP) +#define PM_SCAN_BITMAP(wt, file, present, swap) \ + ((wt) | ((file) << 1) | ((present) << 2) | ((swap) << 3)) + +struct pagemap_scan_private { + struct page_region *vec_buf, cur_buf; + unsigned long vec_buf_len, vec_buf_index, max_pages, found_pages, flags; + unsigned long required_mask, anyof_mask, excluded_mask, return_mask; +}; + +static inline bool is_pte_uffd_wp(pte_t pte) +{ + return (pte_present(pte) && pte_uffd_wp(pte)) || + pte_swp_uffd_wp_any(pte); +} + +static inline void make_uffd_wp_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte) +{ + pte_t ptent = ptep_get(pte); + + if (pte_present(ptent)) { + pte_t old_pte; + + old_pte = ptep_modify_prot_start(vma, addr, pte); + ptent = pte_mkuffd_wp(ptent); + ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); + } else if (is_swap_pte(ptent)) { + ptent = pte_swp_mkuffd_wp(ptent); + set_pte_at(vma->vm_mm, addr, pte, ptent); + } else { + set_pte_at(vma->vm_mm, addr, pte, + make_pte_marker(PTE_MARKER_UFFD_WP)); + } +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline bool is_pmd_uffd_wp(pmd_t pmd) +{ + return (pmd_present(pmd) && pmd_uffd_wp(pmd)) || + (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)); +} + +static inline void make_uffd_wp_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + pmd_t old, pmd = *pmdp; + + if (pmd_present(pmd)) { + old = pmdp_invalidate_ad(vma, addr, pmdp); + pmd = pmd_mkuffd_wp(old); + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } else if (is_migration_entry(pmd_to_swp_entry(pmd))) { + pmd = pmd_swp_mkuffd_wp(pmd); + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } +} +#endif + +#ifdef CONFIG_HUGETLB_PAGE +static inline bool is_huge_pte_uffd_wp(pte_t pte) +{ + return (pte_present(pte) && huge_pte_uffd_wp(pte)) || + pte_swp_uffd_wp_any(pte); +} + +static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t ptent) +{ + if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent)) + return; + + if (is_hugetlb_entry_migration(ptent)) + set_huge_pte_at(vma->vm_mm, addr, ptep, + pte_swp_mkuffd_wp(ptent)); + else if (!huge_pte_none(ptent)) + huge_ptep_modify_prot_commit(vma, addr, ptep, ptent, + huge_pte_mkuffd_wp(ptent)); + else + set_huge_pte_at(vma->vm_mm, addr, ptep, + make_pte_marker(PTE_MARKER_UFFD_WP)); +} +#endif + +static int pagemap_scan_test_walk(unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + + if ((p->flags & PM_SCAN_OP_WRITE) && (!userfaultfd_wp_async(vma) || + !userfaultfd_wp_use_markers(vma))) + return -EPERM; + + if (vma->vm_flags & VM_PFNMAP) + return 1; + + return 0; +} + +static int pagemap_scan_output(bool wt, bool file, bool pres, bool swap, + struct pagemap_scan_private *p, + unsigned long addr, unsigned int n_pages) +{ + unsigned long bitmap = PM_SCAN_BITMAP(wt, file, pres, swap); + struct page_region *cur_buf = &p->cur_buf; + + if (!n_pages) + return -EINVAL; + + if ((p->required_mask & bitmap) != p->required_mask) + return 0; + if (p->anyof_mask && !(p->anyof_mask & bitmap)) + return 0; + if (p->excluded_mask & bitmap) + return 0; + + bitmap &= p->return_mask; + if (!bitmap) + return 0; + + if (cur_buf->bitmap == bitmap && + cur_buf->start + cur_buf->len * PAGE_SIZE == addr) { + cur_buf->len += n_pages; + p->found_pages += n_pages; + } else { + if (cur_buf->len && p->vec_buf_index >= p->vec_buf_len) + return -ENOMEM; + + if (cur_buf->len) { + memcpy(&p->vec_buf[p->vec_buf_index], cur_buf, + sizeof(*p->vec_buf)); + p->vec_buf_index++; + } + + cur_buf->start = addr; + cur_buf->len = n_pages; + cur_buf->bitmap = bitmap; + p->found_pages += n_pages; + } + + if (p->max_pages && (p->found_pages == p->max_pages)) + return PM_SCAN_FOUND_MAX_PAGES; + + return 0; +} + +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, + unsigned long end, struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + bool is_written, flush = false; + pte_t *pte, *orig_pte, ptent; + unsigned long addr = end; + spinlock_t *ptl; + int ret = 0; + + arch_enter_lazy_mmu_mode(); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + ptl = pmd_trans_huge_lock(pmd, vma); + if (ptl) { + unsigned long n_pages = (end - start)/PAGE_SIZE; + + if (p->max_pages && n_pages > p->max_pages - p->found_pages) + n_pages = p->max_pages - p->found_pages; + + is_written = !is_pmd_uffd_wp(*pmd); + + /* + * Break huge page into small pages if the WP operation need to + * be performed is on a portion of the huge page. + */ + if (is_written && IS_PM_SCAN_WP(p->flags) && + n_pages < HPAGE_SIZE/PAGE_SIZE) { + spin_unlock(ptl); + + split_huge_pmd(vma, pmd, start); + goto process_smaller_pages; + } + + if (IS_PM_SCAN_GET(p->flags)) + ret = pagemap_scan_output(is_written, vma->vm_file, + pmd_present(*pmd), + is_swap_pmd(*pmd), + p, start, n_pages); + + if (ret >= 0 && is_written && IS_PM_SCAN_WP(p->flags)) + make_uffd_wp_pmd(vma, addr, pmd); + + if (is_written && IS_PM_SCAN_WP(p->flags)) + flush_tlb_range(vma, start, end); + + spin_unlock(ptl); + + arch_leave_lazy_mmu_mode(); + return ret; + } + +process_smaller_pages: +#endif + + orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } + for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) { + ptent = ptep_get(pte); + is_written = !is_pte_uffd_wp(ptent); + + if (IS_PM_SCAN_GET(p->flags)) + ret = pagemap_scan_output(is_written, vma->vm_file, + pte_present(ptent), + is_swap_pte(ptent), + p, addr, 1); + + if (ret >= 0 && is_written && IS_PM_SCAN_WP(p->flags)) { + make_uffd_wp_pte(vma, addr, pte); + flush = true; + } + } + + if (flush) + flush_tlb_range(vma, start, addr); + + pte_unmap_unlock(orig_pte, ptl); + arch_leave_lazy_mmu_mode(); + + cond_resched(); + return ret; +} + +#ifdef CONFIG_HUGETLB_PAGE +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, + unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + unsigned long n_pages = (end - start)/PAGE_SIZE; + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + struct hstate *h = hstate_vma(vma); + spinlock_t *ptl; + bool is_written; + int ret = 0; + pte_t ptent; + + if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE) + return -EINVAL; + + if (p->max_pages && n_pages > p->max_pages - p->found_pages) + n_pages = p->max_pages - p->found_pages; + + if (IS_PM_SCAN_WP(p->flags)) { + i_mmap_lock_write(vma->vm_file->f_mapping); + ptl = huge_pte_lock(h, vma->vm_mm, ptep); + } + + ptent = huge_ptep_get(ptep); + is_written = !is_huge_pte_uffd_wp(ptent); + + /* + * Partial hugetlb page clear isn't supported + */ + if (is_written && IS_PM_SCAN_WP(p->flags) && + n_pages < HPAGE_SIZE/PAGE_SIZE) { + ret = -ENOSPC; + goto unlock_and_return; + } + + if (IS_PM_SCAN_GET(p->flags)) { + ret = pagemap_scan_output(is_written, vma->vm_file, + pte_present(ptent), is_swap_pte(ptent), + p, start, n_pages); + if (ret < 0) + goto unlock_and_return; + } + + if (is_written && IS_PM_SCAN_WP(p->flags)) { + make_uffd_wp_huge_pte(vma, start, ptep, ptent); + flush_hugetlb_tlb_range(vma, start, end); + } + +unlock_and_return: + if (IS_PM_SCAN_WP(p->flags)) { + spin_unlock(ptl); + i_mmap_unlock_write(vma->vm_file->f_mapping); + } + + return ret; +} +#else +#define pagemap_scan_hugetlb_entry NULL +#endif + +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, + int depth, struct mm_walk *walk) +{ + unsigned long n_pages = (end - addr)/PAGE_SIZE; + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + int ret = 0; + + if (!vma || !IS_PM_SCAN_GET(p->flags)) + return 0; + + if (p->max_pages && n_pages > p->max_pages - p->found_pages) + n_pages = p->max_pages - p->found_pages; + + ret = pagemap_scan_output(false, vma->vm_file, false, false, p, addr, + n_pages); + + return ret; +} + +static const struct mm_walk_ops pagemap_scan_ops = { + .test_walk = pagemap_scan_test_walk, + .pmd_entry = pagemap_scan_pmd_entry, + .pte_hole = pagemap_scan_pte_hole, + .hugetlb_entry = pagemap_scan_hugetlb_entry, +}; + +static inline unsigned long pagemap_scan_check_page_written(struct pagemap_scan_private *p) +{ + return ((p->required_mask | p->anyof_mask | p->excluded_mask) & + PAGE_IS_WRITTEN) ? PM_SCAN_OP_WRITE : 0; +} + +static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start, + struct page_region __user *vec) +{ + /* Detect illegal size, flags, len and masks */ + if (arg->size != sizeof(struct pm_scan_arg)) + return -EINVAL; + if (arg->flags & ~PM_SCAN_OPS) + return -EINVAL; + if (!arg->len) + return -EINVAL; + if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask | + arg->return_mask) & ~PM_SCAN_BITS_ALL) + return -EINVAL; + if (!arg->required_mask && !arg->anyof_mask && + !arg->excluded_mask) + return -EINVAL; + if (!arg->return_mask) + return -EINVAL; + + /* Validate memory range */ + if (!IS_ALIGNED(start, PAGE_SIZE)) + return -EINVAL; + if (!access_ok((void __user *)start, arg->len)) + return -EFAULT; + + if (IS_PM_SCAN_GET(arg->flags)) { + if (!arg->vec) + return -EINVAL; + if (arg->vec_len == 0) + return -EINVAL; + } + + if (IS_PM_SCAN_WP(arg->flags)) { + if (!IS_PM_SCAN_GET(arg->flags) && arg->max_pages) + return -EINVAL; + + if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask) & + ~PAGE_IS_WRITTEN) + return -EINVAL; + } + + return 0; +} + +static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg) +{ + struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg; + unsigned long start, end, walk_start, walk_end; + unsigned long empty_slots, vec_index = 0; + struct mmu_notifier_range range; + struct page_region __user *vec; + struct pagemap_scan_private p; + struct pm_scan_arg arg; + int ret = 0; + + if (copy_from_user(&arg, uarg, sizeof(arg))) + return -EFAULT; + + start = untagged_addr((unsigned long)arg.start); + vec = (struct page_region *)untagged_addr((unsigned long)arg.vec); + + ret = pagemap_scan_args_valid(&arg, start, vec); + if (ret) + return ret; + + end = start + arg.len; + p.max_pages = arg.max_pages; + p.found_pages = 0; + p.required_mask = arg.required_mask; + p.anyof_mask = arg.anyof_mask; + p.excluded_mask = arg.excluded_mask; + p.return_mask = arg.return_mask; + p.flags = arg.flags; + p.flags |= pagemap_scan_check_page_written(&p); + p.cur_buf.start = p.cur_buf.len = p.cur_buf.bitmap = 0; + p.vec_buf = NULL; + p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT; + + /* + * Allocate smaller buffer to get output from inside the page walk + * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As + * we want to return output to user in compact form where no two + * consecutive regions should be continuous and have the same flags. + * So store the latest element in p.cur_buf between different walks and + * store the p.cur_buf at the end of the walk to the user buffer. + */ + if (IS_PM_SCAN_GET(p.flags)) { + p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf), + GFP_KERNEL); + if (!p.vec_buf) + return -ENOMEM; + } + + if (IS_PM_SCAN_WP(p.flags)) { + mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, + mm, start, end); + mmu_notifier_invalidate_range_start(&range); + } + + walk_start = walk_end = start; + while (walk_end < end && !ret) { + if (IS_PM_SCAN_GET(p.flags)) { + p.vec_buf_index = 0; + + /* + * All data is copied to cur_buf first. When more data + * is found, we push cur_buf to vec_buf and copy new + * data to cur_buf. Subtract 1 from length as the + * index of cur_buf isn't counted in length. + */ + empty_slots = arg.vec_len - vec_index; + p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1); + } + + walk_end = (walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK; + if (walk_end > end) + walk_end = end; + + ret = mmap_read_lock_killable(mm); + if (ret) + goto free_data; + ret = walk_page_range(mm, walk_start, walk_end, + &pagemap_scan_ops, &p); + mmap_read_unlock(mm); + + if (ret && ret != -ENOMEM && ret != PM_SCAN_FOUND_MAX_PAGES) + goto free_data; + + walk_start = walk_end; + if (IS_PM_SCAN_GET(p.flags) && p.vec_buf_index) { + if (copy_to_user(&vec[vec_index], p.vec_buf, + p.vec_buf_index * sizeof(*p.vec_buf))) { + /* + * Return error even though the OP succeeded + */ + ret = -EFAULT; + goto free_data; + } + vec_index += p.vec_buf_index; + } + } + + if (p.cur_buf.len) { + if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) { + ret = -EFAULT; + goto free_data; + } + vec_index++; + } + + ret = vec_index; + +free_data: + if (IS_PM_SCAN_WP(p.flags)) + mmu_notifier_invalidate_range_end(&range); + + kfree(p.vec_buf); + return ret; +} + +static long do_pagemap_cmd(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct mm_struct *mm = file->private_data; + + switch (cmd) { + case PAGEMAP_SCAN: + return do_pagemap_scan(mm, arg); + + default: + return -EINVAL; + } +} + const struct file_operations proc_pagemap_operations = { .llseek = mem_lseek, /* borrow this */ .read = pagemap_read, .open = pagemap_open, .release = pagemap_release, + .unlocked_ioctl = do_pagemap_cmd, + .compat_ioctl = do_pagemap_cmd, }; #endif /* CONFIG_PROC_PAGE_MONITOR */ diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index beb7c63d2871..a6e773c3e2b4 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -261,6 +261,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long cp_flags); bool is_hugetlb_entry_migration(pte_t pte); +bool is_hugetlb_entry_hwpoisoned(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); #else /* !CONFIG_HUGETLB_PAGE */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..47879c38ce2f 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -305,4 +305,57 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND) +/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) + +/* Bits are set in the bitmap of the page_region and masks in pm_scan_args */ +#define PAGE_IS_WRITTEN (1 << 0) +#define PAGE_IS_FILE (1 << 1) +#define PAGE_IS_PRESENT (1 << 2) +#define PAGE_IS_SWAPPED (1 << 3) + +/* + * struct page_region - Page region with bitmap flags + * @start: Start of the region + * @len: Length of the region in pages + * bitmap: Bits sets for the region + */ +struct page_region { + __u64 start; + __u64 len; + __u64 bitmap; +}; + +/* + * struct pm_scan_arg - Pagemap ioctl argument + * @size: Size of the structure + * @flags: Flags for the IOCTL + * @start: Starting address of the region + * @len: Length of the region (All the pages in this length are included) + * @vec: Address of page_region struct array for output + * @vec_len: Length of the page_region struct array + * @max_pages: Optional max return pages + * @required_mask: Required mask - All of these bits have to be set in the PTE + * @anyof_mask: Any mask - Any of these bits are set in the PTE + * @excluded_mask: Exclude mask - None of these bits are set in the PTE + * @return_mask: Bits that are to be reported in page_region + */ +struct pm_scan_arg { + __u64 size; + __u64 flags; + __u64 start; + __u64 len; + __u64 vec; + __u64 vec_len; + __u64 max_pages; + __u64 required_mask; + __u64 anyof_mask; + __u64 excluded_mask; + __u64 return_mask; +}; + +/* Supported flags */ +#define PM_SCAN_OP_GET (1 << 0) +#define PM_SCAN_OP_WP (1 << 1) + #endif /* _UAPI_LINUX_FS_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0db13167b1ee..7e60f0f3fd03 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4980,7 +4980,7 @@ bool is_hugetlb_entry_migration(pte_t pte) return false; } -static bool is_hugetlb_entry_hwpoisoned(pte_t pte) +bool is_hugetlb_entry_hwpoisoned(pte_t pte) { swp_entry_t swp;
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl: - Get the information if the pages have been written-to (PAGE_IS_WRITTEN), file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped (PAGE_IS_SWAPPED). - Find pages which have been written-to and/or write protect the pages (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP) This IOCTL can be extended to get information about more PTE bits. Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com> --- Changes in v18: - Rebased on top of next-20230613 - ptep_get() updates - remove pmd_trans_unstable() and add ACTION_AGAIN - Review updates (Micheal) Changes in v17: - Rebased on next-20230606 - Made make_uffd_wp_*_pte() better and minor changes Changes in v16: - Fixed a corner case where kernel writes beyond user buffer by one element - Bring back exclusive PM_SCAN_OP_WP - Cosmetic changes Changes in v15: - Build fix: - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of using x86 specific flush function in do_pagemap_scan() - Remove #ifdef from pagemap_scan_hugetlb_entry() - Use mm instead of undefined vma->vm_mm Changes in v14: - Fix build error caused by #ifdef added at last minute in some configs Changes in v13: - Review updates - mmap_read_lock_killable() instead of mmap_read_lock() - Replace uffd_wp_range() with helpers which increases performance drastically for OP_WP operations by reducing the number of tlb flushing etc - Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range Changes in v12: - Add hugetlb support to cover all memory types - Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch - Review updates to the code Changes in v11: - Find written pages in a better way - Fix a corner case (thanks Paul) - Improve the code/comments - remove ENGAGE_WP + ! GET operation - shorten the commit message in favour of moving documentation to pagemap.rst Changes in v10: - move changes in tools/include/uapi/linux/fs.h to separate patch - update commit message Change in v8: - Correct is_pte_uffd_wp() - Improve readability and error checks - Remove some un-needed code Changes in v7: - Rebase on top of latest next - Fix some corner cases - Base soft-dirty on the uffd wp async - Update the terminologies - Optimize the memory usage inside the ioctl Changes in v6: - Rename variables and update comments - Make IOCTL independent of soft_dirty config - Change masks and bitmap type to _u64 - Improve code quality Changes in v5: - Remove tlb flushing even for clear operation Changes in v4: - Update the interface and implementation Changes in v3: - Tighten the user-kernel interface by using explicit types and add more error checking Changes in v2: - Convert the interface from syscall to ioctl - Remove pidfd support as it doesn't make sense in ioctl --- fs/proc/task_mmu.c | 513 ++++++++++++++++++++++++++++++++++++++++ include/linux/hugetlb.h | 1 + include/uapi/linux/fs.h | 53 +++++ mm/hugetlb.c | 2 +- 4 files changed, 568 insertions(+), 1 deletion(-)