| Message ID | CAP71Wjxwv3i2=UtbzNai79h-f83xcL6G1R8+kY-C-jA==VFy9Q@mail.gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [oe] meta-oe and yocto-check-layer | expand |
Hi Nicolas, On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > hi Armin, Paul, Richard, > > I was looking at getting the compliance report for meta-oe (sumo > branch), and I have found a few issues. > > * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > depends on meta-networking (c-ares). It was fixes in master, with > 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > this fix needs to be backported to sumo as well if we want the YP 2.0 > compliance script to even work. If agreed, once merged, please let me > know so that I can try again to generate a compliance report. Is it appropriate to make such moves in a stable branch? I wouldn't have thought so. > * in order to run the compliance report, i locally added > networking-layer in meta-oe/conf/layer.conf, and it creates a > dependency loop since meta-oe <-> meta-networking. I found out that > yocto-check-layer doesn't like that too much, and brutally fails. The > following patch makes yocto-check-layer work again even with > dependency loop. I am going to do a few more tests and send that over > as a patch. > > diff --git a/scripts/lib/checklayer/__init__.py > b/scripts/lib/checklayer/__init__.py > index 2618416fab..0cc9bf3b6d 100644 > --- a/scripts/lib/checklayer/__init__.py > +++ b/scripts/lib/checklayer/__init__.py > @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, layer, > layers, logger): > logger.debug('Processing dependencies %s for layer %s.' % \ > (depends, layer['name'])) > > + # To avoid never ending recursion, we keep track of layers while > + # they are being processed in this 'static' attribute. > + if not hasattr(recurse_dependencies, "layers"): > + recurse_dependencies.layers = [] > + > for depend in depends.split(): > # core (oe-core) is suppose to be provided > if depend == 'core': > continue > > + if depend in recurse_dependencies.layers: > + continue > + > + recurse_dependencies.layers.append(depend) > + > layer_depend = _find_layer_depends(depend, layers) > if not layer_depend: > logger.error('Layer %s depends on %s and isn\'t found.' % \ Patch looks reasonable to me FWIW. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On 09/24/2018 02:03 PM, Paul Eggleton wrote: > Hi Nicolas, > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: >> hi Armin, Paul, Richard, >> >> I was looking at getting the compliance report for meta-oe (sumo >> branch), and I have found a few issues. >> >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it >> depends on meta-networking (c-ares). It was fixes in master, with >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think >> this fix needs to be backported to sumo as well if we want the YP 2.0 >> compliance script to even work. If agreed, once merged, please let me >> know so that I can try again to generate a compliance report. > Is it appropriate to make such moves in a stable branch? I wouldn't have > thought so. > We have to. Per my understanding and why I tried very hard to make meta-openembedded clean ( appears I failed) is that if you want to be Yocto Compliant and include any layer that does not pass this test, you can not become Yocto Compliant. Or relax your rules!!!. - armin >> * in order to run the compliance report, i locally added >> networking-layer in meta-oe/conf/layer.conf, and it creates a >> dependency loop since meta-oe <-> meta-networking. I found out that >> yocto-check-layer doesn't like that too much, and brutally fails. The >> following patch makes yocto-check-layer work again even with >> dependency loop. I am going to do a few more tests and send that over >> as a patch. >> >> diff --git a/scripts/lib/checklayer/__init__.py >> b/scripts/lib/checklayer/__init__.py >> index 2618416fab..0cc9bf3b6d 100644 >> --- a/scripts/lib/checklayer/__init__.py >> +++ b/scripts/lib/checklayer/__init__.py >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, layer, >> layers, logger): >> logger.debug('Processing dependencies %s for layer %s.' % \ >> (depends, layer['name'])) >> >> + # To avoid never ending recursion, we keep track of layers while >> + # they are being processed in this 'static' attribute. >> + if not hasattr(recurse_dependencies, "layers"): >> + recurse_dependencies.layers = [] >> + >> for depend in depends.split(): >> # core (oe-core) is suppose to be provided >> if depend == 'core': >> continue >> >> + if depend in recurse_dependencies.layers: >> + continue >> + >> + recurse_dependencies.layers.append(depend) >> + >> layer_depend = _find_layer_depends(depend, layers) >> if not layer_depend: >> logger.error('Layer %s depends on %s and isn\'t found.' % \ > Patch looks reasonable to me FWIW. > > Cheers, > Paul > > -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> wrote: > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > Hi Nicolas, > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > >> hi Armin, Paul, Richard, > >> > >> I was looking at getting the compliance report for meta-oe (sumo > >> branch), and I have found a few issues. > >> > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > >> depends on meta-networking (c-ares). It was fixes in master, with > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > >> this fix needs to be backported to sumo as well if we want the YP 2.0 > >> compliance script to even work. If agreed, once merged, please let me > >> know so that I can try again to generate a compliance report. > > Is it appropriate to make such moves in a stable branch? I wouldn't have > > thought so. > > > > We have to. Per my understanding and why I tried very hard to make > meta-openembedded clean ( appears I failed) is that if you want to be > Yocto Compliant and include any layer that does not pass this test, you > can not become Yocto Compliant. I believe that we want meta-openembedded to be compliant, and a good example in general. I will send a backport your way for this change. > > Or relax your rules!!!. > > - armin > >> * in order to run the compliance report, i locally added > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > >> dependency loop since meta-oe <-> meta-networking. I found out that > >> yocto-check-layer doesn't like that too much, and brutally fails. The > >> following patch makes yocto-check-layer work again even with > >> dependency loop. I am going to do a few more tests and send that over > >> as a patch. > >> > >> diff --git a/scripts/lib/checklayer/__init__.py > >> b/scripts/lib/checklayer/__init__.py > >> index 2618416fab..0cc9bf3b6d 100644 > >> --- a/scripts/lib/checklayer/__init__.py > >> +++ b/scripts/lib/checklayer/__init__.py > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, layer, > >> layers, logger): > >> logger.debug('Processing dependencies %s for layer %s.' % \ > >> (depends, layer['name'])) > >> > >> + # To avoid never ending recursion, we keep track of layers while > >> + # they are being processed in this 'static' attribute. > >> + if not hasattr(recurse_dependencies, "layers"): > >> + recurse_dependencies.layers = [] > >> + > >> for depend in depends.split(): > >> # core (oe-core) is suppose to be provided > >> if depend == 'core': > >> continue > >> > >> + if depend in recurse_dependencies.layers: > >> + continue > >> + > >> + recurse_dependencies.layers.append(depend) > >> + > >> layer_depend = _find_layer_depends(depend, layers) > >> if not layer_depend: > >> logger.error('Layer %s depends on %s and isn\'t found.' % \ > > Patch looks reasonable to me FWIW. > > > > Cheers, > > Paul > > > > > -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
hi Armin, On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote: > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> wrote: > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > Hi Nicolas, > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > >> hi Armin, Paul, Richard, > > >> > > >> I was looking at getting the compliance report for meta-oe (sumo > > >> branch), and I have found a few issues. > > >> > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > >> depends on meta-networking (c-ares). It was fixes in master, with > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > >> this fix needs to be backported to sumo as well if we want the YP 2.0 > > >> compliance script to even work. If agreed, once merged, please let me > > >> know so that I can try again to generate a compliance report. > > > Is it appropriate to make such moves in a stable branch? I wouldn't have > > > thought so. > > > > > > > We have to. Per my understanding and why I tried very hard to make > > meta-openembedded clean ( appears I failed) is that if you want to be > > Yocto Compliant and include any layer that does not pass this test, you > > can not become Yocto Compliant. > > I believe that we want meta-openembedded to be compliant, and a good > example in general. I will send a backport your way for this change. Running the compliance script on meta-oe turned out to be an interesting exercise ;) I have found several issues, which I have mentioned in a few different threads, so I will summary here. * oe-core: fix the yocto-check-layer for dependency loop * I have the following local commits in meta-oe: meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) grpc: move it from oe to networking layer meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) With all changes above, the compliance script finds another issue with meta-xfce: AssertionError: Adding layer meta-xfce changed signatures. 7 signatures changed, initial differences (first hash before, second after): vim:do_install: 588d445122dccf317f15b0dd852f3888 -> ec086472d75d663c2fe836b935517810 This is definitely a violation of one our rule since adding meta-xfce changed changes vim recipe. > > > > > Or relax your rules!!!. > > > > - armin > > >> * in order to run the compliance report, i locally added > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > >> yocto-check-layer doesn't like that too much, and brutally fails. The > > >> following patch makes yocto-check-layer work again even with > > >> dependency loop. I am going to do a few more tests and send that over > > >> as a patch. > > >> > > >> diff --git a/scripts/lib/checklayer/__init__.py > > >> b/scripts/lib/checklayer/__init__.py > > >> index 2618416fab..0cc9bf3b6d 100644 > > >> --- a/scripts/lib/checklayer/__init__.py > > >> +++ b/scripts/lib/checklayer/__init__.py > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, layer, > > >> layers, logger): > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > >> (depends, layer['name'])) > > >> > > >> + # To avoid never ending recursion, we keep track of layers while > > >> + # they are being processed in this 'static' attribute. > > >> + if not hasattr(recurse_dependencies, "layers"): > > >> + recurse_dependencies.layers = [] > > >> + > > >> for depend in depends.split(): > > >> # core (oe-core) is suppose to be provided > > >> if depend == 'core': > > >> continue > > >> > > >> + if depend in recurse_dependencies.layers: > > >> + continue > > >> + > > >> + recurse_dependencies.layers.append(depend) > > >> + > > >> layer_depend = _find_layer_depends(depend, layers) > > >> if not layer_depend: > > >> logger.error('Layer %s depends on %s and isn\'t found.' % \ > > > Patch looks reasonable to me FWIW. > > > > > > Cheers, > > > Paul > > > > > > > > -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tuesday, 25 September 2018 9:43:43 PM NZST Nicolas Dechesne wrote: > Running the compliance script on meta-oe turned out to be an > interesting exercise ;) > > I have found several issues, which I have mentioned in a few different > threads, so I will summary here. > > * oe-core: fix the yocto-check-layer for dependency loop > * I have the following local commits in meta-oe: > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) Right, yeah, I noticed that one too a week or so ago and forgot to raise it. I actually think this needs to be fixed - meta-oe (at least by previous design) is not supposed to depend upon any other layer than OE-Core. I believe the dependency is optional though so we should be able to add a PACKAGECONFIG for it and default it to disabled. Assuming there is no violent objection and nobody else gets there first, I'll volunteer to make this fix. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) This causes another circular dependency which we don't want, doesn't it? On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < nicolas.dechesne@linaro.org> wrote: > hi Armin, > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > <nicolas.dechesne@linaro.org> wrote: > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > wrote: > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > Hi Nicolas, > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > >> hi Armin, Paul, Richard, > > > >> > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > >> branch), and I have found a few issues. > > > >> > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > >> this fix needs to be backported to sumo as well if we want the YP > 2.0 > > > >> compliance script to even work. If agreed, once merged, please let > me > > > >> know so that I can try again to generate a compliance report. > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > have > > > > thought so. > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > Yocto Compliant and include any layer that does not pass this test, you > > > can not become Yocto Compliant. > > > > I believe that we want meta-openembedded to be compliant, and a good > > example in general. I will send a backport your way for this change. > > Running the compliance script on meta-oe turned out to be an > interesting exercise ;) > > I have found several issues, which I have mentioned in a few different > threads, so I will summary here. > > * oe-core: fix the yocto-check-layer for dependency loop > * I have the following local commits in meta-oe: > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > grpc: move it from oe to networking layer > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > With all changes above, the compliance script finds another issue with > meta-xfce: > > AssertionError: Adding layer meta-xfce changed signatures. > 7 signatures changed, initial differences (first hash before, second > after): > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > ec086472d75d663c2fe836b935517810 > > This is definitely a violation of one our rule since adding meta-xfce > changed changes vim recipe. > > > > > > > > > Or relax your rules!!!. > > > > > > - armin > > > >> * in order to run the compliance report, i locally added > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > The > > > >> following patch makes yocto-check-layer work again even with > > > >> dependency loop. I am going to do a few more tests and send that > over > > > >> as a patch. > > > >> > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > >> b/scripts/lib/checklayer/__init__.py > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > >> --- a/scripts/lib/checklayer/__init__.py > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > layer, > > > >> layers, logger): > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > >> (depends, layer['name'])) > > > >> > > > >> + # To avoid never ending recursion, we keep track of layers > while > > > >> + # they are being processed in this 'static' attribute. > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > >> + recurse_dependencies.layers = [] > > > >> + > > > >> for depend in depends.split(): > > > >> # core (oe-core) is suppose to be provided > > > >> if depend == 'core': > > > >> continue > > > >> > > > >> + if depend in recurse_dependencies.layers: > > > >> + continue > > > >> + > > > >> + recurse_dependencies.layers.append(depend) > > > >> + > > > >> layer_depend = _find_layer_depends(depend, layers) > > > >> if not layer_depend: > > > >> logger.error('Layer %s depends on %s and isn\'t > found.' % \ > > > > Patch looks reasonable to me FWIW. > > > > > > > > Cheers, > > > > Paul > > > > > > > > > > > > -- > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tue, Sep 25, 2018 at 12:24:31PM +0200, Martin Jansa wrote: > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > This causes another circular dependency which we don't want, doesn't it? Especially if it's caused only by python-protobuf runtime dependency added in: https://patchwork.openembedded.org/patch/146867/ > On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < > nicolas.dechesne@linaro.org> wrote: > > > hi Armin, > > > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > > <nicolas.dechesne@linaro.org> wrote: > > > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > > wrote: > > > > > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > > Hi Nicolas, > > > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > > >> hi Armin, Paul, Richard, > > > > >> > > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > > >> branch), and I have found a few issues. > > > > >> > > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > > >> this fix needs to be backported to sumo as well if we want the YP > > 2.0 > > > > >> compliance script to even work. If agreed, once merged, please let > > me > > > > >> know so that I can try again to generate a compliance report. > > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > > have > > > > > thought so. > > > > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > > Yocto Compliant and include any layer that does not pass this test, you > > > > can not become Yocto Compliant. > > > > > > I believe that we want meta-openembedded to be compliant, and a good > > > example in general. I will send a backport your way for this change. > > > > Running the compliance script on meta-oe turned out to be an > > interesting exercise ;) > > > > I have found several issues, which I have mentioned in a few different > > threads, so I will summary here. > > > > * oe-core: fix the yocto-check-layer for dependency loop > > * I have the following local commits in meta-oe: > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > grpc: move it from oe to networking layer > > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > > > With all changes above, the compliance script finds another issue with > > meta-xfce: > > > > AssertionError: Adding layer meta-xfce changed signatures. > > 7 signatures changed, initial differences (first hash before, second > > after): > > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > > ec086472d75d663c2fe836b935517810 > > > > This is definitely a violation of one our rule since adding meta-xfce > > changed changes vim recipe. > > > > > > > > > > > > > Or relax your rules!!!. > > > > > > > > - armin > > > > >> * in order to run the compliance report, i locally added > > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > > The > > > > >> following patch makes yocto-check-layer work again even with > > > > >> dependency loop. I am going to do a few more tests and send that > > over > > > > >> as a patch. > > > > >> > > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > > >> b/scripts/lib/checklayer/__init__.py > > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > > >> --- a/scripts/lib/checklayer/__init__.py > > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > > layer, > > > > >> layers, logger): > > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > > >> (depends, layer['name'])) > > > > >> > > > > >> + # To avoid never ending recursion, we keep track of layers > > while > > > > >> + # they are being processed in this 'static' attribute. > > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > > >> + recurse_dependencies.layers = [] > > > > >> + > > > > >> for depend in depends.split(): > > > > >> # core (oe-core) is suppose to be provided > > > > >> if depend == 'core': > > > > >> continue > > > > >> > > > > >> + if depend in recurse_dependencies.layers: > > > > >> + continue > > > > >> + > > > > >> + recurse_dependencies.layers.append(depend) > > > > >> + > > > > >> layer_depend = _find_layer_depends(depend, layers) > > > > >> if not layer_depend: > > > > >> logger.error('Layer %s depends on %s and isn\'t > > found.' % \ > > > > > Patch looks reasonable to me FWIW. > > > > > > > > > > Cheers, > > > > > Paul > > > > > > > > > > > > > > > > -- > > _______________________________________________ > > Openembedded-devel mailing list > > Openembedded-devel@lists.openembedded.org > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tue, Sep 25, 2018 at 12:29 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > On Tue, Sep 25, 2018 at 12:24:31PM +0200, Martin Jansa wrote: > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > This causes another circular dependency which we don't want, doesn't it? What are the issues with circular dependency? LAYERDEPENDS only lists each layers own dependencies, which is helpful for integrators to know which layer they need to pull into bblayers.conf. If all layers from LAYERDEPENDS are pulled in, then it is expected that each recipe will build fine. The only circular dependency that I am aware is with yocto-check-layer script , and I sent a patch yesterday to fix this issue. With this patch, yocto-check-layer works fine even when layers have inter dependencies. https://patchwork.openembedded.org/patch/155113/ > > Especially if it's caused only by python-protobuf runtime dependency added in: > > https://patchwork.openembedded.org/patch/146867/ yes. this is the culprit. > > > On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < > > nicolas.dechesne@linaro.org> wrote: > > > > > hi Armin, > > > > > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > > > <nicolas.dechesne@linaro.org> wrote: > > > > > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > > > wrote: > > > > > > > > > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > > > Hi Nicolas, > > > > > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > > > >> hi Armin, Paul, Richard, > > > > > >> > > > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > > > >> branch), and I have found a few issues. > > > > > >> > > > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > > > >> this fix needs to be backported to sumo as well if we want the YP > > > 2.0 > > > > > >> compliance script to even work. If agreed, once merged, please let > > > me > > > > > >> know so that I can try again to generate a compliance report. > > > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > > > have > > > > > > thought so. > > > > > > > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > > > Yocto Compliant and include any layer that does not pass this test, you > > > > > can not become Yocto Compliant. > > > > > > > > I believe that we want meta-openembedded to be compliant, and a good > > > > example in general. I will send a backport your way for this change. > > > > > > Running the compliance script on meta-oe turned out to be an > > > interesting exercise ;) > > > > > > I have found several issues, which I have mentioned in a few different > > > threads, so I will summary here. > > > > > > * oe-core: fix the yocto-check-layer for dependency loop > > > * I have the following local commits in meta-oe: > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > grpc: move it from oe to networking layer > > > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > > > > > With all changes above, the compliance script finds another issue with > > > meta-xfce: > > > > > > AssertionError: Adding layer meta-xfce changed signatures. > > > 7 signatures changed, initial differences (first hash before, second > > > after): > > > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > > > ec086472d75d663c2fe836b935517810 > > > > > > This is definitely a violation of one our rule since adding meta-xfce > > > changed changes vim recipe. > > > > > > > > > > > > > > > > > Or relax your rules!!!. > > > > > > > > > > - armin > > > > > >> * in order to run the compliance report, i locally added > > > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > > > The > > > > > >> following patch makes yocto-check-layer work again even with > > > > > >> dependency loop. I am going to do a few more tests and send that > > > over > > > > > >> as a patch. > > > > > >> > > > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > > > >> b/scripts/lib/checklayer/__init__.py > > > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > > > >> --- a/scripts/lib/checklayer/__init__.py > > > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > > > layer, > > > > > >> layers, logger): > > > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > > > >> (depends, layer['name'])) > > > > > >> > > > > > >> + # To avoid never ending recursion, we keep track of layers > > > while > > > > > >> + # they are being processed in this 'static' attribute. > > > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > > > >> + recurse_dependencies.layers = [] > > > > > >> + > > > > > >> for depend in depends.split(): > > > > > >> # core (oe-core) is suppose to be provided > > > > > >> if depend == 'core': > > > > > >> continue > > > > > >> > > > > > >> + if depend in recurse_dependencies.layers: > > > > > >> + continue > > > > > >> + > > > > > >> + recurse_dependencies.layers.append(depend) > > > > > >> + > > > > > >> layer_depend = _find_layer_depends(depend, layers) > > > > > >> if not layer_depend: > > > > > >> logger.error('Layer %s depends on %s and isn\'t > > > found.' % \ > > > > > > Patch looks reasonable to me FWIW. > > > > > > > > > > > > Cheers, > > > > > > Paul > > > > > > > > > > > > > > > > > > > > -- > > > _______________________________________________ > > > Openembedded-devel mailing list > > > Openembedded-devel@lists.openembedded.org > > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > > > > > -- > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On 9/25/18 2:43 AM, Nicolas Dechesne wrote: > hi Armin, > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > <nicolas.dechesne@linaro.org> wrote: >> >> On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> wrote: >>> >>> >>> >>> On 09/24/2018 02:03 PM, Paul Eggleton wrote: >>>> Hi Nicolas, >>>> >>>> On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: >>>>> hi Armin, Paul, Richard, >>>>> >>>>> I was looking at getting the compliance report for meta-oe (sumo >>>>> branch), and I have found a few issues. >>>>> >>>>> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it >>>>> depends on meta-networking (c-ares). It was fixes in master, with >>>>> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think >>>>> this fix needs to be backported to sumo as well if we want the YP 2.0 >>>>> compliance script to even work. If agreed, once merged, please let me >>>>> know so that I can try again to generate a compliance report. >>>> Is it appropriate to make such moves in a stable branch? I wouldn't have >>>> thought so. >>>> >>> >>> We have to. Per my understanding and why I tried very hard to make >>> meta-openembedded clean ( appears I failed) is that if you want to be >>> Yocto Compliant and include any layer that does not pass this test, you >>> can not become Yocto Compliant. >> >> I believe that we want meta-openembedded to be compliant, and a good >> example in general. I will send a backport your way for this change. > > Running the compliance script on meta-oe turned out to be an > interesting exercise ;) > > I have found several issues, which I have mentioned in a few different > threads, so I will summary here. > > * oe-core: fix the yocto-check-layer for dependency loop > * I have the following local commits in meta-oe: > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > grpc: move it from oe to networking layer > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > With all changes above, the compliance script finds another issue with > meta-xfce: > > AssertionError: Adding layer meta-xfce changed signatures. > 7 signatures changed, initial differences (first hash before, second after): > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > ec086472d75d663c2fe836b935517810 > > This is definitely a violation of one our rule since adding meta-xfce > changed changes vim recipe. yes vim changes should be looked at and consolidated in one layer if possible. > >> >>> >>> Or relax your rules!!!. >>> >>> - armin >>>>> * in order to run the compliance report, i locally added >>>>> networking-layer in meta-oe/conf/layer.conf, and it creates a >>>>> dependency loop since meta-oe <-> meta-networking. I found out that >>>>> yocto-check-layer doesn't like that too much, and brutally fails. The >>>>> following patch makes yocto-check-layer work again even with >>>>> dependency loop. I am going to do a few more tests and send that over >>>>> as a patch. >>>>> >>>>> diff --git a/scripts/lib/checklayer/__init__.py >>>>> b/scripts/lib/checklayer/__init__.py >>>>> index 2618416fab..0cc9bf3b6d 100644 >>>>> --- a/scripts/lib/checklayer/__init__.py >>>>> +++ b/scripts/lib/checklayer/__init__.py >>>>> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, layer, >>>>> layers, logger): >>>>> logger.debug('Processing dependencies %s for layer %s.' % \ >>>>> (depends, layer['name'])) >>>>> >>>>> + # To avoid never ending recursion, we keep track of layers while >>>>> + # they are being processed in this 'static' attribute. >>>>> + if not hasattr(recurse_dependencies, "layers"): >>>>> + recurse_dependencies.layers = [] >>>>> + >>>>> for depend in depends.split(): >>>>> # core (oe-core) is suppose to be provided >>>>> if depend == 'core': >>>>> continue >>>>> >>>>> + if depend in recurse_dependencies.layers: >>>>> + continue >>>>> + >>>>> + recurse_dependencies.layers.append(depend) >>>>> + >>>>> layer_depend = _find_layer_depends(depend, layers) >>>>> if not layer_depend: >>>>> logger.error('Layer %s depends on %s and isn\'t found.' % \ >>>> Patch looks reasonable to me FWIW. >>>> >>>> Cheers, >>>> Paul >>>> >>>> >>> -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tue, Sep 25, 2018 at 03:05:19PM +0200, Nicolas Dechesne wrote: > On Tue, Sep 25, 2018 at 12:29 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > > > On Tue, Sep 25, 2018 at 12:24:31PM +0200, Martin Jansa wrote: > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > > This causes another circular dependency which we don't want, doesn't it? > > What are the issues with circular dependency? LAYERDEPENDS only lists > each layers own dependencies, which is helpful for integrators to know > which layer they need to pull into bblayers.conf. If all layers from > LAYERDEPENDS are pulled in, then it is expected that each recipe will > build fine. It prevents using these layers separately. Now people can include just meta-oe without meta-python (as long as they fix or mask the python-protobuf dependency from protobuf-ptest - which is a bug not a feature). If they have circular dependency then everybody using meta-oe will be forced to use meta-python as well and then why should we keep them in separate layers? It would be the same as adding all meta-python recipes into meta-oe. There was similar issue with meta-perl not so long time ago, before with meta-multimedia and meta-networking.. so if we don't fix these issues properly and just add more dependencies from meta-oe, then whole meta-openembedded as a repository will became one layer soon. > The only circular dependency that I am aware is with yocto-check-layer > script , and I sent a patch yesterday to fix this issue. With this > patch, yocto-check-layer works fine even when layers have inter > dependencies. > > https://patchwork.openembedded.org/patch/155113/ > > > > > Especially if it's caused only by python-protobuf runtime dependency added in: > > > > https://patchwork.openembedded.org/patch/146867/ > > yes. this is the culprit. > > > > > > On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < > > > nicolas.dechesne@linaro.org> wrote: > > > > > > > hi Armin, > > > > > > > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > > > > <nicolas.dechesne@linaro.org> wrote: > > > > > > > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > > > > Hi Nicolas, > > > > > > > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > > > > >> hi Armin, Paul, Richard, > > > > > > >> > > > > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > > > > >> branch), and I have found a few issues. > > > > > > >> > > > > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > > > > >> this fix needs to be backported to sumo as well if we want the YP > > > > 2.0 > > > > > > >> compliance script to even work. If agreed, once merged, please let > > > > me > > > > > > >> know so that I can try again to generate a compliance report. > > > > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > > > > have > > > > > > > thought so. > > > > > > > > > > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > > > > Yocto Compliant and include any layer that does not pass this test, you > > > > > > can not become Yocto Compliant. > > > > > > > > > > I believe that we want meta-openembedded to be compliant, and a good > > > > > example in general. I will send a backport your way for this change. > > > > > > > > Running the compliance script on meta-oe turned out to be an > > > > interesting exercise ;) > > > > > > > > I have found several issues, which I have mentioned in a few different > > > > threads, so I will summary here. > > > > > > > > * oe-core: fix the yocto-check-layer for dependency loop > > > > * I have the following local commits in meta-oe: > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > grpc: move it from oe to networking layer > > > > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > > > > > > > With all changes above, the compliance script finds another issue with > > > > meta-xfce: > > > > > > > > AssertionError: Adding layer meta-xfce changed signatures. > > > > 7 signatures changed, initial differences (first hash before, second > > > > after): > > > > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > > > > ec086472d75d663c2fe836b935517810 > > > > > > > > This is definitely a violation of one our rule since adding meta-xfce > > > > changed changes vim recipe. > > > > > > > > > > > > > > > > > > > > > Or relax your rules!!!. > > > > > > > > > > > > - armin > > > > > > >> * in order to run the compliance report, i locally added > > > > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > > > > The > > > > > > >> following patch makes yocto-check-layer work again even with > > > > > > >> dependency loop. I am going to do a few more tests and send that > > > > over > > > > > > >> as a patch. > > > > > > >> > > > > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > > > > >> b/scripts/lib/checklayer/__init__.py > > > > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > > > > >> --- a/scripts/lib/checklayer/__init__.py > > > > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > > > > layer, > > > > > > >> layers, logger): > > > > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > > > > >> (depends, layer['name'])) > > > > > > >> > > > > > > >> + # To avoid never ending recursion, we keep track of layers > > > > while > > > > > > >> + # they are being processed in this 'static' attribute. > > > > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > > > > >> + recurse_dependencies.layers = [] > > > > > > >> + > > > > > > >> for depend in depends.split(): > > > > > > >> # core (oe-core) is suppose to be provided > > > > > > >> if depend == 'core': > > > > > > >> continue > > > > > > >> > > > > > > >> + if depend in recurse_dependencies.layers: > > > > > > >> + continue > > > > > > >> + > > > > > > >> + recurse_dependencies.layers.append(depend) > > > > > > >> + > > > > > > >> layer_depend = _find_layer_depends(depend, layers) > > > > > > >> if not layer_depend: > > > > > > >> logger.error('Layer %s depends on %s and isn\'t > > > > found.' % \ > > > > > > > Patch looks reasonable to me FWIW. > > > > > > > > > > > > > > Cheers, > > > > > > > Paul > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > _______________________________________________ > > > > Openembedded-devel mailing list > > > > Openembedded-devel@lists.openembedded.org > > > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > > > > > > > > -- > > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tue, Sep 25, 2018 at 7:10 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > On Tue, Sep 25, 2018 at 03:05:19PM +0200, Nicolas Dechesne wrote: > > On Tue, Sep 25, 2018 at 12:29 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > > > > > On Tue, Sep 25, 2018 at 12:24:31PM +0200, Martin Jansa wrote: > > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > > > > This causes another circular dependency which we don't want, doesn't it? > > > > What are the issues with circular dependency? LAYERDEPENDS only lists > > each layers own dependencies, which is helpful for integrators to know > > which layer they need to pull into bblayers.conf. If all layers from > > LAYERDEPENDS are pulled in, then it is expected that each recipe will > > build fine. > > It prevents using these layers separately. > > Now people can include just meta-oe without meta-python (as long as they > fix or mask the python-protobuf dependency from protobuf-ptest - which > is a bug not a feature). > > If they have circular dependency then everybody using meta-oe will be > forced to use meta-python as well and then why should we keep them in > separate layers? It would be the same as adding all meta-python recipes > into meta-oe. It is not because there are circular dependencies that meta-oe requires meta-python, it is because of protobuf , it really depends on meta-python... I am tempted to agree with Paul, we should make the dependency explicitly optional using PACKAGECONFIG instead of requiring users to fix/mask it. > > There was similar issue with meta-perl not so long time ago, before with > meta-multimedia and meta-networking.. so if we don't fix these issues > properly and just add more dependencies from meta-oe, then whole > meta-openembedded as a repository will became one layer soon. agreed. I think it's good to spot these issues and fix them as they come. > > > The only circular dependency that I am aware is with yocto-check-layer > > script , and I sent a patch yesterday to fix this issue. With this > > patch, yocto-check-layer works fine even when layers have inter > > dependencies. > > > > https://patchwork.openembedded.org/patch/155113/ > > > > > > > > Especially if it's caused only by python-protobuf runtime dependency added in: > > > > > > https://patchwork.openembedded.org/patch/146867/ > > > > yes. this is the culprit. > > > > > > > > > On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < > > > > nicolas.dechesne@linaro.org> wrote: > > > > > > > > > hi Armin, > > > > > > > > > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > > > > > <nicolas.dechesne@linaro.org> wrote: > > > > > > > > > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > > > > > Hi Nicolas, > > > > > > > > > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > > > > > >> hi Armin, Paul, Richard, > > > > > > > >> > > > > > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > > > > > >> branch), and I have found a few issues. > > > > > > > >> > > > > > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > > > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > > > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > > > > > >> this fix needs to be backported to sumo as well if we want the YP > > > > > 2.0 > > > > > > > >> compliance script to even work. If agreed, once merged, please let > > > > > me > > > > > > > >> know so that I can try again to generate a compliance report. > > > > > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > > > > > have > > > > > > > > thought so. > > > > > > > > > > > > > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > > > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > > > > > Yocto Compliant and include any layer that does not pass this test, you > > > > > > > can not become Yocto Compliant. > > > > > > > > > > > > I believe that we want meta-openembedded to be compliant, and a good > > > > > > example in general. I will send a backport your way for this change. > > > > > > > > > > Running the compliance script on meta-oe turned out to be an > > > > > interesting exercise ;) > > > > > > > > > > I have found several issues, which I have mentioned in a few different > > > > > threads, so I will summary here. > > > > > > > > > > * oe-core: fix the yocto-check-layer for dependency loop > > > > > * I have the following local commits in meta-oe: > > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > grpc: move it from oe to networking layer > > > > > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > > > > > > > > > With all changes above, the compliance script finds another issue with > > > > > meta-xfce: > > > > > > > > > > AssertionError: Adding layer meta-xfce changed signatures. > > > > > 7 signatures changed, initial differences (first hash before, second > > > > > after): > > > > > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > > > > > ec086472d75d663c2fe836b935517810 > > > > > > > > > > This is definitely a violation of one our rule since adding meta-xfce > > > > > changed changes vim recipe. > > > > > > > > > > > > > > > > > > > > > > > > > Or relax your rules!!!. > > > > > > > > > > > > > > - armin > > > > > > > >> * in order to run the compliance report, i locally added > > > > > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > > > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > > > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > > > > > The > > > > > > > >> following patch makes yocto-check-layer work again even with > > > > > > > >> dependency loop. I am going to do a few more tests and send that > > > > > over > > > > > > > >> as a patch. > > > > > > > >> > > > > > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > > > > > >> b/scripts/lib/checklayer/__init__.py > > > > > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > > > > > >> --- a/scripts/lib/checklayer/__init__.py > > > > > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > > > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > > > > > layer, > > > > > > > >> layers, logger): > > > > > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > > > > > >> (depends, layer['name'])) > > > > > > > >> > > > > > > > >> + # To avoid never ending recursion, we keep track of layers > > > > > while > > > > > > > >> + # they are being processed in this 'static' attribute. > > > > > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > > > > > >> + recurse_dependencies.layers = [] > > > > > > > >> + > > > > > > > >> for depend in depends.split(): > > > > > > > >> # core (oe-core) is suppose to be provided > > > > > > > >> if depend == 'core': > > > > > > > >> continue > > > > > > > >> > > > > > > > >> + if depend in recurse_dependencies.layers: > > > > > > > >> + continue > > > > > > > >> + > > > > > > > >> + recurse_dependencies.layers.append(depend) > > > > > > > >> + > > > > > > > >> layer_depend = _find_layer_depends(depend, layers) > > > > > > > >> if not layer_depend: > > > > > > > >> logger.error('Layer %s depends on %s and isn\'t > > > > > found.' % \ > > > > > > > > Patch looks reasonable to me FWIW. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Paul > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > _______________________________________________ > > > > > Openembedded-devel mailing list > > > > > Openembedded-devel@lists.openembedded.org > > > > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > > > > > > > > > > > -- > > > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com > > -- > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tue, Sep 25, 2018 at 10:17 AM Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote: > > On Tue, Sep 25, 2018 at 7:10 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > > > On Tue, Sep 25, 2018 at 03:05:19PM +0200, Nicolas Dechesne wrote: > > > On Tue, Sep 25, 2018 at 12:29 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > > > > > > > On Tue, Sep 25, 2018 at 12:24:31PM +0200, Martin Jansa wrote: > > > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > > > > > > This causes another circular dependency which we don't want, doesn't it? > > > > > > What are the issues with circular dependency? LAYERDEPENDS only lists > > > each layers own dependencies, which is helpful for integrators to know > > > which layer they need to pull into bblayers.conf. If all layers from > > > LAYERDEPENDS are pulled in, then it is expected that each recipe will > > > build fine. > > > > It prevents using these layers separately. > > > > Now people can include just meta-oe without meta-python (as long as they > > fix or mask the python-protobuf dependency from protobuf-ptest - which > > is a bug not a feature). > > > > If they have circular dependency then everybody using meta-oe will be > > forced to use meta-python as well and then why should we keep them in > > separate layers? It would be the same as adding all meta-python recipes > > into meta-oe. > > It is not because there are circular dependencies that meta-oe > requires meta-python, it is because of protobuf , it really depends on > meta-python... I am tempted to agree with Paul, we should make the > dependency explicitly optional using PACKAGECONFIG instead of > requiring users to fix/mask it. > packageconfig seems a good approach for now > > > > There was similar issue with meta-perl not so long time ago, before with > > meta-multimedia and meta-networking.. so if we don't fix these issues > > properly and just add more dependencies from meta-oe, then whole > > meta-openembedded as a repository will became one layer soon. > > agreed. I think it's good to spot these issues and fix them as they come. > > > > > > The only circular dependency that I am aware is with yocto-check-layer > > > script , and I sent a patch yesterday to fix this issue. With this > > > patch, yocto-check-layer works fine even when layers have inter > > > dependencies. > > > > > > https://patchwork.openembedded.org/patch/155113/ > > > > > > > > > > > Especially if it's caused only by python-protobuf runtime dependency added in: > > > > > > > > https://patchwork.openembedded.org/patch/146867/ > > > > > > yes. this is the culprit. > > > > > > > > > > > > On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < > > > > > nicolas.dechesne@linaro.org> wrote: > > > > > > > > > > > hi Armin, > > > > > > > > > > > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > > > > > > <nicolas.dechesne@linaro.org> wrote: > > > > > > > > > > > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > > > > > > Hi Nicolas, > > > > > > > > > > > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > > > > > > >> hi Armin, Paul, Richard, > > > > > > > > >> > > > > > > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > > > > > > >> branch), and I have found a few issues. > > > > > > > > >> > > > > > > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > > > > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > > > > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > > > > > > >> this fix needs to be backported to sumo as well if we want the YP > > > > > > 2.0 > > > > > > > > >> compliance script to even work. If agreed, once merged, please let > > > > > > me > > > > > > > > >> know so that I can try again to generate a compliance report. > > > > > > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > > > > > > have > > > > > > > > > thought so. > > > > > > > > > > > > > > > > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > > > > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > > > > > > Yocto Compliant and include any layer that does not pass this test, you > > > > > > > > can not become Yocto Compliant. > > > > > > > > > > > > > > I believe that we want meta-openembedded to be compliant, and a good > > > > > > > example in general. I will send a backport your way for this change. > > > > > > > > > > > > Running the compliance script on meta-oe turned out to be an > > > > > > interesting exercise ;) > > > > > > > > > > > > I have found several issues, which I have mentioned in a few different > > > > > > threads, so I will summary here. > > > > > > > > > > > > * oe-core: fix the yocto-check-layer for dependency loop > > > > > > * I have the following local commits in meta-oe: > > > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > > grpc: move it from oe to networking layer > > > > > > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > > > > > > > > > > > With all changes above, the compliance script finds another issue with > > > > > > meta-xfce: > > > > > > > > > > > > AssertionError: Adding layer meta-xfce changed signatures. > > > > > > 7 signatures changed, initial differences (first hash before, second > > > > > > after): > > > > > > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > > > > > > ec086472d75d663c2fe836b935517810 > > > > > > > > > > > > This is definitely a violation of one our rule since adding meta-xfce > > > > > > changed changes vim recipe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Or relax your rules!!!. > > > > > > > > > > > > > > > > - armin > > > > > > > > >> * in order to run the compliance report, i locally added > > > > > > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > > > > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > > > > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > > > > > > The > > > > > > > > >> following patch makes yocto-check-layer work again even with > > > > > > > > >> dependency loop. I am going to do a few more tests and send that > > > > > > over > > > > > > > > >> as a patch. > > > > > > > > >> > > > > > > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > > > > > > >> b/scripts/lib/checklayer/__init__.py > > > > > > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > > > > > > >> --- a/scripts/lib/checklayer/__init__.py > > > > > > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > > > > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > > > > > > layer, > > > > > > > > >> layers, logger): > > > > > > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > > > > > > >> (depends, layer['name'])) > > > > > > > > >> > > > > > > > > >> + # To avoid never ending recursion, we keep track of layers > > > > > > while > > > > > > > > >> + # they are being processed in this 'static' attribute. > > > > > > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > > > > > > >> + recurse_dependencies.layers = [] > > > > > > > > >> + > > > > > > > > >> for depend in depends.split(): > > > > > > > > >> # core (oe-core) is suppose to be provided > > > > > > > > >> if depend == 'core': > > > > > > > > >> continue > > > > > > > > >> > > > > > > > > >> + if depend in recurse_dependencies.layers: > > > > > > > > >> + continue > > > > > > > > >> + > > > > > > > > >> + recurse_dependencies.layers.append(depend) > > > > > > > > >> + > > > > > > > > >> layer_depend = _find_layer_depends(depend, layers) > > > > > > > > >> if not layer_depend: > > > > > > > > >> logger.error('Layer %s depends on %s and isn\'t > > > > > > found.' % \ > > > > > > > > > Patch looks reasonable to me FWIW. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Paul > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > _______________________________________________ > > > > > > Openembedded-devel mailing list > > > > > > Openembedded-devel@lists.openembedded.org > > > > > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > > > > > > > > > > > > > > -- > > > > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com > > > > -- > > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com > -- > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-devel -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
On Tue, Sep 25, 2018 at 07:16:56PM +0200, Nicolas Dechesne wrote: > On Tue, Sep 25, 2018 at 7:10 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > > > On Tue, Sep 25, 2018 at 03:05:19PM +0200, Nicolas Dechesne wrote: > > > On Tue, Sep 25, 2018 at 12:29 PM Martin Jansa <martin.jansa@gmail.com> wrote: > > > > > > > > On Tue, Sep 25, 2018 at 12:24:31PM +0200, Martin Jansa wrote: > > > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > > > > > > This causes another circular dependency which we don't want, doesn't it? > > > > > > What are the issues with circular dependency? LAYERDEPENDS only lists > > > each layers own dependencies, which is helpful for integrators to know > > > which layer they need to pull into bblayers.conf. If all layers from > > > LAYERDEPENDS are pulled in, then it is expected that each recipe will > > > build fine. > > > > It prevents using these layers separately. > > > > Now people can include just meta-oe without meta-python (as long as they > > fix or mask the python-protobuf dependency from protobuf-ptest - which > > is a bug not a feature). > > > > If they have circular dependency then everybody using meta-oe will be > > forced to use meta-python as well and then why should we keep them in > > separate layers? It would be the same as adding all meta-python recipes > > into meta-oe. > > It is not because there are circular dependencies that meta-oe > requires meta-python, it is because of protobuf , it really depends on > meta-python... I am tempted to agree with Paul, we should make the > dependency explicitly optional using PACKAGECONFIG instead of > requiring users to fix/mask it. I meant circular dependency meta-oe -> meta-python -> meta-oe I agree with Paul as well, I was just saying that adding dependency on meta-python to meta-oe doesn't make any sense. > > > > There was similar issue with meta-perl not so long time ago, before with > > meta-multimedia and meta-networking.. so if we don't fix these issues > > properly and just add more dependencies from meta-oe, then whole > > meta-openembedded as a repository will became one layer soon. > > agreed. I think it's good to spot these issues and fix them as they come. > > > > > > The only circular dependency that I am aware is with yocto-check-layer > > > script , and I sent a patch yesterday to fix this issue. With this > > > patch, yocto-check-layer works fine even when layers have inter > > > dependencies. > > > > > > https://patchwork.openembedded.org/patch/155113/ > > > > > > > > > > > Especially if it's caused only by python-protobuf runtime dependency added in: > > > > > > > > https://patchwork.openembedded.org/patch/146867/ > > > > > > yes. this is the culprit. > > > > > > > > > > > > On Tue, Sep 25, 2018 at 11:44 AM Nicolas Dechesne < > > > > > nicolas.dechesne@linaro.org> wrote: > > > > > > > > > > > hi Armin, > > > > > > > > > > > > On Tue, Sep 25, 2018 at 8:59 AM Nicolas Dechesne > > > > > > <nicolas.dechesne@linaro.org> wrote: > > > > > > > > > > > > > > On Mon, Sep 24, 2018 at 11:51 PM akuster808 <akuster808@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/24/2018 02:03 PM, Paul Eggleton wrote: > > > > > > > > > Hi Nicolas, > > > > > > > > > > > > > > > > > > On Monday, 24 September 2018 10:05:02 PM NZST Nicolas Dechesne wrote: > > > > > > > > >> hi Armin, Paul, Richard, > > > > > > > > >> > > > > > > > > >> I was looking at getting the compliance report for meta-oe (sumo > > > > > > > > >> branch), and I have found a few issues. > > > > > > > > >> > > > > > > > > >> * in meta-openembedded/sumo, grpc is in meta-oe layer, while it > > > > > > > > >> depends on meta-networking (c-ares). It was fixes in master, with > > > > > > > > >> 251878e8b6b9 (grpc: move it from oe to networking layer), so I think > > > > > > > > >> this fix needs to be backported to sumo as well if we want the YP > > > > > > 2.0 > > > > > > > > >> compliance script to even work. If agreed, once merged, please let > > > > > > me > > > > > > > > >> know so that I can try again to generate a compliance report. > > > > > > > > > Is it appropriate to make such moves in a stable branch? I wouldn't > > > > > > have > > > > > > > > > thought so. > > > > > > > > > > > > > > > > > > > > > > > > > We have to. Per my understanding and why I tried very hard to make > > > > > > > > meta-openembedded clean ( appears I failed) is that if you want to be > > > > > > > > Yocto Compliant and include any layer that does not pass this test, you > > > > > > > > can not become Yocto Compliant. > > > > > > > > > > > > > > I believe that we want meta-openembedded to be compliant, and a good > > > > > > > example in general. I will send a backport your way for this change. > > > > > > > > > > > > Running the compliance script on meta-oe turned out to be an > > > > > > interesting exercise ;) > > > > > > > > > > > > I have found several issues, which I have mentioned in a few different > > > > > > threads, so I will summary here. > > > > > > > > > > > > * oe-core: fix the yocto-check-layer for dependency loop > > > > > > * I have the following local commits in meta-oe: > > > > > > meta-oe: add meta-python in LAYERDEPENDS (needed for protobuf) > > > > > > grpc: move it from oe to networking layer > > > > > > meta-multimedia: fixup LAYERDEPENDS (for dos2unix issue) > > > > > > > > > > > > With all changes above, the compliance script finds another issue with > > > > > > meta-xfce: > > > > > > > > > > > > AssertionError: Adding layer meta-xfce changed signatures. > > > > > > 7 signatures changed, initial differences (first hash before, second > > > > > > after): > > > > > > vim:do_install: 588d445122dccf317f15b0dd852f3888 -> > > > > > > ec086472d75d663c2fe836b935517810 > > > > > > > > > > > > This is definitely a violation of one our rule since adding meta-xfce > > > > > > changed changes vim recipe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Or relax your rules!!!. > > > > > > > > > > > > > > > > - armin > > > > > > > > >> * in order to run the compliance report, i locally added > > > > > > > > >> networking-layer in meta-oe/conf/layer.conf, and it creates a > > > > > > > > >> dependency loop since meta-oe <-> meta-networking. I found out that > > > > > > > > >> yocto-check-layer doesn't like that too much, and brutally fails. > > > > > > The > > > > > > > > >> following patch makes yocto-check-layer work again even with > > > > > > > > >> dependency loop. I am going to do a few more tests and send that > > > > > > over > > > > > > > > >> as a patch. > > > > > > > > >> > > > > > > > > >> diff --git a/scripts/lib/checklayer/__init__.py > > > > > > > > >> b/scripts/lib/checklayer/__init__.py > > > > > > > > >> index 2618416fab..0cc9bf3b6d 100644 > > > > > > > > >> --- a/scripts/lib/checklayer/__init__.py > > > > > > > > >> +++ b/scripts/lib/checklayer/__init__.py > > > > > > > > >> @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, > > > > > > layer, > > > > > > > > >> layers, logger): > > > > > > > > >> logger.debug('Processing dependencies %s for layer %s.' % \ > > > > > > > > >> (depends, layer['name'])) > > > > > > > > >> > > > > > > > > >> + # To avoid never ending recursion, we keep track of layers > > > > > > while > > > > > > > > >> + # they are being processed in this 'static' attribute. > > > > > > > > >> + if not hasattr(recurse_dependencies, "layers"): > > > > > > > > >> + recurse_dependencies.layers = [] > > > > > > > > >> + > > > > > > > > >> for depend in depends.split(): > > > > > > > > >> # core (oe-core) is suppose to be provided > > > > > > > > >> if depend == 'core': > > > > > > > > >> continue > > > > > > > > >> > > > > > > > > >> + if depend in recurse_dependencies.layers: > > > > > > > > >> + continue > > > > > > > > >> + > > > > > > > > >> + recurse_dependencies.layers.append(depend) > > > > > > > > >> + > > > > > > > > >> layer_depend = _find_layer_depends(depend, layers) > > > > > > > > >> if not layer_depend: > > > > > > > > >> logger.error('Layer %s depends on %s and isn\'t > > > > > > found.' % \ > > > > > > > > > Patch looks reasonable to me FWIW. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Paul > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > _______________________________________________ > > > > > > Openembedded-devel mailing list > > > > > > Openembedded-devel@lists.openembedded.org > > > > > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > > > > > > > > > > > > > > -- > > > > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com > > > > -- > > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py index 2618416fab..0cc9bf3b6d 100644 --- a/scripts/lib/checklayer/__init__.py +++ b/scripts/lib/checklayer/__init__.py @@ -151,11 +151,21 @@ def add_layer_dependencies(bblayersconf, layer, layers, logger): logger.debug('Processing dependencies %s for layer %s.' % \ (depends, layer['name'])) + # To avoid never ending recursion, we keep track of layers while + # they are being processed in this 'static' attribute. + if not hasattr(recurse_dependencies, "layers"): + recurse_dependencies.layers = [] + for depend in depends.split(): # core (oe-core) is suppose to be provided if depend == 'core': continue + if depend in recurse_dependencies.layers: + continue + + recurse_dependencies.layers.append(depend) + layer_depend = _find_layer_depends(depend, layers) if not layer_depend:
hi Armin, Paul, Richard, I was looking at getting the compliance report for meta-oe (sumo branch), and I have found a few issues. * in meta-openembedded/sumo, grpc is in meta-oe layer, while it depends on meta-networking (c-ares). It was fixes in master, with 251878e8b6b9 (grpc: move it from oe to networking layer), so I think this fix needs to be backported to sumo as well if we want the YP 2.0 compliance script to even work. If agreed, once merged, please let me know so that I can try again to generate a compliance report. * in order to run the compliance report, i locally added networking-layer in meta-oe/conf/layer.conf, and it creates a dependency loop since meta-oe <-> meta-networking. I found out that yocto-check-layer doesn't like that too much, and brutally fails. The following patch makes yocto-check-layer work again even with dependency loop. I am going to do a few more tests and send that over as a patch. logger.error('Layer %s depends on %s and isn\'t found.' % \ cheers nico -- _______________________________________________ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel