| Message ID | 20180905165227.29435-1-richard.purdie@linuxfoundation.org |
|---|---|
| State | New |
| Headers | show |
| Series | [1/8] python: Run pybench once for optimisation task in pgo | expand |
On Wed, Sep 5, 2018 at 9:52 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > From: Joshua Lock <joshua.g.lock@intel.com> > > os-release (5) recommends that the os-release file be installed in > /usr/lib/os-release and that /etc/os-release be a relative symlink to it. > > Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> > Signed-off-by: Ross Burton <ross.burton@intel.com> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > meta/recipes-core/os-release/os-release.bb | 9 ++++++--- > 1 file changed, 6 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-core/os-release/os-release.bb b/meta/recipes-core/os-release/os-release.bb > index f9887047561..dae5748ba7a 100644 > --- a/meta/recipes-core/os-release/os-release.bb > +++ b/meta/recipes-core/os-release/os-release.bb > @@ -1,7 +1,7 @@ > inherit allarch > > SUMMARY = "Operating system identification" > -DESCRIPTION = "The /etc/os-release file contains operating system identification data." > +DESCRIPTION = "The /usr/lib/os-release file contains operating system identification data." > LICENSE = "MIT" > INHIBIT_DEFAULT_DEPS = "1" > > @@ -42,6 +42,9 @@ python do_compile () { > do_compile[vardeps] += "${OS_RELEASE_FIELDS}" > > do_install () { > - install -d ${D}${sysconfdir} > - install -m 0644 os-release ${D}${sysconfdir}/ > + install -d ${D}${libdir} ${D}${sysconfdir} > + install -m 0644 os-release ${D}${libdir}/ > + lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea Typo. Please add the rm too when you fix that. > } > + > +FILES_${PN} += "${libdir}/os-release" > -- > 2.17.1 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Wed, 2018-09-05 at 10:16 -0700, Andre McCurdy wrote: > On Wed, Sep 5, 2018 at 9:52 AM, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > From: Joshua Lock <joshua.g.lock@intel.com> > > > > os-release (5) recommends that the os-release file be installed in > > /usr/lib/os-release and that /etc/os-release be a relative symlink > > to it. > > > > Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> > > Signed-off-by: Ross Burton <ross.burton@intel.com> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > --- > > meta/recipes-core/os-release/os-release.bb | 9 ++++++--- > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > diff --git a/meta/recipes-core/os-release/os-release.bb > > b/meta/recipes-core/os-release/os-release.bb > > index f9887047561..dae5748ba7a 100644 > > --- a/meta/recipes-core/os-release/os-release.bb > > +++ b/meta/recipes-core/os-release/os-release.bb > > @@ -1,7 +1,7 @@ > > inherit allarch > > > > SUMMARY = "Operating system identification" > > -DESCRIPTION = "The /etc/os-release file contains operating system > > identification data." > > +DESCRIPTION = "The /usr/lib/os-release file contains operating > > system identification data." > > LICENSE = "MIT" > > INHIBIT_DEFAULT_DEPS = "1" > > > > @@ -42,6 +42,9 @@ python do_compile () { > > do_compile[vardeps] += "${OS_RELEASE_FIELDS}" > > > > do_install () { > > - install -d ${D}${sysconfdir} > > - install -m 0644 os-release ${D}${sysconfdir}/ > > + install -d ${D}${libdir} ${D}${sysconfdir} > > + install -m 0644 os-release ${D}${libdir}/ > > + lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea > > Typo. > > Please add the rm too when you fix that. Too late, was already spotted and fixed. I'm not adding the rm since we're not going through all the do_installs adding rm statements in for things, I am totally against doing that. Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 9/5/18 9:52 AM, Richard Purdie wrote: > A recent binutils patch added the LDGOLD variable but its already used > for controlling EXTRA_OECONF options for gold. Separate the two variables > to different names to avoid build warnings and confusion. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > meta/recipes-devtools/binutils/binutils.inc | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-devtools/binutils/binutils.inc b/meta/recipes-devtools/binutils/binutils.inc > index b56fcd4cd01..81ecbb50cc1 100644 > --- a/meta/recipes-devtools/binutils/binutils.inc > +++ b/meta/recipes-devtools/binutils/binutils.inc > @@ -38,8 +38,8 @@ FILES_${PN}-dev = " \ > # Rather than duplicating multiple entries for these, make one > # list and reuse it. > > -LDGOLD ?= "ld.gold dwp" > -LDGOLD_riscv64 = "" > +LDGOLD_ALTS ?= "ld.gold dwp" > +LDGOLD_ALTS_riscv64 = "" > > USE_ALTERNATIVES_FOR = " \ > addr2line \ > @@ -50,7 +50,7 @@ USE_ALTERNATIVES_FOR = " \ > gprof \ > ld \ > ld.bfd \ > - ${LDGOLD} \ > + ${LDGOLD_ALTS} \ aw my bad. Thanks for taking care of it. > nm \ > objcopy \ > objdump \ > -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Hi Richard, I think my previous patch has covered this case. git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busybox-alternatives Chen Qi (1): busybox: fix the alternatives logic Is there some problem with the patch? Best Regards, Chen Qi On 09/06/2018 12:52 AM, Richard Purdie wrote: > Currently these are in ${PN} and ${PN}-syslog may get replaced by > other packages but update-alternatives would error in the postinst > if other files were installed first. Avoid the problems by putting > the links in the correct package. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > meta/recipes-core/busybox/busybox.inc | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc > index 586d5342e60..c26ef56bf78 100644 > --- a/meta/recipes-core/busybox/busybox.inc > +++ b/meta/recipes-core/busybox/busybox.inc > @@ -367,7 +367,10 @@ python do_package_prepend () { > # Match coreutils > if alt_name == '[': > alt_name = 'lbracket' > - d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name) > + if alt_name == 'klogd' or alt_name == 'syslogd': > + d.appendVar('ALTERNATIVE_%s-syslog' % (pn), ' ' + alt_name) > + else: > + d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name) > d.setVarFlag('ALTERNATIVE_LINK_NAME', alt_name, alt_link_name) > if os.path.exists('%s%s' % (dvar, target)): > d.setVarFlag('ALTERNATIVE_TARGET', alt_name, target) -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Hi RP, On 09/06/2018 12:52 AM, Richard Purdie wrote: > From: Ross Burton <ross.burton@intel.com> > > There is no good reason to loop 10 times on the test since the profile > optimisation won't change between runs, we don't need/want a statistical > average. This is just burning cpu cycles, just make 1 run. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > meta/recipes-devtools/python/python3_3.5.5.bb | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-devtools/python/python3_3.5.5.bb b/meta/recipes-devtools/python/python3_3.5.5.bb > index bfe91fb1c71..8afe15eee6a 100644 > --- a/meta/recipes-devtools/python/python3_3.5.5.bb > +++ b/meta/recipes-devtools/python/python3_3.5.5.bb > @@ -74,7 +74,7 @@ TARGET_CC_ARCH += "-DNDEBUG -fno-inline" > SDK_CC_ARCH += "-DNDEBUG -fno-inline" > EXTRA_OEMAKE += "CROSS_COMPILE=yes" > EXTRA_OECONF += "CROSSPYTHONPATH=${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/ --without-ensurepip --enable-optimizations" > -PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 10" > +PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 1" What's the fault if I don't run it, please ? This is very slow, here are some rough thoughts to speed it up: 1) Make it optional to run, and default to no if there isn't any faults without running it. 2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried it, it can reduce form 12m -> 4m, saved 8 mins: * Without kvm $ bitbake python3 -ccleansstate && time bitbake python3 real 12m28.070s user 0m1.316s sys 0m0.184s * With kvm: real 3m53.781s user 0m1.080s sys 0m0.220s We may need a oe-core/scripts/qemu-kvm-check to check strictly whether kvm can be used or not. Please feel free to give your comments. // Robert > > export CROSS_COMPILE = "${TARGET_PREFIX}" > export _PYTHON_PROJECT_BASE = "${B}" > -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Thu, 2018-09-06 at 16:15 +0800, Robert Yang wrote: > What's the fault if I don't run it, please ? This is very slow, here > are some rough thoughts to speed it up: > > 1) Make it optional to run, and default to no if there isn't any > faults without running it. Without running it, python is not profile optimised and runs slower. > 2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried > it, it can > reduce form 12m -> 4m, saved 8 mins: > * Without kvm > $ bitbake python3 -ccleansstate && time bitbake python3 > real 12m28.070s > user 0m1.316s > sys 0m0.184s > > * With kvm: > real 3m53.781s > user 0m1.080s > sys 0m0.220s > > We may need a oe-core/scripts/qemu-kvm-check to check strictly > whether kvm > can be used or not. > > Please feel free to give your comments. Ross has some patches coming in this area which should improve things a lot so I'd suggest we wait for those. I merged the n10 -> n1 change simply because it was an easy fix to same some time whilst the other patches a debugged. Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Thu, 2018-09-06 at 10:11 +0800, ChenQi wrote: > Hi Richard, > > I think my previous patch has covered this case. > > git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb > ox-alternatives > > Chen Qi (1): > busybox: fix the alternatives logic > > > Is there some problem with the patch? Sorry, I thought I'd replied about this. There were build failures from this patch and I had to drop it from the test branch. I'm afraid with all the build failures we're had recently I'm struggling to find which failures those were... I merged my simpler but less complete fix simply to get patches moving as I'd assumed this one was being debugged. I guess we need the failures for that though... Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 09/06/2018 04:17 PM, richard.purdie@linuxfoundation.org wrote: > On Thu, 2018-09-06 at 16:15 +0800, Robert Yang wrote: >> What's the fault if I don't run it, please ? This is very slow, here >> are some rough thoughts to speed it up: >> >> 1) Make it optional to run, and default to no if there isn't any >> faults without running it. > > Without running it, python is not profile optimised and runs slower. > >> 2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried >> it, it can >> reduce form 12m -> 4m, saved 8 mins: >> * Without kvm >> $ bitbake python3 -ccleansstate && time bitbake python3 >> real 12m28.070s >> user 0m1.316s >> sys 0m0.184s >> >> * With kvm: >> real 3m53.781s >> user 0m1.080s >> sys 0m0.220s >> >> We may need a oe-core/scripts/qemu-kvm-check to check strictly >> whether kvm >> can be used or not. >> >> Please feel free to give your comments. > > Ross has some patches coming in this area which should improve things a > lot so I'd suggest we wait for those. I merged the n10 -> n1 change > simply because it was an easy fix to same some time whilst the other > patches a debugged. Sounds great, thanks. // Robert > > Cheers, > > Richard > -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 09/06/2018 04:19 PM, richard.purdie@linuxfoundation.org wrote: > On Thu, 2018-09-06 at 10:11 +0800, ChenQi wrote: >> Hi Richard, >> >> I think my previous patch has covered this case. >> >> git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives >> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb >> ox-alternatives >> >> Chen Qi (1): >> busybox: fix the alternatives logic >> >> >> Is there some problem with the patch? > Sorry, I thought I'd replied about this. There were build failures from > this patch and I had to drop it from the test branch. I'm afraid with > all the build failures we're had recently I'm struggling to find which > failures those were... > > I merged my simpler but less complete fix simply to get patches moving > as I'd assumed this one was being debugged. I guess we need the > failures for that though... > > Cheers, > > Richard > Got it. I just looked at some build failures, the ones that might be related are those of testimage failures complaining 'No syslog daemon'. I'll check things locally first. Best Regards, Chen Qi -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On Thu, 2018-09-06 at 17:04 +0800, ChenQi wrote: > On 09/06/2018 04:19 PM, richard.purdie@linuxfoundation.org wrote: > > On Thu, 2018-09-06 at 10:11 +0800, ChenQi wrote: > > > Hi Richard, > > > > > > I think my previous patch has covered this case. > > > > > > git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives > > > http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb > > > ox-alternatives > > > > > > Chen Qi (1): > > > busybox: fix the alternatives logic > > > > > > > > > Is there some problem with the patch? > > > > Sorry, I thought I'd replied about this. There were build failures from > > this patch and I had to drop it from the test branch. I'm afraid with > > all the build failures we're had recently I'm struggling to find which > > failures those were... > > > > I merged my simpler but less complete fix simply to get patches moving > > as I'd assumed this one was being debugged. I guess we need the > > failures for that though... > > > > Cheers, > > > > Richard > > > > Got it. > I just looked at some build failures, the ones that might be related are > those of testimage failures complaining 'No syslog daemon'. > I'll check things locally first. It seems that we have problems in master even with my version of the patch. Its not a deterministic problem, it depends upon rpm/dnf package installation order. The dnf.DnfRepoTest.test_dnf_installroot image test does: dnf --repofrompath=oe-testimage-repo-i586,http://192.168.7.7:36895/i586 --repofrompath=oe-testimage-repo-qemux86,http://192.168.7.7:36895/qemux86 --repofrompath=oe-testimage-repo-noarch,http://192.168.7.7:36895/noarch --nogpgcheck install --installroot=/home/root/chroot/test -v -y --rpmverbosity=debug busybox run-postinst which figures out: ================================================================================ | Package Arch Version Repository Size | ================================================================================ | Installing: | busybox i586 1.29.2-r0.0 oe-testimage-repo-i586 352 k | run-postinsts noarch 1.0-r10.0 oe-testimage-repo-noarch 8.7 k | Installing dependencies: | libc6 i586 2.28-r0.0 oe-testimage-repo-i586 1.3 M | update-alternatives-opkg i586 0.3.6-r0.0 oe-testimage-repo-i586 8.5 k | Installing weak dependencies: | busybox-syslog i586 1.29.2-r0.0 oe-testimage-repo-i586 10 k | busybox-udhcpc i586 1.29.2-r0.0 oe-testimage-repo-i586 8.1 k | so far so good. It then tries to install busybox-syslog *before* busybox. During the postinstall of busybox-syslog we see: update-alternatives --install /sbin/klogd klogd /bin/busybox.nosuid 50 | /usr/bin/update-alternatives: line 145: sed: command not found | /usr/bin/update-alternatives: line 145: echo: write error: Broken pipe which makes sense as sed comes from busybox which hasn't installed yet. I added: RDEPENDS_${PN}-syslog = "busybox" to the busybox recipe but this doesn't seem to help. I see in the busybox.spec file for the -syslog package: Requires: /bin/sh Requires: busybox Requires: update-alternatives-opkg Requires(post): /bin/sh Requires(post): busybox Requires(post): update-alternatives-opkg but it still tries to install busybox-syslog *before* busybox. (busybox only RRECOMMENDS busybox-syslog). The autobuilder with the above RDEPENDS added also shows this. Interestingly this test doesn't fail, its the parselogs test later which fails due to the postinstall errors being detected in the dnf log files. This would seem to be an rpm/dnf bug :(. Sometimes we don't see the error as if it chooses to install busybox first, things work ok. THis is currently breaking builds quite badly on master :(. Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Just to follow up, I'm trying separating out busybox-syslog's recommends from busybox into the higher level packagegroups. The patch is in -next and will run tests overnight. Doing this is probably good anyway but it does feel like we're working around an rpm/dnf issue. I'm open to better ideas... Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
On 9/6/18 5:51 PM, Richard Purdie wrote: > Just to follow up, I'm trying separating out busybox-syslog's > recommends from busybox into the higher level packagegroups. The patch > is in -next and will run tests overnight. > > Doing this is probably good anyway but it does feel like we're working > around an rpm/dnf issue. I'm open to better ideas... Circular dependencies can result in non-deterministic behavior. RPM does not have any difference in dependency resolution (ordering) for depends vs recommends. (The only difference is a recommend is not an error.) Due to this, a circular dependency will result in behavior that may change the order of circular dependent items based on external factors such as file list order, date codes or ??? RPM does (or at least used to) permit you to specify a way to break circular dependencies. But this was a static list of known circular dependencies which you would simply specify in the macros file which had to be installed first. The 'correct' way to break this is only specify ACTUAL dependencies, and realistically there should never be a circular dependency based on a pre-post install action. Otherwise, expect non-deterministic breakage. There is also a way to explicitly set dependencies for the install scriptlets themselves, but I don't know how this influences the ordering of the installation. But this would likely require some special syntax for busybox (or other early load environments) which likely would be worse then a package group. (I don't know how or if DNF handles this differently BTW.) --Mark -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
diff --git a/meta/recipes-devtools/python/python3_3.5.5.bb b/meta/recipes-devtools/python/python3_3.5.5.bb index bfe91fb1c71..8afe15eee6a 100644 --- a/meta/recipes-devtools/python/python3_3.5.5.bb +++ b/meta/recipes-devtools/python/python3_3.5.5.bb @@ -74,7 +74,7 @@ TARGET_CC_ARCH += "-DNDEBUG -fno-inline" SDK_CC_ARCH += "-DNDEBUG -fno-inline" EXTRA_OEMAKE += "CROSS_COMPILE=yes" EXTRA_OECONF += "CROSSPYTHONPATH=${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/ --without-ensurepip --enable-optimizations" -PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 10" +PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 1" export CROSS_COMPILE = "${TARGET_PREFIX}" export _PYTHON_PROJECT_BASE = "${B}"