[3/4] kern-tools: fix processing for no branch meta-data

Message ID 78eef00e-4156-bdbf-cda1-688ae0495414@windriver.com
State New
Headers show

Commit Message

Bruce Ashfield Dec. 7, 2016, 4:18 p.m.
On 2016-12-07 10:11 AM, Trevor Woerner wrote:
> This patch does fix the raspberrypi issue, and it has been pushed, thankfully.

>

> But I'm still seeing hard-coded filenames in /tmp that are messing up

> the ability for more than one person to use a given build machine.

> E.g.:

>

> /tmp/patch.standard.arm-versatile-926ejs.queue

> /tmp/patch.standard.mti-malta32.queue

> /tmp/patch.standard.mti-malta64.queue

> /tmp/patch.standard.preempt-rt.base.queue

> /tmp/patch.standard.preempt-rt.queue

> /tmp/patch.standard.queue

> /tmp/patch.unused.queue

>

> I think the malta ones are used for qemumips64, which is still failing

> for this reason (multiple users)


I tracked this down to some of the sub commands using the dirname
of the global patch queue .. and since that is in /tmp/, they are
following along.

With the attached patch, I see nothing else that is named  in /tmp/

If you have the cycles, can you give it a try and let me know ?

Bruce

>

> On Tue, Dec 6, 2016 at 6:02 PM, Paul Barker <paul@paulbarker.me.uk> wrote:

>> On Tue, 06 Dec 2016 22:14:41 +0100

>> Patrick Ohly <patrick.ohly@intel.com> wrote:

>>

>>> On Fri, 2016-12-02 at 16:09 -0500, Bruce Ashfield wrote:

>>>> Lernel meta-data that has patches, but no branches, can trigger an

>>>> error due to no branch specific patch queue.

>>>>

>>>> This error then cascades to more issues since the tools are using

>>>> a named file in /tmp to store and display error messages to the

>>>> user.

>>>>

>>>> We fix both issues though the following kern tools tweaks:

>>>>

>>>>   commit bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c

>>>>   Author: Bruce Ashfield <bruce.ashfield@windriver.com>

>>>>   Date:   Fri Dec 2 13:09:40 2016 -0500

>>>>

>>>>     scc: use mktemp for consolidated output capture

>>>>

>>>>     To provide useful error messages the tools dump pre-processed

>>>>     files and messages to a temporary file. If multiple users are

>>>>     doing builds, this means they either race, or can have permissions

>>>>     issues.

>>>>

>>>>     By creating the temporary file via mktemp, we avoid both issues.

>>>>     (We also make sure to clean these up on exit, or /tmp will get

>>>>     polluted quickly).

>>>>

>>>>   commit a287da4bfe0b4acb8f2b0627bd8e7abd1a1dde26

>>>>   Author: Bruce Ashfield <bruce.ashfield@windriver.com>

>>>>   Date:   Fri Dec 2 13:08:08 2016 -0500

>>>>

>>>>     patch: do not assume a branch specific patch queue is needed

>>>>

>>>>     When processing input files per-branch and global patch queues are

>>>>     generated. If the meta-data has not created any branches in the

>>>>     repo, no branch specific queue is required.

>>>>

>>>>     The tools assumed that one is always valid, and hence would throw a

>>>>     non-zero exit code and stop processing.

>>>>

>>>>     By testing for a named per-branch queue, we avoid this issue.

>>>

>>> Ostro OS runs into the problem while trying to use current OE-core

>>> master:

>>>

>>>  .../patch.cmd: line 29: : No such file or directory

>>>

>>> | ERROR: Function failed: do_kernel_metadata (log file is located ...)

>>>

>>> This commit here fixed it for me. I see that it is already in Ross' mut2

>>> branch, so hopefully that'll land in master soon.

>>>

>>

>> Ditto for meta-raspberrypi, the kernel doesn't build with current

>> oe-core master as discussed on the yocto@ list. This patch is needed

>> to fix things.

>>

>> Thanks,

>> Paul Barker

>> --

>> _______________________________________________

>> Openembedded-core mailing list

>> Openembedded-core@lists.openembedded.org

>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Comments

Trevor Woerner Dec. 7, 2016, 6:05 p.m. | #1
On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield
<bruce.ashfield@windriver.com> wrote:
> With the attached patch, I see nothing else that is named  in /tmp/

>

> If you have the cycles, can you give it a try and let me know ?


Yes, I'm giving it a whirl right now. Thanks!
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Trevor Woerner Dec. 7, 2016, 11:50 p.m. | #2
On Wed, Dec 7, 2016 at 1:05 PM, Trevor Woerner <twoerner@gmail.com> wrote:
> On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield

> <bruce.ashfield@windriver.com> wrote:

>> With the attached patch, I see nothing else that is named  in /tmp/

>>

>> If you have the cycles, can you give it a try and let me know ?

>

> Yes, I'm giving it a whirl right now. Thanks!


That patch looks good, how soon can it land? ;-)

I'm sure there's still something in the build that is creating
temporary files in /tmp. I'm not sure which process is doing it, I
have no reason to suspect it's the kernel tools (in fact I think it
happens too early in the build to be the kernel tools), and the tmp
files that are created are named with temporary names (e.g.
/tmp/tmp.2wavbhTlDU) so they shouldn't interfere with multiple users
building on the same machine. Each build appears to create two such
temp files that aren't cleaned up at the end of the build. Not a big
deal. I don't think it's something new that was introduced recently
(but I'd have to do more investigation to verify).
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Bruce Ashfield Dec. 8, 2016, 3:23 a.m. | #3
On 2016-12-07 6:50 PM, Trevor Woerner wrote:
> On Wed, Dec 7, 2016 at 1:05 PM, Trevor Woerner <twoerner@gmail.com> wrote:

>> On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield

>> <bruce.ashfield@windriver.com> wrote:

>>> With the attached patch, I see nothing else that is named  in /tmp/

>>>

>>> If you have the cycles, can you give it a try and let me know ?

>>

>> Yes, I'm giving it a whirl right now. Thanks!

>

> That patch looks good, how soon can it land? ;-)


For my part, I'll send it first thing tomorrow along with some version
bumps to the kernel.

>

> I'm sure there's still something in the build that is creating

> temporary files in /tmp. I'm not sure which process is doing it, I

> have no reason to suspect it's the kernel tools (in fact I think it

> happens too early in the build to be the kernel tools), and the tmp

> files that are created are named with temporary names (e.g.

> /tmp/tmp.2wavbhTlDU) so they shouldn't interfere with multiple users

> building on the same machine. Each build appears to create two such

> temp files that aren't cleaned up at the end of the build. Not a big

> deal. I don't think it's something new that was introduced recently

> (but I'd have to do more investigation to verify).


I noticed one mktemp file leaking when I was fixing the bug at hand
today. I made a note to loop back and have a look for where an early
exit is skipping clean up.

If it is the tools, I'll fix it along with the patch I did today.

Bruce

>


-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Bruce Ashfield Dec. 8, 2016, 3:27 p.m. | #4
On 2016-12-07 06:50 PM, Trevor Woerner wrote:
> On Wed, Dec 7, 2016 at 1:05 PM, Trevor Woerner <twoerner@gmail.com> wrote:

>> On Wed, Dec 7, 2016 at 11:18 AM, Bruce Ashfield

>> <bruce.ashfield@windriver.com> wrote:

>>> With the attached patch, I see nothing else that is named  in /tmp/

>>>

>>> If you have the cycles, can you give it a try and let me know ?

>>

>> Yes, I'm giving it a whirl right now. Thanks!

>

> That patch looks good, how soon can it land? ;-)

>

> I'm sure there's still something in the build that is creating

> temporary files in /tmp. I'm not sure which process is doing it, I

> have no reason to suspect it's the kernel tools (in fact I think it

> happens too early in the build to be the kernel tools), and the tmp

> files that are created are named with temporary names (e.g.

> /tmp/tmp.2wavbhTlDU) so they shouldn't interfere with multiple users

> building on the same machine. Each build appears to create two such

> temp files that aren't cleaned up at the end of the build. Not a big

> deal. I don't think it's something new that was introduced recently

> (but I'd have to do more investigation to verify).


I did find one mktemp file that wasn't being cleaned up. I fixed
it and updated my commit to include that fix.

It'll all be out later today with my kernel version bumps + CVE
series.

Bruce

>


-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Patch

From 707978ff47533c705eac42619988793f07a66b80 Mon Sep 17 00:00:00 2001
From: Bruce Ashfield <bruce.ashfield@windriver.com>
Date: Wed, 7 Dec 2016 11:14:34 -0500
Subject: [PATCH] kern-tools: ensure that no shared directories are used

We need to avoid using shared/common directories for any files that are
part of specific build, since permissions issues in multi user
environments will cause issues.

Integrating the following commit to solve the issue:

   scc: move unused patch queue under output dir

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/recipes-kernel/kern-tools/kern-tools-native_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
index 0f8a786176f9..3ca6ca22ef20 100644
--- a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
+++ b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
@@ -4,7 +4,7 @@  LIC_FILES_CHKSUM = "file://git/tools/kgit;beginline=5;endline=9;md5=a6c2fa8aef1b
 
 DEPENDS = "git-native"
 
-SRCREV = "bd9e1d6c9b0a34ff3e19a06999aaf57ffadfd04c"
+SRCREV = "626ceac135fa66277c2fa53197be33cc9d4d7614"
 PR = "r12"
 PV = "0.2+git${SRCPV}"
 
-- 
2.5.0