| Message ID | 1445961210-13938-1-git-send-email-ivan.khoronzhuk@linaro.org |
|---|---|
| State | New |
| Headers | show |
ping On 27.10.15 17:53, Ivan Khoronzhuk wrote: > The worker cpu in most cases supposed to be used for one thread. > By default all control threads on CPU0, and odp_cpumask_default_control > returns it. It shouldn't overlap with worker cpumask. > > Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> > --- > platform/linux-generic/odp_cpumask_task.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/platform/linux-generic/odp_cpumask_task.c b/platform/linux-generic/odp_cpumask_task.c > index 535891c..227650a 100644 > --- a/platform/linux-generic/odp_cpumask_task.c > +++ b/platform/linux-generic/odp_cpumask_task.c > @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int num) > num = CPU_COUNT(&cpuset); > > /* build the mask, allocating down from highest numbered CPU */ > - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; --i) { > + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; --i) { > if (CPU_ISSET(i, &cpuset)) { > odp_cpumask_set(mask, i); > cpu++; >
There should be some better way to control this, as arbitrarily excluding a core from consideration for worker threads is a significant penalty for small core count environments. I suggest bringing this up during one of the ARCH calls. On Fri, Oct 30, 2015 at 9:05 AM, Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org > wrote: > ping > > On 27.10.15 17:53, Ivan Khoronzhuk wrote: > >> The worker cpu in most cases supposed to be used for one thread. >> By default all control threads on CPU0, and odp_cpumask_default_control >> returns it. It shouldn't overlap with worker cpumask. >> >> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> >> --- >> platform/linux-generic/odp_cpumask_task.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/platform/linux-generic/odp_cpumask_task.c >> b/platform/linux-generic/odp_cpumask_task.c >> index 535891c..227650a 100644 >> --- a/platform/linux-generic/odp_cpumask_task.c >> +++ b/platform/linux-generic/odp_cpumask_task.c >> @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int >> num) >> num = CPU_COUNT(&cpuset); >> >> /* build the mask, allocating down from highest numbered CPU */ >> - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; --i) { >> + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; --i) { >> if (CPU_ISSET(i, &cpuset)) { >> odp_cpumask_set(mask, i); >> cpu++; >> >> > -- > Regards, > Ivan Khoronzhuk > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp >
On 31.10.15 22:48, Bill Fischofer wrote: > There should be some better way to control this, as arbitrarily excluding a core from consideration for worker threads is a significant penalty for small core count environments. I suggest bringing this up during one of the ARCH calls. Yep. There is suggestion to make it dependent on core count. Like if CPU count is more then 2, then exclude core 0. > > On Fri, Oct 30, 2015 at 9:05 AM, Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org <mailto:ivan.khoronzhuk@linaro.org>> wrote: > > ping > > On 27.10.15 17:53, Ivan Khoronzhuk wrote: > > The worker cpu in most cases supposed to be used for one thread. > By default all control threads on CPU0, and odp_cpumask_default_control > returns it. It shouldn't overlap with worker cpumask. > > Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org <mailto:ivan.khoronzhuk@linaro.org>> > --- > platform/linux-generic/odp_cpumask_task.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/platform/linux-generic/odp_cpumask_task.c b/platform/linux-generic/odp_cpumask_task.c > index 535891c..227650a 100644 > --- a/platform/linux-generic/odp_cpumask_task.c > +++ b/platform/linux-generic/odp_cpumask_task.c > @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int num) > num = CPU_COUNT(&cpuset); > > /* build the mask, allocating down from highest numbered CPU */ > - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; --i) { > + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; --i) { > if (CPU_ISSET(i, &cpuset)) { > odp_cpumask_set(mask, i); > cpu++; > > > -- > Regards, > Ivan Khoronzhuk > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org <mailto:lng-odp@lists.linaro.org> > https://lists.linaro.org/mailman/listinfo/lng-odp > >
I'd think that would be better left as an application option. Excluding a core on a 4 core system is a 25% reduction in potential processing capability--not something ODP should be doing unilaterally. On Sat, Oct 31, 2015 at 4:47 PM, Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org > wrote: > > > On 31.10.15 22:48, Bill Fischofer wrote: > >> There should be some better way to control this, as arbitrarily excluding >> a core from consideration for worker threads is a significant penalty for >> small core count environments. I suggest bringing this up during one of >> the ARCH calls. >> > > Yep. There is suggestion to make it dependent on core count. > Like if CPU count is more then 2, then exclude core 0. > > >> On Fri, Oct 30, 2015 at 9:05 AM, Ivan Khoronzhuk < >> ivan.khoronzhuk@linaro.org <mailto:ivan.khoronzhuk@linaro.org>> wrote: >> >> ping >> >> On 27.10.15 17:53, Ivan Khoronzhuk wrote: >> >> The worker cpu in most cases supposed to be used for one thread. >> By default all control threads on CPU0, and >> odp_cpumask_default_control >> returns it. It shouldn't overlap with worker cpumask. >> >> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org >> <mailto:ivan.khoronzhuk@linaro.org>> >> --- >> platform/linux-generic/odp_cpumask_task.c | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/platform/linux-generic/odp_cpumask_task.c >> b/platform/linux-generic/odp_cpumask_task.c >> index 535891c..227650a 100644 >> --- a/platform/linux-generic/odp_cpumask_task.c >> +++ b/platform/linux-generic/odp_cpumask_task.c >> @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t >> *mask, int num) >> num = CPU_COUNT(&cpuset); >> >> /* build the mask, allocating down from highest numbered >> CPU */ >> - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; >> --i) { >> + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; >> --i) { >> if (CPU_ISSET(i, &cpuset)) { >> odp_cpumask_set(mask, i); >> cpu++; >> >> >> -- >> Regards, >> Ivan Khoronzhuk >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org <mailto:lng-odp@lists.linaro.org> >> https://lists.linaro.org/mailman/listinfo/lng-odp >> >> >> > -- > Regards, > Ivan Khoronzhuk >
I think we have few requirements on cores that we could define in the user-guide. A general rule should be that the application knows what it is doing and ODP should not stop poor choices, it can warn though - like we do about hugepages not being present. Here are some things that look sane to me right now:- - If the system is linux then one core must always be the Linux core - usually core 0 I think by current kernel precedent. ODP linux-generic should warn if core 0 is added to the worker / control set since it matters on this platform - The system boots on its OS core(s) and ODP apps are launched on the cores in the worker and control pools - Only the application can know if it wants to put more than one odp worker per core, and accept any penalty that would be seen with current linux because the OS scheduler called, ODP should warn - Control cores should never overlap worker cores, we intent them to have different properties so this might be worth guarding against in ODP, ODP should warn at least, currently they can all be the same core and an error would break how it works today. - Workers and control cores can be put back in the pool when apps are shut down gracefully - not sure about seg-faults - how will we know that HW resources were released for example, there are platforms that cannot assure this - Cores in the worker and control pool can be removed from the pool and given back to the OS - this allows you to revert back into a normal OS controlled reboot situation. - Workers are isolated cores with no other scheduling from an OS - currently I think this means in linux that at boot time you have to specify the potentially isolated set of cores - Control cores :- these are OS scheduled cores (unlike workers) that generally do not overlap the boot linux core(s). Thus linux house keeping is isolated from the control cores which may be making system calls, but don't want to have linux maintenance tasks running on them. - There is no ODP requirement for a resource manager in any given system, but I think the system should have a "manager" we add such a manager to helpers that can track which cores are ODP workers / control and provide the current set of available cores from the worker and control pools. In this way apps being run / closed down don't all assume they have the whole machine. Clearly any given system does not need to do this if they have their own management but I am thinking of a generic server in the future with apps coming and going. - In linux generic I think we should default to leaving core 0 to linux unless there is only one core. Once you go >1 core ODP is launched on the cores in the worker/control pool as appropriate Questions How do we manage these apps when they are in VMs ? Do we have that resource manager above running in the VM as well and it checks with the host resource manager ? After all VMs can be come and go. On 31 October 2015 at 18:52, Bill Fischofer <bill.fischofer@linaro.org> wrote: > I'd think that would be better left as an application option. Excluding a > core on a 4 core system is a 25% reduction in potential processing > capability--not something ODP should be doing unilaterally. > > On Sat, Oct 31, 2015 at 4:47 PM, Ivan Khoronzhuk < > ivan.khoronzhuk@linaro.org> wrote: > >> >> >> On 31.10.15 22:48, Bill Fischofer wrote: >> >>> There should be some better way to control this, as arbitrarily >>> excluding a core from consideration for worker threads is a significant >>> penalty for small core count environments. I suggest bringing this up >>> during one of the ARCH calls. >>> >> >> Yep. There is suggestion to make it dependent on core count. >> Like if CPU count is more then 2, then exclude core 0. >> >> >>> On Fri, Oct 30, 2015 at 9:05 AM, Ivan Khoronzhuk < >>> ivan.khoronzhuk@linaro.org <mailto:ivan.khoronzhuk@linaro.org>> wrote: >>> >>> ping >>> >>> On 27.10.15 17:53, Ivan Khoronzhuk wrote: >>> >>> The worker cpu in most cases supposed to be used for one thread. >>> By default all control threads on CPU0, and >>> odp_cpumask_default_control >>> returns it. It shouldn't overlap with worker cpumask. >>> >>> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org >>> <mailto:ivan.khoronzhuk@linaro.org>> >>> --- >>> platform/linux-generic/odp_cpumask_task.c | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/platform/linux-generic/odp_cpumask_task.c >>> b/platform/linux-generic/odp_cpumask_task.c >>> index 535891c..227650a 100644 >>> --- a/platform/linux-generic/odp_cpumask_task.c >>> +++ b/platform/linux-generic/odp_cpumask_task.c >>> @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t >>> *mask, int num) >>> num = CPU_COUNT(&cpuset); >>> >>> /* build the mask, allocating down from highest >>> numbered CPU */ >>> - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; >>> --i) { >>> + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; >>> --i) { >>> if (CPU_ISSET(i, &cpuset)) { >>> odp_cpumask_set(mask, i); >>> cpu++; >>> >>> >>> -- >>> Regards, >>> Ivan Khoronzhuk >>> >>> _______________________________________________ >>> lng-odp mailing list >>> lng-odp@lists.linaro.org <mailto:lng-odp@lists.linaro.org> >>> https://lists.linaro.org/mailman/listinfo/lng-odp >>> >>> >>> >> -- >> Regards, >> Ivan Khoronzhuk >> > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp > > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
On Sun, Nov 1, 2015 at 9:00 AM, Mike Holmes <mike.holmes@linaro.org> wrote: > I think we have few requirements on cores that we could define in the > user-guide. > > A general rule should be that the application knows what it is doing and > ODP should not stop poor choices, it can warn though - like we do about > hugepages not being present. > > Here are some things that look sane to me right now:- > > - If the system is linux then one core must always be the Linux core - > usually core 0 I think by current kernel precedent. ODP linux-generic > should warn if core 0 is added to the worker / control set since it matters > on this platform > - The system boots on its OS core(s) and ODP apps are launched on the > cores in the worker and control pools > - Only the application can know if it wants to put more than one odp > worker per core, and accept any penalty that would be seen with current > linux because the OS scheduler called, ODP should warn > - Control cores should never overlap worker cores, we intent them to > have different properties so this might be worth guarding against in > ODP, ODP should warn at least, currently they can all be the same core > and an error would break how it works today. > - Workers and control cores can be put back in the pool when apps are > shut down gracefully - not sure about seg-faults - how will we know > that HW resources were released for example, there are platforms that > cannot assure this > - Cores in the worker and control pool can be removed from the pool > and given back to the OS - this allows you to revert back into a normal OS > controlled reboot situation. > - Workers are isolated cores with no other scheduling from an OS - > currently I think this means in linux that at boot time you have to > specify the potentially isolated set of cores > - Control cores :- these are OS scheduled cores (unlike workers) that > generally do not overlap the boot linux core(s). Thus linux house > keeping is isolated from the control cores which may be making system > calls, but don't want to have linux maintenance tasks running on them. > - There is no ODP requirement for a resource manager in any given > system, but I think the system should have a "manager" we add such a > manager to helpers that can track which cores are ODP workers / > control and provide the current set of available cores from the worker and > control pools. In this way apps being run / closed down don't all assume > they have the whole machine. Clearly any given system does not need to do > this if they have their own management but I am thinking of a generic > server in the future with apps coming and going. > - In linux generic I think we should default to leaving core 0 to linux > unless there is only one core. Once you go >1 core ODP is launched on the > cores in the worker/control pool as appropriate > > Questions > > How do we manage these apps when they are in VMs ? Do we have that > resource manager above running in the VM as well and it checks with the > host resource manager ? After all VMs can be come and go. > > > The VM question is very simple. All ODP references to cores refer to virtual cores. The mapping of virtual cores to real cores is a configuration/management issue, the same as mapping threads to (virtual) cores. In a real environment virtual cores are real cores. In a virtual environment, the VNF orchestrator sets this up. The VNF should have some means of knowing how it has been configured so that it can potentially adapt to the resources at it's disposal, so this becomes a question of extending the ODP configuration APIs to permit such queries. > > > > On 31 October 2015 at 18:52, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> I'd think that would be better left as an application option. Excluding >> a core on a 4 core system is a 25% reduction in potential processing >> capability--not something ODP should be doing unilaterally. >> >> On Sat, Oct 31, 2015 at 4:47 PM, Ivan Khoronzhuk < >> ivan.khoronzhuk@linaro.org> wrote: >> >>> >>> >>> On 31.10.15 22:48, Bill Fischofer wrote: >>> >>>> There should be some better way to control this, as arbitrarily >>>> excluding a core from consideration for worker threads is a significant >>>> penalty for small core count environments. I suggest bringing this up >>>> during one of the ARCH calls. >>>> >>> >>> Yep. There is suggestion to make it dependent on core count. >>> Like if CPU count is more then 2, then exclude core 0. >>> >>> >>>> On Fri, Oct 30, 2015 at 9:05 AM, Ivan Khoronzhuk < >>>> ivan.khoronzhuk@linaro.org <mailto:ivan.khoronzhuk@linaro.org>> wrote: >>>> >>>> ping >>>> >>>> On 27.10.15 17:53, Ivan Khoronzhuk wrote: >>>> >>>> The worker cpu in most cases supposed to be used for one thread. >>>> By default all control threads on CPU0, and >>>> odp_cpumask_default_control >>>> returns it. It shouldn't overlap with worker cpumask. >>>> >>>> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org >>>> <mailto:ivan.khoronzhuk@linaro.org>> >>>> --- >>>> platform/linux-generic/odp_cpumask_task.c | 2 +- >>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>> >>>> diff --git a/platform/linux-generic/odp_cpumask_task.c >>>> b/platform/linux-generic/odp_cpumask_task.c >>>> index 535891c..227650a 100644 >>>> --- a/platform/linux-generic/odp_cpumask_task.c >>>> +++ b/platform/linux-generic/odp_cpumask_task.c >>>> @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t >>>> *mask, int num) >>>> num = CPU_COUNT(&cpuset); >>>> >>>> /* build the mask, allocating down from highest >>>> numbered CPU */ >>>> - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; >>>> --i) { >>>> + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; >>>> --i) { >>>> if (CPU_ISSET(i, &cpuset)) { >>>> odp_cpumask_set(mask, i); >>>> cpu++; >>>> >>>> >>>> -- >>>> Regards, >>>> Ivan Khoronzhuk >>>> >>>> _______________________________________________ >>>> lng-odp mailing list >>>> lng-odp@lists.linaro.org <mailto:lng-odp@lists.linaro.org> >>>> https://lists.linaro.org/mailman/listinfo/lng-odp >>>> >>>> >>>> >>> -- >>> Regards, >>> Ivan Khoronzhuk >>> >> >> >> _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/lng-odp >> >> > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs > > >
diff --git a/platform/linux-generic/odp_cpumask_task.c b/platform/linux-generic/odp_cpumask_task.c index 535891c..227650a 100644 --- a/platform/linux-generic/odp_cpumask_task.c +++ b/platform/linux-generic/odp_cpumask_task.c @@ -33,7 +33,7 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int num) num = CPU_COUNT(&cpuset); /* build the mask, allocating down from highest numbered CPU */ - for (cpu = 0, i = CPU_SETSIZE - 1; i >= 0 && cpu < num; --i) { + for (cpu = 0, i = CPU_SETSIZE - 1; i > 0 && cpu < num; --i) { if (CPU_ISSET(i, &cpuset)) { odp_cpumask_set(mask, i); cpu++;
The worker cpu in most cases supposed to be used for one thread. By default all control threads on CPU0, and odp_cpumask_default_control returns it. It shouldn't overlap with worker cpumask. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- platform/linux-generic/odp_cpumask_task.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)