| Message ID | 1463630733-30902-2-git-send-email-bill.fischofer@linaro.org |
|---|---|
| State | New |
| Headers | show |
On 19 May 2016 at 06:05, Bill Fischofer <bill.fischofer@linaro.org> wrote: > Expand thread section to explain rationale for threads not being ODP > objects and how this relates to addressability considerations for shared > resources. > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > --- > doc/users-guide/users-guide.adoc | 40 > ++++++++++++++++++++++++++++++++++++---- > 1 file changed, 36 insertions(+), 4 deletions(-) > > diff --git a/doc/users-guide/users-guide.adoc > b/doc/users-guide/users-guide.adoc > index 745b85f..fbe745f 100755 > --- a/doc/users-guide/users-guide.adoc > +++ b/doc/users-guide/users-guide.adoc > @@ -288,6 +288,17 @@ _thread mask_ and _scheduler group_ that determine > where they can run and > the type of work that they can handle. These will be discussed in greater > detail later. > > +It is important to note that threads are not ODP objects. This is > intentional. > +The ODP API is designed to abstract objects and functions that are > relevant to > +data plane programming, and that may find accelerated implementation > across > +diverse platforms and architectures. It is not, however, designed to > abstract > +operating system objects and concepts. As a result while ODP _refers_ to > +threads, it does not _define_ threads in terms of their creation or > management > +from an operating system perspective. ODP does provide _helper_ functions > that > +permit Posix threads to be created and managed, however. Applications are > free > +to use these helpers or to manage threads in some other manner of their > +choosing. > + > === Event > Events are what threads process to perform their work. Events can > represent > new work, such as the arrival of a packet that needs to be processed, or > they > @@ -332,6 +343,9 @@ ODP applications if desired. > > Shared memory is represented by handles of abstract type `odp_shm_t`. > > +Note that addressability of shared memory is subject to operating system > +considerations as described in <<Addressing Scope>>. > + > === Buffer > A buffer is a fixed sized block of shared storage that is used by ODP > components and/or applications to realize their function. Buffers contain > @@ -587,10 +601,28 @@ in response to an `odp_queue_deq()` call to get an > event from a queue simply > indicates that the queue is empty. > > === Addressing Scope > -Unless specifically noted in the API, all ODP resources are global to the > ODP > -application, whether it runs as a single process or multiple processes. > ODP > -handles therefore have common meaning within an ODP application but have > no > -meaning outside the scope of the application. > +ODP objects and their associated handles are intended to be global to an > ODP > +instance, _i.e.,_ to the thread that calls `odp_init_global()` and its > +descendants. This is saying that all ODP threads of a given ODP instance are descendant of the ODP instantiation process. I already tried to define ODP threads this way, but it was rejected, and the ODP instance number was added to the API so that any thread/process could join any ODP instance. I still get the feeling that, despite this add-on, we are still saying that a odpthread belongs to the ODP instance it is a descendand of, rather than the one specified in the "instance" parameter passed at odp_local_init() time... The only possible usage of the odp_instance parameter passed at odp_local_init() time would then be on OSes which do not support process hierarchy... Is any such OS known to suport ODP? If not, I thing the odp_instance number should either be suppressed, or clearly stated as meaningless on OS which have process hierarchy, as the process hierarchy seems to take precedence over this parameter in all our discussions. I will post a patch on the glossary just stating that, see if we can at least all agree on that little thing... However, because ODP does not impose a threading model, > +responsibility for achieving this is shared with both the ODP > implementation as > +well as the ODP application. In the odp-linux reference implementation, > for > +example, threads are assumed to be Posix-style threads that all share a > single > *ODP* threads are assumed to be Posix-style threads... This is for monarch only, right? +address space. As a result, all ODP handles and any addresses derived from > +them, are globally sharable throughout the ODP instance. Other > implementations > +may offer the same capabilities or may use a process-based threading > model in > +which individual threads execute in distinct address spaces. > Confused: are talking monarch or some later version? BTW is monarch the name of the API version only or does the "monarch" name also include the linux-generic platform example? > + > +If a platform uses process that follow a Unix-style `fork()` model of > process > +creation, then for best portability ODP applications should be designed so > +that any ODP resource creation (_e.g.,_ pools, shared memory objects, > pktios) > "any ODP resource": That includes everything, e.g. atomics, barrier, timers... as well...right...? +are created during initialization by the thread that creates the ODP > instance > +by calling `odp_init_global()` _before_ any other threads are > +created/launched. This will allow any common objects to be visible in > forked > +processes. > well... yes... But the interesting point is really not mentioned: Either we say: *ALL* odp objects *have to* be created between opd_init_global() and any odp thread creation. That is clear but very restrictive (it would includes everything: for instance, all atomics would have to be known in advance. same for timers or barriers... while it is maybe clear for pools, it is maybe harder to enforce this for these small objects). But at least the definition would be clear... or we say SOME odp objects *have to* be created between opd_init_global() and any odp thread creation, in which case, we have to come with a clear list of which ones we talk about and a precise description of how objects which are not on that list behave, especially when created after fork. This description could of course refer to the OS behavior, but should clearly state what this OS rule applies to (e.g. a how buffer or packet alloc performed after fork() behaves -I guess it depends on when the pool was allocated...). Saying that "most" ODP objects "would possibly" be defined at init time does not really tell much... + > +If a platform uses some other process addressing model, then refer to that > +platform for recommendations to achieve application-desired shared > +addressability. > > === Resources and Names > Many ODP resource objects, such as pools and queues, support an > -- > 2.7.4 > > Thank you for trying, Bill. We need to succeed. I'll keep trying with you. A french writter, Boileau, wrote " Ce qui se conçoit bien, s'énonce clairement". I know you speak some french... I guess this guy would laugh at us right now :-) Christophe
On 19 May 2016 at 03:30, Christophe Milard <christophe.milard@linaro.org> wrote: > On 19 May 2016 at 06:05, Bill Fischofer <bill.fischofer@linaro.org> wrote: > > > Expand thread section to explain rationale for threads not being ODP > > objects and how this relates to addressability considerations for shared > > resources. > > > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > > --- > > doc/users-guide/users-guide.adoc | 40 > > ++++++++++++++++++++++++++++++++++++---- > > 1 file changed, 36 insertions(+), 4 deletions(-) > > > > diff --git a/doc/users-guide/users-guide.adoc > > b/doc/users-guide/users-guide.adoc > > index 745b85f..fbe745f 100755 > > --- a/doc/users-guide/users-guide.adoc > > +++ b/doc/users-guide/users-guide.adoc > > @@ -288,6 +288,17 @@ _thread mask_ and _scheduler group_ that determine > > where they can run and > > the type of work that they can handle. These will be discussed in > greater > > detail later. > > > > +It is important to note that threads are not ODP objects. This is > > intentional. > > +The ODP API is designed to abstract objects and functions that are > > relevant to > > +data plane programming, and that may find accelerated implementation > > across > > +diverse platforms and architectures. It is not, however, designed to > > abstract > > +operating system objects and concepts. As a result while ODP _refers_ to > > +threads, it does not _define_ threads in terms of their creation or > > management > > +from an operating system perspective. ODP does provide _helper_ > functions > > that > > +permit Posix threads to be created and managed, however. Applications > are > > free > > +to use these helpers or to manage threads in some other manner of their > > +choosing. > > + > > === Event > > Events are what threads process to perform their work. Events can > > represent > > new work, such as the arrival of a packet that needs to be processed, or > > they > > @@ -332,6 +343,9 @@ ODP applications if desired. > > > > Shared memory is represented by handles of abstract type `odp_shm_t`. > > > > +Note that addressability of shared memory is subject to operating system > > +considerations as described in <<Addressing Scope>>. > > + > > === Buffer > > A buffer is a fixed sized block of shared storage that is used by ODP > > components and/or applications to realize their function. Buffers > contain > > @@ -587,10 +601,28 @@ in response to an `odp_queue_deq()` call to get an > > event from a queue simply > > indicates that the queue is empty. > > > > === Addressing Scope > > -Unless specifically noted in the API, all ODP resources are global to > the > > ODP > > -application, whether it runs as a single process or multiple processes. > > ODP > > -handles therefore have common meaning within an ODP application but have > > no > > -meaning outside the scope of the application. > > +ODP objects and their associated handles are intended to be global to an > > ODP > > +instance, _i.e.,_ to the thread that calls `odp_init_global()` and its > > +descendants. > > > This is saying that all ODP threads of a given ODP instance are descendant > of the ODP instantiation process. I already tried to define ODP threads > this way, but it was rejected, and the ODP instance number was added to the > API so that any thread/process could join any ODP instance. > I still get the feeling that, despite this add-on, we are still saying that > a odpthread belongs to the ODP instance it is a descendand of, rather than > the one specified in the "instance" parameter passed at odp_local_init() > time... > The only possible usage of the odp_instance parameter passed at > odp_local_init() time would then be on OSes which do not support process > hierarchy... Is any such OS known to suport ODP? > If not, I thing the odp_instance number should either be suppressed, or > clearly stated as meaningless on OS which have process hierarchy, as the > process hierarchy seems to take precedence over this parameter in all our > discussions. > I will post a patch on the glossary just stating that, see if we can at > least all agree on that little thing... > > However, because ODP does not impose a threading model, > > +responsibility for achieving this is shared with both the ODP > > implementation as > > +well as the ODP application. In the odp-linux reference implementation, > > for > > +example, threads are assumed to be Posix-style threads that all share a > > single > > > > *ODP* threads are assumed to be Posix-style threads... > This is for monarch only, right? > > +address space. As a result, all ODP handles and any addresses derived from > > +them, are globally sharable throughout the ODP instance. Other > > implementations > > +may offer the same capabilities or may use a process-based threading > > model in > > +which individual threads execute in distinct address spaces. > > > > Confused: are talking monarch or some later version? BTW is monarch the > name of the API version only or does the "monarch" name also include the > linux-generic platform example? > > > > + > > +If a platform uses process that follow a Unix-style `fork()` model of > > process > > +creation, then for best portability ODP applications should be designed > so > > +that any ODP resource creation (_e.g.,_ pools, shared memory objects, > > pktios) > > > > "any ODP resource": That includes everything, e.g. atomics, barrier, > timers... as well...right...? > > +are created during initialization by the thread that creates the ODP > > instance > > +by calling `odp_init_global()` _before_ any other threads are > > +created/launched. This will allow any common objects to be visible in > > forked > > +processes. > > > > well... yes... But the interesting point is really not mentioned: > > Either we say: > *ALL* odp objects *have to* be created between opd_init_global() and any > odp thread creation. That is clear but very restrictive (it would > includes everything: for instance, all atomics would have to be known in > advance. same for timers or barriers... while it is maybe clear for pools, > it is maybe harder to enforce this for these small objects). But at least > the definition would be clear... > > or we say > SOME odp objects *have to* be created between opd_init_global() and any > odp thread creation, in which case, we have to come with a clear list of > which ones we talk about and a precise description of how objects which are > not on that list behave, especially when created after fork. This > description could of course refer to the OS behavior, but should clearly > state what this OS rule applies to (e.g. a how buffer or packet alloc > performed after fork() behaves -I guess it depends on when the pool was > allocated...). > > Saying that "most" ODP objects "would possibly" be defined at init time > does not really tell much... > > + > > +If a platform uses some other process addressing model, then refer to > that > > +platform for recommendations to achieve application-desired shared > > +addressability. > > > > === Resources and Names > > Many ODP resource objects, such as pools and queues, support an > > -- > > 2.7.4 > > > > > Thank you for trying, Bill. We need to succeed. I'll keep trying with you. > > A french writter, Boileau, wrote " Ce qui se conçoit bien, s'énonce > clairement". I know you speak some french... I guess this guy would laugh > at us right now :-) > What is conceived well is expressed clearly ? > > Christophe > > _______________________________________________ > > lng-odp mailing list > > lng-odp@lists.linaro.org > > https://lists.linaro.org/mailman/listinfo/lng-odp > > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp >
On Thu, May 19, 2016 at 2:30 AM, Christophe Milard < christophe.milard@linaro.org> wrote: > > > On 19 May 2016 at 06:05, Bill Fischofer <bill.fischofer@linaro.org> wrote: > >> Expand thread section to explain rationale for threads not being ODP >> objects and how this relates to addressability considerations for shared >> resources. >> >> Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >> --- >> doc/users-guide/users-guide.adoc | 40 >> ++++++++++++++++++++++++++++++++++++---- >> 1 file changed, 36 insertions(+), 4 deletions(-) >> >> diff --git a/doc/users-guide/users-guide.adoc >> b/doc/users-guide/users-guide.adoc >> index 745b85f..fbe745f 100755 >> --- a/doc/users-guide/users-guide.adoc >> +++ b/doc/users-guide/users-guide.adoc >> @@ -288,6 +288,17 @@ _thread mask_ and _scheduler group_ that determine >> where they can run and >> the type of work that they can handle. These will be discussed in greater >> detail later. >> >> +It is important to note that threads are not ODP objects. This is >> intentional. >> +The ODP API is designed to abstract objects and functions that are >> relevant to >> +data plane programming, and that may find accelerated implementation >> across >> +diverse platforms and architectures. It is not, however, designed to >> abstract >> +operating system objects and concepts. As a result while ODP _refers_ to >> +threads, it does not _define_ threads in terms of their creation or >> management >> +from an operating system perspective. ODP does provide _helper_ >> functions that >> +permit Posix threads to be created and managed, however. Applications >> are free >> +to use these helpers or to manage threads in some other manner of their >> +choosing. >> + >> === Event >> Events are what threads process to perform their work. Events can >> represent >> new work, such as the arrival of a packet that needs to be processed, or >> they >> @@ -332,6 +343,9 @@ ODP applications if desired. >> >> Shared memory is represented by handles of abstract type `odp_shm_t`. >> >> +Note that addressability of shared memory is subject to operating system >> +considerations as described in <<Addressing Scope>>. >> + >> === Buffer >> A buffer is a fixed sized block of shared storage that is used by ODP >> components and/or applications to realize their function. Buffers contain >> @@ -587,10 +601,28 @@ in response to an `odp_queue_deq()` call to get an >> event from a queue simply >> indicates that the queue is empty. >> >> === Addressing Scope >> -Unless specifically noted in the API, all ODP resources are global to >> the ODP >> -application, whether it runs as a single process or multiple processes. >> ODP >> -handles therefore have common meaning within an ODP application but have >> no >> -meaning outside the scope of the application. >> +ODP objects and their associated handles are intended to be global to an >> ODP >> +instance, _i.e.,_ to the thread that calls `odp_init_global()` and its >> +descendants. > > > This is saying that all ODP threads of a given ODP instance are descendant > of the ODP instantiation process. I already tried to define ODP threads > this way, but it was rejected, and the ODP instance number was added to the > API so that any thread/process could join any ODP instance. > We should revisit that because without this assumption I don't think we can define a consistent set of semantics for ODP object visibility. For example, if I create an ODP instance and then pass the instance ID privately to a completely separate program, do we expect it to be able to use that instance ID to join the (remote) ODP instance by calling odp_init_local()? Clearly not. > I still get the feeling that, despite this add-on, we are still saying > that a odpthread belongs to the ODP instance it is a descendand of, rather > than the one specified in the "instance" parameter passed at > odp_local_init() time... > The only possible usage of the odp_instance parameter passed at > odp_local_init() time would then be on OSes which do not support process > hierarchy... Is any such OS known to suport ODP? > I concur. I think it's there more for potential future flexibility than anything else since it's effectively a dummy parameter with a single hard-coded value in odp-linux. I see no harm in it being there, but it's presence doesn't change the fact that ODP instances only cover the odp_global_init() caller and its descendants. > If not, I thing the odp_instance number should either be suppressed, or > clearly stated as meaningless on OS which have process hierarchy, as the > process hierarchy seems to take precedence over this parameter in all our > discussions. > I will post a patch on the glossary just stating that, see if we can at > least all agree on that little thing... > > However, because ODP does not impose a threading model, >> +responsibility for achieving this is shared with both the ODP >> implementation as >> +well as the ODP application. In the odp-linux reference implementation, >> for >> +example, threads are assumed to be Posix-style threads that all share a >> single >> > > *ODP* threads are assumed to be Posix-style threads... > This is for monarch only, right? > That's the only ODP we have at the moment, so yes. > > +address space. As a result, all ODP handles and any addresses derived from >> +them, are globally sharable throughout the ODP instance. Other >> implementations >> +may offer the same capabilities or may use a process-based threading >> model in >> +which individual threads execute in distinct address spaces. >> > > Confused: are talking monarch or some later version? BTW is monarch the > name of the API version only or does the "monarch" name also include the > linux-generic platform example? > Both. Monarch is the current API level and odp-linux implements that level of the ODP API. These statements will still be true of Monarch even after we move on to Tiger Moth, which may or may not introduce additional capabilities. > > >> + >> +If a platform uses process that follow a Unix-style `fork()` model of >> process >> +creation, then for best portability ODP applications should be designed >> so >> +that any ODP resource creation (_e.g.,_ pools, shared memory objects, >> pktios) >> > > "any ODP resource": That includes everything, e.g. atomics, barrier, > timers... as well...right...? > The key word here is creation, not declaration or allocation. Atomics, barriers, and locks are not created, they are declared and initialized, so the caller is controlling the space that they occupy. Similarly packets and timers are allocated from pools that are created, so these objects inherit their visibility from the pools they are drawn from. The requirement is that the pools be created at a point of common addressability for all descendants that reference the pool or similar created object if the objects drawn from them are to be sharable across process boundaries. Queues are created, however they are created from the ODP instance itself, so they automatically have ODP instance-wide visibility. It may be simpler to just enumerate the APIs for which this caveat applies. They are odp_pool_create(), odp_timer_pool_create(), and odp_shm_reserve(). odp_pktio_open() is another case due to the (implied) use of file descriptors. The traffic manager as currently implemented in odp-linux assumes a single address space so we'll have to see what impacts there are there for adding any process mode support, but my preference would be for it to be treated more like queues than pools so that it is implicitly tied directly to the ODP instance. Note that the whole preallocation discussion was to root all of these things to the ODP instance rather than the create() call itself to avoid these subtleties. > > +are created during initialization by the thread that creates the ODP >> instance >> +by calling `odp_init_global()` _before_ any other threads are >> +created/launched. This will allow any common objects to be visible in >> forked >> +processes. >> > > well... yes... But the interesting point is really not mentioned: > > Either we say: > *ALL* odp objects *have to* be created between opd_init_global() and any > odp thread creation. That is clear but very restrictive (it would > includes everything: for instance, all atomics would have to be known in > advance. same for timers or barriers... while it is maybe clear for pools, > it is maybe harder to enforce this for these small objects). But at least > the definition would be clear... > No, for reasons discussed above. There are a very small number of ODP things that fall under this restriction, which I do not believe would be overly onerous for most applications. > > or we say > SOME odp objects *have to* be created between opd_init_global() and any > odp thread creation, in which case, we have to come with a clear list of > which ones we talk about and a precise description of how objects which are > not on that list behave, especially when created after fork. This > description could of course refer to the OS behavior, but should clearly > state what this OS rule applies to (e.g. a how buffer or packet alloc > performed after fork() behaves -I guess it depends on when the pool was > allocated...). > > Saying that "most" ODP objects "would possibly" be defined at init time > does not really tell much... > Agree, we should enumerate the relevant APIs, as I've done above. I can do so in a v2 of the patch. > > + >> +If a platform uses some other process addressing model, then refer to >> that >> +platform for recommendations to achieve application-desired shared >> +addressability. >> >> === Resources and Names >> Many ODP resource objects, such as pools and queues, support an >> -- >> 2.7.4 >> >> > Thank you for trying, Bill. We need to succeed. I'll keep trying with you. > > A french writter, Boileau, wrote " Ce qui se conçoit bien, s'énonce > clairement". I know you speak some french... I guess this guy would laugh > at us right now :-) > > Christophe > > _______________________________________________ >> lng-odp mailing list >> lng-odp@lists.linaro.org >> https://lists.linaro.org/mailman/listinfo/lng-odp >> > >
Am I understanding correctly if I try to rephrase your suggestion, Petri, as follows: 1) " For simplicity all memory should be reserved from the system at global init time.": When calling init_global(), the application provides the total amount of memory it will ever need, including everything (Buffer pools, packet pools, queues...). I guess this means that this amount is either found by try and error when the application is written, or by having some knowledge of the internal ODP implemenation (how much memory different ODP objects actually consume). This provided total amount is also very much ODP implementation dependent as differents implementations may implement some of the ODP objects in HW. 2) The total amount is allocated at global_init() time, hence before any fork(), and any ODP memory allocation that follows (for pools, queues or whatever) is drawn from this initially allocated memory, hence actually guaranteeing sharable pointers over the complete ODP instance 3)" Only restriction would be on if pointers to data can be shared.": The fact that pointers are sharable through the complete ODP instance is hidden from the API which still claims that OS rules apply Is this understanding correct? Christophe. On 23 May 2016 at 09:37, Savolainen, Petri (Nokia - FI/Espoo) < petri.savolainen@nokia.com> wrote: > > > >> > > >> === Addressing Scope > > >> -Unless specifically noted in the API, all ODP resources are global to > > >> the ODP > > >> -application, whether it runs as a single process or multiple > > processes. > > >> ODP > > >> -handles therefore have common meaning within an ODP application but > > have > > >> no > > >> -meaning outside the scope of the application. > > >> +ODP objects and their associated handles are intended to be global to > > an > > >> ODP > > >> +instance, _i.e.,_ to the thread that calls `odp_init_global()` and > its > > >> +descendants. > > > > > > > > > This is saying that all ODP threads of a given ODP instance are > > descendant > > > of the ODP instantiation process. I already tried to define ODP threads > > > this way, but it was rejected, and the ODP instance number was added to > > the > > > API so that any thread/process could join any ODP instance. > > > > > > > We should revisit that because without this assumption I don't think we > > can > > define a consistent set of semantics for ODP object visibility. For > > example, if I create an ODP instance and then pass the instance ID > > privately to a completely separate program, do we expect it to be able to > > use that instance ID to join the (remote) ODP instance by calling > > odp_init_local()? Clearly not. > > > > Yes, if the implementation supports that. E.g. there's no processes or > pthreads in when an SDK runs in bare metal mode. Each cpu starts from C lib > / SDK startup code and stops at some point to wait others. There's some > shared memory set up at init phase. You could start up e.g. two separate > ODP instances of ODP, so that half of the cpus joins the first instance and > the other half joins the other instance. > > odp-linux would not support (at least at first) non-related processes to > form an instance. But API would not limit, if support for it would be added > later. > > > > > > > I still get the feeling that, despite this add-on, we are still saying > > > that a odpthread belongs to the ODP instance it is a descendand of, > > rather > > > than the one specified in the "instance" parameter passed at > > > odp_local_init() time... > > > The only possible usage of the odp_instance parameter passed at > > > odp_local_init() time would then be on OSes which do not support > process > > > hierarchy... Is any such OS known to suport ODP? > > > > > > > I concur. I think it's there more for potential future flexibility than > > anything else since it's effectively a dummy parameter with a single > > hard-coded value in odp-linux. I see no harm in it being there, but it's > > presence doesn't change the fact that ODP instances only cover the > > odp_global_init() caller and its descendants. > > > This is the point. Instance ID enables various thread setups (for linux, > RTOS, bare metal, ...), odp-linux implements some of those. > > > > > > > > > If not, I thing the odp_instance number should either be suppressed, or > > > clearly stated as meaningless on OS which have process hierarchy, as > the > > > process hierarchy seems to take precedence over this parameter in all > > our > > > discussions. > > > An implementation may use the instance ID any way it likes. It may carry a > file name, file descriptor, index to shared memory, magic number, ... etc > information to find the instance or just error check that global_init was > indeed called before local_init was called. No need to state it meaningless > for any implementation. > > > > > > > > Confused: are talking monarch or some later version? BTW is monarch the > > > name of the API version only or does the "monarch" name also include > the > > > linux-generic platform example? > > > > > > > Both. Monarch is the current API level and odp-linux implements that > level > > of the ODP API. These statements will still be true of Monarch even after > > we move on to Tiger Moth, which may or may not introduce additional > > capabilities. > > > > Monarch is API level and various implementations will be maintained for it > for "a long time". E.g. during that time, odp-linux will be optimized > further, new pktios may be added, process mode support will be added, new > implementation specific configuration file / parameters may be introduced, > ... API remains the same but implementations may evolve. > > > > > > > > > > > > > > >> + > > >> +If a platform uses process that follow a Unix-style `fork()` model of > > >> process > > >> +creation, then for best portability ODP applications should be > > designed > > >> so > > >> +that any ODP resource creation (_e.g.,_ pools, shared memory objects, > > >> pktios) > > >> > > > > > > "any ODP resource": That includes everything, e.g. atomics, barrier, > > > timers... as well...right...? > > > > > > > The key word here is creation, not declaration or allocation. Atomics, > > barriers, and locks are not created, they are declared and initialized, > so > > the caller is controlling the space that they occupy. Similarly packets > > and timers are allocated from pools that are created, so these objects > > inherit their visibility from the pools they are drawn from. The > > requirement is that the pools be created at a point of common > > addressability for all descendants that reference the pool or similar > > created object if the objects drawn from them are to be sharable across > > process boundaries. > > > > Queues are created, however they are created from the ODP instance > itself, > > so they automatically have ODP instance-wide visibility. It may be > simpler > > to just enumerate the APIs for which this caveat applies. They are > > odp_pool_create(), odp_timer_pool_create(), and odp_shm_reserve(). > > odp_pktio_open() is another case due to the (implied) use of file > > descriptors. The traffic manager as currently implemented in odp-linux > > assumes a single address space so we'll have to see what impacts there > are > > there for adding any process mode support, but my preference would be for > > it to be treated more like queues than pools so that it is implicitly > tied > > directly to the ODP instance. > > > > Note that the whole preallocation discussion was to root all of these > > things to the ODP instance rather than the create() call itself to avoid > > these subtleties. > > > > > Any allocated object may consume lots of shared memory. E.g. 1B queues (or > timers/timeouts or tm queues or classification rules or ...) would consume > gigabytes of ODP internal memory, and in that perspective a 100MB pool > creation would be in significant addition to the total shared memory > consumption - and memory for that pool could be very well included into > ODP instance itself. So, there's no obvious preference order for odp-linux > to state which objects are created at init time and which dynamically. For > simplicity all memory should be reserved from the system at global init > time. > > For application all handles should be sharable, no matter in which order > the objects were created. Only restriction would be on if pointers to data > can be shared. > > For example, two processes could > * always share a packet handle > * but in odp-linux, could share pointer to packet data (return value of > odp_packet_data()) only if the packet pool was created before both of those > processes were forked (from a common ancestor) > > > -Petri > > >
diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc index 745b85f..fbe745f 100755 --- a/doc/users-guide/users-guide.adoc +++ b/doc/users-guide/users-guide.adoc @@ -288,6 +288,17 @@ _thread mask_ and _scheduler group_ that determine where they can run and the type of work that they can handle. These will be discussed in greater detail later. +It is important to note that threads are not ODP objects. This is intentional. +The ODP API is designed to abstract objects and functions that are relevant to +data plane programming, and that may find accelerated implementation across +diverse platforms and architectures. It is not, however, designed to abstract +operating system objects and concepts. As a result while ODP _refers_ to +threads, it does not _define_ threads in terms of their creation or management +from an operating system perspective. ODP does provide _helper_ functions that +permit Posix threads to be created and managed, however. Applications are free +to use these helpers or to manage threads in some other manner of their +choosing. + === Event Events are what threads process to perform their work. Events can represent new work, such as the arrival of a packet that needs to be processed, or they @@ -332,6 +343,9 @@ ODP applications if desired. Shared memory is represented by handles of abstract type `odp_shm_t`. +Note that addressability of shared memory is subject to operating system +considerations as described in <<Addressing Scope>>. + === Buffer A buffer is a fixed sized block of shared storage that is used by ODP components and/or applications to realize their function. Buffers contain @@ -587,10 +601,28 @@ in response to an `odp_queue_deq()` call to get an event from a queue simply indicates that the queue is empty. === Addressing Scope -Unless specifically noted in the API, all ODP resources are global to the ODP -application, whether it runs as a single process or multiple processes. ODP -handles therefore have common meaning within an ODP application but have no -meaning outside the scope of the application. +ODP objects and their associated handles are intended to be global to an ODP +instance, _i.e.,_ to the thread that calls `odp_init_global()` and its +descendants. However, because ODP does not impose a threading model, +responsibility for achieving this is shared with both the ODP implementation as +well as the ODP application. In the odp-linux reference implementation, for +example, threads are assumed to be Posix-style threads that all share a single +address space. As a result, all ODP handles and any addresses derived from +them, are globally sharable throughout the ODP instance. Other implementations +may offer the same capabilities or may use a process-based threading model in +which individual threads execute in distinct address spaces. + +If a platform uses process that follow a Unix-style `fork()` model of process +creation, then for best portability ODP applications should be designed so +that any ODP resource creation (_e.g.,_ pools, shared memory objects, pktios) +are created during initialization by the thread that creates the ODP instance +by calling `odp_init_global()` _before_ any other threads are +created/launched. This will allow any common objects to be visible in forked +processes. + +If a platform uses some other process addressing model, then refer to that +platform for recommendations to achieve application-desired shared +addressability. === Resources and Names Many ODP resource objects, such as pools and queues, support an
Expand thread section to explain rationale for threads not being ODP objects and how this relates to addressability considerations for shared resources. Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> --- doc/users-guide/users-guide.adoc | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-)