| Message ID | 20230207051105.11575-16-ricardo.neri-calderon@linux.intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | sched: Introduce classes of tasks for load balance | expand |
On Tue, Feb 7, 2023 at 6:02 AM Ricardo Neri <ricardo.neri-calderon@linux.intel.com> wrote: > > Implement the arch_get_ipcc_score() interface of the scheduler. Use the > performance capabilities of the extended Hardware Feedback Interface table > as the IPC score. > > Cc: Ben Segall <bsegall@google.com> > Cc: Daniel Bristot de Oliveira <bristot@redhat.com> > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > Cc: Ionela Voinescu <ionela.voinescu@arm.com> > Cc: Joel Fernandes (Google) <joel@joelfernandes.org> > Cc: Len Brown <len.brown@intel.com> > Cc: Lukasz Luba <lukasz.luba@arm.com> > Cc: Mel Gorman <mgorman@suse.de> > Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > Cc: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> > Cc: Steven Rostedt <rostedt@goodmis.org> > Cc: Tim C. Chen <tim.c.chen@intel.com> > Cc: Valentin Schneider <vschneid@redhat.com> > Cc: x86@kernel.org > Cc: linux-pm@vger.kernel.org > Cc: linux-kernel@vger.kernel.org > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> > --- > Changes since v2: > * None > > Changes since v1: > * Adjusted the returned HFI class (which starts at 0) to match the > scheduler IPCC class (which starts at 1). (PeterZ) > * Used the new interface names. > --- > arch/x86/include/asm/topology.h | 2 ++ > drivers/thermal/intel/intel_hfi.c | 27 +++++++++++++++++++++++++++ > 2 files changed, 29 insertions(+) > > diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h > index ffcdac3f398f..c4fcd9c3c634 100644 > --- a/arch/x86/include/asm/topology.h > +++ b/arch/x86/include/asm/topology.h > @@ -229,8 +229,10 @@ void init_freq_invariance_cppc(void); > > #if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) > void intel_hfi_update_ipcc(struct task_struct *curr); > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu); > > #define arch_update_ipcc intel_hfi_update_ipcc > +#define arch_get_ipcc_score intel_hfi_get_ipcc_score > #endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ > > #endif /* _ASM_X86_TOPOLOGY_H */ > diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c > index 530dcf57e06e..fa9b4a678d92 100644 > --- a/drivers/thermal/intel/intel_hfi.c > +++ b/drivers/thermal/intel/intel_hfi.c > @@ -206,6 +206,33 @@ void intel_hfi_update_ipcc(struct task_struct *curr) > curr->ipcc = msr.split.classid + 1; > } > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) > +{ > + unsigned short hfi_class; It looks like the variable above is only used to save a subtraction or addition of 1 to something going forward. > + int *scores; > + > + if (cpu < 0 || cpu >= nr_cpu_ids) > + return -EINVAL; > + > + if (ipcc == IPC_CLASS_UNCLASSIFIED) > + return -EINVAL; > + > + /* > + * Scheduler IPC classes start at 1. HFI classes start at 0. > + * See note intel_hfi_update_ipcc(). > + */ > + hfi_class = ipcc - 1; > + > + if (hfi_class >= hfi_features.nr_classes) Personally, I would do if (ipcc >= hfi_features.nr_classes + 1) here and -> > + return -EINVAL; > + > + scores = per_cpu_ptr(hfi_ipcc_scores, cpu); > + if (!scores) > + return -ENODEV; > + -> scores[ipcc - 1] below. > + return READ_ONCE(scores[hfi_class]); And why does this need to use READ_ONCE()? > +} > + > static int alloc_hfi_ipcc_scores(void) > { > if (!cpu_feature_enabled(X86_FEATURE_ITD)) > --
On Mon, Mar 27, 2023 at 06:50:13PM +0200, Rafael J. Wysocki wrote: > On Tue, Feb 7, 2023 at 6:02 AM Ricardo Neri > <ricardo.neri-calderon@linux.intel.com> wrote: > > > > Implement the arch_get_ipcc_score() interface of the scheduler. Use the > > performance capabilities of the extended Hardware Feedback Interface table > > as the IPC score. > > > > Cc: Ben Segall <bsegall@google.com> > > Cc: Daniel Bristot de Oliveira <bristot@redhat.com> > > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > > Cc: Ionela Voinescu <ionela.voinescu@arm.com> > > Cc: Joel Fernandes (Google) <joel@joelfernandes.org> > > Cc: Len Brown <len.brown@intel.com> > > Cc: Lukasz Luba <lukasz.luba@arm.com> > > Cc: Mel Gorman <mgorman@suse.de> > > Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > Cc: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> > > Cc: Steven Rostedt <rostedt@goodmis.org> > > Cc: Tim C. Chen <tim.c.chen@intel.com> > > Cc: Valentin Schneider <vschneid@redhat.com> > > Cc: x86@kernel.org > > Cc: linux-pm@vger.kernel.org > > Cc: linux-kernel@vger.kernel.org > > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> > > --- > > Changes since v2: > > * None > > > > Changes since v1: > > * Adjusted the returned HFI class (which starts at 0) to match the > > scheduler IPCC class (which starts at 1). (PeterZ) > > * Used the new interface names. > > --- > > arch/x86/include/asm/topology.h | 2 ++ > > drivers/thermal/intel/intel_hfi.c | 27 +++++++++++++++++++++++++++ > > 2 files changed, 29 insertions(+) > > > > diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h > > index ffcdac3f398f..c4fcd9c3c634 100644 > > --- a/arch/x86/include/asm/topology.h > > +++ b/arch/x86/include/asm/topology.h > > @@ -229,8 +229,10 @@ void init_freq_invariance_cppc(void); > > > > #if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) > > void intel_hfi_update_ipcc(struct task_struct *curr); > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu); > > > > #define arch_update_ipcc intel_hfi_update_ipcc > > +#define arch_get_ipcc_score intel_hfi_get_ipcc_score > > #endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ > > > > #endif /* _ASM_X86_TOPOLOGY_H */ > > diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c > > index 530dcf57e06e..fa9b4a678d92 100644 > > --- a/drivers/thermal/intel/intel_hfi.c > > +++ b/drivers/thermal/intel/intel_hfi.c > > @@ -206,6 +206,33 @@ void intel_hfi_update_ipcc(struct task_struct *curr) > > curr->ipcc = msr.split.classid + 1; > > } > > > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) > > +{ > > + unsigned short hfi_class; > > It looks like the variable above is only used to save a subtraction or > addition of 1 to something going forward. > > > + int *scores; > > + > > + if (cpu < 0 || cpu >= nr_cpu_ids) > > + return -EINVAL; > > + > > + if (ipcc == IPC_CLASS_UNCLASSIFIED) > > + return -EINVAL; > > + > > + /* > > + * Scheduler IPC classes start at 1. HFI classes start at 0. > > + * See note intel_hfi_update_ipcc(). > > + */ > > + hfi_class = ipcc - 1; > > + > > + if (hfi_class >= hfi_features.nr_classes) > > Personally, I would do > > if (ipcc >= hfi_features.nr_classes + 1) > > here and -> > > > + return -EINVAL; > > + > > + scores = per_cpu_ptr(hfi_ipcc_scores, cpu); > > + if (!scores) > > + return -ENODEV; > > + > > -> scores[ipcc - 1] Sure, I can get rid of hfi_class. > > below. > > > + return READ_ONCE(scores[hfi_class]); > > And why does this need to use READ_ONCE()? This is the corresponding read of the WRITE_ONCE in patch 13. The CPU handling the HFI interrupt, very likely a different CPU, updates scores[hfi_class]. My intention is to let that write to complete before reading the score here. > > > +} > > + > > static int alloc_hfi_ipcc_scores(void) > > { > > if (!cpu_feature_enabled(X86_FEATURE_ITD)) > > --
On Wed, Mar 29, 2023 at 1:30 AM Ricardo Neri <ricardo.neri-calderon@linux.intel.com> wrote: > > On Mon, Mar 27, 2023 at 06:50:13PM +0200, Rafael J. Wysocki wrote: > > On Tue, Feb 7, 2023 at 6:02 AM Ricardo Neri > > <ricardo.neri-calderon@linux.intel.com> wrote: > > > > > > Implement the arch_get_ipcc_score() interface of the scheduler. Use the > > > performance capabilities of the extended Hardware Feedback Interface table > > > as the IPC score. > > > > > > Cc: Ben Segall <bsegall@google.com> > > > Cc: Daniel Bristot de Oliveira <bristot@redhat.com> > > > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > > > Cc: Ionela Voinescu <ionela.voinescu@arm.com> > > > Cc: Joel Fernandes (Google) <joel@joelfernandes.org> > > > Cc: Len Brown <len.brown@intel.com> > > > Cc: Lukasz Luba <lukasz.luba@arm.com> > > > Cc: Mel Gorman <mgorman@suse.de> > > > Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > > Cc: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> > > > Cc: Steven Rostedt <rostedt@goodmis.org> > > > Cc: Tim C. Chen <tim.c.chen@intel.com> > > > Cc: Valentin Schneider <vschneid@redhat.com> > > > Cc: x86@kernel.org > > > Cc: linux-pm@vger.kernel.org > > > Cc: linux-kernel@vger.kernel.org > > > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> > > > --- > > > Changes since v2: > > > * None > > > > > > Changes since v1: > > > * Adjusted the returned HFI class (which starts at 0) to match the > > > scheduler IPCC class (which starts at 1). (PeterZ) > > > * Used the new interface names. > > > --- > > > arch/x86/include/asm/topology.h | 2 ++ > > > drivers/thermal/intel/intel_hfi.c | 27 +++++++++++++++++++++++++++ > > > 2 files changed, 29 insertions(+) > > > > > > diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h > > > index ffcdac3f398f..c4fcd9c3c634 100644 > > > --- a/arch/x86/include/asm/topology.h > > > +++ b/arch/x86/include/asm/topology.h > > > @@ -229,8 +229,10 @@ void init_freq_invariance_cppc(void); > > > > > > #if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) > > > void intel_hfi_update_ipcc(struct task_struct *curr); > > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu); > > > > > > #define arch_update_ipcc intel_hfi_update_ipcc > > > +#define arch_get_ipcc_score intel_hfi_get_ipcc_score > > > #endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ > > > > > > #endif /* _ASM_X86_TOPOLOGY_H */ > > > diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c > > > index 530dcf57e06e..fa9b4a678d92 100644 > > > --- a/drivers/thermal/intel/intel_hfi.c > > > +++ b/drivers/thermal/intel/intel_hfi.c > > > @@ -206,6 +206,33 @@ void intel_hfi_update_ipcc(struct task_struct *curr) > > > curr->ipcc = msr.split.classid + 1; > > > } > > > > > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) > > > +{ > > > + unsigned short hfi_class; > > > > It looks like the variable above is only used to save a subtraction or > > addition of 1 to something going forward. > > > > > + int *scores; > > > + > > > + if (cpu < 0 || cpu >= nr_cpu_ids) > > > + return -EINVAL; > > > + > > > + if (ipcc == IPC_CLASS_UNCLASSIFIED) > > > + return -EINVAL; > > > + > > > + /* > > > + * Scheduler IPC classes start at 1. HFI classes start at 0. > > > + * See note intel_hfi_update_ipcc(). > > > + */ > > > + hfi_class = ipcc - 1; > > > + > > > + if (hfi_class >= hfi_features.nr_classes) > > > > Personally, I would do > > > > if (ipcc >= hfi_features.nr_classes + 1) > > > > here and -> > > > > > + return -EINVAL; > > > + > > > + scores = per_cpu_ptr(hfi_ipcc_scores, cpu); > > > + if (!scores) > > > + return -ENODEV; > > > + > > > > -> scores[ipcc - 1] > > Sure, I can get rid of hfi_class. > > > > > below. > > > > > + return READ_ONCE(scores[hfi_class]); > > > > And why does this need to use READ_ONCE()? > > This is the corresponding read of the WRITE_ONCE in patch 13. The CPU > handling the HFI interrupt, very likely a different CPU, updates > scores[hfi_class]. My intention is to let that write to complete before > reading the score here. However, READ_ONCE()/WRITE_ONCE() only affect compiler optimizations AFAICS. What if the CPUs running the code reorder the instructions? In any case, IMV the reason why READ_ONCE() is used needs to be clear to the reviewers from the patch itself (and to a casual reader of the code from the code itself). > > > > > +} > > > + > > > static int alloc_hfi_ipcc_scores(void) > > > { > > > if (!cpu_feature_enabled(X86_FEATURE_ITD)) > > > --
On Wed, Mar 29, 2023 at 02:17:01PM +0200, Rafael J. Wysocki wrote: > On Wed, Mar 29, 2023 at 1:30 AM Ricardo Neri > <ricardo.neri-calderon@linux.intel.com> wrote: > > > > On Mon, Mar 27, 2023 at 06:50:13PM +0200, Rafael J. Wysocki wrote: > > > On Tue, Feb 7, 2023 at 6:02 AM Ricardo Neri > > > <ricardo.neri-calderon@linux.intel.com> wrote: > > > > > > > > Implement the arch_get_ipcc_score() interface of the scheduler. Use the > > > > performance capabilities of the extended Hardware Feedback Interface table > > > > as the IPC score. > > > > > > > > Cc: Ben Segall <bsegall@google.com> > > > > Cc: Daniel Bristot de Oliveira <bristot@redhat.com> > > > > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > > > > Cc: Ionela Voinescu <ionela.voinescu@arm.com> > > > > Cc: Joel Fernandes (Google) <joel@joelfernandes.org> > > > > Cc: Len Brown <len.brown@intel.com> > > > > Cc: Lukasz Luba <lukasz.luba@arm.com> > > > > Cc: Mel Gorman <mgorman@suse.de> > > > > Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > > > Cc: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> > > > > Cc: Steven Rostedt <rostedt@goodmis.org> > > > > Cc: Tim C. Chen <tim.c.chen@intel.com> > > > > Cc: Valentin Schneider <vschneid@redhat.com> > > > > Cc: x86@kernel.org > > > > Cc: linux-pm@vger.kernel.org > > > > Cc: linux-kernel@vger.kernel.org > > > > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> > > > > --- > > > > Changes since v2: > > > > * None > > > > > > > > Changes since v1: > > > > * Adjusted the returned HFI class (which starts at 0) to match the > > > > scheduler IPCC class (which starts at 1). (PeterZ) > > > > * Used the new interface names. > > > > --- > > > > arch/x86/include/asm/topology.h | 2 ++ > > > > drivers/thermal/intel/intel_hfi.c | 27 +++++++++++++++++++++++++++ > > > > 2 files changed, 29 insertions(+) > > > > > > > > diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h > > > > index ffcdac3f398f..c4fcd9c3c634 100644 > > > > --- a/arch/x86/include/asm/topology.h > > > > +++ b/arch/x86/include/asm/topology.h > > > > @@ -229,8 +229,10 @@ void init_freq_invariance_cppc(void); > > > > > > > > #if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) > > > > void intel_hfi_update_ipcc(struct task_struct *curr); > > > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu); > > > > > > > > #define arch_update_ipcc intel_hfi_update_ipcc > > > > +#define arch_get_ipcc_score intel_hfi_get_ipcc_score > > > > #endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ > > > > > > > > #endif /* _ASM_X86_TOPOLOGY_H */ > > > > diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c > > > > index 530dcf57e06e..fa9b4a678d92 100644 > > > > --- a/drivers/thermal/intel/intel_hfi.c > > > > +++ b/drivers/thermal/intel/intel_hfi.c > > > > @@ -206,6 +206,33 @@ void intel_hfi_update_ipcc(struct task_struct *curr) > > > > curr->ipcc = msr.split.classid + 1; > > > > } > > > > > > > > +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) > > > > +{ > > > > + unsigned short hfi_class; > > > > > > It looks like the variable above is only used to save a subtraction or > > > addition of 1 to something going forward. > > > > > > > + int *scores; > > > > + > > > > + if (cpu < 0 || cpu >= nr_cpu_ids) > > > > + return -EINVAL; > > > > + > > > > + if (ipcc == IPC_CLASS_UNCLASSIFIED) > > > > + return -EINVAL; > > > > + > > > > + /* > > > > + * Scheduler IPC classes start at 1. HFI classes start at 0. > > > > + * See note intel_hfi_update_ipcc(). > > > > + */ > > > > + hfi_class = ipcc - 1; > > > > + > > > > + if (hfi_class >= hfi_features.nr_classes) > > > > > > Personally, I would do > > > > > > if (ipcc >= hfi_features.nr_classes + 1) > > > > > > here and -> > > > > > > > + return -EINVAL; > > > > + > > > > + scores = per_cpu_ptr(hfi_ipcc_scores, cpu); > > > > + if (!scores) > > > > + return -ENODEV; > > > > + > > > > > > -> scores[ipcc - 1] > > > > Sure, I can get rid of hfi_class. > > > > > > > > below. > > > > > > > + return READ_ONCE(scores[hfi_class]); > > > > > > And why does this need to use READ_ONCE()? > > > > This is the corresponding read of the WRITE_ONCE in patch 13. The CPU > > handling the HFI interrupt, very likely a different CPU, updates > > scores[hfi_class]. My intention is to let that write to complete before > > reading the score here. > > However, READ_ONCE()/WRITE_ONCE() only affect compiler optimizations > AFAICS. What if the CPUs running the code reorder the instructions? True, this implementation may not be complete and may need a memory barrier. I will look at it more closely. > > In any case, IMV the reason why READ_ONCE() is used needs to be clear > to the reviewers from the patch itself (and to a casual reader of the > code from the code itself). Sure. I will make sure this is clear in the commit message and/or inline comments. Thanks and BR, Ricardo
diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index ffcdac3f398f..c4fcd9c3c634 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -229,8 +229,10 @@ void init_freq_invariance_cppc(void); #if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) void intel_hfi_update_ipcc(struct task_struct *curr); +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu); #define arch_update_ipcc intel_hfi_update_ipcc +#define arch_get_ipcc_score intel_hfi_get_ipcc_score #endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ #endif /* _ASM_X86_TOPOLOGY_H */ diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index 530dcf57e06e..fa9b4a678d92 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -206,6 +206,33 @@ void intel_hfi_update_ipcc(struct task_struct *curr) curr->ipcc = msr.split.classid + 1; } +unsigned long intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) +{ + unsigned short hfi_class; + int *scores; + + if (cpu < 0 || cpu >= nr_cpu_ids) + return -EINVAL; + + if (ipcc == IPC_CLASS_UNCLASSIFIED) + return -EINVAL; + + /* + * Scheduler IPC classes start at 1. HFI classes start at 0. + * See note intel_hfi_update_ipcc(). + */ + hfi_class = ipcc - 1; + + if (hfi_class >= hfi_features.nr_classes) + return -EINVAL; + + scores = per_cpu_ptr(hfi_ipcc_scores, cpu); + if (!scores) + return -ENODEV; + + return READ_ONCE(scores[hfi_class]); +} + static int alloc_hfi_ipcc_scores(void) { if (!cpu_feature_enabled(X86_FEATURE_ITD))
Implement the arch_get_ipcc_score() interface of the scheduler. Use the performance capabilities of the extended Hardware Feedback Interface table as the IPC score. Cc: Ben Segall <bsegall@google.com> Cc: Daniel Bristot de Oliveira <bristot@redhat.com> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Ionela Voinescu <ionela.voinescu@arm.com> Cc: Joel Fernandes (Google) <joel@joelfernandes.org> Cc: Len Brown <len.brown@intel.com> Cc: Lukasz Luba <lukasz.luba@arm.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com> Cc: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Tim C. Chen <tim.c.chen@intel.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com> --- Changes since v2: * None Changes since v1: * Adjusted the returned HFI class (which starts at 0) to match the scheduler IPCC class (which starts at 1). (PeterZ) * Used the new interface names. --- arch/x86/include/asm/topology.h | 2 ++ drivers/thermal/intel/intel_hfi.c | 27 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+)