diff mbox series

image_types.bbclass: Set 70% RAM memory usage limit for xz

Message ID 20190209012850.17425-1-raj.khem@gmail.com
State New
Headers show
Series image_types.bbclass: Set 70% RAM memory usage limit for xz | expand

Commit Message

Khem Raj Feb. 9, 2019, 1:28 a.m. UTC 
when building with opkg backend and huge packages e.g. chromium/llvm all
going in parallel, memory pressure causes xz to catapult with

do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory

since there are many tasks going on in parallel, xz adds to memory pressure
and it wants it all, put an upper limit for memory xz can use

We add a variable XZ_MAXRAM with 70% of RAM limit and can be customized
if builders have more memory one can set it like

XZMAX_RAM = "-M 0"

Signed-off-by: Khem Raj <raj.khem@gmail.com>

Cc: Alejandro del Castillo <alejandro.delcastillo@ni.com>
---
 meta/classes/image_types.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.20.1

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Comments

Andre McCurdy Feb. 9, 2019, 3:27 a.m. UTC | #1 
On Fri, Feb 8, 2019 at 5:29 PM Khem Raj <raj.khem@gmail.com> wrote:
>

> when building with opkg backend and huge packages e.g. chromium/llvm all

> going in parallel, memory pressure causes xz to catapult with

>

> do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory

>

> since there are many tasks going on in parallel, xz adds to memory pressure

> and it wants it all, put an upper limit for memory xz can use

>

> We add a variable XZ_MAXRAM with 70% of RAM limit and can be customized

> if builders have more memory one can set it like

>

> XZMAX_RAM = "-M 0"


This needs a little more explanation. How does a change to command
used to create .xz images help when running out of memory in
do_package_write_ipk? Doesn't the image get created a lot later?

> Signed-off-by: Khem Raj <raj.khem@gmail.com>

> Cc: Alejandro del Castillo <alejandro.delcastillo@ni.com>

> ---

>  meta/classes/image_types.bbclass | 3 ++-

>  1 file changed, 2 insertions(+), 1 deletion(-)

>

> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass

> index 70bd315306..9c8ea0643c 100644

> --- a/meta/classes/image_types.bbclass

> +++ b/meta/classes/image_types.bbclass

> @@ -57,6 +57,7 @@ def imagetypes_getdepends(d):

>  XZ_COMPRESSION_LEVEL ?= "-3"

>  XZ_INTEGRITY_CHECK ?= "crc32"

>  XZ_THREADS ?= "-T 0"

> +XZ_MAXRAM ?= "-M 70%"

>

>  ZIP_COMPRESSION_LEVEL ?= "-9"

>

> @@ -284,7 +285,7 @@ CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25

>  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>  CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> -CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

> +CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} ${XZ_MAXRAM} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

>  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> --

> 2.20.1

>

> --

> _______________________________________________

> Openembedded-core mailing list

> Openembedded-core@lists.openembedded.org

> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Khem Raj Feb. 9, 2019, 6:45 a.m. UTC | #2 
On Fri, Feb 8, 2019 at 7:27 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>

> On Fri, Feb 8, 2019 at 5:29 PM Khem Raj <raj.khem@gmail.com> wrote:

> >

> > when building with opkg backend and huge packages e.g. chromium/llvm all

> > going in parallel, memory pressure causes xz to catapult with

> >

> > do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory

> >

> > since there are many tasks going on in parallel, xz adds to memory pressure

> > and it wants it all, put an upper limit for memory xz can use

> >

> > We add a variable XZ_MAXRAM with 70% of RAM limit and can be customized

> > if builders have more memory one can set it like

> >

> > XZMAX_RAM = "-M 0"

>

> This needs a little more explanation. How does a change to command

> used to create .xz images help when running out of memory in

> do_package_write_ipk? Doesn't the image get created a lot later?

>


its not related to image alone. xz is used in various places e.g.
opkgs switched to
using xz as default compression format recently, so every ipk creation
task involves
xz, and you might be in multiple of ipk creation tasks based on task parallelism

hope that clarifies. If not feel free to ask more

> > Signed-off-by: Khem Raj <raj.khem@gmail.com>

> > Cc: Alejandro del Castillo <alejandro.delcastillo@ni.com>

> > ---

> >  meta/classes/image_types.bbclass | 3 ++-

> >  1 file changed, 2 insertions(+), 1 deletion(-)

> >

> > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass

> > index 70bd315306..9c8ea0643c 100644

> > --- a/meta/classes/image_types.bbclass

> > +++ b/meta/classes/image_types.bbclass

> > @@ -57,6 +57,7 @@ def imagetypes_getdepends(d):

> >  XZ_COMPRESSION_LEVEL ?= "-3"

> >  XZ_INTEGRITY_CHECK ?= "crc32"

> >  XZ_THREADS ?= "-T 0"

> > +XZ_MAXRAM ?= "-M 70%"

> >

> >  ZIP_COMPRESSION_LEVEL ?= "-9"

> >

> > @@ -284,7 +285,7 @@ CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25

> >  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> >  CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

> >  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> > -CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

> > +CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} ${XZ_MAXRAM} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

> >  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

> >  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> >  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> > --

> > 2.20.1

> >

> > --

> > _______________________________________________

> > Openembedded-core mailing list

> > Openembedded-core@lists.openembedded.org

> > http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Khem Raj Feb. 9, 2019, 6:06 p.m. UTC | #3 
On Fri, Feb 8, 2019 at 7:27 PM Andre McCurdy <armccurdy@gmail.com> wrote:
>

> On Fri, Feb 8, 2019 at 5:29 PM Khem Raj <raj.khem@gmail.com> wrote:

> >

> > when building with opkg backend and huge packages e.g. chromium/llvm all

> > going in parallel, memory pressure causes xz to catapult with

> >

> > do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory

> >

> > since there are many tasks going on in parallel, xz adds to memory pressure

> > and it wants it all, put an upper limit for memory xz can use

> >

> > We add a variable XZ_MAXRAM with 70% of RAM limit and can be customized

> > if builders have more memory one can set it like

> >

> > XZMAX_RAM = "-M 0"

>

> This needs a little more explanation. How does a change to command

> used to create .xz images help when running out of memory in

> do_package_write_ipk? Doesn't the image get created a lot later?


I think I have sent v1 patch which was not the one I intended to send.
will send v2 soon which should be complete.

>

> > Signed-off-by: Khem Raj <raj.khem@gmail.com>

> > Cc: Alejandro del Castillo <alejandro.delcastillo@ni.com>

> > ---

> >  meta/classes/image_types.bbclass | 3 ++-

> >  1 file changed, 2 insertions(+), 1 deletion(-)

> >

> > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass

> > index 70bd315306..9c8ea0643c 100644

> > --- a/meta/classes/image_types.bbclass

> > +++ b/meta/classes/image_types.bbclass

> > @@ -57,6 +57,7 @@ def imagetypes_getdepends(d):

> >  XZ_COMPRESSION_LEVEL ?= "-3"

> >  XZ_INTEGRITY_CHECK ?= "crc32"

> >  XZ_THREADS ?= "-T 0"

> > +XZ_MAXRAM ?= "-M 70%"

> >

> >  ZIP_COMPRESSION_LEVEL ?= "-9"

> >

> > @@ -284,7 +285,7 @@ CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25

> >  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> >  CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

> >  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> > -CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

> > +CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} ${XZ_MAXRAM} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

> >  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

> >  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> >  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> > --

> > 2.20.1

> >

> > --

> > _______________________________________________

> > Openembedded-core mailing list

> > Openembedded-core@lists.openembedded.org

> > http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
diff mbox series

Patch

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 70bd315306..9c8ea0643c 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -57,6 +57,7 @@  def imagetypes_getdepends(d):
 XZ_COMPRESSION_LEVEL ?= "-3"
 XZ_INTEGRITY_CHECK ?= "crc32"
 XZ_THREADS ?= "-T 0"
+XZ_MAXRAM ?= "-M 70%"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
 
@@ -284,7 +285,7 @@  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25
 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
+CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} ${XZ_MAXRAM} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
 CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"