diff mbox series

image_types: use pigz to create .gz files

Message ID 20180410143532.31916-1-ross.burton@intel.com
State Accepted
Commit 214fa7fe3b162162d2fa8b31eec28bedd86fcc7d
Headers show
Series image_types: use pigz to create .gz files | expand

Commit Message

Ross Burton April 10, 2018, 2:35 p.m. UTC 
Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native.  Fix this by invoking pigz explicitly.

Signed-off-by: Ross Burton <ross.burton@intel.com>

---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.11.0

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Comments

Martin Jansa April 11, 2018, 10:53 a.m. UTC | #1 
What about using pigz for sstate-cache archives?

On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <ross.burton@intel.com> wrote:

> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)

> the

> image creation has been using gzip instead of pigz, despite still

> depending on

> pigz-native.  Fix this by invoking pigz explicitly.

>

> Signed-off-by: Ross Burton <ross.burton@intel.com>

> ---

>  meta/classes/image_types.bbclass | 2 +-

>  1 file changed, 1 insertion(+), 1 deletion(-)

>

> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.

> bbclass

> index e872ae20982..00a00d318f4 100644

> --- a/meta/classes/image_types.bbclass

> +++ b/meta/classes/image_types.bbclass

> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

>

>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2

> ${COMPRESSIONTYPES}"

>  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_

> SUFFIX}.${type}"

> -CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

> +CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_

> SUFFIX}.${type}"

>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}

> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

> --

> 2.11.0

>

> --

> _______________________________________________

> Openembedded-core mailing list

> Openembedded-core@lists.openembedded.org

> http://lists.openembedded.org/mailman/listinfo/openembedded-core

>
<div dir="ltr">What about using pigz for sstate-cache archives?</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <span dir="ltr">&lt;<a href="mailto:ross.burton@intel.com" target="_blank">ross.burton@intel.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the<br>
image creation has been using gzip instead of pigz, despite still depending on<br>
pigz-native.  Fix this by invoking pigz explicitly.<br>
<br>
Signed-off-by: Ross Burton &lt;<a href="mailto:ross.burton@intel.com">ross.burton@intel.com</a>&gt;<br>

---<br>
 meta/classes/image_types.<wbr>bbclass | 2 +-<br>
 1 file changed, 1 insertion(+), 1 deletion(-)<br>
<br>
diff --git a/meta/classes/image_types.<wbr>bbclass b/meta/classes/image_types.<wbr>bbclass<br>
index e872ae20982..00a00d318f4 100644<br>
--- a/meta/classes/image_types.<wbr>bbclass<br>
+++ b/meta/classes/image_types.<wbr>bbclass<br>
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= &quot;&quot;<br>
<br>
 CONVERSIONTYPES = &quot;gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}&quot;<br>
 CONVERSION_CMD_lzma = &quot;lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type}&quot;<br>
-CONVERSION_CMD_gz = &quot;gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type} &gt; ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type}.gz&quot;<br>
+CONVERSION_CMD_gz = &quot;pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type} &gt; ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type}.gz&quot;<br>
 CONVERSION_CMD_bz2 = &quot;pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type}&quot;<br>
 CONVERSION_CMD_xz = &quot;xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type} &gt; ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type}.xz&quot;<br>
 CONVERSION_CMD_lz4 = &quot;lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_<wbr>SUFFIX}.${type}.lz4&quot;<br>
<span class="HOEnZb"><font color="#888888">--<br>
2.11.0<br>
<br>
--<br>
______________________________<wbr>_________________<br>
Openembedded-core mailing list<br>
<a href="mailto:Openembedded-core@lists.openembedded.org">Openembedded-core@lists.<wbr>openembedded.org</a><br>
<a href="http://lists.openembedded.org/mailman/listinfo/openembedded-core" rel="noreferrer" target="_blank">http://lists.openembedded.org/<wbr>mailman/listinfo/openembedded-<wbr>core</a><br>
</font></span></blockquote></div><br></div>
-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Ross Burton April 11, 2018, 1:36 p.m. UTC | #2 
Wouldn't that introduce a fun circular dependency if pigz-native needs
to be built?

Ross

On 11 April 2018 at 11:53, Martin Jansa <martin.jansa@gmail.com> wrote:
> What about using pigz for sstate-cache archives?

>

> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <ross.burton@intel.com> wrote:

>>

>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)

>> the

>> image creation has been using gzip instead of pigz, despite still

>> depending on

>> pigz-native.  Fix this by invoking pigz explicitly.

>>

>> Signed-off-by: Ross Burton <ross.burton@intel.com>

>> ---

>>  meta/classes/image_types.bbclass | 2 +-

>>  1 file changed, 1 insertion(+), 1 deletion(-)

>>

>> diff --git a/meta/classes/image_types.bbclass

>> b/meta/classes/image_types.bbclass

>> index e872ae20982..00a00d318f4 100644

>> --- a/meta/classes/image_types.bbclass

>> +++ b/meta/classes/image_types.bbclass

>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

>>

>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2

>> ${COMPRESSIONTYPES}"

>>  CONVERSION_CMD_lzma = "lzma -k -f -7

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}

>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

>> --

>> 2.11.0

>>

>> --

>> _______________________________________________

>> Openembedded-core mailing list

>> Openembedded-core@lists.openembedded.org

>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

>

>

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Khem Raj April 11, 2018, 4:03 p.m. UTC | #3 
On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross <ross.burton@intel.com> wrote:
> Wouldn't that introduce a fun circular dependency if pigz-native needs

> to be built?

>


interesting idea certainly, if it ends up with improvements in build
time, we could
add pigz to host dependencies.

> Ross

>

> On 11 April 2018 at 11:53, Martin Jansa <martin.jansa@gmail.com> wrote:

>> What about using pigz for sstate-cache archives?

>>

>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <ross.burton@intel.com> wrote:

>>>

>>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)

>>> the

>>> image creation has been using gzip instead of pigz, despite still

>>> depending on

>>> pigz-native.  Fix this by invoking pigz explicitly.

>>>

>>> Signed-off-by: Ross Burton <ross.burton@intel.com>

>>> ---

>>>  meta/classes/image_types.bbclass | 2 +-

>>>  1 file changed, 1 insertion(+), 1 deletion(-)

>>>

>>> diff --git a/meta/classes/image_types.bbclass

>>> b/meta/classes/image_types.bbclass

>>> index e872ae20982..00a00d318f4 100644

>>> --- a/meta/classes/image_types.bbclass

>>> +++ b/meta/classes/image_types.bbclass

>>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

>>>

>>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

>>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2

>>> ${COMPRESSIONTYPES}"

>>>  CONVERSION_CMD_lzma = "lzma -k -f -7

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}

>>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

>>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

>>> --

>>> 2.11.0

>>>

>>> --

>>> _______________________________________________

>>> Openembedded-core mailing list

>>> Openembedded-core@lists.openembedded.org

>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

>>

>>

> --

> _______________________________________________

> Openembedded-core mailing list

> Openembedded-core@lists.openembedded.org

> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Ross Burton April 11, 2018, 4:07 p.m. UTC | #4 
Woud be interesting to see a benchmark of a full build from scratch
with host gzip vs host pigz...

On 11 April 2018 at 17:03, Khem Raj <raj.khem@gmail.com> wrote:
> On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross <ross.burton@intel.com> wrote:

>> Wouldn't that introduce a fun circular dependency if pigz-native needs

>> to be built?

>>

>

> interesting idea certainly, if it ends up with improvements in build

> time, we could

> add pigz to host dependencies.

>

>> Ross

>>

>> On 11 April 2018 at 11:53, Martin Jansa <martin.jansa@gmail.com> wrote:

>>> What about using pigz for sstate-cache archives?

>>>

>>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <ross.burton@intel.com> wrote:

>>>>

>>>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)

>>>> the

>>>> image creation has been using gzip instead of pigz, despite still

>>>> depending on

>>>> pigz-native.  Fix this by invoking pigz explicitly.

>>>>

>>>> Signed-off-by: Ross Burton <ross.burton@intel.com>

>>>> ---

>>>>  meta/classes/image_types.bbclass | 2 +-

>>>>  1 file changed, 1 insertion(+), 1 deletion(-)

>>>>

>>>> diff --git a/meta/classes/image_types.bbclass

>>>> b/meta/classes/image_types.bbclass

>>>> index e872ae20982..00a00d318f4 100644

>>>> --- a/meta/classes/image_types.bbclass

>>>> +++ b/meta/classes/image_types.bbclass

>>>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

>>>>

>>>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

>>>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2

>>>> ${COMPRESSIONTYPES}"

>>>>  CONVERSION_CMD_lzma = "lzma -k -f -7

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>>>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>>>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>>>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>>>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}

>>>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

>>>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

>>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

>>>> --

>>>> 2.11.0

>>>>

>>>> --

>>>> _______________________________________________

>>>> Openembedded-core mailing list

>>>> Openembedded-core@lists.openembedded.org

>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

>>>

>>>

>> --

>> _______________________________________________

>> Openembedded-core mailing list

>> Openembedded-core@lists.openembedded.org

>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Denys Dmytriyenko April 12, 2018, 1:19 a.m. UTC | #5 
On Wed, Apr 11, 2018 at 05:07:38PM +0100, Burton, Ross wrote:
> Woud be interesting to see a benchmark of a full build from scratch

> with host gzip vs host pigz...


Is pigz so widely used now to be available in every Linux distro like gzip?


> On 11 April 2018 at 17:03, Khem Raj <raj.khem@gmail.com> wrote:

> > On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross <ross.burton@intel.com> wrote:

> >> Wouldn't that introduce a fun circular dependency if pigz-native needs

> >> to be built?

> >>

> >

> > interesting idea certainly, if it ends up with improvements in build

> > time, we could

> > add pigz to host dependencies.

> >

> >> Ross

> >>

> >> On 11 April 2018 at 11:53, Martin Jansa <martin.jansa@gmail.com> wrote:

> >>> What about using pigz for sstate-cache archives?

> >>>

> >>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <ross.burton@intel.com> wrote:

> >>>>

> >>>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)

> >>>> the

> >>>> image creation has been using gzip instead of pigz, despite still

> >>>> depending on

> >>>> pigz-native.  Fix this by invoking pigz explicitly.

> >>>>

> >>>> Signed-off-by: Ross Burton <ross.burton@intel.com>

> >>>> ---

> >>>>  meta/classes/image_types.bbclass | 2 +-

> >>>>  1 file changed, 1 insertion(+), 1 deletion(-)

> >>>>

> >>>> diff --git a/meta/classes/image_types.bbclass

> >>>> b/meta/classes/image_types.bbclass

> >>>> index e872ae20982..00a00d318f4 100644

> >>>> --- a/meta/classes/image_types.bbclass

> >>>> +++ b/meta/classes/image_types.bbclass

> >>>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

> >>>>

> >>>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

> >>>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2

> >>>> ${COMPRESSIONTYPES}"

> >>>>  CONVERSION_CMD_lzma = "lzma -k -f -7

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> >>>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

> >>>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

> >>>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

> >>>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}

> >>>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

> >>>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

> >>>> --

> >>>> 2.11.0

> >>>>

> >>>> --

> >>>> _______________________________________________

> >>>> Openembedded-core mailing list

> >>>> Openembedded-core@lists.openembedded.org

> >>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

> >>>

> >>>

> >> --

> >> _______________________________________________

> >> Openembedded-core mailing list

> >> Openembedded-core@lists.openembedded.org

> >> http://lists.openembedded.org/mailman/listinfo/openembedded-core

> -- 

> _______________________________________________

> Openembedded-core mailing list

> Openembedded-core@lists.openembedded.org

> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Khem Raj April 12, 2018, 1:33 a.m. UTC | #6 
On Wed, Apr 11, 2018 at 6:19 PM, Denys Dmytriyenko <denis@denix.org> wrote:
> On Wed, Apr 11, 2018 at 05:07:38PM +0100, Burton, Ross wrote:

>> Woud be interesting to see a benchmark of a full build from scratch

>> with host gzip vs host pigz...

>

> Is pigz so widely used now to be available in every Linux distro like gzip?

>


Tested distros do seem to provide it as old as ubuntu 14.04 and centos 6
xz is another option for sstate archives the extra time it requires might
be compensated with threaded run in addition the archives will be smaller

>

>> On 11 April 2018 at 17:03, Khem Raj <raj.khem@gmail.com> wrote:

>> > On Wed, Apr 11, 2018 at 6:36 AM, Burton, Ross <ross.burton@intel.com> wrote:

>> >> Wouldn't that introduce a fun circular dependency if pigz-native needs

>> >> to be built?

>> >>

>> >

>> > interesting idea certainly, if it ends up with improvements in build

>> > time, we could

>> > add pigz to host dependencies.

>> >

>> >> Ross

>> >>

>> >> On 11 April 2018 at 11:53, Martin Jansa <martin.jansa@gmail.com> wrote:

>> >>> What about using pigz for sstate-cache archives?

>> >>>

>> >>> On Tue, Apr 10, 2018 at 4:35 PM, Ross Burton <ross.burton@intel.com> wrote:

>> >>>>

>> >>>> Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b)

>> >>>> the

>> >>>> image creation has been using gzip instead of pigz, despite still

>> >>>> depending on

>> >>>> pigz-native.  Fix this by invoking pigz explicitly.

>> >>>>

>> >>>> Signed-off-by: Ross Burton <ross.burton@intel.com>

>> >>>> ---

>> >>>>  meta/classes/image_types.bbclass | 2 +-

>> >>>>  1 file changed, 1 insertion(+), 1 deletion(-)

>> >>>>

>> >>>> diff --git a/meta/classes/image_types.bbclass

>> >>>> b/meta/classes/image_types.bbclass

>> >>>> index e872ae20982..00a00d318f4 100644

>> >>>> --- a/meta/classes/image_types.bbclass

>> >>>> +++ b/meta/classes/image_types.bbclass

>> >>>> @@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""

>> >>>>

>> >>>>  CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

>> >>>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2

>> >>>> ${COMPRESSIONTYPES}"

>> >>>>  CONVERSION_CMD_lzma = "lzma -k -f -7

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>> >>>> -CONVERSION_CMD_gz = "gzip -f -9 -n -c

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>> >>>> +CONVERSION_CMD_gz = "pigz -f -9 -n -c

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"

>> >>>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

>> >>>>  CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}

>> >>>> --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"

>> >>>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}

>> >>>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

>> >>>> --

>> >>>> 2.11.0

>> >>>>

>> >>>> --

>> >>>> _______________________________________________

>> >>>> Openembedded-core mailing list

>> >>>> Openembedded-core@lists.openembedded.org

>> >>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

>> >>>

>> >>>

>> >> --

>> >> _______________________________________________

>> >> Openembedded-core mailing list

>> >> Openembedded-core@lists.openembedded.org

>> >> http://lists.openembedded.org/mailman/listinfo/openembedded-core

>> --

>> _______________________________________________

>> Openembedded-core mailing list

>> Openembedded-core@lists.openembedded.org

>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Richard Purdie April 13, 2018, 12:07 p.m. UTC | #7 
On Wed, 2018-04-11 at 14:36 +0100, Burton, Ross wrote:
> Wouldn't that introduce a fun circular dependency if pigz-native
> needs
> to be built?

For the sstate code we could probably make it use pigz if present. 

We could also only depend on pigz-native for target, or encode the
actual pigz-native dependencies.

So there may be a few options if its worth the performance win.

Cheers,

Richard
diff mbox series

Patch

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e872ae20982..00a00d318f4 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -281,7 +281,7 @@  COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
+CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
 CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"