[edk2,4/5] MdePkg X64: force 'hidden' visibility when building with -fpic

Message ID 1468502169-15248-5-git-send-email-ard.biesheuvel@linaro.org
State Superseded
Headers show

Commit Message

Ard Biesheuvel July 14, 2016, 1:16 p.m.
When building position independent (PIC) ELF objects, the GCC compiler
assumes that each symbol with external linkage may potentially end up
being exported from a shared library, which means that each of those
symbols may be subject to symbol preemption, i.e., the executable
linking to the shared library at runtime may override symbols exported
by the shared library, and every internal reference held by the shared
library itself *must* be made to point to the overridden version instead.

For this reason, PIC code symbol references always go via the Global
Offset Table (GOT), even if the code in question references symbols that
are defined in the same compilation unit. The GOT refers to each symbol
by absolute address, and so each entry is subject to runtime relocation.

Since not every symbol with external linkage is ultimately exported from
a shared library, the GCC compiler allows control over symbol visibility
using attributes, command line arguments and pragmas, where 'hidden'
means that the symbol is only referenced by the shared library itself.
Due to the poor hygiene in EDK2 regarding the use of the 'static'
modifier, many symbols that are local to their compilation unit end up
being referenced indirectly via the GOT when building PIC code.

In UEFI, there are no shared libraries and so there is no need to deal
with symbol preemption, and we can mark every symbol reference 'hidden'.
The only method that applies to all symbol definitions as well as
declarations is the #pragma. So set the visibility 'hidden' pragma when
building PIC code for X64 using GCC.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

---
 MdePkg/Include/X64/ProcessorBind.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

-- 
2.7.4

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Comments

Ard Biesheuvel July 16, 2016, 2:48 p.m. | #1
On 16 July 2016 at 15:54, Shi, Steven <steven.shi@intel.com> wrote:
> Hi Ard,

> Sorry for my late response. Setting all symbols visibility as hidden will block LTO on GCC. GCC LTO need symbols to be public to let linker freely do the link time optimization across local to external. Below is the GCC5 build failure in my side with your patch.

>


Do you have any references that support this? Hidden visibility has
nothing to do with linkage. It only controls the symbols that are
exported from a shared library. Knowing whether a symbol is ever
exported from a shared library allows the compiler to generate better
code.


> "gcc" -o /home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.dll -Os -flto -nostdlib -Wl,-n -Wl,-q -Wl,--gc-sections -z common-page-size=0x40 -Wl,--entry,_ModuleEntryPoint -Wl,-u,_ModuleEntryPoint -Wl,-Map,/home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.map -mcmodel=small -Wl,-melf_x86_64 -Wl,--oformat=elf64-x86-64 -Wl,--start-group,,@/home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/OUTPUT/static_library_files.lst -Wl,--end-group -Wl,--defsym=PECOFF_HEADER_SIZE=0x228 -Wl,--script=/home/jshi19/edk2-fork3/BaseTools/Scripts/GccBase.lds

> "objcopy"  /home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.dll

> objcopy: error: the input file '/home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.dll' has no sections


This looks like an issue with --gc-sections. Could you check the map
file why everything gets discarded?

> make: *** [/home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.dll] Error 1

> GNUmakefile:478: recipe for target '/home/jshi19/edk2-fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.dll' failed

>

>

> Steven Shi

> Intel\SSG\STO\UEFI Firmware

>

> Tel: +86 021-61166522

> iNet: 821-6522

>

>> -----Original Message-----

>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]

>> Sent: Thursday, July 14, 2016 9:16 PM

>> To: Shi, Steven <steven.shi@intel.com>; Zhu, Yonghong

>> <yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>; Kinney,

>> Michael D <michael.d.kinney@intel.com>; Justen, Jordan L

>> <jordan.l.justen@intel.com>; lersek@redhat.com; afish@apple.com; edk2-

>> devel@lists.01.org

>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>

>> Subject: [PATCH 4/5] MdePkg X64: force 'hidden' visibility when building with

>> -fpic

>>

>> When building position independent (PIC) ELF objects, the GCC compiler

>> assumes that each symbol with external linkage may potentially end up

>> being exported from a shared library, which means that each of those

>> symbols may be subject to symbol preemption, i.e., the executable

>> linking to the shared library at runtime may override symbols exported

>> by the shared library, and every internal reference held by the shared

>> library itself *must* be made to point to the overridden version instead.

>>

>> For this reason, PIC code symbol references always go via the Global

>> Offset Table (GOT), even if the code in question references symbols that

>> are defined in the same compilation unit. The GOT refers to each symbol

>> by absolute address, and so each entry is subject to runtime relocation.

>>

>> Since not every symbol with external linkage is ultimately exported from

>> a shared library, the GCC compiler allows control over symbol visibility

>> using attributes, command line arguments and pragmas, where 'hidden'

>> means that the symbol is only referenced by the shared library itself.

>> Due to the poor hygiene in EDK2 regarding the use of the 'static'

>> modifier, many symbols that are local to their compilation unit end up

>> being referenced indirectly via the GOT when building PIC code.

>>

>> In UEFI, there are no shared libraries and so there is no need to deal

>> with symbol preemption, and we can mark every symbol reference 'hidden'.

>> The only method that applies to all symbol definitions as well as

>> declarations is the #pragma. So set the visibility 'hidden' pragma when

>> building PIC code for X64 using GCC.

>>

>> Contributed-under: TianoCore Contribution Agreement 1.0

>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

>> ---

>>  MdePkg/Include/X64/ProcessorBind.h | 10 ++++++++++

>>  1 file changed, 10 insertions(+)

>>

>> diff --git a/MdePkg/Include/X64/ProcessorBind.h

>> b/MdePkg/Include/X64/ProcessorBind.h

>> index 705104af062a..96df78fca07d 100644

>> --- a/MdePkg/Include/X64/ProcessorBind.h

>> +++ b/MdePkg/Include/X64/ProcessorBind.h

>> @@ -27,6 +27,16 @@

>>  #pragma pack()

>>  #endif

>>

>> +#if defined(__GNUC__) && defined(__pic__)

>> +//

>> +// Mark all symbol declarations and references as hidden, meaning they will

>> not

>> +// be exported from a shared library, and thus will not be subject to symbol

>> +// preemption. This allows the compiler to refer to symbols directly using

>> +// relative references rather than via the GOT, which contains absolute

>> symbol

>> +// addresses that are subject to runtime relocation.

>> +//

>> +#pragma GCC visibility push (hidden)

>> +#endif

>>

>>  #if defined(__INTEL_COMPILER)

>>  //

>> --

>> 2.7.4

>

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
Ard Biesheuvel July 16, 2016, 5:25 p.m. | #2
> On 16 jul. 2016, at 17:10, Shi, Steven <steven.shi@intel.com> wrote:

> 

> You clone and can try the GCC LTO build on my below branch, and welcome directly debug on it.

> https://github.com/shijunjing/edk2/tree/llvm_v2

> 


Thanks, I will try it on monday

> 

> 

> Steven Shi

> Intel\SSG\STO\UEFI Firmware

> 

> Tel: +86 021-61166522

> iNet: 821-6522

> 

> 

>> -----Original Message-----

>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]

>> Sent: Saturday, July 16, 2016 10:48 PM

>> To: Shi, Steven <steven.shi@intel.com>

>> Cc: Zhu, Yonghong <yonghong.zhu@intel.com>; Gao, Liming

>> <liming.gao@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>;

>> Justen, Jordan L <jordan.l.justen@intel.com>; lersek@redhat.com;

>> afish@apple.com; edk2-devel@lists.01.org

>> Subject: Re: [PATCH 4/5] MdePkg X64: force 'hidden' visibility when building

>> with -fpic

>> 

>>> On 16 July 2016 at 15:54, Shi, Steven <steven.shi@intel.com> wrote:

>>> Hi Ard,

>>> Sorry for my late response. Setting all symbols visibility as hidden will block

>> LTO on GCC. GCC LTO need symbols to be public to let linker freely do the

>> link time optimization across local to external. Below is the GCC5 build

>> failure in my side with your patch.

>> 

>> Do you have any references that support this? Hidden visibility has

>> nothing to do with linkage. It only controls the symbols that are

>> exported from a shared library. Knowing whether a symbol is ever

>> exported from a shared library allows the compiler to generate better

>> code.

>> 

>> 

>>> "gcc" -o /home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /DEBUG/PeiCore.dll -Os -flto -nostdlib -Wl,-n -Wl,-q -Wl,--gc-sections -z

>> common-page-size=0x40 -Wl,--entry,_ModuleEntryPoint -Wl,-

>> u,_ModuleEntryPoint -Wl,-Map,/home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /DEBUG/PeiCore.map -mcmodel=small -Wl,-melf_x86_64 -Wl,--

>> oformat=elf64-x86-64 -Wl,--start-group,,@/home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /OUTPUT/static_library_files.lst -Wl,--end-group -Wl,--

>> defsym=PECOFF_HEADER_SIZE=0x228 -Wl,--script=/home/jshi19/edk2-

>> fork3/BaseTools/Scripts/GccBase.lds

>>> "objcopy"  /home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /DEBUG/PeiCore.dll

>>> objcopy: error: the input file '/home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /DEBUG/PeiCore.dll' has no sections

>> 

>> This looks like an issue with --gc-sections. Could you check the map

>> file why everything gets discarded?

>> 

>>> make: *** [/home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /DEBUG/PeiCore.dll] Error 1

>>> GNUmakefile:478: recipe for target '/home/jshi19/edk2-

>> fork3/Build/OvmfX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain

>> /DEBUG/PeiCore.dll' failed

>>> 

>>> 

>>> Steven Shi

>>> Intel\SSG\STO\UEFI Firmware

>>> 

>>> Tel: +86 021-61166522

>>> iNet: 821-6522

>>> 

>>>> -----Original Message-----

>>>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]

>>>> Sent: Thursday, July 14, 2016 9:16 PM

>>>> To: Shi, Steven <steven.shi@intel.com>; Zhu, Yonghong

>>>> <yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>;

>> Kinney,

>>>> Michael D <michael.d.kinney@intel.com>; Justen, Jordan L

>>>> <jordan.l.justen@intel.com>; lersek@redhat.com; afish@apple.com;

>> edk2-

>>>> devel@lists.01.org

>>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>

>>>> Subject: [PATCH 4/5] MdePkg X64: force 'hidden' visibility when building

>> with

>>>> -fpic

>>>> 

>>>> When building position independent (PIC) ELF objects, the GCC compiler

>>>> assumes that each symbol with external linkage may potentially end up

>>>> being exported from a shared library, which means that each of those

>>>> symbols may be subject to symbol preemption, i.e., the executable

>>>> linking to the shared library at runtime may override symbols exported

>>>> by the shared library, and every internal reference held by the shared

>>>> library itself *must* be made to point to the overridden version instead.

>>>> 

>>>> For this reason, PIC code symbol references always go via the Global

>>>> Offset Table (GOT), even if the code in question references symbols that

>>>> are defined in the same compilation unit. The GOT refers to each symbol

>>>> by absolute address, and so each entry is subject to runtime relocation.

>>>> 

>>>> Since not every symbol with external linkage is ultimately exported from

>>>> a shared library, the GCC compiler allows control over symbol visibility

>>>> using attributes, command line arguments and pragmas, where 'hidden'

>>>> means that the symbol is only referenced by the shared library itself.

>>>> Due to the poor hygiene in EDK2 regarding the use of the 'static'

>>>> modifier, many symbols that are local to their compilation unit end up

>>>> being referenced indirectly via the GOT when building PIC code.

>>>> 

>>>> In UEFI, there are no shared libraries and so there is no need to deal

>>>> with symbol preemption, and we can mark every symbol reference

>> 'hidden'.

>>>> The only method that applies to all symbol definitions as well as

>>>> declarations is the #pragma. So set the visibility 'hidden' pragma when

>>>> building PIC code for X64 using GCC.

>>>> 

>>>> Contributed-under: TianoCore Contribution Agreement 1.0

>>>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

>>>> ---

>>>> MdePkg/Include/X64/ProcessorBind.h | 10 ++++++++++

>>>> 1 file changed, 10 insertions(+)

>>>> 

>>>> diff --git a/MdePkg/Include/X64/ProcessorBind.h

>>>> b/MdePkg/Include/X64/ProcessorBind.h

>>>> index 705104af062a..96df78fca07d 100644

>>>> --- a/MdePkg/Include/X64/ProcessorBind.h

>>>> +++ b/MdePkg/Include/X64/ProcessorBind.h

>>>> @@ -27,6 +27,16 @@

>>>> #pragma pack()

>>>> #endif

>>>> 

>>>> +#if defined(__GNUC__) && defined(__pic__)

>>>> +//

>>>> +// Mark all symbol declarations and references as hidden, meaning they

>> will

>>>> not

>>>> +// be exported from a shared library, and thus will not be subject to

>> symbol

>>>> +// preemption. This allows the compiler to refer to symbols directly using

>>>> +// relative references rather than via the GOT, which contains absolute

>>>> symbol

>>>> +// addresses that are subject to runtime relocation.

>>>> +//

>>>> +#pragma GCC visibility push (hidden)

>>>> +#endif

>>>> 

>>>> #if defined(__INTEL_COMPILER)

>>>> //

>>>> --

>>>> 2.7.4

>>> 

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
Ard Biesheuvel July 17, 2016, 9:24 a.m. | #3
On 16 July 2016 at 19:25, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>

>

>> On 16 jul. 2016, at 17:10, Shi, Steven <steven.shi@intel.com> wrote:

>>

>> You clone and can try the GCC LTO build on my below branch, and welcome directly debug on it.

>> https://github.com/shijunjing/edk2/tree/llvm_v2

>>

>

> Thanks, I will try it on monday

>


I just tried it, and indeed, it turns out that 'hidden' visibility
interacts poorly with LTO, and we should use 'protected' visibility
instead, which achieves the same thing, i.e., that the compiler is
allowed to assume that symbols are not subject to preemption, and so
it can refer to them relatively rather than indirectly via an absolute
address in the GOT.

I will update this in my v3 series.

Thanks,
Ard.
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Patch

diff --git a/MdePkg/Include/X64/ProcessorBind.h b/MdePkg/Include/X64/ProcessorBind.h
index 705104af062a..96df78fca07d 100644
--- a/MdePkg/Include/X64/ProcessorBind.h
+++ b/MdePkg/Include/X64/ProcessorBind.h
@@ -27,6 +27,16 @@ 
 #pragma pack()
 #endif
 
+#if defined(__GNUC__) && defined(__pic__)
+//
+// Mark all symbol declarations and references as hidden, meaning they will not
+// be exported from a shared library, and thus will not be subject to symbol
+// preemption. This allows the compiler to refer to symbols directly using
+// relative references rather than via the GOT, which contains absolute symbol
+// addresses that are subject to runtime relocation.
+//
+#pragma GCC visibility push (hidden)
+#endif
 
 #if defined(__INTEL_COMPILER)
 //