| Message ID | 20221014065705.5249-7-masahisa.kojima@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | eficonfig: add UEFI Secure Boot key maintenance interface | expand |
Hi, On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima <masahisa.kojima@linaro.org> wrote: > > Provide a unit test for the eficonfig secure boot key > management menu. > > Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > --- > No change since v2 > > newly created in v2 > > test/py/tests/test_eficonfig/conftest.py | 84 +++- > test/py/tests/test_eficonfig/defs.py | 14 + > .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > 3 files changed, 568 insertions(+), 2 deletions(-) > create mode 100644 test/py/tests/test_eficonfig/defs.py > create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py Please can this test be in C? Also, using down-arrow to select menus is brittle. Add a function to select the one you want, e.g. by name. Regards, Simon
Hi Simon, On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: > > Hi, > > On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima > <masahisa.kojima@linaro.org> wrote: > > > > Provide a unit test for the eficonfig secure boot key > > management menu. > > > > Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > > --- > > No change since v2 > > > > newly created in v2 > > > > test/py/tests/test_eficonfig/conftest.py | 84 +++- > > test/py/tests/test_eficonfig/defs.py | 14 + > > .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > > 3 files changed, 568 insertions(+), 2 deletions(-) > > create mode 100644 test/py/tests/test_eficonfig/defs.py > > create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py > > Please can this test be in C? Also, using down-arrow to select menus > is brittle. Add a function to select the one you want, e.g. by name. > Is there a very specific reason why we should do stuff like that in C? Python is way easier to extend and test in our case. Thanks /Ilias > Regards, > Simon
Hi Ilias, On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas <ilias.apalodimas@linaro.org> wrote: > > Hi Simon, > > On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: > > > > Hi, > > > > On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima > > <masahisa.kojima@linaro.org> wrote: > > > > > > Provide a unit test for the eficonfig secure boot key > > > management menu. > > > > > > Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > > > --- > > > No change since v2 > > > > > > newly created in v2 > > > > > > test/py/tests/test_eficonfig/conftest.py | 84 +++- > > > test/py/tests/test_eficonfig/defs.py | 14 + > > > .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > > > 3 files changed, 568 insertions(+), 2 deletions(-) > > > create mode 100644 test/py/tests/test_eficonfig/defs.py > > > create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py > > > > Please can this test be in C? Also, using down-arrow to select menus > > is brittle. Add a function to select the one you want, e.g. by name. > > > > Is there a very specific reason why we should do stuff like that in C? Yes, see here. > Python is way easier to extend and test in our case. In what way? It seems a lot more complicated, plus the brittle nature of this test suggests it will be a hassle to maintain. https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c There is a pending update here too: https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ Regards, SImon
On 10/15/22 03:10, Simon Glass wrote: > Hi Ilias, > > On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas > <ilias.apalodimas@linaro.org> wrote: >> >> Hi Simon, >> >> On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: >>> >>> Hi, >>> >>> On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima >>> <masahisa.kojima@linaro.org> wrote: >>>> >>>> Provide a unit test for the eficonfig secure boot key >>>> management menu. >>>> >>>> Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> >>>> --- >>>> No change since v2 >>>> >>>> newly created in v2 >>>> >>>> test/py/tests/test_eficonfig/conftest.py | 84 +++- >>>> test/py/tests/test_eficonfig/defs.py | 14 + >>>> .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ >>>> 3 files changed, 568 insertions(+), 2 deletions(-) >>>> create mode 100644 test/py/tests/test_eficonfig/defs.py >>>> create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py >>> >>> Please can this test be in C? Also, using down-arrow to select menus >>> is brittle. Add a function to select the one you want, e.g. by name. >>> >> >> Is there a very specific reason why we should do stuff like that in C? > > Yes, see here. > >> Python is way easier to extend and test in our case. > > In what way? It seems a lot more complicated, plus the brittle nature > of this test suggests it will be a hassle to maintain. > > https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c > > There is a pending update here too: > > https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ The discussion touches different aspects: ** What does it take to make a GUI easily testable? ** Relying on cursor movement for testing is fragile. This is why GUIs often assign to each executable element the following: * Access key, e.g <CTRL><S> A key combination that when entered will have the the same effect as selecting the GUI item * Access string, '@SAVE' A string that when typed into the command field will have the same effect as selecting the GUI item Let's look at U-Boot's menu entry definition: struct bootmenu_entry { unsigned short int num; // unique number 0 .. MAX_COUNT char key[3]; // key identifier of number char *title; // title of entry char *command; // hush command of entry enum boot_type type; // boot type of entry u16 bootorder; // order for each boot type struct bootmenu_data *menu; // this bootmenu struct bootmenu_entry *next; // next menu entry (num+1) } Our structure lacks an accessor element that can be used to select a menu item without using cursor actions. Compound keystrokes like <CTRL><F6> are send as multiple bytes on the console, e.g. 1b 5b 31 37 3b 35 7e. We may define a field shortcut of type char *. If the string is received by the menu loop, let it activate the matching menu entry. Let cursor actions (up, down, enter, space '+', '-') interrupt matching the shortcut string. Instead we could also use a convention for the title: If a letter in the title is preceded by '&', this is the shortcut key. This letter will be shown highlighted in the menu and the ampersand will not be shown. This is probably easier to implement. Adding the shortcut facility will allow both for easier testing and faster navigation. ** Choice of programming language ** Several aspects control the choice of the programming language for tests: - Testing single library functions is only possible in C. - Checking contents of internal structures is only possible in C. - Testing the U-Boot's CLI is easily accessible in our Python framework. - Preparation of complex test data is easier to do in Python. - Mixed language tests should be avoided if not strictly necessary. It is much easier to maintain a single code source for a test. ** What to do for secure boot key management? ** Secure boot key management requires complex preparation which fits well into out Python testing framework. Once we provide shortcut keys to U-Boot menus the entries will be easily accessible from Python. As we want to avoid complexity due to mixed language tests we should stick to Python for testing key management at the user level. Additional tests in C for library functions managing keys and verifying signatures would improve our code coverage. Best regards Heinrich
Hi Heinrich, On Sat, 15 Oct 2022 at 13:48, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > On 10/15/22 03:10, Simon Glass wrote: > > Hi Ilias, > > > > On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas > > <ilias.apalodimas@linaro.org> wrote: > >> > >> Hi Simon, > >> > >> On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: > >>> > >>> Hi, > >>> > >>> On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima > >>> <masahisa.kojima@linaro.org> wrote: > >>>> > >>>> Provide a unit test for the eficonfig secure boot key > >>>> management menu. > >>>> > >>>> Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > >>>> --- > >>>> No change since v2 > >>>> > >>>> newly created in v2 > >>>> > >>>> test/py/tests/test_eficonfig/conftest.py | 84 +++- > >>>> test/py/tests/test_eficonfig/defs.py | 14 + > >>>> .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > >>>> 3 files changed, 568 insertions(+), 2 deletions(-) > >>>> create mode 100644 test/py/tests/test_eficonfig/defs.py > >>>> create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py > >>> > >>> Please can this test be in C? Also, using down-arrow to select menus > >>> is brittle. Add a function to select the one you want, e.g. by name. > >>> > >> > >> Is there a very specific reason why we should do stuff like that in C? > > > > Yes, see here. > > > >> Python is way easier to extend and test in our case. > > > > In what way? It seems a lot more complicated, plus the brittle nature > > of this test suggests it will be a hassle to maintain. > > > > https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c > > > > There is a pending update here too: > > > > https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ > > The discussion touches different aspects: > > ** What does it take to make a GUI easily testable? ** > > Relying on cursor movement for testing is fragile. This is why GUIs > often assign to each executable element the following: > > * Access key, e.g <CTRL><S> > A key combination that when entered will have the the same > effect as selecting the GUI item > * Access string, '@SAVE' > A string that when typed into the command field will have > the same effect as selecting the GUI item > > Let's look at U-Boot's menu entry definition: > > struct bootmenu_entry { > unsigned short int num; // unique number 0 .. MAX_COUNT > char key[3]; // key identifier of number > char *title; // title of entry > char *command; // hush command of entry > enum boot_type type; // boot type of entry > u16 bootorder; // order for each boot type > struct bootmenu_data *menu; // this bootmenu > struct bootmenu_entry *next; // next menu entry (num+1) > } > > Our structure lacks an accessor element that can be used to select a > menu item without using cursor actions. > > Compound keystrokes like <CTRL><F6> are send as multiple bytes on the > console, e.g. 1b 5b 31 37 3b 35 7e. > > We may define a field shortcut of type char *. If the string is received > by the menu loop, let it activate the matching menu entry. Let cursor > actions (up, down, enter, space '+', '-') interrupt matching the > shortcut string. > > Instead we could also use a convention for the title: > > If a letter in the title is preceded by '&', this is the shortcut key. > This letter will be shown highlighted in the menu and the ampersand will > not be shown. > > This is probably easier to implement. I agree to the shortcut by '&', I will implement it and update the python test code. Thanks, Masahisa Kojima > > Adding the shortcut facility will allow both for easier testing and > faster navigation. > > ** Choice of programming language ** > > Several aspects control the choice of the programming language for tests: > > - Testing single library functions is only possible in C. > - Checking contents of internal structures is only possible in C. > - Testing the U-Boot's CLI is easily accessible in our Python framework. > - Preparation of complex test data is easier to do in Python. > - Mixed language tests should be avoided if not strictly necessary. > It is much easier to maintain a single code source for a test. > > ** What to do for secure boot key management? ** > > Secure boot key management requires complex preparation which fits well > into out Python testing framework. > > Once we provide shortcut keys to U-Boot menus the entries will be easily > accessible from Python. > > As we want to avoid complexity due to mixed language tests we should > stick to Python for testing key management at the user level. > > Additional tests in C for library functions managing keys and verifying > signatures would improve our code coverage. > > Best regards > > Heinrich
Hi Heinrich, Ilias, Simon, On Mon, 17 Oct 2022 at 13:49, Masahisa Kojima <masahisa.kojima@linaro.org> wrote: > > Hi Heinrich, > > On Sat, 15 Oct 2022 at 13:48, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > > > On 10/15/22 03:10, Simon Glass wrote: > > > Hi Ilias, > > > > > > On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas > > > <ilias.apalodimas@linaro.org> wrote: > > >> > > >> Hi Simon, > > >> > > >> On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: > > >>> > > >>> Hi, > > >>> > > >>> On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima > > >>> <masahisa.kojima@linaro.org> wrote: > > >>>> > > >>>> Provide a unit test for the eficonfig secure boot key > > >>>> management menu. > > >>>> > > >>>> Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > > >>>> --- > > >>>> No change since v2 > > >>>> > > >>>> newly created in v2 > > >>>> > > >>>> test/py/tests/test_eficonfig/conftest.py | 84 +++- > > >>>> test/py/tests/test_eficonfig/defs.py | 14 + > > >>>> .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > > >>>> 3 files changed, 568 insertions(+), 2 deletions(-) > > >>>> create mode 100644 test/py/tests/test_eficonfig/defs.py > > >>>> create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py > > >>> > > >>> Please can this test be in C? Also, using down-arrow to select menus > > >>> is brittle. Add a function to select the one you want, e.g. by name. > > >>> > > >> > > >> Is there a very specific reason why we should do stuff like that in C? > > > > > > Yes, see here. > > > > > >> Python is way easier to extend and test in our case. > > > > > > In what way? It seems a lot more complicated, plus the brittle nature > > > of this test suggests it will be a hassle to maintain. > > > > > > https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c > > > > > > There is a pending update here too: > > > > > > https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ > > > > The discussion touches different aspects: > > > > ** What does it take to make a GUI easily testable? ** > > > > Relying on cursor movement for testing is fragile. This is why GUIs > > often assign to each executable element the following: > > > > * Access key, e.g <CTRL><S> > > A key combination that when entered will have the the same > > effect as selecting the GUI item > > * Access string, '@SAVE' > > A string that when typed into the command field will have > > the same effect as selecting the GUI item > > > > Let's look at U-Boot's menu entry definition: > > > > struct bootmenu_entry { > > unsigned short int num; // unique number 0 .. MAX_COUNT > > char key[3]; // key identifier of number > > char *title; // title of entry > > char *command; // hush command of entry > > enum boot_type type; // boot type of entry > > u16 bootorder; // order for each boot type > > struct bootmenu_data *menu; // this bootmenu > > struct bootmenu_entry *next; // next menu entry (num+1) > > } > > > > Our structure lacks an accessor element that can be used to select a > > menu item without using cursor actions. > > > > Compound keystrokes like <CTRL><F6> are send as multiple bytes on the > > console, e.g. 1b 5b 31 37 3b 35 7e. > > > > We may define a field shortcut of type char *. If the string is received > > by the menu loop, let it activate the matching menu entry. Let cursor > > actions (up, down, enter, space '+', '-') interrupt matching the > > shortcut string. > > > > Instead we could also use a convention for the title: > > > > If a letter in the title is preceded by '&', this is the shortcut key. > > This letter will be shown highlighted in the menu and the ampersand will > > not be shown. > > > > This is probably easier to implement. > > I agree to the shortcut by '&', I will implement it and update the > python test code. The shortcut functionality is a kind of feature enhancement of eficonfig menus. If there is no additional comment and feedback other than the shortcut feature, could you consider to merge this series first? I would like to get more user feedback. I will send the follow up patches to support the shortcut feature after this original series is merged. Thanks, Masahisa Kojima > > Thanks, > Masahisa Kojima > > > > > Adding the shortcut facility will allow both for easier testing and > > faster navigation. > > > > ** Choice of programming language ** > > > > Several aspects control the choice of the programming language for tests: > > > > - Testing single library functions is only possible in C. > > - Checking contents of internal structures is only possible in C. > > - Testing the U-Boot's CLI is easily accessible in our Python framework. > > - Preparation of complex test data is easier to do in Python. > > - Mixed language tests should be avoided if not strictly necessary. > > It is much easier to maintain a single code source for a test. > > > > ** What to do for secure boot key management? ** > > > > Secure boot key management requires complex preparation which fits well > > into out Python testing framework. > > > > Once we provide shortcut keys to U-Boot menus the entries will be easily > > accessible from Python. > > > > As we want to avoid complexity due to mixed language tests we should > > stick to Python for testing key management at the user level. > > > > Additional tests in C for library functions managing keys and verifying > > signatures would improve our code coverage. > > > > Best regards > > > > Heinrich
iHi Heinrich, On Fri, 14 Oct 2022 at 22:43, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > On 10/15/22 03:10, Simon Glass wrote: > > Hi Ilias, > > > > On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas > > <ilias.apalodimas@linaro.org> wrote: > >> > >> Hi Simon, > >> > >> On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: > >>> > >>> Hi, > >>> > >>> On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima > >>> <masahisa.kojima@linaro.org> wrote: > >>>> > >>>> Provide a unit test for the eficonfig secure boot key > >>>> management menu. > >>>> > >>>> Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > >>>> --- > >>>> No change since v2 > >>>> > >>>> newly created in v2 > >>>> > >>>> test/py/tests/test_eficonfig/conftest.py | 84 +++- > >>>> test/py/tests/test_eficonfig/defs.py | 14 + > >>>> .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > >>>> 3 files changed, 568 insertions(+), 2 deletions(-) > >>>> create mode 100644 test/py/tests/test_eficonfig/defs.py > >>>> create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py > >>> > >>> Please can this test be in C? Also, using down-arrow to select menus > >>> is brittle. Add a function to select the one you want, e.g. by name. > >>> > >> > >> Is there a very specific reason why we should do stuff like that in C? > > > > Yes, see here. > > > >> Python is way easier to extend and test in our case. > > > > In what way? It seems a lot more complicated, plus the brittle nature > > of this test suggests it will be a hassle to maintain. > > > > https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c > > > > There is a pending update here too: > > > > https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ > > The discussion touches different aspects: > > ** What does it take to make a GUI easily testable? ** > > Relying on cursor movement for testing is fragile. This is why GUIs > often assign to each executable element the following: > > * Access key, e.g <CTRL><S> > A key combination that when entered will have the the same > effect as selecting the GUI item > * Access string, '@SAVE' > A string that when typed into the command field will have > the same effect as selecting the GUI item > > Let's look at U-Boot's menu entry definition: > > struct bootmenu_entry { > unsigned short int num; // unique number 0 .. MAX_COUNT > char key[3]; // key identifier of number > char *title; // title of entry > char *command; // hush command of entry > enum boot_type type; // boot type of entry > u16 bootorder; // order for each boot type > struct bootmenu_data *menu; // this bootmenu > struct bootmenu_entry *next; // next menu entry (num+1) > } > > Our structure lacks an accessor element that can be used to select a > menu item without using cursor actions. > > Compound keystrokes like <CTRL><F6> are send as multiple bytes on the > console, e.g. 1b 5b 31 37 3b 35 7e. > > We may define a field shortcut of type char *. If the string is received > by the menu loop, let it activate the matching menu entry. Let cursor > actions (up, down, enter, space '+', '-') interrupt matching the > shortcut string. > > Instead we could also use a convention for the title: > > If a letter in the title is preceded by '&', this is the shortcut key. > This letter will be shown highlighted in the menu and the ampersand will > not be shown. > > This is probably easier to implement. > > Adding the shortcut facility will allow both for easier testing and > faster navigation. > > ** Choice of programming language ** > > Several aspects control the choice of the programming language for tests: > > - Testing single library functions is only possible in C. > - Checking contents of internal structures is only possible in C. > - Testing the U-Boot's CLI is easily accessible in our Python framework. > - Preparation of complex test data is easier to do in Python. > - Mixed language tests should be avoided if not strictly necessary. > It is much easier to maintain a single code source for a test. > > ** What to do for secure boot key management? ** > > Secure boot key management requires complex preparation which fits well > into out Python testing framework. > > Once we provide shortcut keys to U-Boot menus the entries will be easily > accessible from Python. > > As we want to avoid complexity due to mixed language tests we should > stick to Python for testing key management at the user level. The test setup should be in Python (and needs to be) but the actual test should be in C, in general. From what I can tell these tests should be split up, one for each test case. Using keyboard shortcuts for menus would certainly help, but you still have the dependency between the menu code in C and the Python code that uses it. The complexity of this dependency is quite significant. Can we not design this all for easier testing? The functionality should be broken down a little more so we can change the configs in a test (e.g. using the eficonfig command), then check that booting does what we expect. Overall I feel that this 'black box' testing is not a good approach and we should try to test the components. It is OK to have a happy-path test but here we are just writing hundreds of lines of Python to get a very high-level signal about correctness. > > Additional tests in C for library functions managing keys and verifying > signatures would improve our code coverage. > > Best regards > > Heinrich Regards, Simon
On 10/19/22 15:17, Simon Glass wrote: > iHi Heinrich, > > On Fri, 14 Oct 2022 at 22:43, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: >> >> On 10/15/22 03:10, Simon Glass wrote: >>> Hi Ilias, >>> >>> On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas >>> <ilias.apalodimas@linaro.org> wrote: >>>> >>>> Hi Simon, >>>> >>>> On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: >>>>> >>>>> Hi, >>>>> >>>>> On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima >>>>> <masahisa.kojima@linaro.org> wrote: >>>>>> >>>>>> Provide a unit test for the eficonfig secure boot key >>>>>> management menu. >>>>>> >>>>>> Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> >>>>>> --- >>>>>> No change since v2 >>>>>> >>>>>> newly created in v2 >>>>>> >>>>>> test/py/tests/test_eficonfig/conftest.py | 84 +++- >>>>>> test/py/tests/test_eficonfig/defs.py | 14 + >>>>>> .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ >>>>>> 3 files changed, 568 insertions(+), 2 deletions(-) >>>>>> create mode 100644 test/py/tests/test_eficonfig/defs.py >>>>>> create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py >>>>> >>>>> Please can this test be in C? Also, using down-arrow to select menus >>>>> is brittle. Add a function to select the one you want, e.g. by name. >>>>> >>>> >>>> Is there a very specific reason why we should do stuff like that in C? >>> >>> Yes, see here. >>> >>>> Python is way easier to extend and test in our case. >>> >>> In what way? It seems a lot more complicated, plus the brittle nature >>> of this test suggests it will be a hassle to maintain. >>> >>> https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c >>> >>> There is a pending update here too: >>> >>> https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ >> >> The discussion touches different aspects: >> >> ** What does it take to make a GUI easily testable? ** >> >> Relying on cursor movement for testing is fragile. This is why GUIs >> often assign to each executable element the following: >> >> * Access key, e.g <CTRL><S> >> A key combination that when entered will have the the same >> effect as selecting the GUI item >> * Access string, '@SAVE' >> A string that when typed into the command field will have >> the same effect as selecting the GUI item >> >> Let's look at U-Boot's menu entry definition: >> >> struct bootmenu_entry { >> unsigned short int num; // unique number 0 .. MAX_COUNT >> char key[3]; // key identifier of number >> char *title; // title of entry >> char *command; // hush command of entry >> enum boot_type type; // boot type of entry >> u16 bootorder; // order for each boot type >> struct bootmenu_data *menu; // this bootmenu >> struct bootmenu_entry *next; // next menu entry (num+1) >> } >> >> Our structure lacks an accessor element that can be used to select a >> menu item without using cursor actions. >> >> Compound keystrokes like <CTRL><F6> are send as multiple bytes on the >> console, e.g. 1b 5b 31 37 3b 35 7e. >> >> We may define a field shortcut of type char *. If the string is received >> by the menu loop, let it activate the matching menu entry. Let cursor >> actions (up, down, enter, space '+', '-') interrupt matching the >> shortcut string. >> >> Instead we could also use a convention for the title: >> >> If a letter in the title is preceded by '&', this is the shortcut key. >> This letter will be shown highlighted in the menu and the ampersand will >> not be shown. >> >> This is probably easier to implement. >> >> Adding the shortcut facility will allow both for easier testing and >> faster navigation. >> >> ** Choice of programming language ** >> >> Several aspects control the choice of the programming language for tests: >> >> - Testing single library functions is only possible in C. >> - Checking contents of internal structures is only possible in C. >> - Testing the U-Boot's CLI is easily accessible in our Python framework. >> - Preparation of complex test data is easier to do in Python. >> - Mixed language tests should be avoided if not strictly necessary. >> It is much easier to maintain a single code source for a test. >> >> ** What to do for secure boot key management? ** >> >> Secure boot key management requires complex preparation which fits well >> into out Python testing framework. >> >> Once we provide shortcut keys to U-Boot menus the entries will be easily >> accessible from Python. >> >> As we want to avoid complexity due to mixed language tests we should >> stick to Python for testing key management at the user level. > > The test setup should be in Python (and needs to be) but the actual > test should be in C, in general. > >>From what I can tell these tests should be split up, one for each test case. > > Using keyboard shortcuts for menus would certainly help, but you still > have the dependency between the menu code in C and the Python code > that uses it. The complexity of this dependency is quite significant. > > Can we not design this all for easier testing? The functionality > should be broken down a little more so we can change the configs in a > test (e.g. using the eficonfig command), then check that booting does > what we expect. > > Overall I feel that this 'black box' testing is not a good approach > and we should try to test the components. It is OK to have a > happy-path test but here we are just writing hundreds of lines of > Python to get a very high-level signal about correctness. From other projects I am used to differentiate between unit tests and integration tests. An integration test is what checks that all the parts work together while a unit test checks a single functionality works as designed. A description can be found in https://circleci.com/blog/unit-testing-vs-integration-testing/#c-consent-modal For an integration test this Python script is fine (except for the menu navigation. As I said in my last mail additional tests on a lower level may be added. But unit tests cannot replace an integration test. Best regards Heinrich > >> >> Additional tests in C for library functions managing keys and verifying >> signatures would improve our code coverage. >> >> Best regards >> >> Heinrich > > Regards, > Simon
Hi Heinrich, On Wed, 19 Oct 2022 at 15:39, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > On 10/19/22 15:17, Simon Glass wrote: > > iHi Heinrich, > > > > On Fri, 14 Oct 2022 at 22:43, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > >> > >> On 10/15/22 03:10, Simon Glass wrote: > >>> Hi Ilias, > >>> > >>> On Fri, 14 Oct 2022 at 09:59, Ilias Apalodimas > >>> <ilias.apalodimas@linaro.org> wrote: > >>>> > >>>> Hi Simon, > >>>> > >>>> On Fri, 14 Oct 2022 at 18:56, Simon Glass <sjg@chromium.org> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> On Fri, 14 Oct 2022 at 00:58, Masahisa Kojima > >>>>> <masahisa.kojima@linaro.org> wrote: > >>>>>> > >>>>>> Provide a unit test for the eficonfig secure boot key > >>>>>> management menu. > >>>>>> > >>>>>> Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> > >>>>>> --- > >>>>>> No change since v2 > >>>>>> > >>>>>> newly created in v2 > >>>>>> > >>>>>> test/py/tests/test_eficonfig/conftest.py | 84 +++- > >>>>>> test/py/tests/test_eficonfig/defs.py | 14 + > >>>>>> .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ > >>>>>> 3 files changed, 568 insertions(+), 2 deletions(-) > >>>>>> create mode 100644 test/py/tests/test_eficonfig/defs.py > >>>>>> create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py > >>>>> > >>>>> Please can this test be in C? Also, using down-arrow to select menus > >>>>> is brittle. Add a function to select the one you want, e.g. by name. > >>>>> > >>>> > >>>> Is there a very specific reason why we should do stuff like that in C? > >>> > >>> Yes, see here. > >>> > >>>> Python is way easier to extend and test in our case. > >>> > >>> In what way? It seems a lot more complicated, plus the brittle nature > >>> of this test suggests it will be a hassle to maintain. > >>> > >>> https://u-boot.readthedocs.io/en/latest/develop/tests_writing.html#python-or-c > >>> > >>> There is a pending update here too: > >>> > >>> https://patchwork.ozlabs.org/project/uboot/patch/20221013122927.636867-15-sjg@chromium.org/ > >> > >> The discussion touches different aspects: > >> > >> ** What does it take to make a GUI easily testable? ** > >> > >> Relying on cursor movement for testing is fragile. This is why GUIs > >> often assign to each executable element the following: > >> > >> * Access key, e.g <CTRL><S> > >> A key combination that when entered will have the the same > >> effect as selecting the GUI item > >> * Access string, '@SAVE' > >> A string that when typed into the command field will have > >> the same effect as selecting the GUI item > >> > >> Let's look at U-Boot's menu entry definition: > >> > >> struct bootmenu_entry { > >> unsigned short int num; // unique number 0 .. MAX_COUNT > >> char key[3]; // key identifier of number > >> char *title; // title of entry > >> char *command; // hush command of entry > >> enum boot_type type; // boot type of entry > >> u16 bootorder; // order for each boot type > >> struct bootmenu_data *menu; // this bootmenu > >> struct bootmenu_entry *next; // next menu entry (num+1) > >> } > >> > >> Our structure lacks an accessor element that can be used to select a > >> menu item without using cursor actions. > >> > >> Compound keystrokes like <CTRL><F6> are send as multiple bytes on the > >> console, e.g. 1b 5b 31 37 3b 35 7e. > >> > >> We may define a field shortcut of type char *. If the string is received > >> by the menu loop, let it activate the matching menu entry. Let cursor > >> actions (up, down, enter, space '+', '-') interrupt matching the > >> shortcut string. > >> > >> Instead we could also use a convention for the title: > >> > >> If a letter in the title is preceded by '&', this is the shortcut key. > >> This letter will be shown highlighted in the menu and the ampersand will > >> not be shown. > >> > >> This is probably easier to implement. > >> > >> Adding the shortcut facility will allow both for easier testing and > >> faster navigation. > >> > >> ** Choice of programming language ** > >> > >> Several aspects control the choice of the programming language for tests: > >> > >> - Testing single library functions is only possible in C. > >> - Checking contents of internal structures is only possible in C. > >> - Testing the U-Boot's CLI is easily accessible in our Python framework. > >> - Preparation of complex test data is easier to do in Python. > >> - Mixed language tests should be avoided if not strictly necessary. > >> It is much easier to maintain a single code source for a test. > >> > >> ** What to do for secure boot key management? ** > >> > >> Secure boot key management requires complex preparation which fits well > >> into out Python testing framework. > >> > >> Once we provide shortcut keys to U-Boot menus the entries will be easily > >> accessible from Python. > >> > >> As we want to avoid complexity due to mixed language tests we should > >> stick to Python for testing key management at the user level. > > > > The test setup should be in Python (and needs to be) but the actual > > test should be in C, in general. > > > >>From what I can tell these tests should be split up, one for each test case. > > > > Using keyboard shortcuts for menus would certainly help, but you still > > have the dependency between the menu code in C and the Python code > > that uses it. The complexity of this dependency is quite significant. > > > > Can we not design this all for easier testing? The functionality > > should be broken down a little more so we can change the configs in a > > test (e.g. using the eficonfig command), then check that booting does > > what we expect. > > > > Overall I feel that this 'black box' testing is not a good approach > > and we should try to test the components. It is OK to have a > > happy-path test but here we are just writing hundreds of lines of > > Python to get a very high-level signal about correctness. > > From other projects I am used to differentiate between unit tests and > integration tests. > > An integration test is what checks that all the parts work together > while a unit test checks a single functionality works as designed. > > A description can be found in > https://circleci.com/blog/unit-testing-vs-integration-testing/#c-consent-modal > > For an integration test this Python script is fine (except for the menu > navigation. > > As I said in my last mail additional tests on a lower level may be > added. But unit tests cannot replace an integration test. We should do the C test first, then. It just has so much more visibility into what is going on. I am OK with an integration test, but it should never find a bug that is not found by the sandbox unit test. The Python tests are much harder to debug! So let's get this sorted out so we don't end up with a complete mess here. Regards, Simon
diff --git a/test/py/tests/test_eficonfig/conftest.py b/test/py/tests/test_eficonfig/conftest.py index f289df0362..6750d33989 100644 --- a/test/py/tests/test_eficonfig/conftest.py +++ b/test/py/tests/test_eficonfig/conftest.py @@ -2,11 +2,12 @@ """Fixture for UEFI eficonfig test """ - import os +import os.path import shutil -from subprocess import check_call +from subprocess import call, check_call, check_output, CalledProcessError import pytest +from defs import * @pytest.fixture(scope='session') def efi_eficonfig_data(u_boot_config): @@ -38,3 +39,82 @@ def efi_eficonfig_data(u_boot_config): shell=True) return image_path + +@pytest.fixture(scope='session') +def efi_boot_env(request, u_boot_config): + """Set up a file system to be used in UEFI secure boot test. + + Args: + request: Pytest request object. + u_boot_config: U-boot configuration. + + Return: + A path to disk image to be used for testing + """ + image_path = u_boot_config.persistent_data_dir + image_path = image_path + '/test_eficonfig_sb.img' + + try: + mnt_point = u_boot_config.build_dir + '/mnt_eficonfig_sb' + check_call('rm -rf {}'.format(mnt_point), shell=True) + check_call('mkdir -p {}'.format(mnt_point), shell=True) + + # suffix + # *.key: RSA private key in PEM + # *.crt: X509 certificate (self-signed) in PEM + # *.esl: signature list + # *.hash: message digest of image as signature list + # *.auth: signed signature list in signature database format + # *.efi: UEFI image + # *.efi.signed: signed UEFI image + + # Create signature database + # PK + check_call('cd %s; openssl req -x509 -sha256 -newkey rsa:2048 -subj /CN=TEST_PK/ -keyout PK.key -out PK.crt -nodes -days 365' + % mnt_point, shell=True) + check_call('cd %s; %scert-to-efi-sig-list -g %s PK.crt PK.esl; %ssign-efi-sig-list -t "2020-04-01" -c PK.crt -k PK.key PK PK.esl PK.auth' + % (mnt_point, EFITOOLS_PATH, GUID, EFITOOLS_PATH), + shell=True) + # PK_null for deletion + check_call('cd %s; touch PK_null.esl; %ssign-efi-sig-list -t "2020-04-02" -c PK.crt -k PK.key PK PK_null.esl PK_null.auth' + % (mnt_point, EFITOOLS_PATH), shell=True) + # KEK + check_call('cd %s; openssl req -x509 -sha256 -newkey rsa:2048 -subj /CN=TEST_KEK/ -keyout KEK.key -out KEK.crt -nodes -days 365' + % mnt_point, shell=True) + check_call('cd %s; %scert-to-efi-sig-list -g %s KEK.crt KEK.esl; %ssign-efi-sig-list -t "2020-04-03" -c PK.crt -k PK.key KEK KEK.esl KEK.auth' + % (mnt_point, EFITOOLS_PATH, GUID, EFITOOLS_PATH), + shell=True) + # db + check_call('cd %s; openssl req -x509 -sha256 -newkey rsa:2048 -subj /CN=TEST_db/ -keyout db.key -out db.crt -nodes -days 365' + % mnt_point, shell=True) + check_call('cd %s; %scert-to-efi-sig-list -g %s db.crt db.esl; %ssign-efi-sig-list -t "2020-04-04" -c KEK.crt -k KEK.key db db.esl db.auth' + % (mnt_point, EFITOOLS_PATH, GUID, EFITOOLS_PATH), + shell=True) + + # dbx_hash (digest of TEST_db certificate) + check_call('cd %s; %scert-to-efi-hash-list -g %s -t "2013-05-27 01:02:03" -s 256 db.crt dbx_hash.crl; %ssign-efi-sig-list -t "2020-04-05" -c KEK.crt -k KEK.key dbx dbx_hash.crl dbx_hash.auth' + % (mnt_point, EFITOOLS_PATH, GUID, EFITOOLS_PATH), + shell=True) + + # Copy image + check_call('cp %s/lib/efi_loader/helloworld.efi %s' % + (u_boot_config.build_dir, mnt_point), shell=True) + + # Sign image + check_call('cd %s; sbsign --key db.key --cert db.crt helloworld.efi' + % mnt_point, shell=True) + + check_call('cd %s; rm -f *.key' % mnt_point, shell=True) + check_call('cd %s; rm -f *.crt' % mnt_point, shell=True) + check_call('cd %s; rm -f *.hash' % mnt_point, shell=True) + check_call('virt-make-fs --partition=gpt --size=+1M --type=vfat {} {}'.format( + mnt_point, image_path), shell=True) + check_call('rm -rf {}'.format(mnt_point), shell=True) + + except CalledProcessError as exception: + pytest.skip('Setup failed: %s' % exception.cmd) + return + else: + yield image_path + finally: + call('rm -f %s' % image_path, shell=True) diff --git a/test/py/tests/test_eficonfig/defs.py b/test/py/tests/test_eficonfig/defs.py new file mode 100644 index 0000000000..b7a2a11851 --- /dev/null +++ b/test/py/tests/test_eficonfig/defs.py @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0+ + +# Owner guid +GUID = '11111111-2222-3333-4444-123456789abc' + +# v1.5.1 or earlier of efitools has a bug in sha256 calculation, and +# you need build a newer version on your own. +# The path must terminate with '/'. +EFITOOLS_PATH = '' + +# "--addcert" option of sbsign must be available, otherwise +# you need build a newer version on your own. +# The path must terminate with '/'. +SBSIGN_PATH = '' diff --git a/test/py/tests/test_eficonfig/test_eficonfig_sbkey.py b/test/py/tests/test_eficonfig/test_eficonfig_sbkey.py new file mode 100644 index 0000000000..727288964d --- /dev/null +++ b/test/py/tests/test_eficonfig/test_eficonfig_sbkey.py @@ -0,0 +1,472 @@ +# SPDX-License-Identifier: GPL-2.0+ +""" Unit test for UEFI menu-driven configuration +""" + +import pytest +import time +from defs import * + +@pytest.mark.boardspec('sandbox') +@pytest.mark.buildconfigspec('cmd_eficonfig') +@pytest.mark.buildconfigspec('cmd_bootefi_bootmgr') +@pytest.mark.buildconfigspec('efi_secure_boot') +def test_efi_eficonfig_sbkey(u_boot_config, u_boot_console, efi_boot_env): + def send_user_input_and_wait(user_str, expect_str): + time.sleep(0.1) # TODO: does not work correctly without sleep + u_boot_console.run_command(cmd=user_str, wait_for_prompt=False, + wait_for_echo=True, send_nl=False) + u_boot_console.run_command(cmd='\x0d', wait_for_prompt=False, + wait_for_echo=False, send_nl=False) + if expect_str is not None: + for i in expect_str: + u_boot_console.p.expect([i]) + + def press_up_down_enter_and_wait(up_count, down_count, enter, expect_str): + # press UP key + for i in range(up_count): + u_boot_console.run_command(cmd='\x1b\x5b\x41', wait_for_prompt=False, + wait_for_echo=False, send_nl=False) + # press DOWN key + for i in range(down_count): + u_boot_console.run_command(cmd='\x1b\x5b\x42', wait_for_prompt=False, + wait_for_echo=False, send_nl=False) + # press ENTER if requested + if enter: + u_boot_console.run_command(cmd='\x0d', wait_for_prompt=False, + wait_for_echo=False, send_nl=False) + # wait expected output + if expect_str is not None: + for i in expect_str: + u_boot_console.p.expect([i]) + + def press_escape_key(wait_prompt): + u_boot_console.run_command(cmd='\x1b', wait_for_prompt=wait_prompt, wait_for_echo=False, send_nl=False) + + def press_enter_key(wait_prompt): + u_boot_console.run_command(cmd='\x0d', wait_for_prompt=wait_prompt, + wait_for_echo=False, send_nl=False) + + def check_current_is_maintenance_menu(): + for i in ('UEFI Maintenance Menu', 'Add Boot Option', 'Edit Boot Option', + 'Change Boot Order', 'Delete Boot Option', 'Secure Boot Configuration', 'Quit'): + u_boot_console.p.expect([i]) + + # Restart the system to clean the previous state + u_boot_console.restart_uboot() + # bind the test disk image for succeeding tests + u_boot_console.run_command(cmd = f'host bind 0 {efi_boot_env}') + + # + # Test Case 1: Enroll non-signed ESL(.esl or .crl) in order of KEK, DB, DBX and PK + # + with u_boot_console.temporary_timeout(500): + u_boot_console.run_command('eficonfig', wait_for_prompt=False) + check_current_is_maintenance_menu() + press_up_down_enter_and_wait(0, 4, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'OFF'): + u_boot_console.p.expect([i]) + + # set KEK.esl to KEK + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 7, True, 'Quit') + # check KEK is expected value + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'KEK', + 'Owner GUID:', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type:', 'X509', 'Subject:', 'TEST_KEK', 'Issuer:', 'TEST_KEK'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # set db.esl to db + press_up_down_enter_and_wait(0, 2, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 1, True, 'Quit') + # check db is expected value + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'db', + 'Owner GUID:', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type:', 'X509', 'Subject:', 'TEST_db', 'Issuer:', 'TEST_db'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # set dbx_hash.crl to dbx + press_up_down_enter_and_wait(0, 3, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 3, True, 'Quit') + # check dbx is expected value + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, None) + # verify CRL, skip hash comparison because it varies in every test + for i in ('Show/Delete Signature Database', 'dbx', + 'Owner GUID:', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type:', 'X509_SHA256 CRL', + 'TimeOfRevocation:', '2013-5-27 01:02:03'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # set PK.esl to PK + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 9, True, 'Quit') + # check PK is expected value + press_up_down_enter_and_wait(0, 1, True, None) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'PK', + 'Owner GUID', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type', 'X509', 'Subject', 'TEST_PK', 'Issuer', 'TEST_PK', + 'Can not delete PK, Press any key to continue'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'ON'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 4, True, 'Quit') + check_current_is_maintenance_menu() + + # + # Test Case 2: Enroll PK first, then non-signed esl fails to enroll + # + + # Restart the system to clean the previous state + u_boot_console.restart_uboot() + # bind the test disk image for succeeding tests + u_boot_console.run_command(cmd = f'host bind 0 {efi_boot_env}') + + with u_boot_console.temporary_timeout(500): + u_boot_console.run_command('eficonfig', wait_for_prompt=False) + check_current_is_maintenance_menu() + press_up_down_enter_and_wait(0, 4, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'OFF'): + u_boot_console.p.expect([i]) + + # set PK.auth to PK + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 8, True, 'Quit') + # check PK is expected value + press_up_down_enter_and_wait(0, 1, True, None) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'PK', + 'Owner GUID', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type', 'X509', 'Subject', 'TEST_PK', 'Issuer', 'TEST_PK', + 'Can not delete PK, Press any key to continue'): + u_boot_console.p.expect([i]) + + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'ON'): + u_boot_console.p.expect([i]) + + # fail to set KEK.esl + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 7, True, 'Quit') + for i in ('ERROR! Failed to update signature database', + 'Press any key to continue'): + u_boot_console.p.expect([i]) + press_escape_key(False) + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # fail to set db.esl + press_up_down_enter_and_wait(0, 2, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 1, True, 'Quit') + for i in ('ERROR! Failed to update signature database', + 'Press any key to continue'): + u_boot_console.p.expect([i]) + press_escape_key(False) + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # fail to set dbx_hash.crl + press_up_down_enter_and_wait(0, 3, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 3, True, 'Quit') + for i in ('ERROR! Failed to update signature database', + 'Press any key to continue'): + u_boot_console.p.expect([i]) + press_escape_key(False) + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # + # Test Case 3: Enroll signed ESL(.auth) in order of PK, KEK, and db, then check status + # + + # Restart the system to clean the previous state + u_boot_console.restart_uboot() + # bind the test disk image for succeeding tests + u_boot_console.run_command(cmd = f'host bind 0 {efi_boot_env}') + + with u_boot_console.temporary_timeout(500): + u_boot_console.run_command('eficonfig', wait_for_prompt=False) + check_current_is_maintenance_menu() + press_up_down_enter_and_wait(0, 4, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'OFF'): + u_boot_console.p.expect([i]) + + # set PK.auth to PK + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 8, True, 'Quit') + # check PK is expected value + press_up_down_enter_and_wait(0, 1, True, None) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'PK', + 'Owner GUID', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type', 'X509', 'Subject', 'TEST_PK', 'Issuer', 'TEST_PK', + 'Can not delete PK, Press any key to continue'): + u_boot_console.p.expect([i]) + + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # set KEK.auth to KEK + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 6, True, 'Quit') + # check KEK is expected value + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'KEK', + 'Owner GUID:', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type:', 'X509', 'Subject:', 'TEST_KEK', 'Issuer:', 'TEST_KEK'): + u_boot_console.p.expect([i]) + + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # set db.auth to db + press_up_down_enter_and_wait(0, 2, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + # check db is expected value + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'db', + 'Owner GUID:', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type:', 'X509', 'Subject:', 'TEST_db', 'Issuer:', 'TEST_db'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'ON'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 4, True, 'Quit') + check_current_is_maintenance_menu() + + # + # Test Case 4: start signed image allowed in db + # + + # Select 'Add Boot Option' + press_up_down_enter_and_wait(0, 0, True, 'Quit') + # Press the enter key to select 'Description:' entry, then enter Description + press_up_down_enter_and_wait(0, 0, True, 'enter description:') + # Send Description user input, press ENTER key to complete + send_user_input_and_wait('hello', 'Quit') + + # Set EFI image(helloworld.efi.signed) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'host 0:1') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 5, True, 'Quit') + for i in ('Description: hello', 'File: host 0:1/helloworld.efi.signed', + 'Initrd File:', 'Optional Data:', 'Save', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 4, True, 'Quit') + press_escape_key(False) + check_current_is_maintenance_menu() + press_escape_key(True) + response = u_boot_console.run_command(cmd = 'bootefi bootmgr') + assert 'Hello, world!' in response + + # + # Test Case 5: can not start the image if it is not signed + # + + u_boot_console.run_command('eficonfig', wait_for_prompt=False) + check_current_is_maintenance_menu() + # Select 'Edit Boot Option' + press_up_down_enter_and_wait(0, 1, True, None) + # Check the curren BootOrder + for i in ('hello', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, None) + # Set EFI image(helloworld.efi) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'host 0:1') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 4, True, 'Quit') + for i in ('Description: hello', 'File: host 0:1/helloworld.efi', + 'Initrd File:', 'Optional Data:', 'Save', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 4, True, 'Quit') + press_escape_key(False) + check_current_is_maintenance_menu() + press_escape_key(True) + response = u_boot_console.run_command(cmd = 'bootefi bootmgr') + assert 'Image not authenticated' in response + + # + # Test Case 6: can not start the signed image if dbx revokes db certificate + # + + u_boot_console.run_command('eficonfig', wait_for_prompt=False) + check_current_is_maintenance_menu() + # Select 'Edit Boot Option' + press_up_down_enter_and_wait(0, 1, True, None) + # Check the curren BootOrder + for i in ('hello', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, 'Quit') + # Set EFI image(helloworld.efi.signed) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'host 0:1') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 5, True, 'Quit') + for i in ('Description: hello', 'File: host 0:1/helloworld.efi.signed', + 'Initrd File:', 'Optional Data:', 'Save', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 4, True, 'Quit') + press_escape_key(False) + check_current_is_maintenance_menu() + press_up_down_enter_and_wait(0, 4, True, 'Quit') + # set dbx_hash.auth to dbx + press_up_down_enter_and_wait(0, 3, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + # check db is expected value + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, None) + # verify CRL, skip hash comparison because it varies in every test + for i in ('Show/Delete Signature Database', 'dbx', + 'Owner GUID:', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type:', 'X509_SHA256 CRL', + 'TimeOfRevocation:', '2013-5-27 01:02:03'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + press_escape_key(False) + check_current_is_maintenance_menu() + press_escape_key(True) + response = u_boot_console.run_command(cmd = 'bootefi bootmgr') + assert 'Image not authenticated' in response + + # + # Test Case 7: clear PK with null key, check secure boot is OFF + # + + u_boot_console.run_command('eficonfig', wait_for_prompt=False) + check_current_is_maintenance_menu() + press_up_down_enter_and_wait(0, 4, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'ON'): + u_boot_console.p.expect([i]) + + # clear PK with null key + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 10, True, 'Quit') + + press_up_down_enter_and_wait(0, 1, True, None) + for i in ('There is no entry in the signature database.', 'Press any key to continue'): + u_boot_console.p.expect([i]) + + press_enter_key(False) + press_up_down_enter_and_wait(0, 2, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'OFF'): + u_boot_console.p.expect([i]) + + # delete dbx + press_up_down_enter_and_wait(0, 3, True, 'Quit') + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_enter_key(False) + for i in ('TimeOfRevocation:', '2013-5-27 01:02:03'): + u_boot_console.p.expect([i]) + press_enter_key(False) + for i in ('Are you sure you want to delete this item?', 'Press ENTER to delete'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('There is no entry in the signature database.', + 'Press any key to continue'): + u_boot_console.p.expect([i]) + press_enter_key(False) + press_up_down_enter_and_wait(0, 2, True, 'Quit') + + # set PK.auth to PK + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 0, True, 'Quit') + press_up_down_enter_and_wait(0, 8, True, 'Quit') + # check PK is expected value + press_up_down_enter_and_wait(0, 1, True, None) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 0, True, None) + for i in ('Show/Delete Signature Database', 'PK', + 'Owner GUID', '11111111-2222-3333-4444-123456789ABC', + 'Signature Type', 'X509', 'Subject', 'TEST_PK', 'Issuer', 'TEST_PK', + 'Can not delete PK, Press any key to continue'): + u_boot_console.p.expect([i]) + press_escape_key(False) + for i in ('11111111-2222-3333-4444-123456789ABC', 'Quit'): + u_boot_console.p.expect([i]) + press_up_down_enter_and_wait(0, 1, True, 'Quit') + press_up_down_enter_and_wait(0, 2, True, 'Quit') + for i in ('UEFI Secure Boot Key Configuration', 'SecureBoot :', 'ON'): + u_boot_console.p.expect([i]) + press_escape_key(False) + check_current_is_maintenance_menu() + press_escape_key(True) + response = u_boot_console.run_command(cmd = 'bootefi bootmgr') + assert 'Hello, world!' in response
Provide a unit test for the eficonfig secure boot key management menu. Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org> --- No change since v2 newly created in v2 test/py/tests/test_eficonfig/conftest.py | 84 +++- test/py/tests/test_eficonfig/defs.py | 14 + .../test_eficonfig/test_eficonfig_sbkey.py | 472 ++++++++++++++++++ 3 files changed, 568 insertions(+), 2 deletions(-) create mode 100644 test/py/tests/test_eficonfig/defs.py create mode 100644 test/py/tests/test_eficonfig/test_eficonfig_sbkey.py