Report DejaGnu ERROR messages in dg-extract-results

Message ID CAKdteOaq=M7NwY3AQ3-0Gn2xE4BFVcZWtnrJJs1Bkk-CH23Mgg@mail.gmail.com
State New
Headers show

Commit Message

Christophe Lyon Sept. 20, 2016, 9:26 a.m.
Hello,

We recently faced a problem where a DejaGnu error went un-noticed
(https://gcc.gnu.org/ml/gcc-patches/2016-08/msg01879.html).

To help identify these problems earlier, here is a patch for
dg-extract-results that makes such error obvious in the final summary:

                === gcc Summary ===

# of DejaGnu errors             1
# of expected passes            104885
# of unexpected failures        43
# of unexpected successes       2
# of expected failures          276
# of unsupported tests          2284


OK?

Thanks,

Christophe
contrib/ChangeLog:

2016-09-20  Christophe Lyon  <christophe.lyon@linaro.org>

	* dg-extract-results.py: Report DejaGnu error in the final
	summary.
	* dg-extract-results.sh: Likewise.

Comments

Christophe Lyon Sept. 24, 2016, 2:39 a.m. | #1
On 23 September 2016 at 13:37, Rainer Orth <ro@cebitec.uni-bielefeld.de> wrote:
> Hi Christophe,

>

>> We recently faced a problem where a DejaGnu error went un-noticed

>> (https://gcc.gnu.org/ml/gcc-patches/2016-08/msg01879.html).

>

> I just noticed that your patch contained an unrelated and bogus change,

> reverted as follows.  This will only show up on targets that lack (a

> sufficiently recent version of) python, Solaris 10 in my case.

>


Oops sorry, it was debug code I forgot to remove.

Thanks!

>         Rainer

>

>

> 2016-09-23  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>

>

>         * dg-extract-results.sh: Revert bogus change.

>

>

>

> --

> -----------------------------------------------------------------------------

> Rainer Orth, Center for Biotechnology, Bielefeld University

>

Patch hide | download patch | download mbox

diff --git a/contrib/dg-extract-results.py b/contrib/dg-extract-results.py
index 7db5e64..4b02a5b 100644
--- a/contrib/dg-extract-results.py
+++ b/contrib/dg-extract-results.py
@@ -134,6 +134,7 @@  class Prog:
         self.end_line = None
         # Known summary types.
         self.count_names = [
+            '# of DejaGnu errors\t\t',
             '# of expected passes\t\t',
             '# of unexpected failures\t',
             '# of unexpected successes\t',
@@ -245,6 +246,10 @@  class Prog:
             segment = Segment (filename, file.tell())
             variation.header = segment
 
+        # Parse the rest of the summary (the '# of ' lines).
+        if len (variation.counts) == 0:
+            variation.counts = self.zero_counts()
+
         # Parse up until the first line of the summary.
         if num_variations == 1:
             end = '\t\t=== ' + tool.name + ' Summary ===\n'
@@ -291,6 +296,11 @@  class Prog:
                 harness.results.append ((key, line))
                 if not first_key and sort_logs:
                     first_key = key
+                if line.startswith ('ERROR: (DejaGnu)'):
+                    for i in range (len (self.count_names)):
+                        if 'DejaGnu errors' in self.count_names[i]:
+                            variation.counts[i] += 1
+                            break
 
             # 'Using ...' lines are only interesting in a header.  Splitting
             # the test up into parallel runs leads to more 'Using ...' lines
@@ -309,9 +319,6 @@  class Prog:
             segment.lines -= final_using
             harness.add_segment (first_key, segment)
 
-        # Parse the rest of the summary (the '# of ' lines).
-        if len (variation.counts) == 0:
-            variation.counts = self.zero_counts()
         while True:
             before = file.tell()
             line = file.readline()
diff --git a/contrib/dg-extract-results.sh b/contrib/dg-extract-results.sh
index 5a8e67e..519d49c 100755
--- a/contrib/dg-extract-results.sh
+++ b/contrib/dg-extract-results.sh
@@ -369,10 +369,11 @@  EOF
 BEGIN {
   variant="$VAR"
   tool="$TOOL"
-  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; kpasscnt=0; kfailcnt=0; unsupcnt=0; unrescnt=0;
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; kpasscnt=0; kfailcnt=0; unsupcnt=0; unrescnt=0; dgerrorcnt=0;
   curvar=""; insummary=0
 }
 /^Running target /		{ curvar = \$3; next }
+/^ERROR: \(DejaGnu\)/		{ if (variant == curvar) dgerrorcnt += 1 }
 /^# of /			{ if (variant == curvar) insummary = 1 }
 /^# of expected passes/		{ if (insummary == 1) passcnt += \$5; next; }
 /^# of unexpected successes/	{ if (insummary == 1) xpasscnt += \$5; next; }
@@ -390,6 +391,7 @@  BEGIN {
 { next }
 END {
   printf ("\t\t=== %s Summary for %s ===\n\n", tool, variant)
+  if (dgerrorcnt != 0) printf ("# of DejaGnu errors\t\t%d\n", dgerrorcnt)
   if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
   if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
   if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
@@ -419,8 +421,9 @@  TOTAL_AWK=${TMP}/total.awk
 cat << EOF > $TOTAL_AWK
 BEGIN {
   tool="$TOOL"
-  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; kfailcnt=0; unsupcnt=0; unrescnt=0
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; kfailcnt=0; unsupcnt=0; unrescnt=0; dgerrorcnt=0
 }
+/^# of DejaGnu errors/		{ dgerrorcnt += \$5 }
 /^# of expected passes/		{ passcnt += \$5 }
 /^# of unexpected failures/	{ failcnt += \$5 }
 /^# of unexpected successes/	{ xpasscnt += \$5 }
@@ -431,7 +434,8 @@  BEGIN {
 /^# of unresolved testcases/	{ unrescnt += \$5 }
 /^# of unsupported tests/	{ unsupcnt += \$5 }
 END {
-  printf ("\n\t\t=== %s Summary ===\n\n", tool)
+  printf ("\n\t\t=== %s MySummary ===\n\n", tool)
+  if (dgerrorcnt != 0) printf ("# of DejaGnu errors\t\t%d\n", dgerrorcnt)
   if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
   if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
   if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)