From patchwork Fri Jul 22 01:39:22 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zygmunt Krynicki X-Patchwork-Id: 3044 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id BB8DA23F52 for ; Fri, 22 Jul 2011 01:39:25 +0000 (UTC) Received: from mail-qy0-f173.google.com (mail-qy0-f173.google.com [209.85.216.173]) by fiordland.canonical.com (Postfix) with ESMTP id E8836A187EA for ; Fri, 22 Jul 2011 01:39:24 +0000 (UTC) Received: by qyk10 with SMTP id 10so4403903qyk.11 for ; Thu, 21 Jul 2011 18:39:24 -0700 (PDT) Received: by 10.224.198.7 with SMTP id em7mr830023qab.112.1311298764426; Thu, 21 Jul 2011 18:39:24 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.229.217.78 with SMTP id hl14cs2686qcb; Thu, 21 Jul 2011 18:39:23 -0700 (PDT) Received: by 10.216.156.207 with SMTP id m57mr1321695wek.44.1311298762837; Thu, 21 Jul 2011 18:39:22 -0700 (PDT) Received: from adelie.canonical.com (adelie.canonical.com [91.189.90.139]) by mx.google.com with ESMTP id v79si3498903weq.122.2011.07.21.18.39.22; Thu, 21 Jul 2011 18:39:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.139 as permitted sender) client-ip=91.189.90.139; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.139 as permitted sender) smtp.mail=bounces@canonical.com Received: from loganberry.canonical.com ([91.189.90.37]) by adelie.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1Qk4i2-0001zx-2E for ; Fri, 22 Jul 2011 01:39:22 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 04D972E889B for ; Fri, 22 Jul 2011 01:39:22 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-dashboard X-Launchpad-Branch: ~linaro-validation/lava-dashboard/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 248 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-dashboard/trunk] Rev 248: Merge 0.6-wip branch Message-Id: <20110722013922.8983.41110.launchpad@loganberry.canonical.com> Date: Fri, 22 Jul 2011 01:39:22 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="13475"; Instance="initZopeless config overlay" X-Launchpad-Hash: b678ff021f91933ffe0e72f60b0b27809e443ca2 Merge authors: Zygmunt Krynicki (zkrynicki) Related merge proposals: https://code.launchpad.net/~linaro-validation/lava-dashboard/0.6-wip/+merge/68142 proposed by: Zygmunt Krynicki (zkrynicki) ------------------------------------------------------------ revno: 248 [merge] committer: Zygmunt Krynicki branch nick: trunk timestamp: Fri 2011-07-22 03:34:46 +0200 message: Merge 0.6-wip branch * New UI synchronized with lava-server, the UI is going to be changed in the next release to be more in line with the official Linaro theme. Currently most changes are under-the-hood, sporting more jQuery UI CSS. * New test browser that allows to see all the registered tests and their test cases. * New data view browser, similar to data view browser. * New permalink system that allows easy linking to bundles, test runs and test results. * New image status views that allow for quick inspection of interesting hardware pack + root filesystem combinations. * New image status detail view with color-coded information about test failures affecting current and historic instances of a particular root filesystem + hardware pack combination. * New image test history view showing all the runs of a particular test on a particular combination of root filesystem + hardware pack. * New table widget for better table display with support for client side sorting and searching. * New option to render data reports without any navigation that is suitable for embedding inside an iframe (by appending &iframe=yes to the URL) * New view for showing text attachments associated with test runs. * New view showing test runs associated with a specific bundle. * New view showing the raw JSON text of a bundle. * New view for inspecting bundle deserialization failures. * Integration with lava-server/RPC2/ for web APIs * Added support for non-anonymous submissions (test results uploaded by authenticated users), including uploading results to personal (owned by person), team (owned by group), public (visible) and private (hidden from non-owners) bundle streams. * Added support for creating non-anonymous bundle streams with dashboard.make_stream() (for authenticated users) removed: dashboard_app/dataview.py dashboard_app/dispatcher.py dashboard_app/tests/other/xml_rpc.py added: dashboard_app/repositories/common.py dashboard_app/repositories/data_view.py dashboard_app/templates/dashboard_app/image_status_detail.html dashboard_app/templates/dashboard_app/image_status_list.html dashboard_app/templates/dashboard_app/image_test_history.html dashboard_app/templatetags/call.py dashboard_app/tests/views/redirects.py modified: dashboard_app/admin.py dashboard_app/models.py dashboard_app/repositories/__init__.py dashboard_app/repositories/data_report.py dashboard_app/static/js/jquery.dashboard.js dashboard_app/templates/dashboard_app/_extension_navigation.html dashboard_app/templates/dashboard_app/bundle_detail.html dashboard_app/templates/dashboard_app/data_view_list.html dashboard_app/templates/dashboard_app/index.html dashboard_app/templates/dashboard_app/report_detail.html dashboard_app/templates/dashboard_app/test_result_detail.html dashboard_app/templates/dashboard_app/test_run_detail.html dashboard_app/tests/__init__.py dashboard_app/tests/other/dataview.py dashboard_app/tests/utils.py dashboard_app/tests/views/test_run_detail_view.py dashboard_app/urls.py dashboard_app/views.py dashboard_app/xmlrpc.py doc/changes.rst doc/index.rst --- lp:lava-dashboard https://code.launchpad.net/~linaro-validation/lava-dashboard/trunk You are subscribed to branch lp:lava-dashboard. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-dashboard/trunk/+edit-subscription === modified file 'dashboard_app/admin.py' --- dashboard_app/admin.py 2011-04-29 11:54:24 +0000 +++ dashboard_app/admin.py 2011-07-19 21:39:22 +0000 @@ -137,8 +137,13 @@ class TestRunAdmin(admin.ModelAdmin): class NamedAttributeInline(generic.GenericTabularInline): model = NamedAttribute - list_display = ('analyzer_assigned_uuid', - 'analyzer_assigned_date', 'import_assigned_date') + list_filter = ('test'), + list_display = ( + 'test', + 'analyzer_assigned_uuid', + 'bundle', + 'analyzer_assigned_date', + 'import_assigned_date') inlines = [NamedAttributeInline] === removed file 'dashboard_app/dataview.py' --- dashboard_app/dataview.py 2011-07-09 15:09:48 +0000 +++ dashboard_app/dataview.py 1970-01-01 00:00:00 +0000 @@ -1,284 +0,0 @@ -# Copyright (C) 2011 Linaro Limited -# -# Author: Zygmunt Krynicki -# -# This file is part of Launch Control. -# -# Launch Control is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License version 3 -# as published by the Free Software Foundation -# -# Launch Control is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Launch Control. If not, see . - -""" -DataViews: Encapsulated SQL query definitions. - -Implementation of the following launchpad blueprint: -https://blueprints.launchpad.net/launch-control/+spec/other-linaro-n-data-views-for-launch-control -""" - -from contextlib import closing -from xml.sax import parseString -from xml.sax.handler import ContentHandler -import logging -import os -import re - - -class DataView(object): - """ - Data view, a container for SQL query and optional arguments - """ - - def __init__(self, name, backend_queries, arguments, documentation, summary): - self.name = name - self.backend_queries = backend_queries - self.arguments = arguments - self.documentation = documentation - self.summary = summary - - def _get_connection_backend_name(self, connection): - backend = str(type(connection)) - if "sqlite" in backend: - return "sqlite" - elif "postgresql" in backend: - return "postgresql" - else: - return "" - - def get_backend_specific_query(self, connection): - """ - Return BackendSpecificQuery for the specified connection - """ - sql_backend_name = self._get_connection_backend_name(connection) - try: - return self.backend_queries[sql_backend_name] - except KeyError: - return self.backend_queries.get(None, None) - - def lookup_argument(self, name): - """ - Return Argument with the specified name - - Raises LookupError if the argument cannot be found - """ - for argument in self.arguments: - if argument.name == name: - return argument - raise LookupError(name) - - @classmethod - def load_from_xml(self, xml_string): - """ - Load a data view instance from XML description - - This raises ValueError in several error situations. - TODO: check what kind of exceptions this can raise - """ - handler = _DataViewHandler() - parseString(xml_string, handler) - return handler.data_view - - @classmethod - def get_connection(cls): - """ - Get the appropriate connection for data views - """ - from django.db import connection, connections - from django.db.utils import ConnectionDoesNotExist - try: - return connections['dataview'] - except ConnectionDoesNotExist: - logging.warning("dataview-specific database connection not available, dataview query is NOT sandboxed") - return connection # NOTE: it's connection not connectionS (the default connection) - - def __call__(self, connection, **arguments): - # Check if arguments have any bogus names - valid_arg_names = frozenset([argument.name for argument in self.arguments]) - for arg_name in arguments: - if arg_name not in valid_arg_names: - raise TypeError("Data view %s has no argument %r" % (self.name, arg_name)) - # Get the SQL template for our database connection - query = self.get_backend_specific_query(connection) - if query is None: - raise LookupError("Specified data view has no SQL implementation " - "for current database") - # Replace SQL aruments with django placeholders (connection agnostic) - template = query.sql_template - template = template.replace("%", "%%") - # template = template.replace("{", "{{").replace("}", "}}") - sql = template.format( - **dict([ - (arg_name, "%s") - for arg_name in query.argument_list])) - # Construct argument list using defaults for missing values - sql_args = [ - arguments.get(arg_name, self.lookup_argument(arg_name).default) - for arg_name in query.argument_list] - with closing(connection.cursor()) as cursor: - # Execute the query with the specified arguments - cursor.execute(sql, sql_args) - # Get and return the results - rows = cursor.fetchall() - columns = cursor.description - return rows, columns - - -class Argument(object): - """ - Data view argument for SQL prepared statements - """ - - def __init__(self, name, type, default, help): - self.name = name - self.type = type - self.default = default - self.help = help - - -class BackendSpecificQuery(object): - """ - Backend-specific query and argument list - """ - - def __init__(self, backend, sql_template, argument_list): - self.backend = backend - self.sql_template = sql_template - self.argument_list = argument_list - - -class _DataViewHandler(ContentHandler): - """ - ContentHandler subclass for parsing DataView documents - """ - - def _end_text(self): - """ - Stop collecting text and produce a stripped string with deduplicated whitespace - """ - full_text = re.sub("\s+", " ", u''.join(self._text)).strip() - self.text = None - return full_text - - def _start_text(self): - """ - Start collecting text - """ - self._text = [] - - def startDocument(self): - # Text can be None or a [] that accumulates all detected text - self._text = None - # Data view object - self.data_view = DataView(None, {}, [], None, None) - # Internal variables - self._current_backend_query = None - - def endDocument(self): - # TODO: check if we have anything defined - if self.data_view.name is None: - raise ValueError("No data view definition found") - - def startElement(self, name, attrs): - if name == "data-view": - self.data_view.name = attrs["name"] - elif name == "summary" or name == "documentation": - self._start_text() - elif name == "sql": - self._start_text() - self._current_backend_query = BackendSpecificQuery(attrs.get("backend"), None, []) - self.data_view.backend_queries[self._current_backend_query.backend] = self._current_backend_query - elif name == "value": - if "name" not in attrs: - raise ValueError(" requires attribute 'name'") - self._text.append("{" + attrs["name"] + "}") - self._current_backend_query.argument_list.append(attrs["name"]) - elif name == "argument": - if "name" not in attrs: - raise ValueError(" requires attribute 'name'") - if "type" not in attrs: - raise ValueError(" requires attribute 'type'") - if attrs["type"] not in ("string", "number", "boolean", "timestamp"): - raise ValueError("invalid value for argument 'type' on ") - argument = Argument(name=attrs["name"], type=attrs["type"], - default=attrs.get("default", None), - help=attrs.get("help", None)) - self.data_view.arguments.append(argument) - - def endElement(self, name): - if name == "sql": - self._current_backend_query.sql_template = self._end_text() - self._current_backend_query = None - elif name == "documentation": - self.data_view.documentation = self._end_text() - elif name == "summary": - self.data_view.summary = self._end_text() - - def characters(self, content): - if isinstance(self._text, list): - self._text.append(content) - - -class DataViewRepository(object): - - _instance = None - - def __init__(self): - self.data_views = [] - - def __iter__(self): - return iter(self.data_views) - - def __getitem__(self, name): - for item in self: - if item.name == name: - return item - else: - raise KeyError(name) - - def load_from_directory(self, directory): - for name in os.listdir(directory): - pathname = os.path.join(directory, name) - if os.path.isfile(pathname) and pathname.endswith(".xml"): - self.load_from_file(pathname) - - def load_from_file(self, pathname): - try: - with open(pathname, "rt") as stream: - text = stream.read() - data_view = DataView.load_from_xml(text) - self.data_views.append(data_view) - except Exception as exc: - logging.error("Unable to load data view from %s: %s", pathname, exc) - - @classmethod - def get_instance(cls): - from django.conf import settings - if cls._instance is None: - cls._instance = cls() - cls._instance.load_default() - - # I development mode always reload data views - if getattr(settings, "DEBUG", False) is True: - cls._instance.data_views = [] - cls._instance.load_default() - return cls._instance - - def load_default(self): - from django.conf import settings - for dirname in getattr(settings, "DATAVIEW_DIRS", []): - self.load_from_directory(dirname) - - -__all__ = [ - "Argument", - "BackendSpecificQuery", - "DataView", - "DataViewRepository", -] === removed file 'dashboard_app/dispatcher.py' --- dashboard_app/dispatcher.py 2011-05-07 23:08:30 +0000 +++ dashboard_app/dispatcher.py 1970-01-01 00:00:00 +0000 @@ -1,188 +0,0 @@ -# Copyright (C) 2010 Linaro Limited -# -# Author: Zygmunt Krynicki -# -# This file is part of Launch Control. -# -# Launch Control is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License version 3 -# as published by the Free Software Foundation -# -# Launch Control is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Launch Control. If not, see . - -""" -XML-RPC Dispatcher for the dashboard -""" -import SimpleXMLRPCServer -import logging -import sys -import xmlrpclib - -class _NullHandler(logging.Handler): - def emit(self, record): - pass - - -class FaultCodes: - """ - Common fault codes. - - See: http://xmlrpc-epi.sourceforge.net/specs/rfc.fault_codes.php - """ - class ParseError: - NOT_WELL_FORMED = -32700 - UNSUPPORTED_ENCODING = -32701 - INVALID_CHARACTER_FOR_ENCODING = -32702 - class ServerError: - INVALID_XML_RPC = -32600 - REQUESTED_METHOD_NOT_FOUND = -32601 - INVALID_METHOD_PARAMETERS = -32602 - INTERNAL_XML_RPC_ERROR = -32603 - APPLICATION_ERROR = -32500 - SYSTEM_ERROR = -32400 - TRANSPORT_ERROR = -32300 - - -class DjangoXMLRPCDispatcher(SimpleXMLRPCServer.SimpleXMLRPCDispatcher): - """ - Slightly extended XML-RPC dispatcher class suitable for embedding in - django applications. - """ - #TODO: Implement _marshaled_dispatch() and capture XML errors to - # translate them to appropriate standardized fault codes. There - # might be some spill to the view code to make this complete. - - #TODO: Implement and expose system.getCapabilities() and advertise - # support for standardised fault codes. - # See: http://tech.groups.yahoo.com/group/xml-rpc/message/2897 - def __init__(self): - # it's a classic class, no super - SimpleXMLRPCServer.SimpleXMLRPCDispatcher.__init__(self, - allow_none=True) - self.logger = logging.getLogger(__name__) - self.logger.addHandler(_NullHandler()) - - def _lookup_func(self, method): - """ - Lookup implementation of method named `method`. - - Returns implementation of `method` or None if the method is not - registered anywhere in the dispatcher. - - This new function is taken directly out of the base class - implementation of _dispatch. The point is to be able to - detect a situation where method is not known and return - appropriate XML-RPC fault. With plain dispatcher it is - not possible as the implementation raises a plain Exception - to signal this error condition and capturing and interpreting - arbitrary exceptions is flaky. - """ - func = None - try: - # check to see if a matching function has been registered - func = self.funcs[method] - except KeyError: - if self.instance is not None: - # check for a _dispatch method - if hasattr(self.instance, '_dispatch'): - # FIXME: pyflakes, params is undefined - return self.instance._dispatch(method, params) - else: - # call instance method directly - try: - func = SimpleXMLRPCServer.resolve_dotted_attribute( - self.instance, - method, - self.allow_dotted_names - ) - except AttributeError: - pass - return func - - def _dispatch(self, method, params): - """ - Improved dispatch method from the base dispatcher. - - The primary improvement is exception handling: - - xml-rpc faults are passed back to the caller - - missing methods return standardized fault code ( - FaultCodes.ServerError.REQUESTED_METHOD_NOT_FOUND) - - all other exceptions in the called method are translated - to standardized internal xml-rpc fault code - (FaultCodes.ServerError.INTERNAL_XML_RPC_ERROR). In - addition such errors cause _report_incident() to be - called. This allows to hook a notification mechanism for - deployed servers where exceptions are, for example, mailed - to the administrator. - """ - func = self._lookup_func(method) - if func is None: - raise xmlrpclib.Fault( - FaultCodes.ServerError.REQUESTED_METHOD_NOT_FOUND, - "No such method: %r" % method) - try: - # TODO: check parameter types before calling - return func(*params) - except xmlrpclib.Fault: - # Forward XML-RPC Faults to the client - raise - except: - # Treat all other exceptions as internal errors - # This prevents the clients from seeing internals - exc_type, exc_value, exc_tb = sys.exc_info() - incident_id = self._report_incident(method, params, exc_type, exc_value, exc_tb) - string = ("Dashboard has encountered internal error. " - "Incident ID is: %s" % (incident_id,)) - raise xmlrpclib.Fault( - FaultCodes.ServerError.INTERNAL_XML_RPC_ERROR, - string) - - def _report_incident(self, method, params, exc_type, exc_value, exc_tb): - """ - Report an exception that happened - """ - self.logger.exception("Internal error when dispatching " - "XML-RPC method: %s%r", method, params) - # TODO: store the exception somewhere and assign fault codes - return None - - def system_methodSignature(self, method): - if method.startswith("_"): - return "" - if self.instance is not None: - func = getattr(self.instance, method, None) - else: - func = self.funcs.get(method) - # When function is not known return empty string - if func is None: - return "" - # When signature is not known return "undef" - # See: http://xmlrpc-c.sourceforge.net/introspection.html - return getattr(func, 'xml_rpc_signature', "undef") - - -def xml_rpc_signature(*sig): - """ - Small helper that attaches "xml_rpc_signature" attribute to the - function. The attribute is a list of values that is then reported - by system_methodSignature(). - - This is a simplification of the XML-RPC spec that allows to attach a - list of variants (like I may accept this set of arguments, or that - set or that other one). This version has only one set of arguments. - - Note that it's a purely presentational argument for our - implementation. Putting bogus values here won't spoil the day. - - The first element is the signature of the return type. - """ - def decorator(func): - func.xml_rpc_signature = sig - return func - return decorator === modified file 'dashboard_app/models.py' --- dashboard_app/models.py 2011-07-12 14:38:26 +0000 +++ dashboard_app/models.py 2011-07-22 00:57:44 +0000 @@ -26,6 +26,7 @@ import os import simplejson import traceback +import contextlib from django.contrib.auth.models import User from django.contrib.contenttypes import generic @@ -43,6 +44,7 @@ from dashboard_app.managers import BundleManager from dashboard_app.repositories import RepositoryItem from dashboard_app.repositories.data_report import DataReportRepository +from dashboard_app.repositories.data_view import DataViewRepository # Fix some django issues we ran into from dashboard_app.patches import patch @@ -364,6 +366,9 @@ def get_absolute_url(self): return ("dashboard_app.views.bundle_detail", [self.bundle_stream.pathname, self.content_sha1]) + def get_permalink(self): + return reverse("dashboard_app.views.redirect_to_bundle", args=[self.content_sha1]) + def save(self, *args, **kwargs): if self.content: try: @@ -749,6 +754,9 @@ self.bundle.content_sha1, self.analyzer_assigned_uuid]) + def get_permalink(self): + return reverse("dashboard_app.views.redirect_to_test_run", args=[self.analyzer_assigned_uuid]) + def get_summary_results(self): stats = self.test_results.values('result').annotate( count=models.Count('result')).order_by() @@ -932,6 +940,20 @@ def __unicode__(self): return "Result {0}/{1}".format(self.test_run.analyzer_assigned_uuid, self.relative_index) + @models.permalink + def get_absolute_url(self): + return ("dashboard_app.views.test_result_detail", [ + self.test_run.bundle.bundle_stream.pathname, + self.test_run.bundle.content_sha1, + self.test_run.analyzer_assigned_uuid, + self.relative_index, + ]) + + def get_permalink(self): + return reverse("dashboard_app.views.redirect_to_test_result", + args=[self.test_run.analyzer_assigned_uuid, + self.relative_index]) + @property def result_code(self): """ @@ -973,15 +995,6 @@ duration = property(_get_duration, _set_duration) - @models.permalink - def get_absolute_url(self): - return ("dashboard_app.views.test_result_detail", [ - self.test_run.bundle.bundle_stream.pathname, - self.test_run.bundle.content_sha1, - self.test_run.analyzer_assigned_uuid, - self.relative_index, - ]) - def related_attachment_available(self): """ Check if there is a log file attached to the test run that has @@ -1001,6 +1014,105 @@ order_with_respect_to = 'test_run' +class DataView(RepositoryItem): + """ + Data view, a container for SQL query and optional arguments + """ + + repository = DataViewRepository() + + def __init__(self, name, backend_queries, arguments, documentation, summary): + self.name = name + self.backend_queries = backend_queries + self.arguments = arguments + self.documentation = documentation + self.summary = summary + + def __unicode__(self): + return self.name + + def __repr__(self): + return "" % (self.name,) + + @models.permalink + def get_absolute_url(self): + return ("dashboard_app.views.data_view_detail", [self.name]) + + def _get_connection_backend_name(self, connection): + backend = str(type(connection)) + if "sqlite" in backend: + return "sqlite" + elif "postgresql" in backend: + return "postgresql" + else: + return "" + + def get_backend_specific_query(self, connection): + """ + Return BackendSpecificQuery for the specified connection + """ + sql_backend_name = self._get_connection_backend_name(connection) + try: + return self.backend_queries[sql_backend_name] + except KeyError: + return self.backend_queries.get(None, None) + + def lookup_argument(self, name): + """ + Return Argument with the specified name + + Raises LookupError if the argument cannot be found + """ + for argument in self.arguments: + if argument.name == name: + return argument + raise LookupError(name) + + @classmethod + def get_connection(cls): + """ + Get the appropriate connection for data views + """ + from django.db import connection, connections + from django.db.utils import ConnectionDoesNotExist + try: + return connections['dataview'] + except ConnectionDoesNotExist: + logging.warning("dataview-specific database connection not available, dataview query is NOT sandboxed") + return connection # NOTE: it's connection not connectionS (the default connection) + + def __call__(self, connection, **arguments): + # Check if arguments have any bogus names + valid_arg_names = frozenset([argument.name for argument in self.arguments]) + for arg_name in arguments: + if arg_name not in valid_arg_names: + raise TypeError("Data view %s has no argument %r" % (self.name, arg_name)) + # Get the SQL template for our database connection + query = self.get_backend_specific_query(connection) + if query is None: + raise LookupError("Specified data view has no SQL implementation " + "for current database") + # Replace SQL aruments with django placeholders (connection agnostic) + template = query.sql_template + template = template.replace("%", "%%") + # template = template.replace("{", "{{").replace("}", "}}") + sql = template.format( + **dict([ + (arg_name, "%s") + for arg_name in query.argument_list])) + # Construct argument list using defaults for missing values + sql_args = [ + arguments.get(arg_name, self.lookup_argument(arg_name).default) + for arg_name in query.argument_list] + with contextlib.closing(connection.cursor()) as cursor: + # Execute the query with the specified arguments + cursor.execute(sql, sql_args) + # Get and return the results + rows = cursor.fetchall() + columns = cursor.description + return rows, columns + + class DataReport(RepositoryItem): """ Data reports are small snippets of xml that define @@ -1013,6 +1125,16 @@ self._html = None self._data = kwargs + def __unicode__(self): + return self.title + + def __repr__(self): + return "" % (self.name,) + + @models.permalink + def get_absolute_url(self): + return ("dashboard_app.views.report_detail", [self.name]) + def _get_raw_html(self): pathname = os.path.join(self.base_path, self.path) try: @@ -1041,13 +1163,6 @@ self._html = template.render(context) return self._html - def __unicode__(self): - return self.title - - @models.permalink - def get_absolute_url(self): - return ("dashboard_app.views.report_detail", [self.name]) - @property def title(self): return self._data['title'] @@ -1067,3 +1182,90 @@ @property def author(self): return self._data.get('author') + + +class ImageHealth(object): + + def __init__(self, rootfs_type, hwpack_type): + self.rootfs_type = rootfs_type + self.hwpack_type = hwpack_type + + @models.permalink + def get_absolute_url(self): + return ("dashboard_app.views.image_status_detail", [ + self.rootfs_type, self.hwpack_type]) + + def get_tests(self): + return Test.objects.filter(test_runs__in=self.get_test_runs()).distinct() + + def current_health_for_test(self, test): + test_run = self.get_current_test_run(test) + test_results = test_run.test_results + fail_count = test_results.filter( + result=TestResult.RESULT_FAIL).count() + pass_count = test_results.filter( + result=TestResult.RESULT_PASS).count() + total_count = test_results.count() + return { + "test_run": test_run, + "total_count": total_count, + "fail_count": fail_count, + "pass_count": pass_count, + "other_count": total_count - (fail_count + pass_count), + "fail_percent": fail_count * 100.0 / total_count if total_count > 0 else None, + } + + def overall_health_for_test(self, test): + test_run_list = self.get_all_test_runs_for_test(test) + test_result_list = TestResult.objects.filter(test_run__in=test_run_list) + fail_result_list = test_result_list.filter(result=TestResult.RESULT_FAIL) + pass_result_list = test_result_list.filter(result=TestResult.RESULT_PASS) + + total_count = test_result_list.count() + fail_count = fail_result_list.count() + pass_count = pass_result_list.count() + fail_percent = fail_count * 100.0 / total_count if total_count > 0 else None + return { + "total_count": total_count, + "total_run_count": test_run_list.count(), + "fail_count": fail_count, + "pass_count": pass_count, + "other_count": total_count - fail_count - pass_count, + "fail_percent": fail_percent, + } + + def get_test_runs(self): + return TestRun.objects.filter( + bundle__bundle_stream__pathname="/anonymous/lava-daily/" + ).filter( + attributes__name='rootfs.type', + attributes__value=self.rootfs_type + ).filter( + attributes__name='hwpack.type', + attributes__value=self.hwpack_type) + + def get_current_test_run(self, test): + return self.get_all_test_runs_for_test(test).order_by('-analyzer_assigned_date')[0] + + def get_all_test_runs_for_test(self, test): + return self.get_test_runs().filter(test=test) + + @classmethod + def get_rootfs_list(self): + rootfs_list = [ + attr['value'] + for attr in NamedAttribute.objects.filter( + name='rootfs.type').values('value').distinct()] + try: + rootfs_list.remove('android') + except ValueError: + pass + return rootfs_list + + @classmethod + def get_hwpack_list(self): + hwpack_list = [ + attr['value'] + for attr in NamedAttribute.objects.filter( + name='hwpack.type').values('value').distinct()] + return hwpack_list === modified file 'dashboard_app/repositories/__init__.py' --- dashboard_app/repositories/__init__.py 2011-05-03 22:07:19 +0000 +++ dashboard_app/repositories/__init__.py 2011-07-22 00:04:21 +0000 @@ -32,7 +32,8 @@ """ def __new__(mcls, name, bases, namespace): - cls = super(RepositoryItemMeta, mcls).__new__(mcls, name, bases, namespace) + cls = super(RepositoryItemMeta, mcls).__new__( + mcls, name, bases, namespace) if "repository" in namespace: repo = cls.repository repo.item_cls = cls @@ -46,7 +47,7 @@ Each repository item is loaded from a XML file. """ - __metaclass__ = RepositoryItemMeta + __metaclass__ = RepositoryItemMeta _base_path = None @@ -57,7 +58,6 @@ def base_path(self): return self._base_path - class DoesNotExist(Exception): pass @@ -89,12 +89,12 @@ def all(self): return self - + def get(self, **kwargs): query = self.filter(**kwargs) if len(query) == 1: return query[0] - if not query: + if not query: raise self.model.DoesNotExist() else: raise self.model.MultipleValuesReturned() @@ -127,7 +127,7 @@ __metaclass__ = abc.ABCMeta def __init__(self): - self.item_cls = None # later patched by RepositoryItemMeta + self.item_cls = None # later patched by RepositoryItemMeta self._items = [] self._did_load = False @@ -154,9 +154,10 @@ try: items = os.listdir(directory) except (OSError, IOError) as exc: - logging.exception("Unable to enumreate directory: %s: %s", directory, exc) + logging.exception("Unable to enumreate directory: %s: %s", + directory, exc) else: - for name in items: + for name in items: pathname = os.path.join(directory, name) if os.path.isfile(pathname) and pathname.endswith(".xml"): self.load_from_file(pathname) @@ -176,7 +177,8 @@ item._load_from_external_representation(pathname) self._items.append(item) except Exception as exc: - logging.exception("Unable to load object into repository %s: %s", pathname, exc) + logging.exception("Unable to load object into repository %s: %s", + pathname, exc) @abc.abstractproperty def settings_variable(self): === added file 'dashboard_app/repositories/common.py' --- dashboard_app/repositories/common.py 1970-01-01 00:00:00 +0000 +++ dashboard_app/repositories/common.py 2011-07-22 00:05:03 +0000 @@ -0,0 +1,47 @@ +# Copyright (C) 2011 Linaro Limited +# +# Author: Zygmunt Krynicki +# +# This file is part of Launch Control. +# +# Launch Control is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License version 3 +# as published by the Free Software Foundation +# +# Launch Control is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Launch Control. If not, see . + + +from xml.sax.handler import ContentHandler +import re + + +class BaseContentHandler(ContentHandler): + + def _end_text(self): + """ + Stop collecting text and produce a stripped string with de-duplicated + whitespace + """ + full_text = re.sub("\s+", " ", u''.join(self._text)).strip() + self.text = None + return full_text + + def _start_text(self): + """ + Start collecting text + """ + self._text = [] + + def characters(self, content): + if isinstance(self._text, list): + self._text.append(content) + + def startDocument(self): + # Text can be None or a [] that accumulates all detected text + self._text = None === modified file 'dashboard_app/repositories/data_report.py' --- dashboard_app/repositories/data_report.py 2011-05-03 18:16:04 +0000 +++ dashboard_app/repositories/data_report.py 2011-07-22 00:07:21 +0000 @@ -18,34 +18,19 @@ from xml.sax import parseString -from xml.sax.handler import ContentHandler -import re from dashboard_app.repositories import Repository, Undefined, Object - - -class _DataReportHandler(ContentHandler): +from dashboard_app.repositories.common import BaseContentHandler + + +class _DataReportHandler(BaseContentHandler): """ - ContentHandler subclass for parsing DataView documents + ContentHandler subclass for parsing DataReport documents """ - def _end_text(self): - """ - Stop collecting text and produce a stripped string with deduplicated whitespace - """ - full_text = re.sub("\s+", " ", u''.join(self._text)).strip() - self.text = None - return full_text - - def _start_text(self): - """ - Start collecting text - """ - self._text = [] - def startDocument(self): - # Text can be None or a [] that accumulates all detected text - self._text = None + # Classic-classes + BaseContentHandler.startDocument(self) # Data report object self.obj = Object() @@ -70,10 +55,6 @@ self.obj.title = self._end_text() elif name == "path": self.obj.path = self._end_text() - - def characters(self, content): - if isinstance(self._text, list): - self._text.append(content) class DataReportRepository(Repository): === added file 'dashboard_app/repositories/data_view.py' --- dashboard_app/repositories/data_view.py 1970-01-01 00:00:00 +0000 +++ dashboard_app/repositories/data_view.py 2011-07-22 00:11:36 +0000 @@ -0,0 +1,133 @@ +# Copyright (C) 2011 Linaro Limited +# +# Author: Zygmunt Krynicki +# +# This file is part of Launch Control. +# +# Launch Control is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License version 3 +# as published by the Free Software Foundation +# +# Launch Control is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Launch Control. If not, see . + +""" +DataViews: Encapsulated SQL query definitions. + +Implementation of the following launchpad blueprint: +https://blueprints.launchpad.net/launch-control/+spec/other-linaro-n-data-views-for-launch-control +""" + + +from xml.sax import parseString + +from dashboard_app.repositories import Repository, Undefined, Object +from dashboard_app.repositories.common import BaseContentHandler + + +class _DataViewHandler(BaseContentHandler): + + """ + ContentHandler subclass for parsing DataView documents + """ + + def startDocument(self): + # Classic-classes + BaseContentHandler.startDocument(self) + # Data view object + self.obj = Object() + # Set default values + self.obj.name = Undefined + self.obj.backend_queries = {} + self.obj.arguments = [] + self.obj.documentation = None + self.obj.summary = None + # Internal variables + self._current_backend_query = None + + def endDocument(self): + # TODO: check if we have anything defined + if self.obj.name is Undefined: + raise ValueError("No data view definition found") + + def startElement(self, name, attrs): + if name == "data-view": + self.obj.name = attrs["name"] + elif name == "summary" or name == "documentation": + self._start_text() + elif name == "sql": + self._start_text() + self._current_backend_query = BackendSpecificQuery( + attrs.get("backend"), None, []) + self.obj.backend_queries[ + self._current_backend_query.backend] = self._current_backend_query + elif name == "value": + if "name" not in attrs: + raise ValueError(" requires attribute 'name'") + self._text.append("{" + attrs["name"] + "}") + self._current_backend_query.argument_list.append(attrs["name"]) + elif name == "argument": + if "name" not in attrs: + raise ValueError(" requires attribute 'name'") + if "type" not in attrs: + raise ValueError(" requires attribute 'type'") + if attrs["type"] not in ("string", "number", "boolean", "timestamp"): + raise ValueError("invalid value for argument 'type' on ") + argument = Argument(name=attrs["name"], type=attrs["type"], + default=attrs.get("default", None), + help=attrs.get("help", None)) + self.obj.arguments.append(argument) + + def endElement(self, name): + if name == "sql": + self._current_backend_query.sql_template = self._end_text() + self._current_backend_query = None + elif name == "documentation": + self.obj.documentation = self._end_text() + elif name == "summary": + self.obj.summary = self._end_text() + + +class Argument(object): + """ + Data view argument for SQL prepared statements + """ + + def __init__(self, name, type, default, help): + self.name = name + self.type = type + self.default = default + self.help = help + + +class BackendSpecificQuery(object): + """ + Backend-specific query and argument list + """ + + def __init__(self, backend, sql_template, argument_list): + self.backend = backend + self.sql_template = sql_template + self.argument_list = argument_list + + +class DataViewRepository(Repository): + + @property + def settings_variable(self): + return "DATAVIEW_DIRS" + + def load_from_xml_string(self, text): + handler = _DataViewHandler() + parseString(text, handler) + return self.item_cls(**handler.obj.__dict__) + + +__all__ = [ + "DataViewRepository", +] === modified file 'dashboard_app/static/js/jquery.dashboard.js' --- dashboard_app/static/js/jquery.dashboard.js 2011-07-09 15:09:19 +0000 +++ dashboard_app/static/js/jquery.dashboard.js 2011-07-19 12:20:02 +0000 @@ -76,7 +76,7 @@ } if (column.name == "UUID") { /* This is a bit hacky but will work for now */ - cell_link = _url + ".." + "/test-runs/" + cell + "/"; + cell_link = _url + ".." + "/permalink/test-run/" + cell + "/"; } html += ""; if (cell_link) { === modified file 'dashboard_app/templates/dashboard_app/_extension_navigation.html' --- dashboard_app/templates/dashboard_app/_extension_navigation.html 2011-07-12 02:34:12 +0000 +++ dashboard_app/templates/dashboard_app/_extension_navigation.html 2011-07-22 00:57:44 +0000 @@ -1,6 +1,8 @@ {% load i18n %} +
+ + + + + + + + + + + + + + + + + + + + + + {% for test in image_health.get_tests %} + {% call image_health.current_health_for_test test as current_test_health %} + {% call image_health.overall_health_for_test test as overall_test_health %} + 0 %} + {% if current_test_health.pass_count == 0 %} + style="background-color: rgba(255, 0, 0, 0.5)" + {% else %} + style="background-color: rgba(255, 165, 0, 0.5)" + {% endif %} + {% else %} + {% if current_test_health.pass_count > 0 %} + style="background-color: rgba(173, 255, 47, 0.5)" + {% endif %} + {% endif %} + > + + + + + + + + + + + + + {% endcall %} + {% endcall %} + {% endfor %} + +
TestTotalsMost Recent Test RunDescription
PASSFAILFAIL rateTest RunsTest ResultsPASSFAILFAIL rateTest Results
{{ test.test_id }}{{ overall_test_health.pass_count }}{{ overall_test_health.fail_count }}{{ overall_test_health.fail_percent|default_if_none:0|floatformat }}%{{ overall_test_health.total_run_count }}{{ overall_test_health.total_count }}{{ current_test_health.pass_count|default:0 }}{{ current_test_health.fail_count|default:0 }}{{ current_test_health.fail_percent|default_if_none:0|floatformat }}%{{ current_test_health.total_count|default:0 }}{{ test.name|default:"not set" }}
+{% endblock %} === added file 'dashboard_app/templates/dashboard_app/image_status_list.html' --- dashboard_app/templates/dashboard_app/image_status_list.html 1970-01-01 00:00:00 +0000 +++ dashboard_app/templates/dashboard_app/image_status_list.html 2011-07-22 00:57:44 +0000 @@ -0,0 +1,38 @@ +{% extends "dashboard_app/_content.html" %} +{% load call %} + + +{% block content %} + + + + + + {% for hwpack in hwpack_list %} + + {% endfor %} + + + + {% for rootfs in rootfs_list %} + + + {% for hwpack in hwpack_list %} + + {% endfor %} + + {% endfor %} + +
{{ hwpack }}
{{ rootfs }} + {% call ImageHealth rootfs hwpack as image_health %} + {{ image_health.get_test_runs.count }} test runs + {% endcall %} +
+{% endblock %} === added file 'dashboard_app/templates/dashboard_app/image_test_history.html' --- dashboard_app/templates/dashboard_app/image_test_history.html 1970-01-01 00:00:00 +0000 +++ dashboard_app/templates/dashboard_app/image_test_history.html 2011-07-22 00:57:44 +0000 @@ -0,0 +1,15 @@ +{% extends "dashboard_app/_content.html" %} + + +{% block content %} + +{% include "dashboard_app/_test_run_list_table.html" %} +{% endblock %} === modified file 'dashboard_app/templates/dashboard_app/index.html' --- dashboard_app/templates/dashboard_app/index.html 2011-07-12 02:34:12 +0000 +++ dashboard_app/templates/dashboard_app/index.html 2011-07-22 01:09:09 +0000 @@ -1,10 +1,58 @@ {% extends "dashboard_app/_content.html" %} + {% block content %} -

TODO

-
    -
  • Briefly mention key dashboard features
    • -
  • Link to readthedocs dashboard manual
    • -
  • Add sensible dashboard index page (recent/interesting stuff)
    • +

    Welcome

    +

    The Validation Dashboard is your window to +test results, regardless of how your run your tests you +can upload the results here and analyze them with simple +built-in reports as well as arbitrary custom reports and +data mining queries.

    + +

    Key Features

    +
      +
    • Online repository of test results, with simple to use, web APIs and + command line tools for uploading test results.
      • +
    • Test results are packaged in documents (bundles) that you can easily sync + across systems, model is similar to the one used by git
      • +
    • Test results can refer to software and hardware context so that you know + exactly what software and hardware combination fails
      • +
    • Data mining and reporting allows users to create custom tailored reports + based on the data in the system
      • +
    • Distributed work-flow model, with some data privacy out of the box, fully + private installation can be deployed in minutes.
      • +
    + +

    Documentation & Get Started

    +

    To get started quickly follow the link below, if you feel that an important +content is missing please report a bug or ask a question. Please make sure to report dashboard version (you are +currently using version {{lava.extensions.as_mapping.dashboard_app.version}})

    +

    All documentation is hosted on ReadTheDocs.org.

    + +

    Developers

    +
      +
    • How to put test results of my test suite into the Dashboard?
      • +
    • How to integrate my testing toolkit with the Dashboard?
      • +
    • How to allow users of my application to send anonymous qualitative and + quantitative (tests and benchmarks) data from their systems?
      • +
    + +

    Managers

    +
      +
    • What kind of reporting features are available out of the box?
      • +
    • How to create additional reports?
      • +
    • What kind of data is available in the system
      • +
    + +

    System Administrators

    +
      +
    • System requirements
      • +
    • How to deploy or upgrade the dashboard?
      • +
    • How to backup and restore the data
    {% endblock %} === modified file 'dashboard_app/templates/dashboard_app/report_detail.html' --- dashboard_app/templates/dashboard_app/report_detail.html 2011-07-13 11:07:18 +0000 +++ dashboard_app/templates/dashboard_app/report_detail.html 2011-07-19 21:46:36 +0000 @@ -48,3 +48,12 @@ }); {% endblock %} + +{% block body %} +{% if is_iframe %} +{{ report.get_html|safe}} +{% else %} +{{ block.super }} +{% endif %} +{% endblock %} + === modified file 'dashboard_app/templates/dashboard_app/test_result_detail.html' --- dashboard_app/templates/dashboard_app/test_result_detail.html 2011-07-12 02:34:12 +0000 +++ dashboard_app/templates/dashboard_app/test_result_detail.html 2011-07-18 16:29:23 +0000 @@ -4,15 +4,18 @@ {% block sidebar %} -

    {% trans "Hints" %}

    -

    -{% blocktrans %} -This is all the information that launch control has about this result. Log -analyzers can provide additional information by scrubbing it from the log file. -Information that is global to a test run can be attached to test run attributes -instead. -{% endblocktrans %} -

    +
    +
    + + Note: This is all the information that the dashboard has + about this result. Log analyzers can provide additional information by + scrubbing it from the log file. Information that is global to a test run + can be attached to test run attributes instead. +
    +
    +
    {% if test_result.test_run.test_results.count > 1 %}

    Other results

    Results from the same test run are available here

    @@ -42,11 +45,25 @@ {% block content %} -

    {% trans "Detailed information about test result" %}

    -

    {% trans "Launch Control has the following information about this test result" %}

    +

    {% trans "Test Result Details" %}

    {% trans "Result ID:" %}
    -
    {{ test_result }}
    +
    + {{ test_result }} +
    +
    + + {% trans "Note:" %} + {% blocktrans %} + You can navigate to this test result, regardless of the bundle stream it is + located in, by using this + {% endblocktrans %} + {% trans "permalink" %} +
    +
    +
    {% trans "Test case:" %}
    {% if test_result.test_case %} @@ -54,7 +71,7 @@ {% else %} {% trans "unknown test case" %} {% endif %} - {% trans "from test" %} {{ test_result.test_run.test }} + {% trans "from test" %} {{ test_result.test_run.test }}
    {% trans "Test outcome:" %}
    {{ test_result.get_result_display }}
    === modified file 'dashboard_app/templates/dashboard_app/test_run_detail.html' --- dashboard_app/templates/dashboard_app/test_run_detail.html 2011-07-13 17:28:46 +0000 +++ dashboard_app/templates/dashboard_app/test_run_detail.html 2011-07-18 16:29:23 +0000 @@ -43,9 +43,9 @@ {% block sidebar %}
    {% trans "Test Run UUID" %}
    -
    {{ test_run.analyzer_assigned_uuid }}
    +
    {{ test_run.analyzer_assigned_uuid }} {% trans "permalink" %}
    {% trans "Test Name" %}
    -
    {{ test_run.test }}
    +
    {{ test_run.test }}
    {% trans "OS Distribution" %}
    {{ test_run.sw_image_desc|default:"Unspecified" }}
    {% trans "Bundle SHA1" %}
    === added file 'dashboard_app/templatetags/call.py' --- dashboard_app/templatetags/call.py 1970-01-01 00:00:00 +0000 +++ dashboard_app/templatetags/call.py 2011-07-22 00:29:54 +0000 @@ -0,0 +1,64 @@ +from django import template + + +register = template.Library() + + +class CallNode(template.Node): + def __init__(self, func, args, name, nodelist): + self.func = func + self.args = args + self.name = name + self.nodelist = nodelist + + def __repr__(self): + return "" + + def _lookup_func(self, context): + parts = self.func.split('.') + current = context[parts[0]] + for part in parts[1:]: + if part.startswith("_"): + raise ValueError( + "Function cannot traverse private implementation attributes") + current = getattr(current, part) + return current + + def render(self, context): + try: + func = self._lookup_func(context) + values = [template.Variable(arg).resolve(context) for arg in self.args] + context.push() + context[self.name] = func(*values) + output = self.nodelist.render(context) + context.pop() + return output + except Exception as ex: + import logging + logging.exception("Unable to call %s with %r: %s", + self.func, self.args, ex) + raise + +def do_call(parser, token): + """ + Adds a value to the context (inside of this block) for caching and easy + access. + + For example:: + + {% call func 1 2 3 as result %} + {{ result }} + {% endcall %} + """ + bits = list(token.split_contents()) + if len(bits) < 2 or bits[-2] != "as": + raise template.TemplateSyntaxError( + "%r expected format is 'call func [args] [as name]'" % bits[0]) + func = bits[1] + args = bits[2:-2] + name = bits[-1] + nodelist = parser.parse(('endcall',)) + parser.delete_first_token() + return CallNode(func, args, name, nodelist) + +do_call = register.tag('call', do_call) === modified file 'dashboard_app/tests/__init__.py' --- dashboard_app/tests/__init__.py 2011-07-13 11:24:08 +0000 +++ dashboard_app/tests/__init__.py 2011-07-22 00:55:51 +0000 @@ -24,11 +24,11 @@ 'other.deserialization', 'other.login', 'other.test_client', - 'other.xml_rpc', 'regressions.LP658917', 'views.bundle_stream_list_view', 'views.test_run_detail_view', 'views.test_run_list_view', + 'views.redirects', ] def load_tests_from_submodules(_locals): === modified file 'dashboard_app/tests/other/dataview.py' --- dashboard_app/tests/other/dataview.py 2011-07-07 11:38:53 +0000 +++ dashboard_app/tests/other/dataview.py 2011-07-22 00:29:54 +0000 @@ -16,12 +16,10 @@ # You should have received a copy of the GNU Affero General Public License # along with Launch Control. If not, see . -import unittest - from mocker import Mocker, expect from testtools import TestCase -from dashboard_app.dataview import DataView +from dashboard_app.models import DataView class DataViewHandlerTests(TestCase): @@ -45,7 +43,7 @@ def setUp(self): super(DataViewHandlerTests, self).setUp() - self.dataview = DataView.load_from_xml(self.text) + self.dataview = DataView.repository.load_from_xml_string(self.text) def test_name_parsed_ok(self): self.assertEqual(self.dataview.name, "foo") @@ -97,7 +95,6 @@ Test for DataView.get_connection() """ # Mock connections['dataview'] to return special connection - from django.db.utils import ConnectionDoesNotExist mocker = Mocker() connections = mocker.replace("django.db.connections") special_connection = mocker.mock() === removed file 'dashboard_app/tests/other/xml_rpc.py' --- dashboard_app/tests/other/xml_rpc.py 2011-05-23 17:02:43 +0000 +++ dashboard_app/tests/other/xml_rpc.py 1970-01-01 00:00:00 +0000 @@ -1,132 +0,0 @@ -# Copyright (C) 2010 Linaro Limited -# -# Author: Zygmunt Krynicki -# -# This file is part of Launch Control. -# -# Launch Control is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License version 3 -# as published by the Free Software Foundation -# -# Launch Control is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Launch Control. If not, see . - -""" -Unit tests of the Dashboard application -""" -import xmlrpclib - -from django_testscenarios.ubertest import TestCaseWithScenarios - -from dashboard_app.dispatcher import ( - DjangoXMLRPCDispatcher, - FaultCodes, - xml_rpc_signature, - ) - - -class TestAPI(object): - """ - Test API that gets exposed by the dispatcher for test runs. - """ - - @xml_rpc_signature() - def ping(self): - """ - Return "pong" message - """ - return "pong" - - def echo(self, arg): - """ - Return the argument back to the caller - """ - return arg - - def boom(self, code, string): - """ - Raise a Fault exception with the specified code and string - """ - raise xmlrpclib.Fault(code, string) - - def internal_boom(self): - """ - Raise a regular python exception (this should be hidden behind - an internal error fault) - """ - raise Exception("internal boom") - - -class DjangoXMLRPCDispatcherTestCase(TestCaseWithScenarios): - - def setUp(self): - super(DjangoXMLRPCDispatcherTestCase, self).setUp() - self.dispatcher = DjangoXMLRPCDispatcher() - self.dispatcher.register_instance(TestAPI()) - - def xml_rpc_call(self, method, *args): - """ - Perform XML-RPC call on our internal dispatcher instance - - This calls the method just like we would have normally from our view. - All arguments are marshaled and un-marshaled. XML-RPC fault exceptions - are raised like normal python exceptions (by xmlrpclib.loads) - """ - request = xmlrpclib.dumps(tuple(args), methodname=method) - response = self.dispatcher._marshaled_dispatch(request) - # This returns return value wrapped in a tuple and method name - # (which we don't have here as this is a response message). - return xmlrpclib.loads(response)[0][0] - - -class DjangoXMLRPCDispatcherTests(DjangoXMLRPCDispatcherTestCase): - - def test_standard_fault_code_for_missing_method(self): - try: - self.xml_rpc_call("method_that_hopefully_does_not_exist") - except xmlrpclib.Fault as ex: - self.assertEqual( - ex.faultCode, - FaultCodes.ServerError.REQUESTED_METHOD_NOT_FOUND) - else: - self.fail("Calling missing method did not raise an exception") - - def test_ping(self): - retval = self.xml_rpc_call("ping") - self.assertEqual(retval, "pong") - - def test_echo(self): - self.assertEqual(self.xml_rpc_call("echo", 1), 1) - self.assertEqual(self.xml_rpc_call("echo", "string"), "string") - self.assertEqual(self.xml_rpc_call("echo", 1.5), 1.5) - - def test_boom(self): - self.assertRaises(xmlrpclib.Fault, - self.xml_rpc_call, "boom", 1, "str") - - -class DjangoXMLRPCDispatcherFaultCodeTests(DjangoXMLRPCDispatcherTestCase): - - scenarios = [ - ('method_not_found', { - 'method': "method_that_hopefully_does_not_exist", - 'faultCode': FaultCodes.ServerError.REQUESTED_METHOD_NOT_FOUND, - }), - ('internal_error', { - 'method': "internal_boom", - 'faultCode': FaultCodes.ServerError.INTERNAL_XML_RPC_ERROR, - }), - ] - - def test_standard_fault_codes(self): - try: - self.xml_rpc_call(self.method) - except xmlrpclib.Fault as ex: - self.assertEqual(ex.faultCode, self.faultCode) - else: - self.fail("Exception not raised") === modified file 'dashboard_app/tests/utils.py' --- dashboard_app/tests/utils.py 2011-05-30 16:53:27 +0000 +++ dashboard_app/tests/utils.py 2011-07-22 00:29:54 +0000 @@ -1,7 +1,6 @@ """ Django-specific test utilities """ -import os import xmlrpclib from django.conf import settings === added file 'dashboard_app/tests/views/redirects.py' --- dashboard_app/tests/views/redirects.py 1970-01-01 00:00:00 +0000 +++ dashboard_app/tests/views/redirects.py 2011-07-18 16:29:23 +0000 @@ -0,0 +1,75 @@ +# Copyright (C) 2010 Linaro Limited +# +# Author: Zygmunt Krynicki +# +# This file is part of Launch Control. +# +# Launch Control is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License version 3 +# as published by the Free Software Foundation +# +# Launch Control is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Launch Control. If not, see . + +from django.core.urlresolvers import reverse +from django_testscenarios.ubertest import TestCase + +from dashboard_app.tests import fixtures + + +class RedirectTests(TestCase): + + _PATHNAME = "/anonymous/" + _BUNDLE_TEXT = """ +{ + "test_runs": [ + { + "test_results": [ + { + "test_case_id": "test-case-0", + "result": "pass" + } + ], + "analyzer_assigned_date": "2010-10-15T22:04:46Z", + "time_check_performed": false, + "analyzer_assigned_uuid": "00000000-0000-0000-0000-000000000001", + "test_id": "examples" + } + ], + "format": "Dashboard Bundle Format 1.0" +} + """ + _BUNDLE_NAME = "whatever.json" + + def setUp(self): + super(RedirectTests, self).setUp() + self.bundle = fixtures.create_bundle(self._PATHNAME, self._BUNDLE_TEXT, self._BUNDLE_NAME) + self.bundle.deserialize() + self.assertTrue(self.bundle.is_deserialized) + + def test_bundle_permalink(self): + response = self.client.get( + reverse("dashboard_app.views.redirect_to_bundle", + args=(self.bundle.content_sha1, ))) + self.assertRedirects(response, self.bundle.get_absolute_url()) + + def test_test_run_permalink(self): + test_run = self.bundle.test_runs.all()[0] + response = self.client.get( + reverse("dashboard_app.views.redirect_to_test_run", + args=(test_run.analyzer_assigned_uuid, ))) + self.assertRedirects(response, test_run.get_absolute_url()) + + def test_test_result_permalink(self): + test_run = self.bundle.test_runs.all()[0] + test_result = test_run.test_results.all()[0] + response = self.client.get( + reverse("dashboard_app.views.redirect_to_test_result", + args=(test_run.analyzer_assigned_uuid, + test_result.relative_index))) + self.assertRedirects(response, test_result.get_absolute_url()) === modified file 'dashboard_app/tests/views/test_run_detail_view.py' --- dashboard_app/tests/views/test_run_detail_view.py 2011-07-12 02:33:19 +0000 +++ dashboard_app/tests/views/test_run_detail_view.py 2011-07-22 00:29:54 +0000 @@ -20,7 +20,6 @@ from django_testscenarios.ubertest import (TestCase, TestCaseWithScenarios) from dashboard_app.models import BundleStream, TestRun from django.contrib.auth.models import (User, Group) -from django.core.urlresolvers import reverse from dashboard_app.tests.utils import TestClient === modified file 'dashboard_app/urls.py' --- dashboard_app/urls.py 2011-07-13 12:29:53 +0000 +++ dashboard_app/urls.py 2011-07-22 00:57:44 +0000 @@ -55,4 +55,10 @@ url(r'^streams(?P/[a-zA-Z0-9/_-]+)bundles/(?P[0-9a-z]+)/(?P[a-zA-Z0-9-]+)/hardware-context/$', 'test_run_hardware_context'), url(r'^streams(?P/[a-zA-Z0-9/_-]+)bundles/(?P[0-9a-z]+)/(?P[a-zA-Z0-9-]+)/software-context/$', 'test_run_software_context'), url(r'^streams(?P/[a-zA-Z0-9/_-]+)test-runs/$', 'test_run_list'), + url(r'^permalink/test-run/(?P[a-zA-Z0-9-]+)/$', 'redirect_to_test_run'), + url(r'^permalink/test-result/(?P[a-zA-Z0-9-]+)/(?P[0-9]+)/$', 'redirect_to_test_result'), + url(r'^permalink/bundle/(?P[0-9a-z]+)/$', 'redirect_to_bundle'), + url(r'^image_status/$', 'image_status_list'), + url(r'^image_status/(?P[a-zA-Z0-9_-]+)\+(?P[a-zA-Z0-9_-]+)/$', 'image_status_detail'), + url(r'^image_status/(?P[a-zA-Z0-9_-]+)\+(?P[a-zA-Z0-9_-]+)/test-history/(?P[^/]+)/$', 'image_test_history'), ) === modified file 'dashboard_app/views.py' --- dashboard_app/views.py 2011-07-13 17:28:57 +0000 +++ dashboard_app/views.py 2011-07-22 00:57:44 +0000 @@ -20,25 +20,21 @@ Views for the Dashboard application """ -from django.contrib.auth.decorators import login_required -from django.contrib.csrf.middleware import csrf_exempt -from django.contrib.sites.models import Site from django.db.models.manager import Manager from django.db.models.query import QuerySet -from django.http import (HttpResponse, Http404) -from django.shortcuts import render_to_response +from django.http import Http404 +from django.shortcuts import render_to_response, redirect, get_object_or_404 from django.template import RequestContext from django.views.generic.list_detail import object_list, object_detail -from dashboard_app.dataview import DataView, DataViewRepository -from dashboard_app.dispatcher import DjangoXMLRPCDispatcher from dashboard_app.models import ( Attachment, Bundle, BundleStream, DataReport, + DataView, + ImageHealth, Test, - TestCase, TestResult, TestRun, ) @@ -276,7 +272,7 @@ @BreadCrumb( - "Result {relative_index}", + "Details of result {relative_index}", parent=test_run_detail, needs=['pathname', 'content_sha1', 'analyzer_assigned_uuid', 'relative_index']) def test_result_detail(request, pathname, content_sha1, analyzer_assigned_uuid, relative_index): @@ -384,31 +380,39 @@ raise Http404('No report matches given name.') return render_to_response( "dashboard_app/report_detail.html", { - 'bread_crumb_trail': BreadCrumbTrail.leading_to(report_detail, name=report.name, title=report.title), + "is_iframe": request.GET.get("iframe") == "yes", + 'bread_crumb_trail': BreadCrumbTrail.leading_to( + report_detail, + name=report.name, + title=report.title), "report": report, }, RequestContext(request)) @BreadCrumb("Data views", parent=index) def data_view_list(request): - repo = DataViewRepository.get_instance() return render_to_response( "dashboard_app/data_view_list.html", { 'bread_crumb_trail': BreadCrumbTrail.leading_to(data_view_list), - "data_view_list": repo.data_views + "data_view_list": DataView.repository.all(), }, RequestContext(request)) -@BreadCrumb("Details of {name}", parent=data_view_list, needs=['name']) +@BreadCrumb( + "Details of {name}", + parent=data_view_list, + needs=['name']) def data_view_detail(request, name): - repo = DataViewRepository.get_instance() try: - data_view = repo[name] - except KeyError: + data_view = DataView.repository.get(name=name) + except DataView.DoesNotExist: raise Http404('No data view matches the given query.') return render_to_response( "dashboard_app/data_view_detail.html", { - 'bread_crumb_trail': BreadCrumbTrail.leading_to(data_view_detail, name=data_view.name, summary=data_view.summary), + 'bread_crumb_trail': BreadCrumbTrail.leading_to( + data_view_detail, + name=data_view.name, + summary=data_view.summary), "data_view": data_view }, RequestContext(request)) @@ -437,3 +441,80 @@ extra_context={ 'bread_crumb_trail': BreadCrumbTrail.leading_to(test_detail, test_id=test_id) }) + + +def redirect_to_test_run(request, analyzer_assigned_uuid): + test_run = get_restricted_object_or_404( + TestRun, + lambda test_run: test_run.bundle.bundle_stream, + request.user, + analyzer_assigned_uuid=analyzer_assigned_uuid) + return redirect(test_run.get_absolute_url()) + + +def redirect_to_test_result(request, analyzer_assigned_uuid, relative_index): + test_result = get_restricted_object_or_404( + TestResult, + lambda test_result: test_result.test_run.bundle.bundle_stream, + request.user, + test_run__analyzer_assigned_uuid=analyzer_assigned_uuid, + relative_index=relative_index) + return redirect(test_result.get_absolute_url()) + + +def redirect_to_bundle(request, content_sha1): + bundle = get_restricted_object_or_404( + Bundle, + lambda bundle: bundle.bundle_stream, + request.user, + content_sha1=content_sha1) + return redirect(bundle.get_absolute_url()) + + +@BreadCrumb("Image Status Matrix", parent=index) +def image_status_list(request): + return render_to_response( + "dashboard_app/image_status_list.html", { + 'hwpack_list': ImageHealth.get_hwpack_list(), + 'rootfs_list': ImageHealth.get_rootfs_list(), + 'ImageHealth': ImageHealth, + 'bread_crumb_trail': BreadCrumbTrail.leading_to(image_status_list) + }, RequestContext(request)) + + +@BreadCrumb( + "Image Status for {rootfs_type} + {hwpack_type}", + parent=image_status_list, + needs=["rootfs_type", "hwpack_type"]) +def image_status_detail(request, rootfs_type, hwpack_type): + image_health = ImageHealth(rootfs_type, hwpack_type) + return render_to_response( + "dashboard_app/image_status_detail.html", { + 'image_health': image_health, + 'bread_crumb_trail': BreadCrumbTrail.leading_to( + image_status_detail, + rootfs_type=rootfs_type, + hwpack_type=hwpack_type), + }, RequestContext(request)) + + +@BreadCrumb( + "Test history for {test_id}", + parent=image_status_detail, + needs=["rootfs_type", "hwpack_type", "test_id"]) +def image_test_history(request, rootfs_type, hwpack_type, test_id): + image_health = ImageHealth(rootfs_type, hwpack_type) + test = get_object_or_404(Test, test_id=test_id) + test_run_list = image_health.get_test_runs().filter(test=test) + return render_to_response( + "dashboard_app/image_test_history.html", { + 'test_run_list': test_run_list, + 'test': test, + 'image_health': image_health, + 'bread_crumb_trail': BreadCrumbTrail.leading_to( + image_test_history, + rootfs_type=rootfs_type, + hwpack_type=hwpack_type, + test=test, + test_id=test_id), + }, RequestContext(request)) === modified file 'dashboard_app/xmlrpc.py' --- dashboard_app/xmlrpc.py 2011-07-07 11:39:05 +0000 +++ dashboard_app/xmlrpc.py 2011-07-22 01:04:53 +0000 @@ -24,15 +24,20 @@ import logging import xmlrpclib -from django.contrib.auth.models import User +from django.contrib.auth.models import User, Group from django.db import IntegrityError, DatabaseError -from linaro_django_xmlrpc.models import ExposedAPI -from linaro_django_xmlrpc.models import Mapper +from linaro_django_xmlrpc.models import ( + ExposedAPI, + Mapper, + xml_rpc_signature, +) from dashboard_app import __version__ -from dashboard_app.dataview import DataView, DataViewRepository -from dashboard_app.dispatcher import xml_rpc_signature -from dashboard_app.models import Bundle, BundleStream +from dashboard_app.models import ( + Bundle, + BundleStream, + DataView, +) class errors: @@ -57,10 +62,8 @@ All public methods are automatically exposed as XML-RPC methods """ - data_view_connection = DataView.get_connection() - @xml_rpc_signature('str') def version(self): """ @@ -137,8 +140,8 @@ ------------------------------ The following rules govern bundle stream upload access rights: - all anonymous streams are accessible - - personal streams are accessible by owners - - team streams are accessible by team members + - personal streams are accessible to owners + - team streams are accessible to team members """ try: @@ -201,8 +204,8 @@ ------------------------------ The following rules govern bundle stream download access rights: - all anonymous streams are accessible - - personal streams are accessible by owners - - team streams are accessible by team members + - personal streams are accessible to owners + - team streams are accessible to team members """ try: bundle = Bundle.objects.get(content_sha1=content_sha1) @@ -256,8 +259,8 @@ ------------------------------ The following rules govern bundle stream download access rights: - all anonymous streams are accessible - - personal streams are accessible by owners - - team streams are accessible by team members + - personal streams are accessible to owners + - team streams are accessible to team members """ bundle_streams = BundleStream.objects.accessible_by_principal(self.user) return [{ @@ -319,8 +322,8 @@ ------------------------------ The following rules govern bundle stream download access rights: - all anonymous streams are accessible - - personal streams are accessible by owners - - team streams are accessible by team members + - personal streams are accessible to owners + - team streams are accessible to team members """ try: bundle_stream = BundleStream.objects.accessible_by_principal(self.user).get(pathname=pathname) @@ -418,20 +421,55 @@ if name is None: name = "" try: - user, group, slug, is_public, is_anonymous = BundleStream.parse_pathname(pathname) + user_name, group_name, slug, is_public, is_anonymous = BundleStream.parse_pathname(pathname) except ValueError as ex: raise xmlrpclib.Fault(errors.FORBIDDEN, str(ex)) - if user is None and group is None: + + # Start with those to simplify the logic below + user = None + group = None + if is_anonymous is False: + if self.user is not None: + assert is_anonymous is False + assert self.user is not None + if user_name is not None: + if user_name != self.user.username: + raise xmlrpclib.Fault( + errors.FORBIDDEN, + "Only user {user!r} could create this stream".format(user=user_name)) + user = self.user # map to real user object + elif group_name is not None: + try: + group = self.user.groups.get(name=group_name) + except Group.DoesNotExist: + raise xmlrpclib.Fault( + errors.FORBIDDEN, + "Only a member of group {group!r} could create this stream".format(group=group_name)) + else: + assert is_anonymous is False + assert self.user is None + raise xmlrpclib.Fault( + errors.FORBIDDEN, "Only anonymous streams can be constructed (you are not signed in)") + else: + assert is_anonymous is True + assert user_name is None + assert group_name is None # Hacky but will suffice for now user = User.objects.get_or_create(username="anonymous-owner")[0] - try: - bundle_stream = BundleStream.objects.create(user=user, group=group, slug=slug, is_public=is_public, is_anonymous=is_anonymous, name=name) - except IntegrityError: - raise xmlrpclib.Fault(errors.CONFLICT, "Stream with the specified pathname already exists") + try: + bundle_stream = BundleStream.objects.create( + user=user, + group=group, + slug=slug, + is_public=is_public, + is_anonymous=is_anonymous, + name=name) + except IntegrityError: + raise xmlrpclib.Fault( + errors.CONFLICT, + "Stream with the specified pathname already exists") else: - # TODO: Make this constraint unnecessary - raise xmlrpclib.Fault(errors.FORBIDDEN, "Only anonymous streams can be constructed") - return bundle_stream.pathname + return bundle_stream.pathname def data_views(self): """ @@ -461,7 +499,6 @@ ----------------- None """ - repo = DataViewRepository.get_instance() return [{ 'name': data_view.name, 'summary': data_view.summary, @@ -472,7 +509,7 @@ "help": arg.help, "default": arg.default } for arg in data_view.arguments] - } for data_view in repo] + } for data_view in DataView.repository.all()] def data_view_info(self, name): """ @@ -516,10 +553,9 @@ 404 Name does not designate a data view """ - repo = DataViewRepository.get_instance() try: - data_view = repo[name] - except KeyError: + data_view = DataView.repository.get(name=name) + except DataView.DoesNotExist: raise xmlrpclib.Fault(errors.NOT_FOUND, "Data view not found") else: query = data_view.get_backend_specific_query(self.data_view_connection) @@ -564,10 +600,9 @@ ----------------- TBD """ - repo = DataViewRepository.get_instance() try: - data_view = repo[name] - except KeyError: + data_view = DataView.repository.get(name=name) + except DataView.DoesNotExist: raise xmlrpclib.Fault(errors.NOT_FOUND, "Data view not found") try: rows, columns = data_view(self.data_view_connection, **arguments) @@ -584,7 +619,6 @@ } - # Mapper used by the legacy URL legacy_mapper = Mapper() legacy_mapper.register_introspection_methods() === modified file 'doc/changes.rst' --- doc/changes.rst 2011-06-29 22:11:18 +0000 +++ doc/changes.rst 2011-07-22 01:31:19 +0000 @@ -1,6 +1,45 @@ Version History *************** +.. _version_0_6: + +Version 0.6 +=========== + +This version was released as 2011.07 in the Linaro monthly release process. + +Release highlights: + +* New UI synchronized with lava-server, the UI is going to be changed in the + next release to be more in line with the official Linaro theme. Currently + most changes are under-the-hood, sporting more jQuery UI CSS. +* New test browser that allows to see all the registered tests and their test + cases. +* New data view browser, similar to data view browser. +* New permalink system that allows easy linking to bundles, test runs and test results. +* New image status views that allow for quick inspection of interesting + hardware pack + root filesystem combinations. +* New image status detail view with color-coded information about test failures + affecting current and historic instances of a particular root filesystem + + hardware pack combination. +* New image test history view showing all the runs of a particular test on a + particular combination of root filesystem + hardware pack. +* New table widget for better table display with support for client side + sorting and searching. +* New option to render data reports without any navigation that is suitable for + embedding inside an iframe (by appending &iframe=yes to the URL) +* New view for showing text attachments associated with test runs. +* New view showing test runs associated with a specific bundle. +* New view showing the raw JSON text of a bundle. +* New view for inspecting bundle deserialization failures. +* Integration with lava-server/RPC2/ for web APIs +* Added support for non-anonymous submissions (test results uploaded by + authenticated users), including uploading results to personal (owned by + person), team (owned by group), public (visible) and private (hidden from + non-owners) bundle streams. +* Added support for creating non-anonymous bundle streams with + dashboard.make_stream() (for authenticated users) + .. _version_0_5: Version 0.5 === modified file 'doc/index.rst' --- doc/index.rst 2011-06-29 22:11:18 +0000 +++ doc/index.rst 2011-07-22 01:31:19 +0000 @@ -5,7 +5,7 @@ .. automodule:: dashboard_app .. seealso:: To get started quickly see :ref:`usage` -.. seealso:: See what's new in :ref:`version_0_5` +.. seealso:: See what's new in :ref:`version_0_6` Features ========