From patchwork Wed Dec 12 22:35:19 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Doan X-Patchwork-Id: 13519 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id D4F6A23E02 for ; Wed, 12 Dec 2012 22:35:22 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id 45C8BA19B6A for ; Wed, 12 Dec 2012 22:35:22 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id c10so3053502ieb.39 for ; Wed, 12 Dec 2012 14:35:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=CkILIf9fldMq0nkHpLexwHIqyJJWS+kzilecX7rTGOw=; b=bZHP9t+diyXaw1MYDzRcfkI8PtLVVLYQwKqIJwF1GOsNRiXGFXEwOD3fcDLnTtZmYJ eElwR8qfbsjXqeT28zY8//uZopNPOsEX2q+AyNLkec71NzvWVKeYxkAc+XVzK8BPQQpy hL0VnDJEQjW6GNUoce2F7DEYDFDmfpSYhvTmCoFJYJJftkPkyAviFwgu+rBVYXudM9Yy 4y98SeDwFjWP/L7iyZ2+LX7kd3OxC0OSXXQs8mWxpdIPVhFQLXttRPX7xRcThtKcCCyV +fnsWYt4zYN1ZKlPv3u+ZOlx9apD9YRzm2WF+coVtfRIEMWJuvpsVmyq0vSuV5tnOOa6 0k8w== Received: by 10.50.213.69 with SMTP id nq5mr7606622igc.70.1355351721737; Wed, 12 Dec 2012 14:35:21 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.67.148 with SMTP id n20csp229060igt; Wed, 12 Dec 2012 14:35:21 -0800 (PST) Received: by 10.194.119.5 with SMTP id kq5mr4562282wjb.48.1355351720507; Wed, 12 Dec 2012 14:35:20 -0800 (PST) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id u19si8929720wjw.47.2012.12.12.14.35.19 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 12 Dec 2012 14:35:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1Tiutb-0001Sa-Pi for ; Wed, 12 Dec 2012 22:35:19 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id B3D13E0046 for ; Wed, 12 Dec 2012 22:35:19 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-scheduler X-Launchpad-Branch: ~linaro-validation/lava-scheduler/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 230 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-scheduler/trunk] Rev 230: remove celery requirement and allow remote "workers" Message-Id: <20121212223519.28875.75392.launchpad@ackee.canonical.com> Date: Wed, 12 Dec 2012 22:35:19 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="16361"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: c9fd9ddc2e43b6d39f9fac3c0e77d95071277301 X-Gm-Message-State: ALoCoQlN6Gqh3PwGmXKBDuHKrjH7tjp7TjCqOOpxrH5ICpWLqzw5IJOYIlDwu1HPzspgcACwuD73 Merge authors: Andy Doan (doanac) Related merge proposals: https://code.launchpad.net/~doanac/lava-scheduler/celery-removal/+merge/137476 proposed by: Andy Doan (doanac) ------------------------------------------------------------ revno: 230 [merge] committer: Andy Doan branch nick: lava-scheduler timestamp: Wed 2012-12-12 16:28:16 -0600 message: remove celery requirement and allow remote "workers" This removes the need for celery and also makes the scheduler daemon only accept jobs for boards configured in its local instance. This allows us to run remote workers in lava. The key thing is the change to the lava-deployment-tool that incorporates sshfs shared file system so that we have a central place for job log files. added: lava_scheduler_app/migrations/0028_auto__del_field_devicetype_use_celery.py modified: lava_scheduler_app/management/commands/schedulermonitor.py lava_scheduler_app/models.py lava_scheduler_daemon/board.py lava_scheduler_daemon/dbjobsource.py lava_scheduler_daemon/service.py --- lp:lava-scheduler https://code.launchpad.net/~linaro-validation/lava-scheduler/trunk You are subscribed to branch lp:lava-scheduler. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-scheduler/trunk/+edit-subscription === modified file 'lava_scheduler_app/management/commands/schedulermonitor.py' --- lava_scheduler_app/management/commands/schedulermonitor.py 2012-07-09 02:56:07 +0000 +++ lava_scheduler_app/management/commands/schedulermonitor.py 2012-12-03 05:03:38 +0000 @@ -16,7 +16,6 @@ # You should have received a copy of the GNU Affero General Public License # along with LAVA Scheduler. If not, see . -import os import simplejson @@ -37,12 +36,10 @@ source = DatabaseJobSource() dispatcher, board_name, json_file = args - log_to_stdout = os.getenv("CELERY_CONFIG_MODULE", False) - job = Job( simplejson.load(open(json_file)), dispatcher, - source, board_name, reactor, daemon_options=daemon_options, - log_to_stdout=log_to_stdout) + source, board_name, reactor, daemon_options=daemon_options) + def run(): job.run().addCallback(lambda result: reactor.stop()) reactor.callWhenRunning(run) === added file 'lava_scheduler_app/migrations/0028_auto__del_field_devicetype_use_celery.py' --- lava_scheduler_app/migrations/0028_auto__del_field_devicetype_use_celery.py 1970-01-01 00:00:00 +0000 +++ lava_scheduler_app/migrations/0028_auto__del_field_devicetype_use_celery.py 2012-12-03 05:12:52 +0000 @@ -0,0 +1,151 @@ +# -*- coding: utf-8 -*- +import datetime +from south.db import db +from south.v2 import SchemaMigration +from django.db import models + + +class Migration(SchemaMigration): + + def forwards(self, orm): + # Deleting field 'DeviceType.use_celery' + db.delete_column('lava_scheduler_app_devicetype', 'use_celery') + + + def backwards(self, orm): + # Adding field 'DeviceType.use_celery' + db.add_column('lava_scheduler_app_devicetype', 'use_celery', + self.gf('django.db.models.fields.BooleanField')(default=False), + keep_default=False) + + + models = { + 'auth.group': { + 'Meta': {'object_name': 'Group'}, + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '80'}), + 'permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'}) + }, + 'auth.permission': { + 'Meta': {'ordering': "('content_type__app_label', 'content_type__model', 'codename')", 'unique_together': "(('content_type', 'codename'),)", 'object_name': 'Permission'}, + 'codename': ('django.db.models.fields.CharField', [], {'max_length': '100'}), + 'content_type': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['contenttypes.ContentType']"}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'name': ('django.db.models.fields.CharField', [], {'max_length': '50'}) + }, + 'auth.user': { + 'Meta': {'object_name': 'User'}, + 'date_joined': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}), + 'email': ('django.db.models.fields.EmailField', [], {'max_length': '75', 'blank': 'True'}), + 'first_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}), + 'groups': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Group']", 'symmetrical': 'False', 'blank': 'True'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'is_active': ('django.db.models.fields.BooleanField', [], {'default': 'True'}), + 'is_staff': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'is_superuser': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'last_login': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}), + 'last_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}), + 'password': ('django.db.models.fields.CharField', [], {'max_length': '128'}), + 'user_permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'}), + 'username': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '30'}) + }, + 'contenttypes.contenttype': { + 'Meta': {'ordering': "('name',)", 'unique_together': "(('app_label', 'model'),)", 'object_name': 'ContentType', 'db_table': "'django_content_type'"}, + 'app_label': ('django.db.models.fields.CharField', [], {'max_length': '100'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'model': ('django.db.models.fields.CharField', [], {'max_length': '100'}), + 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}) + }, + 'dashboard_app.bundle': { + 'Meta': {'ordering': "['-uploaded_on']", 'object_name': 'Bundle'}, + '_gz_content': ('django.db.models.fields.files.FileField', [], {'max_length': '100', 'null': 'True', 'db_column': "'gz_content'"}), + '_raw_content': ('django.db.models.fields.files.FileField', [], {'max_length': '100', 'null': 'True', 'db_column': "'content'"}), + 'bundle_stream': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'bundles'", 'to': "orm['dashboard_app.BundleStream']"}), + 'content_filename': ('django.db.models.fields.CharField', [], {'max_length': '256'}), + 'content_sha1': ('django.db.models.fields.CharField', [], {'max_length': '40', 'unique': 'True', 'null': 'True'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'is_deserialized': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'uploaded_by': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'uploaded_bundles'", 'null': 'True', 'to': "orm['auth.User']"}), + 'uploaded_on': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.utcnow'}) + }, + 'dashboard_app.bundlestream': { + 'Meta': {'object_name': 'BundleStream'}, + 'group': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.Group']", 'null': 'True', 'blank': 'True'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'is_anonymous': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'is_public': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'name': ('django.db.models.fields.CharField', [], {'max_length': '64', 'blank': 'True'}), + 'pathname': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '128'}), + 'slug': ('django.db.models.fields.CharField', [], {'max_length': '64', 'blank': 'True'}), + 'user': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.User']", 'null': 'True', 'blank': 'True'}) + }, + 'lava_scheduler_app.device': { + 'Meta': {'object_name': 'Device'}, + 'current_job': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'+'", 'null': 'True', 'on_delete': 'models.SET_NULL', 'to': "orm['lava_scheduler_app.TestJob']", 'blank': 'True', 'unique': 'True'}), + 'device_type': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['lava_scheduler_app.DeviceType']"}), + 'device_version': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '200', 'null': 'True', 'blank': 'True'}), + 'health_status': ('django.db.models.fields.IntegerField', [], {'default': '0'}), + 'hostname': ('django.db.models.fields.CharField', [], {'max_length': '200', 'primary_key': 'True'}), + 'last_health_report_job': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'+'", 'null': 'True', 'on_delete': 'models.SET_NULL', 'to': "orm['lava_scheduler_app.TestJob']", 'blank': 'True', 'unique': 'True'}), + 'status': ('django.db.models.fields.IntegerField', [], {'default': '1'}), + 'tags': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['lava_scheduler_app.Tag']", 'symmetrical': 'False', 'blank': 'True'}) + }, + 'lava_scheduler_app.devicestatetransition': { + 'Meta': {'object_name': 'DeviceStateTransition'}, + 'created_by': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.User']", 'null': 'True', 'on_delete': 'models.SET_NULL', 'blank': 'True'}), + 'created_on': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}), + 'device': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'transitions'", 'to': "orm['lava_scheduler_app.Device']"}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'job': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['lava_scheduler_app.TestJob']", 'null': 'True', 'on_delete': 'models.SET_NULL', 'blank': 'True'}), + 'message': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}), + 'new_state': ('django.db.models.fields.IntegerField', [], {}), + 'old_state': ('django.db.models.fields.IntegerField', [], {}) + }, + 'lava_scheduler_app.devicetype': { + 'Meta': {'object_name': 'DeviceType'}, + 'display': ('django.db.models.fields.BooleanField', [], {'default': 'True'}), + 'health_check_job': ('django.db.models.fields.TextField', [], {'default': 'None', 'null': 'True', 'blank': 'True'}), + 'name': ('django.db.models.fields.SlugField', [], {'max_length': '50', 'primary_key': 'True'}) + }, + 'lava_scheduler_app.tag': { + 'Meta': {'object_name': 'Tag'}, + 'description': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'name': ('django.db.models.fields.SlugField', [], {'unique': 'True', 'max_length': '50'}) + }, + 'lava_scheduler_app.testjob': { + 'Meta': {'object_name': 'TestJob'}, + '_results_bundle': ('django.db.models.fields.related.OneToOneField', [], {'null': 'True', 'db_column': "'results_bundle_id'", 'on_delete': 'models.SET_NULL', 'to': "orm['dashboard_app.Bundle']", 'blank': 'True', 'unique': 'True'}), + '_results_link': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '400', 'null': 'True', 'db_column': "'results_link'", 'blank': 'True'}), + 'actual_device': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'related_name': "'+'", 'null': 'True', 'blank': 'True', 'to': "orm['lava_scheduler_app.Device']"}), + 'definition': ('django.db.models.fields.TextField', [], {}), + 'description': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '200', 'null': 'True', 'blank': 'True'}), + 'end_time': ('django.db.models.fields.DateTimeField', [], {'null': 'True', 'blank': 'True'}), + 'group': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.Group']", 'null': 'True', 'blank': 'True'}), + 'health_check': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'is_public': ('django.db.models.fields.BooleanField', [], {'default': 'False'}), + 'log_file': ('django.db.models.fields.files.FileField', [], {'default': 'None', 'max_length': '100', 'null': 'True', 'blank': 'True'}), + 'priority': ('django.db.models.fields.IntegerField', [], {'default': '50'}), + 'requested_device': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'related_name': "'+'", 'null': 'True', 'blank': 'True', 'to': "orm['lava_scheduler_app.Device']"}), + 'requested_device_type': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'related_name': "'+'", 'null': 'True', 'blank': 'True', 'to': "orm['lava_scheduler_app.DeviceType']"}), + 'start_time': ('django.db.models.fields.DateTimeField', [], {'null': 'True', 'blank': 'True'}), + 'status': ('django.db.models.fields.IntegerField', [], {'default': '0'}), + 'submit_time': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}), + 'submit_token': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['linaro_django_xmlrpc.AuthToken']", 'null': 'True', 'on_delete': 'models.SET_NULL', 'blank': 'True'}), + 'submitter': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'+'", 'to': "orm['auth.User']"}), + 'tags': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['lava_scheduler_app.Tag']", 'symmetrical': 'False', 'blank': 'True'}), + 'user': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.User']", 'null': 'True', 'blank': 'True'}) + }, + 'linaro_django_xmlrpc.authtoken': { + 'Meta': {'object_name': 'AuthToken'}, + 'created_on': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'}), + 'description': ('django.db.models.fields.TextField', [], {'default': "''", 'blank': 'True'}), + 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), + 'last_used_on': ('django.db.models.fields.DateTimeField', [], {'null': 'True'}), + 'secret': ('django.db.models.fields.CharField', [], {'default': "'ed8uhywjegfx2khwzfkcu0x3jwfbee4dbbp1dbs8mj0er7717uzx42c84bgdi9iss5xymgsz1cg1508cgcc7ypj0ux6zn1d8rbayo6jee49fhoopu7mlcailg69i6mbw'", 'unique': 'True', 'max_length': '128'}), + 'user': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'auth_tokens'", 'to': "orm['auth.User']"}) + } + } + + complete_apps = ['lava_scheduler_app'] \ No newline at end of file === modified file 'lava_scheduler_app/models.py' --- lava_scheduler_app/models.py 2012-11-30 00:11:57 +0000 +++ lava_scheduler_app/models.py 2012-12-03 05:12:52 +0000 @@ -56,10 +56,6 @@ health_check_job = models.TextField( null=True, blank=True, default=None, validators=[validate_job_json]) - use_celery = models.BooleanField(default=False, - help_text=("Denotes the job should be run via the celery "\ - "schedulermonitor rather than the local one")) - display = models.BooleanField(default=True, help_text=("Should this be displayed in the GUI or not. This can be " \ "useful if you are removing all devices of this type but don't " \ @@ -148,9 +144,6 @@ def get_device_health_url(self): return ("lava.scheduler.labhealth.detail", [self.pk]) - def use_celery(self): - return self.device_type.use_celery - def recent_jobs(self): return TestJob.objects.select_related( "actual_device", === modified file 'lava_scheduler_daemon/board.py' --- lava_scheduler_daemon/board.py 2012-09-23 23:28:45 +0000 +++ lava_scheduler_daemon/board.py 2012-12-03 05:03:38 +0000 @@ -83,7 +83,7 @@ def __init__(self, job_data, dispatcher, source, board_name, reactor, - daemon_options, log_to_stdout=False): + daemon_options): self.job_data = job_data self.dispatcher = dispatcher self.source = source @@ -98,7 +98,6 @@ self._time_limit_call = None self._killing = False self.job_log_file = None - self._log_to_stdout = log_to_stdout def _checkCancel(self): if self._killing: @@ -138,9 +137,6 @@ self.cancel("killing job for exceeding timeout") def run(self): - if self._log_to_stdout: - return self._run(sys.stdout) - d = self.source.getLogFileForJobOnBoard(self.board_name) return d.addCallback(self._run).addErrback( catchall_errback(self.logger)) @@ -166,7 +162,6 @@ d.addBoth(self._exited) return d - def _exited(self, exit_code): self.logger.info("job finished on %s", self.job_data['target']) if self._json_file is not None: @@ -204,7 +199,7 @@ def __init__(self, job_data, dispatcher, source, board_name, reactor, - daemon_options, use_celery=False): + daemon_options): self.logger = logging.getLogger(__name__ + '.MonitorJob') self.job_data = job_data self.dispatcher = dispatcher @@ -212,7 +207,6 @@ self.board_name = board_name self.reactor = reactor self.daemon_options = daemon_options - self.use_celery = use_celery self._json_file = None def run(self): @@ -222,19 +216,14 @@ with os.fdopen(fd, 'wb') as f: json.dump(json_data, f) - childFDs = {0:0, 1:1, 2:2} - if self.use_celery: - args = [ - 'setsid', 'lava', 'celery-schedulermonitor', - self.dispatcher, str(self.board_name), self._json_file] - else: - args = [ - 'setsid', 'lava-server', 'manage', 'schedulermonitor', - self.dispatcher, str(self.board_name), self._json_file, - '-l', self.daemon_options['LOG_LEVEL']] - if self.daemon_options['LOG_FILE_PATH']: - args.extend(['-f', self.daemon_options['LOG_FILE_PATH']]) - childFDs = None + childFDs = {0: 0, 1: 1, 2: 2} + args = [ + 'setsid', 'lava-server', 'manage', 'schedulermonitor', + self.dispatcher, str(self.board_name), self._json_file, + '-l', self.daemon_options['LOG_LEVEL']] + if self.daemon_options['LOG_FILE_PATH']: + args.extend(['-f', self.daemon_options['LOG_FILE_PATH']]) + childFDs = None self.logger.info('executing "%s"', ' '.join(args)) self.reactor.spawnProcess( SchedulerMonitorPP(d, self.board_name), 'setsid', @@ -299,7 +288,7 @@ job_cls = MonitorJob def __init__(self, source, board_name, dispatcher, reactor, daemon_options, - use_celery=False, job_cls=None): + job_cls=None): self.source = source self.board_name = board_name self.dispatcher = dispatcher @@ -312,7 +301,6 @@ self._stopping_deferreds = [] self.logger = logging.getLogger(__name__ + '.Board.' + board_name) self.checking = False - self.use_celery = use_celery def _state_name(self): if self.running_job: @@ -386,7 +374,7 @@ self.logger.info("starting job %r", job_data) self.running_job = self.job_cls( job_data, self.dispatcher, self.source, self.board_name, - self.reactor, self.daemon_options, self.use_celery) + self.reactor, self.daemon_options) d = self.running_job.run() d.addCallbacks(self._cbJobFinished, self._ebJobFinished) === modified file 'lava_scheduler_daemon/dbjobsource.py' --- lava_scheduler_daemon/dbjobsource.py 2012-11-13 18:48:34 +0000 +++ lava_scheduler_daemon/dbjobsource.py 2012-12-03 05:03:38 +0000 @@ -1,5 +1,6 @@ import datetime import logging +import os import urlparse from dashboard_app.models import Bundle @@ -20,6 +21,8 @@ from zope.interface import implements +import lava_dispatcher.config as dispatcher_config + from lava_scheduler_app.models import ( Device, DeviceStateTransition, @@ -89,8 +92,14 @@ return self.deferToThread(wrapper, *args, **kw) def getBoardList_impl(self): - return [ {'hostname': d.hostname, 'use_celery': d.use_celery()} - for d in Device.objects.all()] + cfgdir = os.path.join(os.environ['VIRTUAL_ENV'], 'etc/lava-dispatcher') + configured_boards = [ + x.hostname for x in dispatcher_config.get_devices(cfgdir)] + boards = [] + for d in Device.objects.all(): + if d.hostname in configured_boards: + boards.append({'hostname': d.hostname}) + return boards def getBoardList(self): return self.deferForDB(self.getBoardList_impl) === modified file 'lava_scheduler_daemon/service.py' --- lava_scheduler_daemon/service.py 2012-07-08 16:48:10 +0000 +++ lava_scheduler_daemon/service.py 2012-12-03 05:03:38 +0000 @@ -25,24 +25,19 @@ self._cbUpdateBoards).addErrback(catchall_errback(self.logger)) def _cbUpdateBoards(self, board_cfgs): - '''board_cfgs is an array of dicts {hostname=name, use_celery=...} ''' + '''board_cfgs is an array of dicts {hostname=name} ''' new_boards = {} for board_cfg in board_cfgs: board_name = board_cfg['hostname'] - use_celery = board_cfg['use_celery'] if board_cfg['hostname'] in self.boards: board = self.boards.pop(board_name) - if use_celery != board.use_celery: - board.use_celery = use_celery - self.logger.info("use_celery changed for %s to '%s'" % \ - (board_name, use_celery)) new_boards[board_name] = board else: self.logger.info("Adding board: %s" % board_name) new_boards[board_name] = Board( self.source, board_name, self.dispatcher, self.reactor, - self.daemon_options, use_celery) + self.daemon_options) new_boards[board_name].start() for board in self.boards.values(): self.logger.info("Removing board: %s" % board.board_name) @@ -61,5 +56,3 @@ self.logger.info( "waiting for %s boards", len(self.boards) - len(dead_boards)) return defer.gatherResults(ds) - -