1. 01 Jan, 2009 1 commit
  2. 31 Dec, 2008 1 commit
    • Barry Warsaw's avatar
      More test repairs. · 996f7ea8
      Barry Warsaw authored
      * Move the lmtp configurations to lazr.config
      * Fixes to master.py
      * In qrunner.py, don't override initialize()
      * Make sure subprocesses get configured correctly during tests
  3. 30 Sep, 2008 1 commit
    • Barry Warsaw's avatar
      More clean up. · af67970b
      Barry Warsaw authored
      - Remove Utils.reap()
      - Flesh out interfaces.
      - PEP 8 names.
  4. 25 Sep, 2008 1 commit
  5. 23 Mar, 2008 1 commit
  6. 21 Mar, 2008 1 commit
  7. 12 Mar, 2008 1 commit
    • Barry Warsaw's avatar
      Add a working (though not yet complete) test for the LMTP runner. · 4e2070ca
      Barry Warsaw authored
      Refactor the TestableMaster class so that doctests don't need to do all the
      threading and event fiddling themselves.  Hide all that in the test class.
      In bin/master, add the -r/--runner option to allow overriding the set of queue
      runners to start from the command line.  This is much more convenient than
      fiddling the config file (which is still supported), and it allows us to start
      a subset of the runners from the TestableMaster class.
      Refactor the calculation of queue runner shortcut names into
      Configuration.add_qrunner().  This way, it's easy to share between bin/qrunner
      and bin/master.
      Use pkg_resource in bin/testall to create the test configuration file.  We're
      still using mailman.__file__ as the target to os.walk() though, so this
      conversion isn't complete.
      In bin/testall, update to the new convention of putting .options and
      .arguments on the parser instance, so we only need to pass back one thing.
      In test_documentation.py, use config.verbosity instead of
      config.options.verbosity.  Wind that through to bin/testall while we're at
      Update loginit.py so that defaults for all the logging options can be found in
      a [*] section.  This is better than [DEFAULT] because the latter requires
      %-interpolation and still requires each sublogger to be specified explicitly.
      Now we just set values in the [*] section and have them apply to all loggers.
      Allow for passing bin/testall's -v and -e options to any subprocesses that get
      created, e.g. via the TestableMaster.  Now testall will write a logging.cfg
      file with a [*] entry as appropriate.
      StockDatabase._reset() requires that the store be rolled back before it's
      reset.  Otherwise if there are outstanding transactions, the reset will fail
      with an OperationalError.
      Update the LMTPRunner.  Have it return a 501 error if the message has defects.
      Most defective messages are spam.  Make the LMTPRunner startable and
      stoppable.  Just because it's asyncore based doesn't mean that's not possible
      Enable the LMTP runner in tests.
  8. 27 Feb, 2008 1 commit
  9. 25 Feb, 2008 1 commit
    • Barry Warsaw's avatar
      Rework the basic infrastructure for qrunner process control. Split out the · 6965bd89
      Barry Warsaw authored
      functionality of mailmanctl into a separate master watcher script.  mailmanctl
      has not yet been updated but that'll happen next.
      Fix DELIVERY_MODULE to name a handler instead of a module.
      Change make_instance to use pkg_resources instead of module.__file__.
      Change the qrunner and master processes coordination so that the qrunners are
      not restarted on SIGINT, because otherwise C-c just doesn't work.  Now SIGUSR1
      is how we'll implement 'mailman restart'.
      Add a database commit so that initializing the schema doesn't lock the sqlite
      database.  Also, don't try to initialize the schema if the tables already
      exist.  Use some sqlite magic to do this test.
      Move mailman.cfg.in into a new package Mailman/extras inside the tree.  Also,
      MAILMAN_UID and MAILMAN_GID should be integers not strings.
      Convert the command runner to use an IHandler instance instead of handler
      module.  Similarly for the outgoing runner, DELIVERY_MODULE now names an
      IHandler instance instead of a handler module.
  10. 19 Feb, 2008 1 commit
  11. 08 Feb, 2008 1 commit
  12. 29 Sep, 2007 1 commit
    • Barry Warsaw's avatar
      Reorganize the qrunner infrastructure. First, the package has been renamed · 3e9ed398
      Barry Warsaw authored
      from Mailman.Queue to Mailman.queue (note the case change to be more PEP 8
      The Switchboard and Runner classes have been moved into the package
      __init__.py and the previous class modules have been removed.
      The switchboard cache is removed; I don't think it was ultimately buying us
      much.  Now, just import the Switchboard class and instantiate it directly.
      Added an IRunner interface.
      Renamed the ArchRunner to ArchiveRunner.
      bin/qrunner and bin/mailmanctl are updated accordingly.  For the former, it no
      long accepts -r=All to run all qrunners.  You can still use the short name
      (e.g. --runner=incoming) to run the built-in queue runners, but this design
      will eventually allow for plugin qrunners by allowing them to be run
      specifying the full package path to the class.  It also now accepts a leading
      dot to indicate a qrunner class relative to the Mailman.queue package.
  13. 05 Jan, 2007 1 commit
    • bwarsaw's avatar
      In HTTPRunner, when we see a KeyboardInterrupt during the serve_forever(), · 39fa9eaa
      bwarsaw authored
      don't re-raise the exception since that will show up in the log files.
      Instead just exit with a code equal to SIGTERM.
      Rework the way qrunners are specified in the mailman.cfg file.  Always start
      the default number of the default set of qrunners, but allow mailman.cfg to
      delete some with the del_qrunner() function.  Rename add_runner() to
      add_qrunner() and make this actually work <wink>.  Both take the shortened
      qrunner name as the first argument (e.g. 'Arch' instead of 'ArchRunner').
      Automatically start the MaildirRunner if USE_MAILDIR = Yes; same goes for
      LMTPRunner and USE_LMTP = Yes.  In both cases, you do not need to also use
      add_qrunner() in your mailman.cfg file to enable them.  You still do need to
      put "add_qrunner('HTTP')" in your mailman.cfg if you want to enable the wsgi
      server.  This may end up being added to the default set.
  14. 29 Dec, 2006 1 commit
    • bwarsaw's avatar
      Merged revisions 8113-8121 via svnmerge from · f4a456a8
      bwarsaw authored
        r8114 | bwarsaw | 2006-12-06 00:16:54 -0500 (Wed, 06 Dec 2006) | 44 lines
        Initial take on using SQLAlchemy to store list data in lieu of Python pickles.
        While all the list data (including OldStyleMemberships attributes) are stored
        in the database, many attributes are stored as PickleTypes binary data.  This
        isn't idea but it gets things working until a more sophisticated schema can be
        MailList class is now a new-style class, as is required by SQLAlchemy.  This
        makes several things, er, interesting.  Rip out all the low-level pickle
        reading and writing stuff.  Hook SA transaction events into Lock() and
        Unlock().  Move the hooking of the _memberadaptor into InitTempVars(), which
        gets called by the SQLAlchemy hooks (MailList.__init__() never is).
        Add an initialize.py module which centralizes all the initialization bits that
        command line scripts have to do, including configuration, logging, and atabase
        This change also converts bin/withlist to mmshell wrapper.
        Update to SQLAlchemy 0.3.1.
        Revamp paths.py.in considerably.  There were several problems with the old
        way.  We no longer disable default loading of site-packages so we don't need
        to add Python's site-packages back to sys.path.  Also, because
        site.addsitedir() causes things like .pth paths to be /appended/ to sys.path,
        they actually won't override any site-installed packages.  E.g. if SQLAlchemy
        is installed in the system Python, our version will not override.  IIUC,
        setuptools-based packages can be configured to work properly in the face of
        package versions, however not all packages we currently depend on are
        setuptools-based.  So instead, we steal a bit of stuff from site.py but change
        things so the prepend .pth stuff to sys.path.
        Update several modules to use True/False and whitespace normalization.
        Convert from mm_cfg to config object.  Modernize a few coding constructs.
        Add a couple of exceptions to handle database problems.
        In the export script, include the widget type in the elements.  This helped in
        my stupid little throw away conversion script, but I think it will be more
        generally useful.
        Add an interact.py module which refactors interactive interpreter access.
        Mostly this is used by withlist -i, but it lets us import Mailman.interact and
        drop into a prompt just about anywhere (e.g. debugging).
        r8115 | bwarsaw | 2006-12-07 09:13:56 -0500 (Thu, 07 Dec 2006) | 22 lines
        Start to flesh out more of the SQLAlchemy mechanisms.
        Added a MailList.__new__() which hooks instantiation to use a query on
        dbcontext to get an existing mailing list.  A 'no-args' call means we're doing
        a Create(), though eventually that will change too.
        For now, disable the CheckVersion() call.  Eventually this will be folded into
        schema migration.
        list_exists(): Rewrite to use the dbcontext query to determine if the named
        mailing list exists or not.  Requires the fqdn_listname.
        Eradicate two failed member adaptors: BDBMemberAdaptor and SAMemberships.
        Change the way the DBContext holds onto tables.  It now keeps a dictionary
        mapping the table's name to the SA Table instance.  This makes it easier to
        look up and use the individual tables.
        Add 'web_page_url' as an attribute managed by SA, and remove a debugging
        r8116 | bwarsaw | 2006-12-11 07:27:47 -0500 (Mon, 11 Dec 2006) | 29 lines
        Rework the whole dbcontext and transaction framework.  SA already handles
        nested transactions so we don't have to worry about them.  However, we do have
        the weird situation where some transactions are tied to MailList
        .Lock()/.Unlock()/.Save() and some are tied to non-mlist actions.  So now we
        use an @txn decorator to put methods in a session transaction, but then we
        also hook into the above MailList methods as possibly sub-transactions.  We
        use a weakref subclass to manage the MailList interface, with a dictionary
        mapping MailList fqdn_listnames against transactions.  The weakrefs come in by
        giving us a callback when a MailList gets derefed such that we're guaranteed
        to rollback any outstanding transaction.
        Also, we have one global DBContext instance but rather than force the rest of
        Mailman to deal with context objects, instead we expose API methods on that
        object into the Mailman.database module, which the rest of the code will use.
        Such methods must be prepended with 'api_' to get exposed this way.
        bin/rmlist now works with the SA-backend.  I refactored the code here so that
        other code (namely, the test suite) can more easily and consistently remove a
        mailing list.  This isn't the best place for it ultimately, but it's good
        enough for now.
        New convenience functions Utils.split_listname(), .fqdn_listname().
        Convert testall to use Mailman.initialize.initialize().  Not all tests work,
        but I'm down to only 8 failures and 7 errors.  Also, do a better job of
        recovering from failures in setUp().
        MailList.__new__() now takes keyword arguments.
        r8117 | bwarsaw | 2006-12-11 22:58:06 -0500 (Mon, 11 Dec 2006) | 7 lines
        Unit test repairs; even though the unit tests are still pretty fragile,
        everything now passes with the SQLAlchemy storage of list data.
        Added missing 'personalize' column.  Converted mailmanctl and qrunner to
        initialize() interface.  Fixed _cookie_path() to not fail if SCRIPT_NAME is
        not in the environment.
        r8118 | bwarsaw | 2006-12-27 18:45:41 -0500 (Wed, 27 Dec 2006) | 21 lines
        Utils.list_names(): Use a database query to get all the list names.
        dbcontext.py: Added api_get_list_names() to support Utils.list_names().
        listdata.py: Added two additional MailList attributes which need to be stored
        in the database.  The first is 'admin_member_chunksize' which isn't modifiable
        from the web.  The second is 'password' which holds the list's password.
        HTMLFormatObject: item strings can now be unicodes.
        bin/list_lists.py: Must call initialize() to get the database properly
        initialized, not just config.load().  This will be a common theme.
            - Remove md5 and crypt support
            - Added mailman.debug logger, though it will be only used during
            - The 'secret' can be a unicode now.
            - A few coding style updates; repr() instead of backticks, 'key in dict'
              instead of 'dict.has_key(key)'
        r8119 | bwarsaw | 2006-12-27 19:13:09 -0500 (Wed, 27 Dec 2006) | 2 lines
        genaliases.py: config.load() -> initialize()
        r8120 | bwarsaw | 2006-12-27 19:17:26 -0500 (Wed, 27 Dec 2006) | 9 lines
        Blocked revisions 8113 via svnmerge
          r8113 | bwarsaw | 2006-12-05 23:54:30 -0500 (Tue, 05 Dec 2006) | 3 lines
          Initialized merge tracking via "svnmerge" with revisions "1-8112" from 
        r8121 | bwarsaw | 2006-12-28 23:34:52 -0500 (Thu, 28 Dec 2006) | 20 lines
        Remove SIGTERM handling from all the CGI scripts.  This messes with HTTPRunner
        because when you issue "mailmanctl stop" after the signal handler has been
        installed, the process will get a SIGTERM, the signal handler will run, and
        the process will exit with a normal zero code.  This will cause mailmanctl to
        try to restart the HTTPRunner.
        I don't think we need that stuff at all when running under wsgi with a
        SQLAlchemy backend.  If mailmanctl kills the HTTPRunner in the middle of the
        process, I believe (but have not tested) that the transaction should get
        properly rolled back at process exit.  We need to make sure about this, and
        also we need to test the signal handling functionality under traditional CGI
        environment (if we even still want to support that).
        Also, make sure that we don't try to initialize the loggers twice in qrunner.
        This was the cause of all the double entries in logs/qrunner.
        Fix a coding style nit in mailmanctl.py.
        De-DOS-ify line endings in loginit.py.
  15. 08 Jul, 2006 1 commit
    • bwarsaw's avatar
      Massive conversion process so that Mailman can be run from a user specified · cbef3114
      bwarsaw authored
      configuration file.  While the full conversion is not yet complete, everything
      that seems to be required to run mailmanctl, qrunner, rmlist, and newlist have
      been updated.
      Basically, modules should no longer import mm_cfg, but instead they should
      import Mailman.configuration.config.  The latter is an object that's
      guaranteed to exist, but not guaranteed to be initialized until some top-level
      script calls config.load().  The latter should be called with the argument to
      -C/--config which is a new convention the above scripts have been given.
      In most cases, where mm_cfg.<variable> is used config.<variable> can be used,
      but the exceptions are where the default value must be available before
      config.load() is called.  Sometimes you can import Mailman.Default and get the
      variable from there, but other times the code has to be changed to work around
      this limitation.  Take each on a case-by-case basis.
      Note that the various directories calculated from VAR_PREFIX, EXEC_PREFIX, and
      PREFIX are now calculated in config.py, not in Defaults.py.  This way a
      configuration file can override the base directories and everything should
      work correctly.
      Other changes here include:
      - mailmanctl, qrunner, and update are switched to optparse and $-strings, and
        changed to the mmshell architecture
      - An etc directory has been added to /usr/local/mailman and a
        mailman.cfg.sample file is installed there.  Sites should now edit an
        etc/mailman.cfg file to do their configurations, although the mm_cfg file is
        still honored.  The formats of the two files are identical.
      - list_lists is given the -C/--config option
      - Some coding style fixes in bin/update, but not extensive
      - Get rid of nested scope hacks in qrunner.py
      - A start on getting EmailBase tests working (specifically test_message),
        although not yet complete.