1. 07 Sep, 2020 1 commit
    • Deimos's avatar
      Remove remnants of Redis breached-passwords check · 68870119
      Deimos authored
      We've been using pts_lbsearch on the text file for a few weeks now, and
      it's working fine. Checks generally seem to take about 10 ms, and that's
      totally fine for the relatively uncommon events of registrations and
      password changes.
      This removes everything related to the previous Redis-based method,
      which means we no longer need the second Redis server or the ReBloom
  2. 12 Aug, 2020 1 commit
    • Deimos's avatar
      Use pts_lbsearch to check for breached passwords · 26b1d4dd
      Deimos authored
      This replaces the current method of using a Bloom filter in Redis to
      check for breached passwords with searching the text file directly using
      pts_lbsearch (https://github.com/pts/pts-line-bisect/).
      I'm not removing the Redis-based method yet because I want to test the
      performance of this first, but this is *far* simpler and doesn't have
      the possibility for false positives like the Bloom filter does.
  3. 10 Aug, 2020 1 commit
  4. 15 Jul, 2020 1 commit
  5. 14 Jul, 2020 1 commit
    • Deimos's avatar
      Remove welcome message sent on registration · a9d312d1
      Deimos authored
      This message is getting pretty outdated now, and should probably be done
      in a different way regardless so that it doesn't need to be in the code,
      especially since forks won't want the same message (or any message).
      A better approach would probably be a consumer or cronjob watching for
      new registrations in the event stream.
  6. 16 Jun, 2020 1 commit
  7. 22 May, 2020 1 commit
  8. 15 May, 2020 2 commits
    • Deimos's avatar
      Exclude forums.terraria.org from url transforms · d5453cf2
      Deimos authored
      This forum uses the same invalid url scheme as the Paradox forums, so
      needs to be excluded as well.
    • Deimos's avatar
      Add simple metrics to event stream consumer jobs · b011be34
      Deimos authored
      This adds some very simple metrics to all of the background jobs that
      consume the event streams. Currently, the only "real" metric is a
      counter tracking how many messages have been processed by that consumer,
      but a lot of the value will come from being able to utilize the
      automatic "up" metric provided by Prometheus to monitor and make sure
      that all of the jobs are running.
      I decided to use ports starting from 25010 for these jobs - this is
      completely arbitrary, it's just a fairly large range of unassigned
      ports, so shouldn't conflict with anything.
      I'm not a fan of how much hard-coding is involved here for the different
      ports and jobs in the Prometheus config, but it's also not a big deal.
  9. 29 Feb, 2020 1 commit
    • Deimos's avatar
      Rework permissions/ACL system · 94b9e1bf
      Deimos authored
      This is a major rework of the permissions system to enable various new
      capabilities and clean up some of the oddities that were there.
      - The concept of "admin" permission is removed. All permissions must be
        granted individually.
      - Permissions can now be granted on a group-specific level, such as
        giving a user the ability to tag topics only in a specific group.
      - Permissions can also be denied for a specific group (or all groups),
        enabling uses like "tag topics in all groups except ~music".
      - Removed the two cases where "all permissions" were granted: users on
        themselves and the sender and recipient on messages. This was
        dangerous, we should always grant permissions explicitly.
      - Eliminated all the granular permissions for changing a user's settings
        (which were all granted implicitly), and replaced with an overall
        "change_settings" permission.
  10. 11 Feb, 2020 1 commit
  11. 28 Jan, 2020 1 commit
    • Deimos's avatar
      Enable "mark new comments" for all users · 24849973
      Deimos authored
      Previously, this feature was disabled by default. However, despite being
      one of the best features on the site, only about 10% of users ever
      enabled it, and even very involved/frequent users often didn't realize
      it existed.
      My original thought about why it should be opt-in only is that I thought
      it had a meaningful privacy impact, but it really doesn't. User visits
      to topics are already tracked through server logs and similar data, so
      the feature doesn't really make any difference.
      This commit enables the feature for everyone, removes the separate
      Settings page, and moves the "Collapse old comments" sub-setting onto
      the main Settings page.
  12. 25 Jan, 2020 2 commits
    • Deimos's avatar
      Add more SiteInfo entries for common sites · 0d589f24
      Deimos authored
    • Deimos's avatar
      Add error-handling for event stream consumers · 3e37c1ec
      Deimos authored
      Previously, if an event stream consumer hit an error when processing a
      message, it would crash and restart, and the message that caused the
      error would be left in "pending" status for that consumer forever while
      the consumer continued processing new messages.
      This commit adds some more deliberate handling of messages that cause
      * When a consumer starts, it will try to read pending messages first. In
        a case where an error was transient, this should mean that the message
        that previously caused a crash will be processed on retry.
      * If a particular message causes the consumer to crash 3 times, it will
        be considered "dead" and moved out of the consumer's pending list into
        one specifically for dead messages. These dead queues can be monitored
        and inspected manually to look into failures, while the consumer can
        still continue processing new messages.
      * After clearing or processing all pending messages, consumers go back
        to waiting for and processing new messages.
  13. 21 Jan, 2020 1 commit
    • Deimos's avatar
      Eliminate RabbitMQ · 3811ec39
      Deimos authored
      This removes RabbitMQ as well as everything else attached to it:
      Erlang; the Prometheus collector; the pg-amqp-bridge and all PostgreSQL
      functions and triggers; and the amqpy Python package and the Tildes code
      that used it.
      Note that this commit does not actually uninstall or delete any of these
      packages or services, so if you have a running instance that you want to
      keep (instead of re-provisioning from scratch), you will need to
      manually remove them if you want them completely gone.
  14. 20 Jan, 2020 1 commit
    • Deimos's avatar
      Replace RabbitMQ uses with Redis streams · bcb5a3e0
      Deimos authored
      RabbitMQ was used to support asynchronous/background processing tasks,
      such as determining word count for text topics and scraping the
      destinations or relevant APIs for link topics. This commit replaces
      RabbitMQ's role (as the message broker) with Redis streams.
      This included building a new "PostgreSQL to Redis bridge" that takes
      over the previous role of pg-amqp-bridge: listening for NOTIFY messages
      on a particular PostgreSQL channel and translating them to messages in
      appropriate Redis streams.
      One particular change of note is that the names of message "sources"
      were adjusted a little and standardized. For example, the routing key
      for a message caused by a new comment was previously "comment.created",
      but is now "comments.insert". Similarly, "comment.edited" became
      "comments.update.markdown". The new naming scheme uses the table name,
      proper name for the SQL operation, and column name instead of the
      previous unpredictable terms.
  15. 14 Jan, 2020 1 commit
  16. 06 Jan, 2020 1 commit
  17. 19 Dec, 2019 1 commit
  18. 17 Dec, 2019 1 commit
    • Deimos's avatar
      Update Prospector to 1.2.0 · 5c1a4dde
      Deimos authored
      This required a few minor changes/fixes:
      * Change the name of an ignored pylint check about logging interpolation
      * Add check=True to all subprocess.run() calls - this probably always
        should have been used so the scripts will crash if a command fails
      * Remove a couple of unnecessary list comprehensions
      * Ignore some warnings caused by mypy @hybrid_property workaround
  19. 13 Dec, 2019 1 commit
  20. 12 Dec, 2019 2 commits
  21. 10 Dec, 2019 1 commit
  22. 06 Dec, 2019 3 commits
  23. 02 Dec, 2019 1 commit
  24. 29 Nov, 2019 1 commit
    • Deimos's avatar
      Add IP-based ratelimit to Stripe donation page · f3eca36c
      Deimos authored
      There's still someone trying to use the Tildes donation page to check
      stolen credit cards occasionally. The new version of Checkout seems to
      be blocking them all successfully, but I might as well not make it easy
      on them.
  25. 28 Nov, 2019 1 commit
  26. 26 Nov, 2019 3 commits
  27. 13 Nov, 2019 2 commits
    • Deimos's avatar
      Add better control of content metadata by type · af66a760
      Deimos authored
      Previously, the content metadata displayed next to a topic's content
      type (like "Article: 1800 words") was fairly generic and could result in
      strange data being displayed if a scraper fetched it for an
      inappropriate type (for example, displaying word count for videos).
      This creates an enum to hold all the different content metadata fields,
      and moves some logic into that class to handle deciding which fields to
      show for different types, and the formatting logic for values.
    • Deimos's avatar
      Improve warnings when replying to old posts · 33124e2b
      Deimos authored
      Previously, the warning would only ever say "over a week old", even when
      the topic/comment was much older than that. This adds a new function to
      create a vague timedelta description for longer periods, and also
      enables the Javascript to use it as well through adding the description
      as a data attr on the reply button when a warning is needed, instead of
      duplicating the logic in JS.
  28. 05 Nov, 2019 1 commit
  29. 17 Oct, 2019 1 commit
  30. 10 Oct, 2019 1 commit
  31. 09 Oct, 2019 1 commit
    • Deimos's avatar
      Refactor tag list columns to use TypeDecorator · 43be910d
      Deimos authored
      I've always been pretty unhappy with the ugly way tags were implemented,
      using @hybrid_property and needing to do strange things all over the
      place to deal with converting underscores to spaces and vice versa, as
      well as other idiosyncracies.
      There are still a few oddities here and there, but overall this is much
  32. 08 Oct, 2019 1 commit