in short: our server crashed the other day and we’re not really sure why, so we had to restore a backup

it went down at about 1:40 pm yesterday and we gave it until early this morning to diagnose what the problem was ourselves and get feedback from our server host (Digital Ocean) on what the problem might be. unfortunately we couldn’t do that and none of their suggestions helped, so this morning we pulled the trigger on restoring the backup. as a consequence a few days of posts and registrations are now lost.

so: if you registered some time in the past 7 days, you will need to re-register.

(as a registration aside, we approve registrations very quickly so if you have registered your account should be approved within 24 hours of joining the site)

it’s unfortunate to lose all the posts and registrations of a week but we don’t pretend to have 100% uptime (nor do we really have the funding to guarantee that, lol) so it is what it is. thanks for using the site, folks, hopefully this won’t happen again anytime soon

  • Chris Remington
    link
    fedilink
    1610 months ago

    I can add what the Digital Ocean support team told me. Namely, the server suffered from some type of “kernel panic” that is very rare. The Digital Ocean support team do not have access to our back-end user interface nor can they access our server otherwise.

    They guided me, step by step, through several attempts to restore the server from this “kernel panic”. We made eight different attempts which all failed and we don’t know what caused this issue in the first place.

    Luckily, we have server backups enabled and this was the last resort.

    • @Hirom@beehaw.org
      link
      fedilink
      6
      edit-2
      10 months ago

      A kernel panic may prevent a VM from booting but shouldn’t prevent accessing the disk image for recovery purpose.

      Maybe the kernel panic corrupted the disk image, or maybe Digital Ocean doesn’t have a recovery mechanism that work when the VM doesn’t boot.

      • Chris Remington
        link
        fedilink
        410 months ago

        Maybe the kernel panic corrupted the disk image…

        More than likely, this is what happened.