in short: our server crashed the other day and we’re not really sure why, so we had to restore a backup

it went down at about 1:40 pm yesterday and we gave it until early this morning to diagnose what the problem was ourselves and get feedback from our server host (Digital Ocean) on what the problem might be. unfortunately we couldn’t do that and none of their suggestions helped, so this morning we pulled the trigger on restoring the backup. as a consequence a few days of posts and registrations are now lost.

so: if you registered some time in the past 7 days, you will need to re-register.

(as a registration aside, we approve registrations very quickly so if you have registered your account should be approved within 24 hours of joining the site)

it’s unfortunate to lose all the posts and registrations of a week but we don’t pretend to have 100% uptime (nor do we really have the funding to guarantee that, lol) so it is what it is. thanks for using the site, folks, hopefully this won’t happen again anytime soon

  • Chris Remington
    link
    fedilink
    1610 months ago

    I can add what the Digital Ocean support team told me. Namely, the server suffered from some type of “kernel panic” that is very rare. The Digital Ocean support team do not have access to our back-end user interface nor can they access our server otherwise.

    They guided me, step by step, through several attempts to restore the server from this “kernel panic”. We made eight different attempts which all failed and we don’t know what caused this issue in the first place.

    Luckily, we have server backups enabled and this was the last resort.

    • @Hirom@beehaw.org
      link
      fedilink
      6
      edit-2
      10 months ago

      A kernel panic may prevent a VM from booting but shouldn’t prevent accessing the disk image for recovery purpose.

      Maybe the kernel panic corrupted the disk image, or maybe Digital Ocean doesn’t have a recovery mechanism that work when the VM doesn’t boot.

      • Chris Remington
        link
        fedilink
        410 months ago

        Maybe the kernel panic corrupted the disk image…

        More than likely, this is what happened.

  • anji
    link
    fedilink
    1310 months ago

    Curiously, because I access Beehaw through my own instance, I didn’t notice. Those posts and replies are still there in my database. Makes me wish ActivityPub applications were truly distributed/decentralized and it could recover from this.

    Thanks for running Beehaw btw.

  • @nachtigallA
    link
    fedilink
    English
    910 months ago

    Damn that’s weird. feddit.de recently had a crash too which resulted in the loss of a few posts. Do we already know the reason behind the crash @wintermute@feddit.de ? (Sorry I can’t check it on Matrix. It’s annoying that Element supports only one account and I hate other clients too much, and I am too lazy to login to the web client :D)

    • @wintermuteA
      link
      fedilink
      English
      10
      edit-2
      10 months ago

      The pictrs container filled up the disk with temp files.

      The issue was closed, but the problem still persists.

    • alyaza [they/she]OPM
      link
      fedilink
      English
      1310 months ago

      if there are articles that got eaten by the past 7 days that you think are worth reposting yeah, i’d repost them