Author Topic: The Recent Forum/IRC down Time  (Read 6671 times)

0 Members and 1 Guest are viewing this topic.

Offline Geekboy1011

  • The Oneironaut
  • Donator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2031
  • Rating: +119/-2
  • Dream that Awakening dream
    • View Profile
The Recent Forum/IRC down Time
« on: March 08, 2014, 04:07:19 pm »
Over the last few days there have been a series of downtime on both the irc network and the forum. We apologize for this. Our server manged to have a logging issue where we filled up our disk space and caused MySQL and some other processes to crash. Nickserv information on IRC was lost and thankfully nothing on the forum was lost to these incidents.

What we have done to fix it. We have started implementing local backups of everything to allow us to quickly fix the issues should they arise again. As well as taken a look at the issues with the logs and made the proper config changes to them (Why are rolling logs not the default anyone please?)

 Thank you,
     The Omnimaga Administration Team

Offline Juju

  • Incredibly sexy mare
  • Coder Of Tomorrow
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 5730
  • Rating: +500/-19
  • Weird programmer
    • View Profile
    • juju2143's shed
Re: The Recent Forum/IRC down Time
« Reply #1 on: March 08, 2014, 04:14:56 pm »
Glad it works again!

Remember the day the walrus started to fly...

I finally cleared my sig after 4 years you're happy now?
THEGAME
This signature is ridiculously large you've been warned.

The cute mare that used to be in my avatar is Yuki Kagayaki, you can follow her on Facebook and Tumblr.

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: The Recent Forum/IRC down Time
« Reply #2 on: March 08, 2014, 04:26:53 pm »
Good to hear it is working again :)

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: The Recent Forum/IRC down Time
« Reply #3 on: March 08, 2014, 05:03:41 pm »
At least I'm glad it's not just the server technical specs being too low to handle OIRC2, SMF 2.0 and Omninet simultaneously. That would have sucked since this would have meant having to host Omnom on Soru or Albert's servers. :P

Offline Eeems

  • Mr. Dictator
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 6268
  • Rating: +318/-36
  • little oof
    • View Profile
    • Eeems
Re: The Recent Forum/IRC down Time
« Reply #4 on: March 08, 2014, 05:05:19 pm »
At least I'm glad it's not just the server technical specs being too low to handle OIRC2, SMF 2.0 and Omninet simultaneously. That would have sucked since this would have meant having to host Omnom on Soru or Albert's servers. :P
Our server specs are fine. More then adequate for everything. It's kinda difficult when a log file grows to 13GB+ in size though.
/e

Offline Vogtinator

  • LV9 Veteran (Next: 1337)
  • *********
  • Posts: 1193
  • Rating: +108/-5
  • Instruction counter
    • View Profile
Re: The Recent Forum/IRC down Time
« Reply #5 on: March 08, 2014, 05:56:55 pm »
WOW, 13GB is massive. Which log file was it?
On my servers systemd handles logging just fine. You can even set a nice size limit for the syslog.
And for other services which log into own files there's logrotate.

Offline Eeems

  • Mr. Dictator
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 6268
  • Rating: +318/-36
  • little oof
    • View Profile
    • Eeems
Re: The Recent Forum/IRC down Time
« Reply #6 on: March 08, 2014, 05:58:15 pm »
WOW, 13GB is massive. Which log file was it?
On my servers systemd handles logging just fine. You can even set a nice size limit for the syslog.
And for other services which log into own files there's logrotate.
Mysql binary logs.
I've disabled it and we will have to setup something else for backups now then replication.
/e

Offline Geekboy1011

  • The Oneironaut
  • Donator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2031
  • Rating: +119/-2
  • Dream that Awakening dream
    • View Profile
Re: The Recent Forum/IRC down Time
« Reply #7 on: March 08, 2014, 05:59:29 pm »
Why not just disable it for issue tables? Or is it not that simple?

Offline Eeems

  • Mr. Dictator
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 6268
  • Rating: +318/-36
  • little oof
    • View Profile
    • Eeems
Re: The Recent Forum/IRC down Time
« Reply #8 on: March 08, 2014, 06:27:10 pm »
Why not just disable it for issue tables? Or is it not that simple?
Then we wouldn't be able to backup the important parts (messages)
/e

Offline Geekboy1011

  • The Oneironaut
  • Donator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2031
  • Rating: +119/-2
  • Dream that Awakening dream
    • View Profile
Re: The Recent Forum/IRC down Time
« Reply #9 on: March 08, 2014, 06:33:48 pm »
It's the forum dB that's having issues?

Offline Eeems

  • Mr. Dictator
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 6268
  • Rating: +318/-36
  • little oof
    • View Profile
    • Eeems
Re: The Recent Forum/IRC down Time
« Reply #10 on: March 08, 2014, 07:10:47 pm »
It's the forum dB that's having issues?
No binary logging was used to backup, and if I was to just disable it for the ones that would cause the filesystem to fill up then we wouldn't be able to properly back up.
/e