Author Topic: withgusto Networks - Status Updates and Discussion  (Read 59668 times)

0 Members and 1 Guest are viewing this topic.

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #105 on: October 17, 2014, 12:35:49 pm »
How much did renewal cost?
Renewal @ Namecheap was $9/year, cheaper if bought in bulk (~$35 for 5 years!).
There's definitely some alternatives out there, but I haven't experimented with them yet...
(Startcom's 2 year cert looks interesting - $60 for 2 years, with wildcard + multiple domains - but it looks kinda fishy...)
Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: withgusto Networks - Status Updates and Discussion
« Reply #106 on: October 17, 2014, 04:39:16 pm »
Also there's an RSS feed now.

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #107 on: October 17, 2014, 08:40:18 pm »
Migration Update - 10/17/2014

Importance: URGENT
Author: albert
Date: Fri, October 17th, 2014 at 9:36:06 AM PDT
Blog post: http://withg.org/blog/index.php?post=1413563766

Migration Update - MANDATORY FORMS
==================================

Almost there!
-------------
We're almost done with tooling for the migration, so migration is
imminent!

(HW peeps might've heard about tooling before! In this context,
tooling is designing tools to help manage the migration - no worries,
this is NOT like last time - these are very specific tools designed to
do very specific things for the migration process.)

MANDATORY FORM
--------------
That said, you MUST fill out the "I'm alive" form ASAP, if you haven't
already!

http://bit.ly/withgalive

NOT filling this out can lead your account to be "archived". The
unarchival process does exist, but it will be a hassle, since you will
be considered inactive in our system.

(Essentially, you're filling out a form like the sign-up form to get
your account back. Your data is still there, but your account just
doesn't exist.)

We're not trying to be evil - there's a LOT of users on this server
that we would consider significantly inactive! Freeing up resources
allows us to make the migration smoother, and for the resulting server
to be much faster and lighter!

So please - help us help YOU - fill it out!

http://bit.ly/withgalive

RSS/Pushbullet Support!
-----------------------
Random aside - our blog now supports RSS and Pushbullet Channels!
(Thanks to Sorunome for adding the support!)

RSS: http://withg.org/blog/?rss
Pushbullet Channel: https://www.pushbullet.com/channel?tag=withgusto

Subscribe to receive updates on your RSS reader or mobile device
(via Pushbullet)!




Also there's an RSS feed now.
(Yes, thank you Sorunome! :D)
Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: withgusto Networks - Status Updates and Discussion
« Reply #108 on: October 17, 2014, 10:49:32 pm »
How much did renewal cost?
Renewal @ Namecheap was $9/year, cheaper if bought in bulk (~$35 for 5 years!).
There's definitely some alternatives out there, but I haven't experimented with them yet...
(Startcom's 2 year cert looks interesting - $60 for 2 years, with wildcard + multiple domains - but it looks kinda fishy...)
Ah that's good. I just remembered horror stories about such stuff costing several hundreds of dollars before but IIRC it was upfront costs and I forgot how much it was yearly.

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #109 on: November 23, 2014, 03:05:00 pm »
Migration Update - DELAYED - 11/23/2014

Importance: IMPORTANT
Author: albert
Date: Sun, November 23rd, 2014 at 12:04:09 PM PST
Blog post: http://withg.org/blog/index.php?post=1416773049

Migration Update - Expect Delays!
==================================

Delays...
---------
Apologies, but it looks like we'll have to delay again, due to the admin
being short on time. This time, however, we want to make sure this
doesn't become an indefinite delay, so we've planned ahead and allocated
due dates for each step of the migration process.

You can check it out here:
https://trello.com/b/ZmSFCaVE/migration-2014-2015

This also includes the current status as well.

MANDATORY FORM
--------------
If you haven't already, you MUST fill out the "I'm alive" form ASAP
http://bit.ly/withgalive

NOT filling this out can lead your account to be "archived". The
unarchival process does exist, but it will be a hassle, since you will
be considered inactive in our system.

(Essentially, you're filling out a form like the sign-up form to get
your account back. Your data is still there, but your account just
doesn't exist.)

We're not trying to be evil - there's a LOT of users on this server
that we would consider significantly inactive! Freeing up resources
allows us to make the migration smoother, and for the resulting server
to be much faster and lighter!

So please - help us help YOU - fill it out!

http://bit.ly/withgalive

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: withgusto Networks - Status Updates and Discussion
« Reply #110 on: November 23, 2014, 05:04:37 pm »
Knew that the move wasn't going to happen >:D
* Sorunome runs

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #111 on: November 23, 2014, 05:44:25 pm »
Knew that the move wasn't going to happen >:D
* Sorunome runs
Oh gee thanks for having faith in me... :p /s

All jokes aside, I'm disappointed that I couldn't deliver the migration on time (and no surprise, since I'm in school)... but rest assured, the new server DOES exist, and we're working to make this a success!

This time, we're going to make sure that the migration moves past tooling! :p
(Last time, it died because it was difficult to make an automated migration tool - this time, we're limiting the tool's scope, and making multiple tools to handle certain cases.)

See you in 2015! :D (First milestone is January 12th, 2015!)
Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #112 on: January 27, 2015, 10:20:38 am »
Server Issues + Reboot - 1/27/2015

Importance: URGENT
Author: albert
Date: Tuesday, January 27th, 2015 at 7:16:47 AM PST
Blog post: http://withg.org/blog/index.php?post=1422371807

Server Issues + Reboot
=======================

Server Connectivity Issues
-------------------------
You may have noticed some network connectivity issues with our
services, including a laggy shell, downtime for IRC/ZNC, etc.

We are currently experiencing server connectivity issues due to issues
on our VPS provider's side of things. So far, we've had problems from
1/25 - 1/27. We believe that the VPS provider is just experiencing some
networking issues (including a possible DDoS), so we ask for hopefully
some patience as we try and ride this one out.

For reference, this is the amount of downtimes that have occurred:



Emergency Reboot Performed
---------------------------
In light of significant server downtime, we performed an emergency
reboot around 2015-01-26 10:31 AM EDT (yesterday morning) to attempt to
resolve the network issues.

As a result, your background processes have been terminated during the
reboot. If you have any background processes, you will need to start
them back up.

General services (shell, web server [Nginx/PHP5/NGINX], IRC, ZNC)
should now be up. If there are any issues with these services, please
let us know ASAP!

We're really sorry for the inconvenience any of this may have caused.

Thanks for flying withgusto! We're just experiencing some turbulence,
so hold tight!

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: withgusto Networks - Status Updates and Discussion
« Reply #113 on: January 27, 2015, 10:55:07 am »
* Sorunome prods alberthrocks to start migrating :P

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!

Offline Eeems

  • Mr. Dictator
  • Administrator
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 6268
  • Rating: +318/-36
  • little oof
    • View Profile
    • Eeems
Re: withgusto Networks - Status Updates and Discussion
« Reply #114 on: January 27, 2015, 11:34:09 am »
Glad to know somebody else is watching those pingdom emails ;P

EDIT: Some more in depth downtime information:

« Last Edit: January 27, 2015, 11:43:00 am by Eeems »
/e

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #115 on: February 07, 2015, 07:10:22 pm »
Due to a DDoS attack, withg.org is down for the moment. It may not be back up for a while. We'll keep you posted as things update. Sorry about that!

« Last Edit: February 07, 2015, 09:53:13 pm by alberthrocks »
Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: withgusto Networks - Status Updates and Discussion
« Reply #116 on: February 08, 2015, 04:59:18 am »
DDoS attacks are no fun :(

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!

Offline alberthrocks

  • Moderator
  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 876
  • Rating: +103/-10
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #117 on: February 08, 2015, 07:29:04 pm »
Update on Server Downtime

withgusto is now back up... but ZNC will continue to remain down for another day or so. SSH and other services requiring a direct connection will remain down as well. Sorry about that.

I'll say it now - we screwed up. The DDoS was pretty bad, but it definitely could have been preventable. In fact, a free solution like CloudFlare was really all we needed to prevent it.

What was the attack? Well, a certain user's public folder was harassed by many online bots, to the point where 10000+ requests to the user's folder were made per day in the past few days, totaling 40386 requests overall... just in February! (29581 of those requests were made to a single 18 MB file!) Obviously, this is enough to trigger a DDoS null-route.

Security has been lacking as of late, and I feel that we definitely could do better. As of today, CloudFlare is enabled on all withgusto domains... with the exception of a secret domain, which is being purposely withheld for now. We are planning on developing much better mitigation methods in the future so that this kind of attack won't take us out again.

At the same time, we should also examine the cause of the attack. Usually, these kind of attacks are caused by conflicts/arguments with other people, hence causing a DDoS. As some may know, we host a ZNC on withgusto. We'd like to continue ZNC service, but at the same time develop policies to avoid getting into this kind of situation again. These policies are in development, but include ZNC usage guidelines, as well as security guidelines for setting up ZNC. Even if we do get better DDoS protection in the future, it's way better to prevent a DDoS than to fight one!

Again, we apologize for the massive downtime. It was unexpected, but we definitely could have done better to prevent this kind of problem from occurring.

That's it - stay tuned for more information as we recover.

EDIT: For everyone that uses withgusto services (IRC, ZNC, etc.), please visit here and test out which location downloads the fastest. Once you have figured it out, post your best location here!
« Last Edit: February 08, 2015, 07:37:46 pm by alberthrocks »
Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/


Proud member of ClrHome!

Miss my old signature? Here it is!
Spoiler For Signature:
Alternate "New" IRC post notification bot (Newy) down? Go here to reset it! http://withg.org/albert/cpuhero/

Withgusto Networks Founder and Administrator
Main Server Status: http://withg.org/status/
Backup Server Status: Not available
Backup 2/MC Server Status: http://mc.withg.org/status/

Activity remains limited due to busyness from school et al. Sorry! :( Feel free to PM, email, or if you know me well enough, FB me if you have a question/concern. :)

Don't expect me to be online 24/7 until summer. Contact me via FB if you feel it's urgent.


Proud member of ClrHome!

Spoiler For "My Projects! :D":
Projects:

Computer/Web/IRC Projects:
C______c: 0% done (Doing planning and trying to not forget it :P)
A_____m: 40% done (Need to develop a sophisticated process queue, and a pretty web GUI)
AtomBot v3.0: 0% done (Planning stage, may do a litmus test of developer wants in the future)
IdeaFrenzy: 0% done (Planning and trying to not forget it :P)
wxWabbitemu: 40% done (NEED MOAR FEATURES :P)

Calculator Projects:
M__ C_____ (an A____ _____ clone): 0% done (Need to figure out physics and Axe)
C2I: 0% done (planning, checking the demand for it, and dreaming :P)

Offline pimathbrainiac

  • Occasionally I make projects
  • Members
  • LV10 31337 u53r (Next: 2000)
  • **********
  • Posts: 1731
  • Rating: +136/-23
  • dagaem
    • View Profile
Re: withgusto Networks - Status Updates and Discussion
« Reply #118 on: February 08, 2015, 09:56:31 pm »
The East Server works best for me. I hope ZNC gets back up soon!
I am Bach.

Offline Sorunome

  • Fox Fox Fox Fox Fox Fox Fox!
  • Support Staff
  • LV13 Extreme Addict (Next: 9001)
  • *************
  • Posts: 7920
  • Rating: +374/-13
  • Derpy Hooves
    • View Profile
    • My website! (You might lose the game)
Re: withgusto Networks - Status Updates and Discussion
« Reply #119 on: February 09, 2015, 12:15:01 pm »
Hey, no worries alberthrocks, every body makes mistakes, what counts is that you can figure that out and prevent such things in the future ;)

THE GAME
Also, check out my website
If OmnomIRC is screwed up, blame me!
Click here to give me an internet!