Update on Site Response Time

[meissen]

Hey guys,

Larry mentioned to me that the site had been running slow for him but I never experienced the same slowness until today so I couldn't really validate what the cause was. About 20 minutes ago I finally experienced it first hand and was able to investigate.

What I found was that a spammer was literally flooding our website with registration attempts - attempting to register every 3 seconds - which was overwhelming the server to the point that the website was slowing down.

I was able to temporarily mitigate the attack by blocking the IP address, but unfortunately vBulletin does not have a built in measure for countering such an issue. I've reached out to the vBulletin community forum to see if anyone has any ideas on what to do to resolve the issue permanently. Looking through our logs I was able to see that this has been going on intermittently where the bot would attempt to push through thousands of registrations and then stop for awhile, then pick up again 12 to 24 hours later.

In any case - if you guys are experiencing such slowness or issues, please don't hesitate to let me know.

[popo8]

PHEW!!!

I knew I wasnt nuts!


Thank You Bro... Great work...

[BIG CAT]

it might be more work but what about having any new member email admin to get a secure link to register.

[meissen]

The problem is that they're not completing the registration - they're just attempting to register but right before the last part to submit the registration they cancel out and it's flooding the site.

[firebird_1995]

Can you trace the IP address to a geographical location? Maybe it matches a former members location....

[BIG CAT]

Can you trace the IP address to a geographical location? Maybe it matches a former members location....

Monty over on yb had his it guy block all of China and russia because of spamming.

[meissen]

They do a flood of at least a thousand attempts each "session" and each "session" (usually separated by at least 12 hours) are different IP addresses entirely. This isn't your normal disgruntled member - they're using a chinese domain for their e-mail address and the frequency is far more than any normal guy could do - they'd have to have some sophisticated program that is doing it.

Edit: While one flood "session" was using a Chinese IP address, the second to most recent "session" was from France.

[BIG CAT]

The problem is that they're not completing the registration - they're just attempting to register but right before the last part to submit the registration they cancel out and it's flooding the site.

What I was trying to say was make any new members get a link sent from a admin. Not something that's Automatic.

[meissen]

I know what you're saying but we already have the e-mail validation link in place (though it IS automatic). The problem is that they're not getting that far in the registration so even if each member was moderated it wouldn't solve the problem. They're not completing the registration process which would trigger the e-mail link to validate their account (or require administrator moderation). They're not even doing that - they're cancelling out before hitting submit so it never creates the account or triggers that e-mail.

In any case, I liked the idea of blocking all the foreign spammer countries so I just IP banned the following countries:
Afghanistan
China
Algeria
Hong Kong
India
Iraq
Kazakhstan
Nigeria
Panama
Russia
Romania
Ukraine
Taiwan
Indonesia
Bulgaria
Vietnam
Slovakia
Moldova
Turkey
Philippines
Brazil
Latvia

[96LT1355Z28]

Nice, work! Two question though, is this just for new registrations? Can people in the military stationed in Iraq or Afganistan still get on the site?

[meissen]

Good point - I'll remove the blocks for Iraq and Afghanistan as it's a universal block applying to any attempted connection to the server.

[Fastbird]

Brian, now that I'm on a quick cable internet connection again, I'm experiencing the same issues. Very slow latency. Just now actually. Happened last night too, any time I tried a response to a thread it took FOREVER.

[Blackbird96WS6]

Yeah, this has been bugging me for the last week or so. At least I know it's not just me now lol. And it's pretty bad right now...

[CamaroZGuy]

Past few days i have experienced it between about 9pm-11pm, just figured it was a high traffic time.

Sent from my DROID RAZR using Tapatalk 2

[Blackbird96WS6]

Past few days i have experienced it between about 9pm-11pm, just figured it was a high traffic time.

Sent from my DROID RAZR using Tapatalk 2

I hop on every hour or so while I'm at work since I'm on a computer anyway and it's pretty much always like this all day long. Whoever is doing this half-assed DDOS attack needs to be taken out back and beaten senseless.

[HAND HERPES]

Which car manufacturer made the LT1 engine?

Which engine is this forum based off of?



Well the security isn't all there with the registration.

As a programmer (who is making the bot), they would just throw in a statement to check if one of those exist then they will just auto fill it with the correct answer. So pretty much, that anti-spam feature does nothing lol

I would add a captcha such as reCAPTCHA (http://www.google.com/recaptcha)instead. Much more secure.

[popo8]

Well the security isn't all there with the registration.

As a programmer (who is making the bot), they would just throw in a statement to check if one of those exist then they will just auto fill it with the correct answer. So pretty much, that anti-spam feature does nothing lol

I would add a captcha such as reCAPTCHA (http://www.google.com/recaptcha)instead. Much more secure.

Maybe we should make it "please enter the full vin nunber of ur LT1 car."

Bazinga!!!!


Larry (Popo8) Co-owner
LTXtech.com

[Blackbird96WS6]

Lol but then how would you cross check it so that it isn't just a random group of letters and numbers mashed together?

[Chris]

In response to the site load time, If I click on the response box it takes about 20-30 seconds before I can type. I just though it was my shit computer though

[HAND HERPES]

Lol but then how would you cross check it so that it isn't just a random group of letters and numbers mashed together?

This.

[popo8]

Lol but then how would you cross check it so that it isn't just a random group of letters and numbers mashed together?

U forget what i do for a living???


Larry (Popo8) Co-owner
LTXtech.com

[Chris]

All we would need is the check of a few vin digits not the whole vin

[Blackbird96WS6]

U forget what i do for a living???


Larry (Popo8) Co-owner
LTXtech.com

Lol ok, just sounds like a lot of unnecessary extra work for you when you could automate it with reCAPTCHA, but hey, who am I to tell you what to do with your free time :jest:

[HAND HERPES]

All we would need is the check of a few vin digits not the whole vin

Too easy still if the guy has a clue.


Lol ok, just sounds like a lot of unnecessary extra work for you when you could automate it with reCAPTCHA, but hey, who am I to tell you what to do with your free time :jest:

Agreed. Take it from the computer guys popo!

[thake618]

i noticed slow response on the site. i thought my brand new computer was slowing down but its only on this site even with 8gb

[MeanTA]

Can we make it where. new members must make a introduction thread. to unlock full site access?

[popo8]

Lol ok, just sounds like a lot of unnecessary extra work for you when you could automate it with reCAPTCHA, but hey, who am I to tell you what to do with your free time :jest:

I was really only saying it so xbrandonx couldnt log in.

You know the kid with the 6cyl and no job


Larry (Popo8) Co-owner
LTXtech.com

[firebird_1995]

LOL!!

[Blackbird96WS6]

Can we make it where. new members must make a introduction thread. to unlock full site access?

That wouldn't affect the amount of traffic on the server. They're pinging it nonstop with the sign up crap which is bogging it down across the entire site.


I was really only saying it so xbrandonx couldnt log in.

You know the kid with the 6cyl and no job


Larry (Popo8) Co-owner
LTXtech.com

Lmao

[thake618]

lol out of all things to spam they spam a car fourm.....pathetic

[MeanTA]

What if we require a blood sample and finger print sent to Popo. Then we send a link to registration form.

[popo8]

LOL!!




Lmao

Thank you.. Ill be here for a long time.

[popo8]

Blood/DNA has nearly 6month turn around.


What if we require a blood sample and finger print sent to Popo. Then we send a link to registration form.

[Chris]

What about a piss test?

[MeanTA]

Blood/DNA has nearly 6month turn around.

Im sure you could put a rush on them. Just tell the lab you got a site emergency. And valuable ltx posting time is on the line:)
Finger prints are pretty fast though. Either way it would weed out the terrorist.

[thake618]

not a good idea think of it lmao....with terrorism rates the way they are they would come to America with a bomb strapped to their chest and demand your blood to use to submit to popo hahah
Blood/DNA has nearly 6month turn around.

[popo8]

HAHAHA.. Im just a PEON in real life... no say. no pull.


Im sure you could put a rush on them. Just tell the lab you got a site emergency. And valuable ltx posting time is on the line:)
Finger prints are pretty fast though. Either way it would weed out the terrorist.


not a good idea think of it lmao....with terrorism rates the way they are they would come to America with a bomb strapped to their chest and demand your blood to use to submit to popo hahah

[thake618]

haha theres only one correct way to take care of this........

[popo8]

Yessss?????


haha theres only one correct way to take care of this........

[thake618]

lol i forgot but theres only one way haha

[MeanTA]

HAHAHA.. Im just a PEON in real life... no say. no pull.

Your a bad ass. And you know it.

http://www.youtube.com/watch?v=EMvL5GkTWe8

[thake618]

lol in all seriousness what about a new way to register not directly on the site like you register it gets sent to an admin and he gives u ur access information. mabey a little much but something lol

[popo8]

Ok...



maybe a little bit. :)


Your a bad ass. And you know it.

http://www.youtube.com/watch?v=EMvL5GkTWe8

hahahahahaha!

[chile]

Maybe we should make it "please enter the full vin nunber of ur LT1 car."

Bazinga!!!!


Larry (Popo8) Co-owner
LTXtech.com

I know that by heart!

[popo8]

How do we know your not cheating and just reading back your tattoo. lol


I know that by heart!

[meissen]

Well the security isn't all there with the registration.

As a programmer (who is making the bot), they would just throw in a statement to check if one of those exist then they will just auto fill it with the correct answer. So pretty much, that anti-spam feature does nothing lol

I would add a captcha such as reCAPTCHA (http://www.google.com/recaptcha)instead. Much more secure.

I'll change it to reCAPTCHA and see what happens. I have reCAPTCHA on my Meet & Greet registration page and I still get a handful of spammers a week that get through.

Edit: Unfortunately, it looks like that isn't the only issue as we haven't had a registration flood since I posted this thread and did the IP bans. I'm still experiencing slowness on the site too so I'll have to investigate it more. Unfortunately, GM blocks access to be able to RDP into my server so I can't investigate until after work.

[Blackbird96WS6]

I'll change it to reCAPTCHA and see what happens. I have reCAPTCHA on my Meet & Greet registration page and I still get a handful of spammers a week that get through.

Edit: Unfortunately, it looks like that isn't the only issue as we haven't had a registration flood since I posted this thread and did the IP bans. I'm still experiencing slowness on the site too so I'll have to investigate it more. Unfortunately, GM blocks access to be able to RDP into my server so I can't investigate until after work.

Think it could possibly be related to the influx of new users? How much traffic was your server built to support? And don't forget the first rule of IT, when in doubt, reboot it and see if it fixes itself! lol

[meissen]

Think it could possibly be related to the influx of new users? How much traffic was your server built to support? And don't forget the first rule of IT, when in doubt, reboot it and see if it fixes itself! lol

:lol: Yeah, I'm already planning on a reboot tonight or this weekend. To be honest I'm not sure how many users it's built to support - it's a Core2Quad Q9650 with 8gb DDR2 -- pretty beefy. Memory is staying right down at 2gb in use, it's the CPU that is bouncing between 20% and 95% last night while I was watching it. But it was more sinuous utilization than sustained high levels of cpu. When it was just MiFBody's server I had set up a few administrative backup tasks to run in the evening and late night hours but I'll have to see if that's causing some contention with resources now that it's supporting two sites.

[Blackbird96WS6]

:lol: Yeah, I'm already planning on a reboot tonight or this weekend. I also have some items that were set up to run when the server was just MiFBody. To be honest I'm not sure how many users it's built to support - it's a Core2Quad Q9650 with 8gb DDR2 -- pretty beefy. Memory is staying right down at 2gb in use, it's the CPU that is bouncing between 20% and 95% last night while I was watching it. But it was more sinuous utilization than sustained high levels of cpu.

You could try a program like "What's My Computer Doing?" or something to try and track down what's spiking the CPU usage, then go from there.

[SexyTransAm]

In response to the site load time, If I click on the response box it takes about 20-30 seconds before I can type. I just though it was my shit computer though

same here most the time. but pages still load fine

[meissen]

You could try a program like "What's My Computer Doing?" or something to try and track down what's spiking the CPU usage, then go from there.

Well, unfortunately I can see that the two main hogs are the apache and mysql instances, but hopefully knocking out some of the other stuff will help offset.

[popo8]

Well, unfortunately I can see that the two main hogs are the apache and mysql instances,.....

IDK what that means.

[meissen]

IDK what that means.

webserver and database :D

[zooguy]

its been running slow as i read through this thread and taking forever to load the next page and when i got the last page it is fine now ... so at 19:22 cst some thing changed and its now running faster

[popo8]

614am... Site is up to speed for now.


Larry (Popo8) Co-owner
LTXtech.com

[meissen]

Let me know how it's been running since last night around 10pm, I made some tweaks to hopefully speed things up a tad.

[Blackbird96WS6]

I'd say it's running pretty damn good for right now lol. I just woke up and opened up about 15 tabs at once to get caught up on reading from last night and they all loaded within about 20 seconds :D

[popo8]

Seems much more consistent to me... Ur site is still faster...but im only comparing cause u asked. Lol


Larry (Popo8) Co-owner
LTXtech.com

[meissen]

Yeah I've been trying to figure out why mifbody is faster than ltxtech, but at least the speed has been more consistent.

[NuckinFutz93]

I just got data error page with the blue V looking logo on it. It took me like 5 try to get the page to open and now its slow, real slow.

[meissen]

Sorry -- the server was rebooted at 8:35pm EST. Has it returned to good speed now? It's super speedy for me now that it's been rebooted.

[NuckinFutz93]

Its back to normal now.

[popo8]

Lets give this guy a huge thanks....


Seems right back on track brotha...


We wouldve never made it this far without him. I praise him alot...cause he deserves it!


Larry (Popo8) Co-owner
LTXtech.com

[meissen]

Lets give this guy a huge thanks....


Seems right back on track brotha...


We wouldve never made it this far without him. I praise him alot...cause he deserves it!


Larry (Popo8) Co-owner
LTXtech.com

:lol: Keep it up and I'm not going to be able to walk around my house because my head will be too big to fit through the door frames :lolsign:

Thanks Larry!

[Blackbird96WS6]

So the site seems to be lagging up a little bit again. It sort of comes and goes at the moment though, not quite as bad as the last round, but it's not snappy like it was yesterday.

[CamaroZGuy]

So the site seems to be lagging up a little bit again. It sort of comes and goes at the moment though, not quite as bad as the last round, but it's not snappy like it was yesterday.

it is lagging hardcore for me at time. and i gave up on trying to use tapatalk today.

[chile]

its currently good for me thru computer and tapatalk.

[Chris]

I was getting a script error on my computer still slow on my phone though. I think its all the new success

Sent from my SAMSUNG-SGH-I747 using Tapatalk 2

[popo8]

I sent meissen the weirdest message the site ever gave me... tonight it told me my password was over 180 days old.. I needed to change it.

[Blackbird96WS6]

I sent meissen the weirdest message the site ever gave me... tonight it told me my password was over 180 days old.. I needed to change it.

Oh god lol I hate changing my passwords. Let's hope that was a random ass glitch.

[popo8]

Oh god lol I hate changing my passwords. Let's hope that was a random ass glitch.

Well I changed it cause it was not letting me back in.. and I went into the admin area and tried to change it back... and I didnt have authority to... lol

Im gonna need to find someone higher up. lol.,

[CamaroZGuy]

past few mins, site seems to have smoothed out a bit.

[popo8]

past few mins, site seems to have smoothed out a bit.

I agree.. thought the same thing.

Dont Jinx it.

Again I am sorry with the issues....

[Blackbird96WS6]

Everybody knock on wood! lol

Don't sweat it Larry, technology is never as reliable as you'd like it to be lol. Sometimes shit goes bonkers that you just have no control of.

[popo8]

Everybody knock on wood! lol

Don't sweat it Larry, technology is never as reliable as you'd like it to be lol. Sometimes shit goes bonkers that you just have no control of.


Thank you for understanding.

[firebird_1995]

The auto refresh isn't causing issues is it? The shenanigans thread has almost 10 views to every post.

[Blackbird96WS6]

The auto refresh isn't causing issues is it? The shenanigans thread has almost 10 views to every post.

You know, I'd be curious to test that and see if you aren't on to something. That'd be pretty fucking hilarious if the Shenanigans thread's popularity single-handedly brought the site to its knees LOL.

[popo8]

You know, I'd be curious to test that and see if you aren't on to something. That'd be pretty fucking hilarious if the Shenanigans thread's popularity single-handedly brought the site to its knees LOL.

Well if thats the case.. we could close the current one out... and start the sequel.

[firebird_1995]

I think its more the fact that in addition to the guys posting there are a few following along and everyone has that window open and it refreshes every 5 seconds. It would probably do it in the sequel as well. If it is causing the problem it might be time to revisit the shout box idea

[Blackbird96WS6]

Well if thats the case.. we could close the current one out... and start the sequel.

Nooooo lol it's so epic! It must live on! Plus I don't know if the size of the thread is necessarily what would create any lag. It would be more based on the volume of traffic hitting the thread as it refreshes every 5 seconds, since so many people are in it now and it's moving so fast lol. And I'm purely talking theoretically here, too. Meissen would probably be a lot more knowledgeable in that area to see if it holds any merit. Still makes me smile at the thought, though :D But at the same time, I really don't think that's the issue.

[popo8]

Nooooo lol it's so epic! It must live on! Plus I don't know if the size of the thread is necessarily what would create any lag. It would be more based on the volume of traffic hitting the thread as it refreshes every 5 seconds, since so many people are in it now and it's moving so fast lol. And I'm purely talking theoretically here, too. Meissen would probably be a lot more knowledgeable in that area to see if it holds any merit. Still makes me smile at the thought, though :D But at the same time, I really don't think that's the issue.


LOL... it is EPIC...

Hopefully he takes a look in here and sees what the ideas you guys mentioned are.

[meissen]

Good point guys - I didn't think about the auto refresh being an issue. I know some aren't going to like this but I whittled it down to autorefresh every 20 seconds instead of 5 and I also disabled automatic un-idling from mouse movement. Let's see how the site works after doing that. If it responds well I'll try enabling the automatic un-idling from mouse movement and see how that works. I made the same change to MiFBody so that it wouldn't be auto refreshing threads over there every 5 seconds either.

PS - Sorry I'm always late to the party. You guys post so late at night and I'm in bed by 9pm EST on nights I have to work the next morning :lolsign: I'm usually off the computer by 8-ish, so I usually don't see posts until the next morning.

[popo8]

Good point guys - I didn't think about the auto refresh being an issue. I know some aren't going to like this but I whittled it down to autorefresh every 20 seconds instead of 5 and I also disabled automatic un-idling from mouse movement. Let's see how the site works after doing that. If it responds well I'll try enabling the automatic un-idling from mouse movement and see how that works. I made the same change to MiFBody so that it wouldn't be auto refreshing threads over there every 5 seconds either.

PS - Sorry I'm always late to the party. You guys post so late at night and I'm in bed by 9pm EST on nights I have to work the next morning :lolsign: I'm usually off the computer by 8-ish, so I usually don't see posts until the next morning.

thanku Brian...and no sweat...between different time zones and lifestyles its totally understandable.

As of 1053 in PA I can tell u its quick thru tapa talk....


Larry (Popo8) Co-owner
LTXtech.com

[Blackbird96WS6]

I'd say that's definitely making an improvement, the sites much snappier than I've ever seen it before lol. Yay for brainstorming!

[CamaroZGuy]

site is runing great this morning even being on a DoD computer/network....

[popo8]

..... Yay for brainstorming!

Seriously though.. thats how success happens. ONE BRAIN (or in this case BRIAN) can only do so much... but when we all put our heads together... its incredible.

Working good on the PD computer also Brian....

[meissen]

:thumbsup: Happy to hear guys! At GM our network "comes and goes" through the proxy -- one minute everything is snappy, then you click a link and the site stops responding but it's really the connection to the proxy that's throttling down so every site, including google, won't respond. So that makes my troubleshooting efforts during the day when I'm at work pretty hard because I can't really say the site is slow without blaming GM's network first :lolsign:

[popo8]

just wanted to say it still appears to be goin well. Maybe me destroying shenanigans helped also. lol.


Larry (Popo8) Co-owner
LTXtech.com

[meissen]

I think there's still some juice to be squeezed out -

At the very bottom of the page you guys will now see load time and # of queries. While I was at work yesterday I did a comparison between MiFBody and LTxTech -- they're on the same physical server, same database server, same apache server, etc... Yet for some reason MiFBody's load times were always quicker than LTxTech and I really don't understand why. I thought maybe the style in use here is a little more intensive, but I tried with the default vBulletin 4 style and the times were the same as before. So I have more investigating to do - in my mind they should be very comparable times.

Forum Home
LTxTech - 0.9599 seconds with 23 queries
MiFBody - 0.7373 seconds with 15 queries


Individual Forum
LTxTech - 1.0303 seconds with 22 queries
- 0.8094 seconds with 22 queries
- 0.7901 seconds with 22 queries
MiFBody - 0.475 seconds with 22 queries
- 0.4693 seconds with 22 queries
- 0.4249 seconds with 22 queries


Back to Forum Home
LTxTech - 0.9702 seconds with 23 queries
MiFBody - 0.7466 seconds with 15 queries


Individual Thread
LTxTech - 1.2893 seconds with 36 queries
- 1.405 seconds with 32 queries
- 1.1317 seconds with 32 queries
- 1.2134 seconds with 32 queries
MiFBody - 1.1792 seconds with 37 queries
- 1.1115 seconds with 33 queries
- 0.956 seconds with 33 queries
- 1.0216 seconds with 33 queries

[popo8]

I think there's still some juice to be squeezed out -

At the very bottom of the page you guys will now see load time and # of queries. While I was at work yesterday I did a comparison between MiFBody and LTxTech -- they're on the same physical server, same database server, same apache server, etc... Yet for some reason MiFBody's load times were always quicker than LTxTech and I really don't understand why. I thought maybe the style in use here is a little more intensive, but I tried with the default vBulletin 4 style and the times were the same as before. So I have more investigating to do - in my mind they should be very comparable times.

Forum Home
LTxTech - 0.9599 seconds with 23 queries
MiFBody - 0.7373 seconds with 15 queries


Individual Forum
LTxTech - 1.0303 seconds with 22 queries
- 0.8094 seconds with 22 queries
- 0.7901 seconds with 22 queries
MiFBody - 0.475 seconds with 22 queries
- 0.4693 seconds with 22 queries
- 0.4249 seconds with 22 queries


Back to Forum Home
LTxTech - 0.9702 seconds with 23 queries
MiFBody - 0.7466 seconds with 15 queries


Individual Thread
LTxTech - 1.2893 seconds with 36 queries
- 1.405 seconds with 32 queries
- 1.1317 seconds with 32 queries
- 1.2134 seconds with 32 queries
MiFBody - 1.1792 seconds with 37 queries
- 1.1115 seconds with 33 queries
- 0.956 seconds with 33 queries
- 1.0216 seconds with 33 queries

i noticed i was seeing that last night.


Larry (Popo8) Co-owner
LTXtech.com

[popo8]

Site is way slow for me all of the sudden. Firefox, home computer.

[meissen]

Hrm - I'll have to take a look at what was running. It seemed like we finally had it figured out? :(

[popo8]

Hrm - I'll have to take a look at what was running. It seemed like we finally had it figured out? :(

There is a very good chance it was my computer if no one else noticed.


Larry (Popo8) Co-owner
LTXtech.com

[chile]

I took had a little hiccup but its bit to normal for me.