Keyword

Κ2 heavy load on large database, gateway timeouts

More
8 years 6 months ago #148538 by Nick
Yes the googlebot made huge traffic, but now it slowed down, after the changes i made in the crawl delay, and also the same applied for the bingbot, but again with the same settings both bots, crawl every 1-2 minutes.
Apart from that bot traffic probably the real problem appeared now, as after i slowed down the bots the traffic was almost doubled and actually tripled, since we had about 30.000 unique ΙP's per day, and now we have almost 80-90.000. The case is that all that traffic are not really visits, but in the access log they are coming from 3-4 blogs, some of them they have us in their affiliate pages.
The entries are like that:

37.6.xxx.xxx - - [19/Oct/2015:14:33:01 +0300] "GET /favicon.ico HTTP/1.0" 404 373 "blablabla.blogspot.com/2013/08/john-hopkins.html" "Mozilla/5.0 (Windows NT 6.1; rv:41.0) Gecko/20100101 Firefox/41.0"

79.166.xxx.xxx - - [19/Oct/2015:14:33:07 +0300] "GET /favicon.ico HTTP/1.0" 404 373 "blablabla2.blogspot.com/search?updated-max=2015-10-18T21:00:00%2B03:00&max-results=30&start=31&by-date=false" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0"

Those entries are thousands and occupy almost 95% of the daily traffic, and we have different IP's almost every second. As you can see they are targeting the favorite icon that i have already removed from the root, but this actually doesn't change things a lot. Actually i tried to block the referrers that this traffic is coming from, but the only difference is that instead of 404 Not Found, they are getting 403 Access Denied, and things got much worse when i blocked them, since the whole server collapsed from the overload.

From my hosting company they told me that these visits are legal, and the problem is in the K2 cache and it actually needs to be cleaned. Actually the K2 cache has about 60.000 files that occupy 3GB, because each article generates 6 different image sizes, so 10.000 articles generates all those files. But i told them that i cannot clean the cache, because those are the images of the articles. But the question again is why the website is working pretty fast in the subfolder, and in root it timesout. The same database, the same files and the same installation are in both cases. Actually i tried to remove all the cache folder and no images were loading, but again the problem was the same. So maybe the problem is in the visits, and we need to upgrade to a bigger VPS with more memory and stuff to handle this load.

Please Log in or Create an account to join the conversation.

  • Krikor Boghossian
  • Krikor Boghossian's Avatar
  • Offline
  • Platinum Member
More
8 years 6 months ago #148545 by Krikor Boghossian
Replied by Krikor Boghossian on topic Κ2 heavy load on large database, gateway timeouts
The favicon is requested automatically and you should include it in your site's root folder.

The cache folder contains the images/ media of the site. It is not static html files. Finally I cannot answer the last question (the subfolder issue), since I do not really know your setup. I would agree that a larger VPS might solve this issue.

JoomlaWorks Support Team
---
Please search the forum before posting a new topic :)

Please Log in or Create an account to join the conversation.

More
8 years 6 months ago - 8 years 6 months ago #148553 by Nick
The favicon doesn't change anything and if i place it in the root folder, it just takes a 200 status code instead of 404. But all those requests are not actual visits. Because in awstats and in google analytics those days we have approximately 300-400 visits. To be more specific yesterday in awstats we had 359 visits, and in google analytics 100 visits. Those differences are acceptable because they count the unique visitor in different time span. But in the access log i count over 70.000 unique IP's. Because i manage other websites also, and i am seeing similar statistics, in another website for yesterday we had 326 visits in awstats, and 84 visits in google analytics. In the access log we had only 15.000 unique IP's. The status codes with the favicon request are only a few, and the referrers are all articles from the originating website and only a few links from other ones. As we can see there is a huge difference here, as we had similar stats here in analytics and in awstats but in the access log there is almost 5 times the traffic. So that's why i suspect we need a bigger VPS with more RAM to handle this traffic. What do you want to know about the setup, i can send you a link to visit the website. In short the website occupies about 7.5Gb, and the SQL almost 2Gb. It has 10.000 articles in K2, 9 categories, 4000 tags, and 100 comments. As for the VPS it has 3 Cores, 1024Mb ram, 80GB HD and 350GB Bandwidth. In the subfolder it works, although and in here we have some timeouts, but very few. When i move it in root it simply cannot work, it floods with the requests and the whole VPS actually is unaccessible, not only the website.
Last edit: 8 years 6 months ago by Nick.

Please Log in or Create an account to join the conversation.

  • Krikor Boghossian
  • Krikor Boghossian's Avatar
  • Offline
  • Platinum Member
More
8 years 6 months ago #148581 by Krikor Boghossian
Replied by Krikor Boghossian on topic Κ2 heavy load on large database, gateway timeouts
The db can be cleared by purging the finder plugin's results.
Finally I would recommend hiring a sysadmin to perform a check on your server setup and perhaps optimise it.

JoomlaWorks Support Team
---
Please search the forum before posting a new topic :)

Please Log in or Create an account to join the conversation.

More
8 years 6 months ago #148589 by Nick
I have already purged the finder entries, and from 1.8Gb the database size dropped to 300Mb, but it broke the menus of the administration and some other stuff, so i had to restore them back. But again with those entries removed, nothing changed in the performance, the timeouts were the same, like nothing was changed, that's why i think the problem are the resources of the server and not the database. Actually i don't know to optimize the database, from my side, i tried everything in Joomla and K2 settings and other plugins and components, to make them as light as i could but that wasn't enough. I think a sys admin would be ideal, but here they are charging a lot. It is a lot cheaper to upgrade the VPS, and see if it can handle the load. If that also fails, then i could hire a sys admin.

Please Log in or Create an account to join the conversation.

  • Krikor Boghossian
  • Krikor Boghossian's Avatar
  • Offline
  • Platinum Member
More
8 years 6 months ago #148600 by Krikor Boghossian
Replied by Krikor Boghossian on topic Κ2 heavy load on large database, gateway timeouts
You need to purge them from the administrator, not from the db.
If you are on a cloud environment, updating your VPS would take 10 - 15 mins so I agree on this one, although an audit would be beneficial.

JoomlaWorks Support Team
---
Please search the forum before posting a new topic :)

Please Log in or Create an account to join the conversation.

More
8 years 6 months ago #148608 by Nick
From the administration i couldn't do it, it is a huge query and it times out everytime. I have to block all the traffic and and try it at late night, where there are a few visitors, so i could try to remove the entries afterwards. Unfortunately i am not on cloud, so the upgrade it isn't so easy, but i told my hosting company to make an upgrade to see how it goes. To move on cloud would be ideal, but they are charging a lot for those services, and i don't know if the single packets could cover us...

Please Log in or Create an account to join the conversation.

  • Krikor Boghossian
  • Krikor Boghossian's Avatar
  • Offline
  • Platinum Member
More
8 years 6 months ago #148610 by Krikor Boghossian
Replied by Krikor Boghossian on topic Κ2 heavy load on large database, gateway timeouts
FYI cloud VPSs in DigitalOcean, linode and similar services start from $5.

JoomlaWorks Support Team
---
Please search the forum before posting a new topic :)

Please Log in or Create an account to join the conversation.

More
8 years 6 months ago #148620 by Nick
Yes, i saw some ads about DigitalOcean, it looks pretty cheap. I will give them a try anyway. Thanks a lot...

Please Log in or Create an account to join the conversation.

  • Krikor Boghossian
  • Krikor Boghossian's Avatar
  • Offline
  • Platinum Member
More
8 years 6 months ago #148631 by Krikor Boghossian
Replied by Krikor Boghossian on topic Κ2 heavy load on large database, gateway timeouts
There are several 10$ coupons to get you started for free.

JoomlaWorks Support Team
---
Please search the forum before posting a new topic :)

Please Log in or Create an account to join the conversation.


Powered by Kunena Forum