Keyword

K2 and it's image handling

  • Uzi
  • Uzi's Avatar Topic Author
  • Offline
  • Premium Member
More
1 year 10 months ago #170235 by Uzi
K2 and it's image handling was created by Uzi
Hello,

I have a very large installation with K2 and a lot of pictures. The cache folder in the media folder is around 100.000 images large. I had some trouble with downloading all these pictures via FTP because most servers have a limit on the amount of items you can list. So I did some digging and found out a way to tackle this problem.

I have a comment regarding the image handling. Isn't it an idea to split up the image saving into folders with for example the year and the month (media/k2/items/src/2018/11/image.jpg , and , media/k2/items/cache/2018/11/image_S.jpg etc)

Because this way the server won't get slowed (Linux has a hard time with folders that are extremely large, filled with individual files) and you can access the files a lot easier with most configurations of the servers out there. I think also that K2 gets a speed boost if you go this route. Maybe something to think about?

Also I think it's an idea to built in a external server for spreading of the load of the website. For example use the Amazon S3 servers for your images. With a large website, images that gets pulled from the same server tend do dramatically slow the website down. A native solution where you directly upload the images to an external server would be nice.

Also regarding SIGPro, I really would like to have an option to use a external server to directly upload the images to, and also let the plugin directly generate the thumbnails (this gets done every 120 minutes standard). I think this could also drastically make the plugin more valuable and useful for websites with lots of images.

Is this something that is on the list of to do? Maybe I could help in getting this to work? I really like the K2 component and the Sigpro gallery plugin, but these 2 things are one of the most significant problems I encounter.

Thanks for reading.

Please Log in or Create an account to join the conversation.

  • Fotis
  • Fotis's Avatar
  • Offline
  • Administrator
  • JoomlaWorks Support Team
More
1 year 10 months ago #170263 by Fotis
Replied by Fotis on topic K2 and it's image handling
First off, the file limitation you mention is not Linux related. It's FTP related. FTP can list 1999 files tops if I recall properly. But you shouldn't use FTP, instead use SFTP which has no limitation. Even better, don't use manual uploading, downloading. Use rsync. Much more efficient and faster.

If you have space issues, you can safely clean up your "/media/k2/items/src/" folder. This is probably larger than your cache folder there and (as the name implies) it keeps the source folders uploaded. If you don't wanna do that, you can always consider "mounting" (locally or via NFS, CIFS) an external drive in /media/k2/items/ so you can offload these files to another server or storage system.

All this was common knowledge when I designed K2 back in 2008 and I don't plan on changing that behaviour. We operate some pretty big sites around the globe with more than 100,000 items and storage ranging from a few dozen GBs (with careful image planning) to a few hundred GBs (large sites that have been migrated to K2 with over 10 years of history). Never had a single problem. Using rsync we could always migrate servers without hassle (with server to server direct transfers).

As for SIGPro, I've already mentioned that the plan is to use a universal "driver" for any cloud storage to be utilized for remote (gallery) file hosting. I'll do my best for v4 of SIGPro.

If you use & love K2, please take a moment to add your review and rate it
at the Joomla Extensions Directory: extensions.joomla.org/extension/k2/


IMPORTANT: Please search the forum before posting a question!

JoomlaWorks Support Team Member

Please Log in or Create an account to join the conversation.

  • Uzi
  • Uzi's Avatar Topic Author
  • Offline
  • Premium Member
More
1 year 10 months ago #170281 by Uzi
Replied by Uzi on topic K2 and it's image handling
I know the file limitation is FTP related, but large folder with loads of files do hinder the perfomance of Linux: serverfault.com/questions/147731/do-large-folder-sizes-slow-down-io-performance It's better to distribute those files in subfolders, as I mentioned in my original post.

Ok, looking forward to the v4 of Sigpro.

Please Log in or Create an account to join the conversation.

  • Fotis
  • Fotis's Avatar
  • Offline
  • Administrator
  • JoomlaWorks Support Team
More
1 year 10 months ago #170292 by Fotis
Replied by Fotis on topic K2 and it's image handling
Not really. K2 does not force the server to scan files. It knows which files to request directly. So there is no performance issue. By default, modern linux servers can keep more than 60,000 file path mappings in memory (which is a lot).

If you use & love K2, please take a moment to add your review and rate it
at the Joomla Extensions Directory: extensions.joomla.org/extension/k2/


IMPORTANT: Please search the forum before posting a question!

JoomlaWorks Support Team Member

Please Log in or Create an account to join the conversation.

More
1 year 10 months ago #170299 by william white
Replied by william white on topic K2 and it's image handling
As I read posts from out in the wild from other sites, this is one thing that always gets misunderstood about K2.
The way i see it K2 is a mousetrap that comes with the cheese! Keep up the good work, looking forward to the coming advances.

Please Log in or Create an account to join the conversation.

  • Fotis
  • Fotis's Avatar
  • Offline
  • Administrator
  • JoomlaWorks Support Team
More
1 year 10 months ago #170316 by Fotis
Replied by Fotis on topic K2 and it's image handling
Thanks William! :)

If you use & love K2, please take a moment to add your review and rate it
at the Joomla Extensions Directory: extensions.joomla.org/extension/k2/


IMPORTANT: Please search the forum before posting a question!

JoomlaWorks Support Team Member

Please Log in or Create an account to join the conversation.

More
1 year 9 months ago #170581 by Ad
Replied by Ad on topic K2 and it's image handling
Hello,

I have a news portal with K2. The cache folder is the only thing that worries me in K2. I publish ~5-6 items per day (~2.000 per year). My cache folder has ~15.000 files but in a few years folder will have ~100.000 files or more. I worry about performance and hosting provider's reaction.

In my opinion, separating files in many folders (for example '/2018/12/' like in Wordpress) would be a good idea. Is it possible to implement this solution in the future releases of K2?

At the end, thank you for really great component. I am waiting for K2 Pro :)

Please Log in or Create an account to join the conversation.

  • Fotis
  • Fotis's Avatar
  • Offline
  • Administrator
  • JoomlaWorks Support Team
More
1 year 9 months ago #170588 by Fotis
Replied by Fotis on topic K2 and it's image handling
From a technical perspective, Linux servers don't care if a folder has 1000 files or a million files. Being on a single folder means they can be transferred very fast to another host.

From a practical perspective, even in a few years, the size of the cache folder will be smaller than any regular Joomla folder where you'd allow users to upload images. That's because images in K2 are compressed and resized automatically, but users usually don't care when uploading images elsewhere. And you can safely delete the "src" folder there, which keeps the source images being uploaded into K2. We did that in case a redesign in the future requires larger images, so with a plugin, you could utilize the source folder to re-generate the cached ones. Our experience has also shown that people never really care to browse for these images. Most of the times, e.g. in large publications, there is an archive kept in /images which is used by content authors to re-publish images in K2.

Having run K2 in sites since 2008 (a year before it was publicly released), I can confirm that in around 10 years, K2 has proven more optimal in terms of disk occupied, compared to plain ol' Joomla and of course WordPress. The latter can prove a PITA especially if you change image dimensions after a redesign. In K2, it's a piece of cake. We even got a plugin for that: github.com/getk2/rebuildK2imageCache

With Joomla/K2, we've had sites with more than half a million items occupy less than 70-80GBs in total. And we've seen WordPress sites with ~20,000 posts occupy 250GBs!

If you use & love K2, please take a moment to add your review and rate it
at the Joomla Extensions Directory: extensions.joomla.org/extension/k2/


IMPORTANT: Please search the forum before posting a question!

JoomlaWorks Support Team Member

Please Log in or Create an account to join the conversation.

More
1 year 9 months ago #170593 by Ad
Replied by Ad on topic K2 and it's image handling
Thank you for your reply :)

Please Log in or Create an account to join the conversation.

  • Fotis
  • Fotis's Avatar
  • Offline
  • Administrator
  • JoomlaWorks Support Team
More
1 year 9 months ago #170603 by Fotis
Replied by Fotis on topic K2 and it's image handling
You're welcome.

If you use & love K2, please take a moment to add your review and rate it
at the Joomla Extensions Directory: extensions.joomla.org/extension/k2/


IMPORTANT: Please search the forum before posting a question!

JoomlaWorks Support Team Member

Please Log in or Create an account to join the conversation.


Powered by Kunena Forum