Building The Joomla.com Hosting Platform

joomla_blog_pic

When SiteGround was chosen by OSM to be the official hosting provider for Joomla.com – the new free website service by Joomla, we were really excited. We knew that creating an infrastructure that needs to host hundreds of thousands of sites will not be an easy task. This was a huge and complicated project, but our passion for complex technical challenges was what made us participate in the RFP in the first place. Of course, we were additionally motivated by our love for Joomla and by our previous long-year experience of hosting thousands of Joomla sites on our own servers. Many people from SiteGround and from the Joomla! community have done some amazing job on the different aspects of this new service like creating the website, designing the templates and developing the extensions. However, in this blog post I will concentrate especially on the technical challenges that our DevOps team encountered while designing, developing and implementing the hosting platform for Joomla.com.

The Goals

If you check the official Joomla.com site’s footer you’ll see that users get access to modified version of Joomla! CMS hosted on a special hosting infrastructure built by SiteGround. We had to develop this special hosting infrastructure because this project has some unique requirements:

  1. It must be extremely efficient, so that hosting millions of websites for free is sustainable.
  2. The websites should be blazing fast and highly secure, so that the user never associate Joomla CMS and SiteGround hosting with negative experience.
  3. Starting your new Joomla website should be as easy and fast as possible, so that people do not get discouraged on the first step.
  4. The infrastructure should be highly scalable, so that it grows seamlessly with the number of its users.

Below I will go over the technical decisions we made to achieve these goals.

One Joomla! to Rule Them All (efficiency and safety)

The main efficiency challenge that our team had to overcome was that hosting a big number of Joomla websites, even if they are just inactive default installs, requires quite a lot of space. Just to give you some numbers, the latest Joomla! Installation archive (3.3.6) contains 6952 folders and files. On a Linux server every file or folder uses an inode. The size in MBs of these files is not big, but the number of inodes on a server is limited. Even people that are not big mathematics fans will understand that installing a new Joomla! app for every new client will quickly cause the drives on the servers to become full.

We needed to think of an original solution how to minimize the number of the inodes needed and what we came out with was that only one Joomla! CMS application will be shared by all the clients. The idea was to create a Joomla! CMS skeleton that contains the core Joomla! files and then link all sites to this one skeleton. We identified the folders and files that should be part of the skeleton and we created it. Of course, we also considered the folders, such as the images directory, that should be unique for every site. Once the skeleton was ready, creating new accounts was easy because we just had to create symbolic links to it.

On top of the resource usage efficiency that resulted from this decision, we achieved two additional things that highly increased the security of the system. (1) Using a single Joomla made the upgrade process really easy. Once a new Joomla! version is released we just upgrade the skeleton’s codebase and all clients are instantly upgraded to the new release. (2) Using a single Joomla meant that the users themselves will have limited access to mess up with the system files. The upload of external extensions and templates is not allowed as a result and this makes the security of the system much easier to maintain.

Custom Caching (efficiency and speed)

Another one of the key technical solutions that allowed us to achieve even more efficiency from the system and to ensure that the users’ websites are really fast is the custom caching that we developed specifically for this platform. We used the famous Nginx web server and PHP-FPM to cache information and speed up the delivery of static resources and dynamically generated data. We installed Nginx and configured the PHP-FPM and then we found out that we’ll have to patch them in order to use them. The first problem that we faced was that we did not have the ability to purge the cache only for a specific domain name. We downloaded the ngx_cache_purge module, extended it and tested it on our end. The issue with PHP-FPM was that it lacks support for chroot and this was essential for us due to security reasons. Again, it took us some time to do it but our DevOps engineers patched and extended the code and we compiled our own PHP-FPM.

Dynamic DNS (immediate provisioning)

Now that we have created fast, secure and efficient environment, we had to think of a way to allow clients to use their new Joomla! sites right after they created their accounts. When one creates a new Joomla.com account he/she chooses a unique sub-domain name. We all know that the DNS sometimes needs time to update all DNS records in all databases used all over the world. Many internet service providers cache DNS data and in general it is not cool to wait to use your newly created free Joomla! site. To solve this issue we decided to use something called Dynamic DNS. This technology allows us to quickly map domain names to IP addresses and maintain certain accessibility levels using well-known addresses. The result was immediate account activation and access to your website without the need to wait for any propagation time.

Linux Containers (infinite scalability)

As I mentioned, we had to find a solution that should be highly scalable. At the same time we needed something that is easy to configure and automate the whole process. We decided to use Linux containers mainly because they offer almost unlimited scalability options. For example, we can configure a container to automatically scale-up if it uses more than 80% of the available RAM. The same is valid for the storage capacity and the available CPU power. On every container we also use two MySQL instances to ensure that the servers will be able to handle more concurrent connections.

Now that Joomla.com is already functioning we have started welcoming a big number of Joomla! sites on the infrastructure described above. I am pleased to say that so far we haven’t faced any major issues and the platform is handling flawlessly the traffic towards the existing sites, as well as the launch of the new ones. We are looking forward to the bright future ahead of the project!

Enterprise Cloud Solutions Architect

My challenging job is closely related to all kinds of Free and Open-Source Software products (some of my favorites are WordPress, Joomla!, Magento, Varnish and Apache mod_security). As a Web security and performance freak I am always hyper focused on solving all kinds of issues and improving our services.

16 Comments

  1. Reply February 10, 2015 / 06:30 AlessandroSiteGround Team

    Hello,

    do you think to use the custom configuration, Nginx + php-fpm also the plans of Shared Hosting in the future? Or, do you will continue to use Apache + Varnish Cache?

    What are the comparison from these two configurations, from your test?

    Thank you,
    Best regards

    • Reply February 11, 2015 / 07:56 HristoSiteGround Team

      NGINX is a newer technology and has its benefits. We're considering migrating to it but at this point I cannot confim that or tell you when it will become a fact.

  2. Reply February 10, 2015 / 21:07 Neil RobertsonSiteGround Team

    I have tried out creating a new website on joomla.com and the process is indeed very quick and simple.

    joomla.com is good competition for wordpress.com and gives Joomla a boost in the WordPress vs Joomla discussion.

    Great work SiteGround!

    • Reply February 11, 2015 / 08:01 HristoSiteGround Team

      Thanks for the kind words!

  3. Reply February 11, 2015 / 05:31 Alessandro MuraroSiteGround Team

    That must have been quite an effort, it would be great if you could share more about this process!

  4. Reply February 11, 2015 / 05:58 Alessandro MuraroSiteGround Team

    Also, are you using nginx?

  5. Reply February 11, 2015 / 05:58 Alessandro MuraroSiteGround Team

    Oh yes you do 😀

  6. Reply February 11, 2015 / 06:00 Eddie MaySiteGround Team

    Just to echo Neil's comments, I also created a new website on joomla.com and was similarly pleased with the speed and simplicity of the process.

    So, well done to all concerned & best wishes for the future.

  7. Reply February 11, 2015 / 07:31 Jackson SiroSiteGround Team

    Am a webdesigner from Nairobi, Kenya and still doing a diploma in Computer Studies. I used to fear Joomla 3 years ago and never dared play with it till I got a little internship of doing webdesign. I had to learn a lot within a short time and I must say Joomla is now my number one cms when I get new client who needs a new website. Joomla has all the extensibility we need that are lacking on other cms like wordpress and drupal. For instance you can have one website with different components installed on it so that it serves a lot of things at once.

  8. Reply February 11, 2015 / 09:26 HristoSiteGround Team

    Glad that you're enjoying this!

  9. Reply February 15, 2015 / 20:39 Iain MaceSiteGround Team

    Daniel - I have an existing account with SG - is it possible to integrate this service with existing accounts. The simplicity of joomla.com could provide a nice solution for some smaller sites. When looking at the upgrade options it just quotes the standard upgrade accounts (gogeek etc) but there is no info about how it works once the account is upgraded? I have asked a few times but the support crew haven't been briefed on the finer details (appreciate it is a new product). Is there anyway to find out how it would work for multiple accounts?

    • Reply February 16, 2015 / 02:42 Daniel KanchevSiteGround Team

      Hi Iain,

      Once an account is upgraded it is moved from the joomla.com infrastructure to our shared hosting servers. This means that shared hosting accounts are fully featured accounts and the sites hosted on the joomla.com infrastructure are limited in terms of available Joomla! features.

  10. Reply February 24, 2015 / 10:31 SorenSiteGround Team

    Extension developers should be able to apply to be included for installation in Joomla.com. I wouldn't mind if Siteground earned money reviewing the applications and there was an annual fee for inclusion, but Joomla is so much better with extensions.

    • Reply February 25, 2015 / 07:41 HristoSiteGround Team

      Hey Soren,

      That is an interesting idea and thank you for suggesting it! However, as a free service, Joomla.com is appropriate for people with standard website needs. There are several pre-installed extensions that are designed to meet those needs. People that outgrow them can easily export or upgrade their websites to a self-hosted environment.

      I agree that Joomla is better with extensions, but for such solution it would be extremely difficult to maintain a big database of available extensions.

  11. Reply February 24, 2015 / 16:36 SaulSiteGround Team

    Can't wait to try the new site but I had a quick question:

    The fine print indicated that site ground is allowed to use a "limited" version of joomla by open source matters.

    Can you tell us what exactly is limited? I'd love to recommend this to my clients that just want a simple site but I need to know what they will be able to do and what they won't.

    Thanks and thank you for providing a wonderful free service.

    I might switch to you completely and migrate my websites over but I still haven't finished evaluating you completely. I am currently still hosting my sites at TMDHosting and only have one site on my siteground hosting account.

    • Reply February 25, 2015 / 07:43 HristoSiteGround Team

      Joomla.com actually provides you with the full functioanlity of the Joomla CMS software. However, we've limited the installation of 3rd party templates and extensions. For more information on that matter, please check our tutorial on the differences between Joomla.com and having a hosted solution: http://tutorials.joomla.com/basics/16-joomla-com-vs-joomla-org

Reply

* (Required)