Comparing Cloud providers for new product development

Choosing the Cloud is more difficult now as vendors regularly drop prices and offer new features. I’m sure there is no clear winner and each will have each own strength and weakness. So, it is better to set the context before we compare.

Context

  • Team is small but all are seasoned developer.
  • Going to release multiple times per day.
  • At least two environments. Dev, Production.
  • At least need to develop one web application and one mobile app.
  • PaaS is always preferred than IaaS.
  • Only AWS, Azure or Google.
  • Main development platform will be either .Net Core or Golang.

Basic Architecture

image

This is very typical architecture for public facing web application. Of course, it could be split into many micro services - it would be multiple API services calling each other.I think this diagram is good enought for the very high level architecture.

The following table describes the components and related cloud features.

  AWS Azure Google Cloud Platform
Web API Elastic Beanstalk Azure WebSites AppEngine
Database Amazon RDS Azure SQL Datastore/Cloud SQL
Storage Amazon S3 Azure Storage Cloud Storage
Jobs Amazon Lambda Azure Functions AppEngine/Cloud Functions
CI third-party VsTeam Services third-party

Web API

  Elastic Beanstalk Azure WebSites AppEngine
Managed 3 4 5
Price* $70.08 $204 $25.55

Managed services mean server is maintained by provider. The higher number, the better. I gave Azure WebSites to 4 because it doesn’t have other features compare to AppEngine such as version splitting, no downtime deployment, no free centralize cache (memcache/redis), etc.

*price is estimated for two environments

  • AWS: 1 year no-upfront (1x t2.small, 1x t2.medium)
  • Azure: (1x B1, 1x S2)
  • AppEngine: (1x free quota, 1x n1-standard-1 flex-vm) assuming that we got more traffic than what free quota provides.

Database

Database Amazon RDS Azure SQL Datastore/Cloud SQL
Price $77.38 $79.98 $59.65
  • AWS: (1x db.t2.micro, 1x db.m3.medium)
  • Azure: (1x B, 1x S2) it is very difficult to convert DTU to server. So, I just guess it from this review.
  • Cloud SQL: (1x f1.micro - share CPU, 0.6 GB, 1x n1-standard-1 - 1CPU, 3.75 GB)

Storage

The base line is

  • 10 GB storage
  • PUT/LIST ops 1million
  • GET object ops 10million
  • 1TB Egress bandwidth (South-east-asia)
Storage Amazon S3 Azure Storage Cloud Storage
Price $130.21 $141.93 $122.88

Jobs

For our usecase, there won’t be many jobs running and we could re-use resource from Web such as WebJobs or AppEngine tasks. So, I’ll skip the comparison.

CI

We could use CircleCI (1 concurrent build is free) for AWS and Google while we use Visual Studio Team Services for Azure. So, I’ll also skip it.

Process

I will try to break down into a few software development processes so that we can compare each process for different cloud providers.

image

Yes. This is SDLC for our team. We are targeting to release multiple times per day. Small, fast and incremental releases.

The following table contains some components which hasn’t covered by the basic architecture components.

  AWS Azure Google Cloud Platform
Coding Go dotnet Go
Deployment CodeDeploy VsTeam Service Deployment Manager
Monitoring third-party Application Insights StackDrivers

Coding

Both Go and dotnet core are great. It is all up to your team. Even if your team know about dotnet, the new dotnet is completely new. You have to re-learn a lot. I warn you. :smile:

Deployment

Visual Studio Team services included everything from CI to deployment and it is well integrate with Azure. I think it has some edge over other providers.

Monitoring.

There are only two contenders, Azure and Google Cloud Platform. I didn’t have good experience with Application Insights - to be fair it is still in preview.

Container

How about Container?

Container has been very porpular in recent years and many company bet on that as future software packing. Even Microsoft partner with Docker to bring Docker to Windows (Nano Server). It won’t be too long for Microsoft own Container Service will land to Azure. Anyway, for now - there is only two horses (AWS and GCP) in this race.

I personally haven’t tried but both of them are just sit on top of VMs. If comparing VMs, Google Cloud Platform has some edge on that and their container engine is K8 is opensource - that will bring portability if you need to host in on-premise servers.

BizSpark

You may notice that Azure is a bit more expensive than others but they have nice BizSpark program. With that, you will get $150/month credit for 3 years.

That will help you to run without spending money for 3 years but after that you still need to pay.

Conclusion

If you have reach this far, I believe you notice that I’m a bit toward Google Cloud Platform because of their pricing model (Per-Minute Billing and Automatic Discounts) and their PaaS offering such as AppEngine, StackDriver and Container Engine.

But. You will never wrong for choosing one because each has their own strength. The real pros and cons is depends on YOU. (Your Team, Skillset, Business deal/startup accelerator, etc.).

The best way to choose is draw your architecture and do some research on that with all three providers.

Set up github pages with jekyll on Windows

I re-setup jekyll on my PC as I didn’t upgrade since I did it 3 years ago. It is much easier to setup jekyll on windows - thanks to bundler and github-pages gem.

Here is the steps:

  1. Install Ruby and DevKit. I install ruby 2.0.0 series because there are some issues in nokogiri gem with ruby 2.2 series
  2. Install bundler by running the command gem install bundler
  3. Create/add a file called Gemfile with this content

    source 'https://rubygems.org' gem 'github-pages' gem 'wdm', '>= 0.1.0' if Gem.win_platform?

  4. Install all the dependencies by simply running the command bundle install

:tada: :tada: :tada: :tada:

Run the jekyll

Use the command bundle exec jekyll serve in the root of repository and the site should be available at http://localhost:4000.

Github - two-factor authentication and cloning with https

If you are on windows, cloning with https may be a better option for you. You don’t have to set up ssh agent (which sounds alien for most windows user). You also don’t have to enter password every time you use, you can cache your password by following github’s article. In short, make sure you have msysgit 1.8.1 and above and set credential helper as follows:

git config --global credential.helper wincred

Two factors authentication

But after you enable 2FA, the above method doesn’t work anymore. We need a few more steps to make https url work with 2FA.

  1. Go to Github application settings github application settings
  2. Click Generate new token
  3. Note down your token.
  4. Go to Windows Credentials Manager Windows credentials manager
  5. Click Add a generic credential
  6. Enter your credential with the generated token. Note that address format is git:https://<username>@github.com Add credential

That’s all, you should now be able to push without entering any username and password.

How to set up squid as forward proxy in Azure

I was playing with squid to set up as forward proxy. If you are not sure about proxy, there a great answer at stackoverflow. In this post, I will use Azure as Cloud platform, but it should also work on Amazon as well.

Set up a Linux VM

We will first create a linux VM using Azure portal and later we use PuTTy to access.

From azure portal, I’ll go NEW | COMPUTE | VIRTUAL MACHINE | FROM GALLERY and choose Ubuntu.

gallery

I will just choose user name with password. If you prefer SSH key, you can also use that too.

vm configuration

I’ll create new CLOUD SERVICE as well - if you are wondering what cloud service is, it is just the container of one or more virtual machines. Please note that there is CLOUD SERVICE DNS NAME - we will use that name to connect the vm.

cloud service

Now, I’ll use PuTTy to connect the vm (We just need Putty.exe). The host name will be cloud-service-name.cloudapp.net - in my case, the host name is squidpxy.cloudapp.net.

putty

After connect, you should be able to log in with user name and password.

Install and configure squid

Before install anything, we will update the system itself first.

$ sudo apt-get update -y
$ sudo apt-get upgrade -y

We will install Squid and some utilities that we need later.

$ sudo apt-get install squid apache2-utils

We are going to use HTTP Digest authentication to authenticate users using a local password file. Let’s create the password file.

$ cd /etc/squid3/
$ sudo touch passwd
$ sudo chown proxy:proxy passwd
$ sudo chmod 640 passwd

We don’t need to change the owner and permission to make squid works, but it’s good security practice. If you check the file,

$ ls -l passwd

you should see this - of course, the date and time will be different.

-rw-r----- 1 proxy proxy 0 May 29 09:02 passwd

Now, we are going to add user to password file by using htdigest from apache2-utils.

$ sudo htdigest /etc/squid3/passwd krt jittuu
Adding user jittuu in realm krt
New password:
Re-type new password:

We can test the new user with squid digest auth as below. (of course, md5 hash will be different. If the same, we are using the same password. :smiling_imp:)

$ sudo /usr/lib/squid3/digest_file_auth -c /etc/squid3/passwd
"jittuu":"krt"
01b21a5c47050b4e56d6c1c5540acd8f

It is the time to configure squid. The default configuration file is in /etc/squid3/squid.conf with thousands of line - because it is heavily documented configuration file. I think it is better to create new file than editing the default config.

$ sudo mv /etc/squid3/squid.conf /etc/squid3/squid.conf.origin
$ sudo touch /etc/squid3/squid.conf

Luckily, we are just setting forward proxy without any caching. For now, these are all we need:

auth_param digest program /usr/lib/squid3/digest_file_auth -c /etc/squid3/passwd
auth_param digest realm krt
auth_param digest children 5

acl auth_users proxy_auth REQUIRED

http_access allow auth_users
http_access deny all

http_port 3128

I will just use squid default port: 3128, but I strongly recommend to change other random port. After we configure, we need restart the squid with this command.

$ sudo service squid3 restart

At azure, we still need to open the endpoint of the vm. Go to VIRTUAL MACHINES | <vm> | ENDPOINTS

endpoint

For now, we will just use standalone endpoint and specify the endpoint details with the port squid use.

standalone

open port

OK. That’s all to install and configure squid.

Accessing via squid

I’ll use Firefox browser to use with proxy. Go to Options | Advanced | Network | Settings

firefox network setting

I’ll use Manual proxy configuration

firefox manual proxy

When the browser prompt the dialog box, enter user name and password. You should be able to browse via proxy now. You could test your IP at http://whatismyipaddress.com/. It should be different if you test your IP with different browser.

But if you visit http://www.whatismyip.com/, you will see like:

no privacy

It is because the server still can detect that you are browser via proxy.

Privacy

To protect the privacy, we can strip proxy header by adding the following settings to squid.conf.

forwarded_for delete
via off

We restart squid to reload the config.

$ sudo service squid3 restart

Now, if you visit http://www.whatismyip.com/, the server should not able to detect you are behind the proxy.

privacy

I hope this post will help someone who want to set up squid as forward proxy in azure.

Using Azure Explorer on Windows Server 2012

TL;DR

  1. add http://installers.cerebrata.com into the Trusted sites
  2. add https://cerebratainstallers.blob.core.windows.net into the Trusted sites
  3. go to download and install

More

I use Azure Explorer from cerebrata to work with azure storage. Today, I need to use it at my azure VM with Windows Server 2012.

I opened IE and go to download Azure Explorer - then I got this error.

error 1

As error suggest, I added http://installers.cerebrata.com into the Trusted sites. And I tried to download again. This time I got this error;

error 2

I check Details.. and found these two interesting log:

Application url			: https://cerebratainstallers.blob.core.windows.net/installers/Azure%20Explorer/production/1.0.0.529/Cerebrata.AzureExplorer.UI.exe.manifest
Server		: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

Deployment and application do not have matching security zones.

This time I add https://cerebratainstallers.blob.core.windows.net into the Trusted sites and download again. Now it shows me Security Warning and I just click install since I know what I’m installing. :)

security warning

When the installation finish, …. it works!

working