All our base are belong to Google (with a few gotchas)

Original posted date: 12 October 2015


The Esplorio team has moved into the same town resulting in me sharing a flat with Essa, and we took our servers along with us (I kid, I kid)

A while back we managed to get into the Google Launch programme, which includes a $100k voucher of Google Compute Engine (GCE) credits. The idea is that Google will assist these new exciting startups to scale with many different resources they have at their command, and beefy servers are just one of their specialties. There are a few technical gotchas I will mention at the end so if you want to skip the BS, go all the way down to The gotchas

The move

These credits sat around for quite some time because our whole team (3+1) were dead focused on getting the iOS app out until about 3 weeks ago when we asked our friend George Hickman to join Esplorio once more to help us with this huge switch involving a lot of different moving parts:

  • API servers serving the webapp and iOS app
  • Web frontend server
  • A single-node 8GB database box we need to convert to a proper distributed cluster as it was designed to run (Couchbase minimum requirements state 16GB of RAM for each node in the cluster)
  • A myriad of other servers to process geodata, images and queued tasks

By the end of the move, with George's tremendous help, we rewrote all of our deployment scripts using the awesome Apache Libcloud. Spinning up a whole database cluster only takes one single deploy.db_cluster:node_count=100 line in the terminal. After all the scripts were rewritten, it took me another couple of days to complete the switch, tighten our firewalls, and scrub all the old servers. As careful as I was, some parts of the system still went down for about half an hour because of a DNS change.

We now even have a staging database cluster, which makes us feel a bit more like a proper software company, and plenty of firepower to prep for growth :fingers crossed:

It was a great experience. After months and months of writing way too much Javascript and Swift, I've eventually got my hands around some much needed DevOps stuff. It also serves as a reminder for myself: Esplorio is totally not a simple system to run!

The gotchas

However, we had several problems that we encountered during our move:

1. Unusual traffic from China

When we first set up some test GCE boxes, we noticed some suspicious traffic hitting our Django servers. Since Django has the ALLOWED_HOSTS check, fortunately it filters out various invalid hosts from hitting most of our endpoints, and on top of that it sends us alerts of these repeated spoofing attempts to hit our servers like this:

ERROR: Invalid HTTP_HOST header: ''.You may need to add u'' to ALLOWED_HOSTS

No stack trace available

Request repr() unavailable.

(The HOST header may vary:,,,

After some investigation, it turns out all of these requests came from a program called GoAgent, which snoops around Google App Engine (GAE) servers and use them as a free resource to create a proxy service. As you would have guessed, it is apparently used by many Chinese to bypass the Great Firewall. Our Compute Engine boxes must have fallen in the same IP range that GAE boxes use, and we've had thousands of these requests coming our way.

We decided to filter out these requests before it reaches our Django instances, returning a HTTP 444 (bad request error, without any response) right when they hit our HTTP server.

2. Funky network setup by Google

To bring our database over to the new infrastructure without any downtime, we used a technique in Couchbase called XDCR (Cross-DataCenter Replication). The process is to first build the new cluster, and then set up an automatic unidirectional copy of the data from the old cluster into the new one where every single document in the old cluster will be sent over as part of the copy (each copy request is thus called an XDCR op). Once all the data is in place, one can simply flip the switch for the application to use the new cluster, and all the precious data will be there in the new cluster, ready to use. When all of the left-over XDCR ops finish, we can make a backup of the old server and then archive it.

In order for this to happen successfully, all nodes within the 2 clusters need to be able to talk to each other. We first set the new cluster up so that they can all talk to each other using Google's internal IP addresses, leaving only one box exposed to the old cluster, because we thought if we point the XDCR target to this "leader" box, it'd be enough. XDCR failed, of course, because Couchbase clusters treat each node equally and so all of these individual nodes need to be able to talk to each other. I did some further digging into the GCE network structure, and found that Google have done some funky setup where the IP address of eth0 is the internal one, and the external address is apparently generated and configured elsewhere. The idea is that all the nodes are connected to the Internet not directly but via a different layer, and as a result external IP addresses can be changed either at creation time or even on the fly. It's quite cool.

I predict our database cluster would perform even better if we use an all-internal setup, however it is a task for another day.

3. Quotas

The last gotcha we hit during the migration was the quotas. I assume this is enforced by Google to prevent abuse of their system. Basically, our whole setup required a total of a few dozen CPUs and a number of terabytes of SSD to run so we had to ask for quota raises twice. This was, thankfully, not much of a big trouble since we are a totally legit startup (yay!) and Google's support was very quick and receptive about it.


I normally consider myself a (somewhat) full-stack developer, but much of the fun I've had still comes from back-end and DevOps. Building these new servers was a bit like assembling many parts of a big puzzle, and the end result was very satisfying. Now on to a tonne of other stuff waiting for me to complete while I procrastinate by writing this blog entry...

San Francisco stories #5

Original posted date: 10 October 2015

Not your kind of people

Supermoon observers

POV of an outsider

On one hand, I got to meet exceptional individuals on my trip to San Francisco and the Bay Area. These people are the ones in the driving who push the limits of technology, constantly on the forefront of innovation, trying to have a shot at the impossible, and keeping the wealth flowing in. No exaggeration: they left me in awe of their intelligence and talents. Oh yes, there are way way more talents to tap into outside of Silicon Valley - but these superhumans are a totally different breed, seriously...

On the other hand, there are homeless folks roaming almost every street we walked/drove by, and many of them are mentally ill. One night, when I was waiting for the BART in SF centre to get to our place in Oakland, an old man came around, kept saying a lot of gibberish to everybody nearby - amongst which I made out the part where he said he was a vet in 'Nam (funnily enough...). He then proceeded to sing a song that I could not really understand either, danced along with it in a deranged way, spoke some more gibberish and walked away. I had a glance into his eyes and found them... well, soulless. It would have made a great photograph, but it was just all really sad so I hesitated and decided not to take the shot. Ever since, I have been reading more and more about the homeless and mental health problems of San Francisco - fascinating stuff.

All of it was just like a dream - SF is now 8 timezones away. A jet lag is like leaving your heart and soul somewhere else. It is time to readjust.

From High Wycombe, England

San Francisco stories #4

Original posted date: 22 September 2015


What to do in San Francisco:

After TechCrunch Disrupt Day 2, go to a random drink where you talk about travel startups, hotel pricing algorithm, your dream Arduino builds, chatting up with girls about your tech stack.

I can't imagine doing the last bit in many other places I've been to without getting weird look, and people slowly walking away from this nutty Asian guy.

It's time to just do it

Original posted date: 15 September 2015

Next destination: San Francisco

3 months ago, the reaction would be "Yeah, let's go to Disrupt SF!", now it has changed to "Holy shit, it's next week..."

We've now got a solid platform, with an upcoming iOS app that we have always wanted to build reaching launch date, and (positive - woot!) feedback flooding in from beta users and early adopters. In the past 4 months, I have written a lot of thousands of lines of code in at least 5 different programming languages, and currently on an over month-long GitHub streak with 70 pull requests within the last 7 days alone. Recruiters who spams me with LinkedIn messages about "new exciting challenges", please take note: if something can pique my interest, it needs to be that Esplorio-level challenging and addicting.


There is still a crazy amount of work to do. The more we get finished, the more new stuff there is. However, it is safe to say that I am having the time of my life right now, and the best is yet to come.

Let's hope all the long hours and all those "Okay, I'll stay back to finish this, maybe one fewer day/night out/game..." moments will eventually pay off. See you again soon, San Francisco.

If you are going to attend TechCrunch Disrupt SF 2015, you can find us showing off all the awesome things we've built in the Startup Alley on the 21st Sep. Otherwise, I'll stay in town with my team until the 30th if you fancy a catchup

PS: Fun fact. While I'm writing these lines, Esplorio servers are under attack by Chinese hackers

CNN Conference 2015 - Hội thảo du học Chuyên Ngoại Ngữ 2015

Original posted date: 31 July 2015

Vietnamese post - CNN Conference có gì hot?

Series các câu hỏi nổi bật nhận được trong những ngày qua về CNN Conference

"Chưa chán à?"


"Làm gì ở đấy?"

Đến xem chơi.

"CNN Conference là gì?"

Hội thảo du học hàng năm của trường Chuyên Ngoại Ngữ

"Hội thảo của Chuyên Ngữ có gì hay hơn các hội thảo khác?"

Các hội thảo khác, dù nhiều tiếp tế súng ống đạn dược hơn, nhưng thông thường chỉ có thể tập trung sâu vào một mảng du học lớn. CNN Conference được chạy bởi CNNers, và thiết kế cho CNNers. Mà đã là dân Chuyên Ngữ thì có đặc điểm là đi đến nước nào cũng có.

Có về lần này mới biết trend của các em bây giờ là không chỉ apply nhiều trường trong 1 nước như kiểu ngày xưa mình làm, mà còn trải applications ra các nước khác nhau. Và hiếm có cái hội thảo nào bao được hơn 10 quốc gia trong 1 ngày.

"Học sinh trường ngoài có đến được không?"

Có chứ, đến chơi nhé?

"Phụ huynh thì sao?"

Đương nhiên là ok rồi.

Muốn đi phải làm gì?

Đăng ký ở đây rồi 7h45 sáng có mặt

RSVP ở trên facebook cho vui

"Tóm tắt lại trong ngày có gì?"

Sáng có panel discussion hay ho của đại diện 1 số quốc gia, chiều đến có đủ các booth của 10 nước, tha hồ nói chuyện với các bạn/anh chị du hí và du học sinh cool cool.

Sau đây là poster hoành tráng nhưng Facebook resize lại còn 1 mẩu bé như con kiến, xin được trả lại công lý cho nó:


... and here goes bonus round 1:


... and bonus round 2:


#muchhappi #socri

Spread your wing and fly.