The problem: (AKA Challenge as Barney Stinson would call it)

As the Chinese market is really enthusiastic about unitedstyles we are currently planning to roll out a version for this market. It’s not the first localisation we have done, as you might have heard we’ve launched a Russian, Dutch and Japanese version already. However, the China localisation is something else. As many of you know, the Internet in china is heavily monitored and some sites aren’t’ approachable at all, this concept of heavy monitoring is often jokingly referred to as ‘the great firewall of China’ or GFW. Because of this GFW, websites outside China are slower to load and at one point recently, they didn’t load at all.

The news to launch in China is very exciting for us as a company, but for the IT department, it raised questions immediately. By now we have applied our ICP license (license to publish a site) and attained localized hosting. Thus we can serve our static and dynamic content from within China using CDN. However, for the database we are running into some tough decisions.

In a quick setup we immediately saw that the database speed for USA site would be too slow. (Our out-of-china servers are hosted on Amazon.com cloud environment)

We’ve created three hypothesis before we started:

1. Would it make sense to host database in Japan? (or Singapore)
2. Would speed increase if we use a different port then the standard one?
3. Would it be feasable to tunnel the SQL in SSL, SSH or VPN connection?

I’ll discuss them one by one:

1. Would it make sense to host database in Japan? (or Singapore)

I’ve heard that another International clothing brand setup shop by hosting the site just outside China, got the recommendation to try Japan just like they did. Unfortunately, results (with Amazon) are laughable. Again, this is a query of exactly 5000 rows run inside China, to a mysql server outside China.

Due to some international events, all Internet in Shanghai has been slow for the last two days. A good time to test. But as you can see, would this case happen again, our site would go to a stop. Unfortunately, hosting in Japan didn’t seem like a stable way to continue.

2. Would speed increase if we use a different port then the standard one?

The idea behind this hypothesis came from some experiences with the gfw. For example, we noticed that changing the useragent of a browser will decrease your browsing speed. So perhaps it would be as simple as changing a server port to achieve faster speeds. We had to put this to the test. Our policy doesn’t allow us to test on the live environment, so we setup a simular one in the same datacenter. Unfortunately, changing the port created identical results so we abandoned this hypothesis.

3. Would it be feasable to tunnel the SQL in SSL, SSH or VPN connection?

MySQL data connections are quite sophisticated. Perhaps it’s this sophistication that is slowing down the query speed. What if we encrypt or socket-proxy the connection, or even use a VPN to connect out of China? We’ve tested all these methods and found that that only the VPN solution would speed up the database query speed two fold. This is subject to many variables but we can see this is a potential way. Unfortunately, it won’t be suitable of production mode as it’s too risky for high volume traffic.

Verdict: No alternative -> Seems we have to host in China
It seems to us, after a few tests, that one really need to host the database within China to guarantee a good user experience. This leads us to a difficult decision as we are reluctant to simply duplicate what we abroad within China. This ofcourse, would be a last resort, short term: customers would potentially have to register on both sites, internal confusion about which website orders are placed, potential duplicate keys etc etc. And on the long run: we would be never able to merge both databases back to one… A nightmare.

One idea that was raised was to seperate some tables and host parts inside, and parts outside China. However, this will not get rid of the slow connection. We really need a server within China.

Conceptual solution: Multi-Master Circular Replication

Ok, I’ve titled this conceptual solution because we haven’t gotten into the nuts and bolts here, this idea has been in the back of my mind and now it seems like a potential solution. As many of you know, MySQL replication offers us tools to have a master-slave setup. This setup is known and often used to speed up query manipulations or to create live backups. These are sometimes referred to as star-shaped setups.

Traditional master->slave replication, netexpertise.eu

As you can see in the illustration. the master receives writes and the slave databases execute the binary query log of the master. When activity happens (like creating a user account), the necessary data is applied on the master and it will copy to the IO thread in it’s log and tells all slaves that things have changed. The slave databases will one by one execute the queries to stay in sync with the master. Ok, now we got this concept out of the way, let’s continue:

Back to the China challenge: Something we are looking at now, is to use a setup with two masters: one in USA and one in China. This would be referred as a multi master setup. We setup these databases to keep eye out for each other, whilst acting as master, we can actually achieve localized servers in both continents.

The first question that comes to mind is: How to avoid that both masters happen to write the same ID at the same time? I hear you ask: What if data is duplicate? For example, two users that register at the exact same time, the table ID isn’t unique? Duplicate ID’s are avoided with auto_increment_increment, a feature to skip ID’s. So one could setup one server to be even ID numbers, the other uneven.

CC onlamp.com, 'advanced mysql replication' (2006)

My question to you? Do you have simular experiences and is this wise to do in our case? There are several up and downsides that we considered but of-course user experience is at upmost importance! Anyway, time for weekend now, would love to hear your opinions.

First about the trip: We had our entire provision, including instant noodles, bread and chocolate in our backpacks and left room for one celebration meal at return.

The plan was as follows: leave in the early morning by bus in Shanghai and aim to be at the our sleeping place (close to the summit) before the dark. The next day we would get up early to hike to the (200m?) to the top and see the sunrise, and then hike one of the designated paths back to civilization.

The trip was quite spontaneous so we didn’t plan too far ahead. We made it a sport to keep costs at a minimum, while still enjoying the outdoors. We travelled the 500km by bus and decided to sleep in a tent instead of a hotel. Smell the nature…

Going there, In the back of the bus two Huang Shan residents quickly turned out to be quite friendly. One turned out to be an excellent tour guide for his home town. He spent more then an hour filling us in on every detail about our upcoming journey.

Friendly people explaining us where to go next

Due to traffic, we arrived barely on time and had to keep pace up, we arrived at the hotel just before sunset. The tent was already setup for us, so there was no problem. We had a meal and looked around before we slept.

At the camping site. Simple dinner, followed by beer and potato chips

The next morning we got up at 4am. Frankly, we were amiming for 4:30 but our neighbors woke up with a lot of noise, but it was perfect actually, we climbed (with flashlights) to the top of the mountain and could settle a nice place as we were one of the first spectators.

Sunrise

After sunrise, we looked around for a bit. It’s tranquility was amazing. Still a few hazy clouds, just hanging on the base of the mountains, creating a bit of a mystical landscape.

After sunrise, 5:40 am in the morning.

Incredible peaks, beautiful sights

The highlight for me was the Lotus peak. Although it got a bit more cloudy and it was a very steep stair climb, we were lucky with the weather and that it wasn’t crowded with people. The view was magnificent. We really are thinking about buying a more serious camera soon.

On top of lotus peak, 1,864m

During the way down, the sun came out from time to time. Steep, steep stairs down. Careful now!

Afterwards we enjoyed a meal and a footmassage down in the village, before taking the bus back to Shanghai.

If anyone reading this is planning to go, it seemed that our decision to stay the night and see the sunrise saved the day, would highly recommend it. We didn’t have to wait anywhere we came. When we came down the summit it quickly became busy and when we walked by the queue, waiting times were reaching 2.5 hours to go up by cable cart.

Unitedstyles is still a very lean organization (14 full-time employees), but we now decided that we need to hire an additional senior IT person in our Shanghai office. We put the profile on a few career sites today, so of course I also put it on my blog. Please forward to anybody that would be a fit for this position!

IT operations and development manager

About us

Unitedstyles has developed a fashion customization platform on unitedstyles.com and also in an iframe on other websites. Users can design, share and buy their own creations, soon they should be able to sell using our affiliate system. Unitedstyles is based in Shanghai and was a finalist at Techcrunch Beijing 2011.

Job description

You should have experience managing servers with over 50,000 visitors both in and outside china. Proficiency in CSS and Javascript is a plus. You should appreciate the details that make a front-end user experience memorable, and demonstrate enthusiasm for efficient server implementation. Besides that, enthusiasm to use technology to improve our backend processes.

Fundamentals

• Experienced with Linux/Unix/BSD in CDN. Database management in MySQL.

• Architecting MySQL databases a plus.

• Stability of website and scaling of website.

•  Security of webshops is a plus.

•  PHP and javascript expert.

• Ability to work proactively.

•  A desire to improve yourself as an engineer.

• Should be able to speak both Chinese and English

Salary

Depending on experience, up to RMB 15,000/month (gross salary)

Education

BA/BS degree in Math or Computer Science a plus.

Contact

Email: joop@unitedstyles.com