Nginx comes to rescue !!


Measure Data Usage of a Client/User in a Scalable manner @ server 

Problem Statement was very clear, we wanted to measure the data usage of users to understand the data efficiency of our product and usage patterns.

This was inherently a hard problem to solve and if you look around everyone has built custom solutions to solve this problem, especially when you need to measure at an API/Session level. As the application information is embedded in packet and the measuring is happening at an upper layer.

Fortunately in our case the User and API information is at the URI level :).

We use Netty as our application server and we tried multiple ways to measure the data usage at an API level but couldn’t do so. We tried using the SimpleChannelHandler and built some counters which would be incremented based on the MessageEvent. But due to various reasons it didn’t work, thats for another post.

Anyway as we grew as a business the importance of this metric/feature increased drastically. I re-looked at all the options and then there was a Eureka moment. What about Nginx ? Use Nginx as a pass through proxy and measure the incoming and outgoing traffic.

Experiment : was very simple, setup a Nginx instance before the Netty server as a pass through proxy and setup appropriate log_format. The one of importance are body_bytes_sent, request_length, bytes_sent. Ran a client and wireshark, measured the API calls via Wireshark and Nginx Log and they were exactly the same. Problem solved.

Next Step: Build a service which parses the Nginx log and pushes the metrics to a data source from various nodes. Make this data source accessible to the reporting module so that we can run all kinds of Analytics and make the product better.

Stats: While doing this experiment, I measured that the there is an overhead of around 300bytes on each API call, which is due to headers. Which will lead me into the next problem of optimising our API usage. 

customising my dialler for smartphones

Last few days I had few messages which had phone number in them, none of them were in the format which the default dialler of the phone could understand and dial the number.

Then starts a tussle between remembering the number and typing into the dialler or using notepad where you convert the number into a proper country format and then copy the number and paste into the dialler.

Its incredible that its such a common problem and almost everyone faces it, still no one has solved it. The solution is pretty simple as well, use a library like libphone which converts a raw phone number into canonical phone number based on rules. Also the phone is aware of its MCC & MNC hence we can easily convert the raw phone numbers into canonical. Most of the apps when they ask phone number they do a similar task to normalize the phone numbers.

Once this task is done, then its easy to pass the appropriate number to the dialler which can immediately make the call without any fiddling.

Obviously there would be cases where the normal rules would not fit and you still would have to make changes to phone number but that should be rare.

The biggest challenge is to get something done like this on iOS which is a closed ecosphere, you cant change the dialler or do some smart things while you open the dialler app. On the other hand Android should be pretty straight forward.

Hopefully after reading this blog a smart engineer with some spare time, develops this utility 🙂