Watson, What's next? - Ultimate knowledge economy.

IBM has built a new super computer program called Watson and it is capable of playing Jeopardy quiz game against humans and winning them hands down. This amazing super computer is a result of deepQA project which "understands" human natural language and answers the question from its knowledge built again by processing millions of articles written in natural language. So, this is how the Watson plays the game. Just like other human contestants, it listens to the "answer" on select topic, digitizes the voice, process and understand the hint. Searches its knowledge base, comes up with correct "question" before others and presses the button. Then when asked, Watson would 'speak out' the question in natural language. If the "question" was correct contestants get points else they get to loose points.

In other words, computer program can read and understand any printed book, news paper archive and correlate events hence build its 'knowledge'. Here is an interesting video by lead scientist David Ferrucci of deepQA project. What interested me at first is that Watson is improving its answering accuracy since three years and now it can beat any grandmasters of Jeopardy game. Another piece of detail is Watson is built by networking what seems to be server-grade computers. It runs 3000 Cores simultaneously. It is NOT a keyword matching search engine.

Ok, now let me speculate little bit here what might happen next:

1) Immediate: Dawn of cog-engine
IBM might roll out its next version of "search engine" whose results are far better than Google's. They might probably call it "Cog Engine" than search engine. Who knows, Google might already be working on something similar already. It is not IBM or google, point here is "search" as we know it today might change dramatically.


2)Immediate to near future: Worthless "Research"
"Research" would loose its meaning. Anyone who has access to Watson type programs could produce the results in a day or two what today's researcher would produce with years of effort. So, it is imperative that either human researcher find something really complex to research on OR they research on improving watson algo itself!


3) Immediate to near future: Hand-held Watson
You would be able to conduct the research/learning/QA from your mobile devices. Mobile phones would be so powerful that they could easily do the "knowledge retrival" part. "Knowledge gathering" part may be done by relatively powerful servers on the cloud.

4) Far future: Download F-16 flying skill
Some one will eventually figure out how human brain analyses the natural language and makes decision vis-a-vis Watson. Watson is made to mimic the brain in the first place any way. Once that is done, the "knowledge" can be freely moved across machines and humans. Think about it, computers can learn a subject and move the "knowledge" to humans' brains and vice versa.

5) Digitize the skill:
Logical next step would be to digitize the "skill set". Till today, humans acquire knowledge individually. That means, One has to learn a subject by reading books, practising the art etc basically by putting in time and effort. He or she eventually becomes expert in the subject. Then He/She dies. Only way to pass on the expertise or knowledge to other people is by writing a book or articles or by giving speech and so on. So, there is no "continuity" in the knowledge gaining process. Now, if one can digitize and pass on the "expertise" to others as he/she left it without needing to write a book and others to read and work on it to get to that point.
Imagine you could download Linus Torvalds' operating system knowledge and Larry Page/Sergey Brin's analytical knowledge and Steve Job's product finish and marketing right into your brain!
No one would ever need to write a book, article, papers, exams and so on. you could directly submit your brain map for evaluation when you graduate out of college! Implications are profound.

6) Cog-market: sell your skill for the stay at ski resort
Imagine you have knowledge market, where various skill sets in various maturity level gets traded like stocks in today's stock/commodity market. one could learn any language in a jiffy. There could be some rating agency that grades the quality people's knowledge. So, people will be become Hardware that constantly work and improve the humanity's software - knowledge system that is.

7) Far into future: The Big Leap
We humans, have taken 10,000 years since we organized ourselves as civilization to develop steam engine (invention in mid of 18th century)and hence industrial revolution. It just took us less than 50years since we got our practical computer to get to where we are now. Previously, it would take decades to conceptualizing a product , developing a product to seeing the product in market. Now-a-days, it is the matter of years some time within a year.
Rightly, we are already on exponential growth trajectory in improving our collective cognizance. This is primarily because we are building stuff on the knowledge we have gained so far. The predictions which I have dared to make, if comes true... if at all, then the exponential growth is not just going to grow faster. It is going to be a complete game changer. our growth path would be hyper-exponential or hyper-hyperbolic.

The exponential growth seems very impressive given where we started from. But in absolute terms, humanity is still sub Type-1 Kardashev scale. We are at 0.7 in kardashev scale while aliens/extra terrestrials are expected to be of Type-II or even Type-III civilizations. Ultimately this hyper-hyperbolic growth would help propel our civilization from the Type-I into Type-II.

I would think it would all or most of it happen in next 3, 4 or 5 decades of time. We are going to live in interesting times. The Chinese curse, "May you live in interesting times" is more truer for this generation than anybody else.

mini Howto: write your own iptables target module

Howto Extend the iptables

I searched the existing iptables target/match modules whether they would help me collect the packet length statistics of LAN. I set out to use iptables infrastructure because iptables offered a nice packet management framework. I did not find a readily available target so, I decided to write my own.
What i found was it is surprisingly easy to write your own target modules.

Design:
My target module should inspect the length of the packets it receives. Iptables rules can be setup to send packets from specific source/destination/port number/application/time of day etc to the newly written module. The module, builds a array of 32 integers, which is modifiable at compile-time. The array elements act as a packet counter. array[0] would count the packets whose length varies from 0-63 and so on. My sampling step size is 64, which is again programmable at compile-time.

I used proc entry in order to access these counters from user space. For time being I have avoided netlink socket(elegant solution) and kprintf(beginner solution) both would work.

The module has entry and exit functions. Entry function creates proc entry and registers a target function. Write the target function itself and fill up the xt_target structure. Use this structure to register with iptables as a new target. Exit function de-registers the target and removes the proc entry.

The target function gets the skb pointer to work on. Yes, now we can do anything with the packet. The target function gets called by iptables with the packet (skb) as an argument and target module's argument as typed in iptables command as second argument. To keep things simple, I have decided to have my module without second argument. so, it just receives the packet.I get the packet length and quantize the length into 32 steps, and increase the appropriate counter.

To export the constructed histogram, I have written the output function which is invoked whenever user attempts to read the /proc/pkt-hist file. This function just prints out the array.

PS: Extending the iptables involves writing actual target or match modules which does the job in kernel space AND writing supporting user space library to parse the arguments and let the iptables know what to do when a target is typed in the command line for e.g
iptables -I FORWARD -s 192.168.x.y/32 -d 0.0.0.0/0 -j PHIST


Steps:
1) Make sure you have latest kernel code downloaded and netfilter option enabled. I had linux-2.6.32. Also make sure the /usr/src/linux points to /usr/src/linux-2.6.32/ where i have the linux source is available.

2) Get the iptables source code and untar it. Do ./configure;make;make install - usual procedure. I had iptables-1.4.3.2.

3) Implement the target: easiest way to write your own target is to take a reference from existing target modules. go into /usr/src/linux/net/netfilter/. You will find all the match and target modules supported by that netfilter(kernel). All match modules are in named in small letters while targets are written in caps. for eg, xt_mark.c is a match module, while xt_MARK.c is a target module.
I have copied the xt_NOTRACK.c which is a very simple target module and fits my bill. I have removed all code inside the target function and written my own. Renamed all the functions, structures appropriately. In entry function i have created /proc/pkt-hist file. I have also added required call back code for /proc/pkt-hist.

4) Modify KConfig: Now, in /usr/src/linux/net/netfilter/ open Kconfig in vi editor. search for NOTRACK, you will find
" config NETFILTER_XT_TARGET_NOTRACK
tristate '"NOTRACK" target support'
depends on IP_NF_RAW || IP6_NF_RAW
depends ....
"
If you recollect, this is what appears in "make menuconfig" program. So, copy-paste this entire section into Kconfig as below:
config NETFILTER_XT_TARGET_PHIST
tristate '"PHIST" target support'
depends on NF_CONNTRACK
help
The PHIST target collects the packet distribution
in pre-set steps.

If you want to compile it as a module, say M here and read
. If unsure, say `N'.


5) Implement user space library: go into /usr/src/iptables-1.4.3.2/extensions
and you will find libxt_NOTRACK.c, clone the same into libxt_PHIST.c and modify the names of functions appropriately.

6) compile and fly
Compile Target:
In /usr/src/linux/net/netfilter/, edit Makefile. Search for NOTRACK. you will find the following
obj-$(CONFIG_NETFILTER_XT_TARGET_NOTRACK) += xt_NOTRACK.o
Now, clone this line for PHIST. see below:
obj-$(CONFIG_NETFILTER_XT_TARGET_PHIST) += xt_PHIST.o

Now, At /usr/src/linux/ do make modules;make module_install. you will find xt_PHIST.ko in /usr/src/linux/net/netfilter/
Compile iptables userspace: repeat step (2). you will find libxt_PHIST.so in /usr/src/iptables-1.4.3.2/extensions/

7) Now try typing
iptables -I FORWARD -s 192.168.x.y/32 -d 0.0.0.0/0 -j PHIST
Now, iptables finds the xt_PHIST.ko and loads it and send the packet satifsying the rule to the target code which is just written.

So, the target module worked like a charm may be due to it is not doing much. But it was a breeze to write a new target. I guess writing match module wont be entirely different from what is described here.

Packet Length Distribution in LAN

I this post, I have copy-pasted from internet (ofcourse, citing the author) the packet distribution in internet. I always wanted to do that experiment myself. Last week, finally i could do just that. This is just a first experiment where I have captured all my own packets going into LAN, not purely internet.

Setup was similar to what i have done in my previous post. Replace "LAN cloud" with my PC and "wan cloud" with my office servers and internet. Linux box is a dual NIC-PC which runs FC10. I have configured the "Linux box" as a bridge. I wrote a iptables target module and passed all packets emerging from my PC through the module.

The Target module I wrote, looked and sampled the packet length of all packetes it receives and built a histogram. The histogram was built with the step size of 64bytes. After running it for couple of days, the result looks like below:

Pkt Histogram
==============
Length %of total packets
=========================
[ 0- 63]=18.91%
[ 64- 127]=0.84%
[ 128- 191]=5.76%
[ 192- 255]=0.47%
[ 256- 319]=0.8%
[ 320- 383]=0.05%
[ 384- 447]=0.08%
[ 448- 511]=0.09%
[ 512- 575]=0.13%
[ 576- 639]=0.17%
[ 640- 703]=0.09%
[ 704- 767]=0.06%
[ 768- 831]=0.05%
[ 832- 895]=0.03%
[ 896- 959]=0.03%
[ 960-1023]=0.03%
[1024-1087]=0.04%
[1088-1151]=0.04%
[1152-1215]=0.04%
[1216-1279]=0.2%
[1280-1343]=0.02%
[1344-1407]=0.62%
[1408-1471]=0.03%
[1472-1535]=71.44%
[1536-1599]= 0%
[1600-1663]= 0%
[1664-1727]= 0%
[1728-1791]= 0%
[1792-1855]= 0%
[1856-1919]= 0%
[1920-1983]= 0%
[1984-2047]= 0%

Now, I am tempted to the same exercise on the packet flowing purely into internet from my office. I will do the same and report in coming days. so, keep visiting this page or just subscribe to the RSS feed of my page.

PS:
My Work-PC is also a Linux machine running FC13. The packet captured reflects regular web browsing -http/https, email, and NFS (my home is mounted on nfs) and NIS.

Coming up next: How to write a iptables target module... Stay tuned.

An Amateur's Analysis of Nifty 50

How to time the market

Is it a right time to enter market? is there a potential upside from here? If you ask these questions to value-investors, they would say look at P/E. If P/E is high, the stock is said to be expensive and if low, the stock is said to be cheap. Buy a stock when it is cheap and sell it when it is expensive. simple, right? But the problem is how to establish if a P/E number is 'high' or 'low'? Pundits would ask you to take a look at the industry in which the stock in question belong and compare it with its peers.

To me, any single stock is too much of a risk. Even if you are an employee in a company there is a high probability that you wont know what is going on 'in-side' the company. Remember Satyam/LIC HF/Enron? Nor I have time to construct and prune a portfolio. It is too much of an effort. So, I would put my money into index ETF, which i think is a simple investment strategy that would not drain my time at the same time, gets me decent returns by the virtue of equity exposure. My favourite vehicle Niftybees is quoting about 589,which would be 1/10th of index value (with tracking error of about 1%). I don't want to blindly buy niftybees to get just "Rupee cost averaging". I want to restraint from buying whenever market is too expensive (hence build up cash) and resume to do SIP when it is not. As on 29 Nov 2010, nifty is trading at 5830 with current year's P/E of 23.

So, the question is, Does it make sense to put more money into this ETF or should i wait on the sidelines? Let us see if we could analyse the nifty index and try to arrive some answers.

Before I start, Let me tell you I am going to treat nifty index as a stock. I have downloaded the Nifty Index P/E from www.nseindia.com site from Jan 2002 till today.

At first cut, I get following

Jan 2006 - today
Average: 18.1 Min:10.7 Max:28.3 Std Dev: 3.8
This shows that over a long term, P/E has moved between the band of 11 and 28. Very wide band indeed.

Jan - Dec 2008
Average: 18.7 Min: 10.7 Max: 28.3 Std Dev: 4

Jan - Dec 2009
Average:18.7 Min:13 Max:23.2 Std Dev:3.7

Jan 2010 - 26 Nov 2010
Average:22.8 Min:20 Max:26 Std Dev:1.5

This shows me that today's P/E of 23.3 is more than average p/e so market is expensive. This also tells me that overall, 2010 valuations are stretched comparing long term.

Let us drill deeper. I plotted p/e distribution across the time line.
Key Take-away 1: Over the period of last 3 years, the is progressively become more expensive.
Key take-away 2: In 2010, mostly PE of 23 is sustained.

Now, 23.3 is very expensive if i compare last 7-8 years. But, it is close to average for the year. So, what will i do? probably I will stay on cash, waiting for correction to kick-in. Or I can invest little and keep the remaining in cash, look for right P/E level to enter.

I would probably do the later.

Disclaimer:
* I am not a qualified financial advisor nor a money manager. I manage my own money.
* This is not an advert for Niftybees - I do own some Niftybees.
* Opinion expressed here are mine it could be out-right foolish OR extremely genius, depending on how much you know on this subject.
* Calculations shown here may be prone to errors (i call bugs), after all I am a software engineer. So, trust this at your own risk.
* Nifty is a benchmark index provided by Nationsal Stock Exchange (NSE), India. It comprises of top 50 stocks in terms of free float listed on NSE. These 50 stocks accounts for 50%+ in free float market capitalization.
* Used Openoffice's Calc tool to crunch the numbers.

MS Office is online now! Will it change Google Docs for better?

Disclaimer: I am a Linux programmer, user and evangelist. I use open office and MS Office for work related documentation. When I bought a PC for my home use, I was very reluctant to buy ms office for the personal use as it was too pricey. Office is more expensive than MS Win7 by almost 1.6x to 2x. So, I chose not to buy the MS Office.

The Google docs service is around for quite some time now. It is a useful tool for non-m$ users like me. I could open email attachments online convert one doc format to other. So, I do not need to spend money just to open my mail attachments.

But the usefulness stops there. I don't think anyone can do some serious 'office work' on Google docs. For example, how do you do simple function like 'save as'? You will have to do 'make a copy' of a doc. Then delete the prev copy. I feel it is just good enough for very basic document/spreadsheet editing.

Recently Google added realtime doc sharing which could probably compete with plethora of web meeting tools like teamviewer, cisco webex etc. This is probably one 'new' feature that got added in so long time.

Today, this news item drew my attention. If ms office is offered online, that will give good competition to g.docs. as a result it is reasonable to expect Google would spend more money on the project and get some useable software. So, welcome Microsoft...

How to Ratelimit Physical Port (L2)

If you have a LAN (PC or Network)and you want to restrict the amount of data rate the LAN receives/transmits, with tc (Traffic Control) command you can achieve this very easily. All you need is that you need to make sure entire traffic passes throug a Linux machine in Bridge configuration.

You might want to type in the following command at your terminal.

# tc qdisc add dev eth0 root tbf rate 0.5mbit burst 5kb latency 70ms minburst 1540

This implements a 512kbps 'pipe'. In other words, entire LAN segment shown in the diagram gets to share 512kbps.

This command also implements bucket size of 5KBytes.

Packets would wait for 70milli seconds before they get dropped.

Applications
1) This is useful in lab environments where a DUT is to be tested with restricted bandwidth. DUT could be a base station simulator or any other network element.
2) In case you want to simulate the congestion or choked bandwidth within LAN, this comes handy to simulate a smaller pipe.

Whats inside UTStarcom's BSNL Branded WA3002GA ADSL2+Router

I have a ADSL modem at my disposal at my office. I am going to test my learnings from my personal Broadcom Beetel router in my home.

This ADSL modem is supplied by BSNL. This device has got 4 LAN Ethernet ports and has a Wifi-AP built in. Wan side, it connects to a ADSL enabled POTS line.

I have logged into GUI. surprise surprise, I saw the exactly the same look and feel of my Airtel -supplied-beetel modem. Of course It shows some wifi menu, vpn config menu and more capable firewall menus. Apart from these same font, same color, same table everything same. I checked the bottom of device, it shows it is made by UTStarcom and manufactured in China. My home modem has No info though. It should be either Beetel modem is also manufactured by UTStarcom OR Broadcom chip vendor has supplied the same code base to these OEMs.

I ventured telent-ing into the box. I see 'some' difference here though. I see fixed menu showing. In my home modem, you have to press ? to get the list of commands supported. I typed 'sh' to get a unrestricted busybox shell -so that is same. Just like beetel modem this box's software bulid doesn't support 'ls' command so used 'echo *' to list the files. Software sports 2.6.8 kernel and gcc version 3.4.2, uClibc library. The software version of the box has 'bsnl' in it. It suggests me that code base for both BSNL and Beetel modems are same and separately built.

The Hardware is has got some more muscle

Wifi Chipset,
4 LAN ports
14MB RAM
3MB Flash

It makes sense as my office is using business class ADSL .

I have one more ADSL+everything kinda box with me. it is imported!! I am going to work on it and make it work with BSNL Line. Hope it works, Watch this space for updates.