‘Mind at Play’ -Shannon Claude & the Information Highway
Reflections on Shannon’s Biography and other TelCo stories..
The book had brought back many a hazy moments and recollections . The purpose and the towering foundations of the many UG subjects seemed to shine out better.
Entropy, unpredictability and surprises are the ways of the Universe. Our professor would Introduce this topic... If I said the “Sun will raise in the East tomorrow”, there in no information. However if I said “Sun will raise in the West”, there is a lot of information. From News Channels to writers and musicians, there is a race for bringing information and novelty to the table.
What is the history of Telecom. What is Shannon’s contribution in Information theory and other fields? How did the Universe conspire with him? What is the role of English? And some reflections of his model in other fields of Life..
Connecting the World
And it took them 6 voyages to complete the wiring between Europe and America.
“The first wiring of the world began in 1850, only six years after Samuel Morse demonstrated the reality of telegraphy. Soon came a cable across the Atlantic. Queen Victoria sent a message to President James Buchanan. Some of the words reached Washington that day; the rest came through on the next day.”
“In July 1865 the ship set out from England with a crew of 500, a dozen oxen for hauling, a cow for fresh milk, a herd of pigs for bacon -and a thickly insulated 2,800-mile cable that weighed 5,000 tons. They had almost finished laying it when the cable snapped. The next year they succeeded.”
And to combat the effect of noise in the waters, the signal was pumped up and sent with a high voltage in the magnitude of thousands of Volts.
“It took a hundred years to connect a billion people by wire. It has taken only ten years to connect the next billion people.”
And slowly from copper wires to radio waves, microwaves, satellite signals and fiber optics, we live in a connected world of today.
Invention of the Telephone
Back in the 1870s when electricity was the cutting edge technology and communication was limited to dash and dots of telegraphy, Alex Graham Bell a young man in his 20s had the audacity of transmitting human voice over electricity.
At the transmitter side, the diaphragm was connected with an electromagnet. Like a trampoline, the diaphragm moved up and down with the sound waves. This inturn generated electric signals on the electromagnet and was transmitted over the wires. The receiver side electromagnet on receiving electric signals cause the diaphragm to excite and that inturn produce voice.
Idea of Compression and Encryption
Imagine if a river had to be laid between you and your sibling inorder to make a call. And your voice would float on the river and reach the other end. No matter whether there was silence or speech or a song, information was continously being pumped to the river. And anybody on the river side could sniff and hear the conversation.
Now imagine.. Instead of sending a truckload of information, you zip your voice packets into a suitcase and send them every second.. Smaller suitcase for a silence.. Bigger suitcase for a dailog. and since it is a suitcase, you can secure it with a key which is also available the other side. And the same river can now accomodate 100 more phone calls due to the compressed suitcases.
And with technology, from a river you migrate to a 10 lane highway enabling many ducatis and ferraris. This ends the story telling part of the Voice Communication.
Developments in Shannon’s time
Thompson was building a Harmonic analyser. The tidal waves in the ocean were influenced by a number of parameters. The month, the moon’s gravity, Earth’s orbit etc. 3 years of American weather could be predicted in about 3 months of time with his new contraption. The mathematicians laughed off on the idea that a machine could do something that needed the brains of the mathematicians.
Earliest calculus machines were mechanical.. The gears and pistons did the integration and math. Then it progressed to electrical circuits and later to transistors. Influenced by Mr Bool, Shannon proposed a logical way of problem solving, deduction and simplifying the logical representation before implementing circuitry be it mechanics, electrical or transistors underneath.
Come world war and it called for accurate mathematics. The urgent need for machines solving calculus.. The moving targets, the rotation of the Earth, the angle of attack. Need for encryption.. And thus fueling the advancements and acceleration in Information handling.
Shannon’s Famous Entropy Coding: Compression & Error Recovery
Crossword puzzles and games such as Hangman and Wheel of Fortune exploit redundancy by assuming that humans can guess letters in a word or phrase based on their previous knowledge of the language. Shannon’s ingenious idea was to exploit this natural measure of redundancy. At the end, it is about transmitting as less bytes as possible and reconstructing it at the other end correctly.
h = −(log 1 /27) = 4.75
In our messaging scheme, if we were to transmit 27 characters of English and assume all the characters are independent and have equal probability, we would require 4.75 bits per character to transmit.
If we consider the probability distribution of these letters, the Entropy would reduce to 4.219.
Further, if we were to consider the stickiness in the 2 letter clusters and 3 letter clusters, the bit requirement from the earlier 4.75 is reduced to 2.77.
Estimates by Shannon based on more and more corelation and human experiments have yielde d values as low as 0.6 to 1.3 bits per symbol.
After addressing the entropy and “what codes to send for efficieny purpose”, Shannon introduced redundant error codes so that the Signal could be retrieved even if there were some losses.
Patterns in English. Shannon’s winds of oppurtunity?
A large part of Shannon’s thoughts on compression started of with the redundancies in the written messages in English. What if Shannon was from a family whose language patterns were very different? Like Chinese, Hindi, Russian? May be another Shannon would have done it?
Inventions like Electricity and Telephony had no direct bearing on linguistics and compression schemes. And redundancies are also seen in audio, text and video formats. In my opinion, the redundancies standout in written English much more than other family of languages. This could have been a premise and a stepping stone.
Entropy in English
I wonder why the Information Scientists did not go to the roots and trace the origins of the English Alphabet.
In my opinion, English literals are unstructured, lossy, an approximation, linear and hence simple. The written code words act as a skeleton and the nuances of articulation, vowel lengths and morphing of sounds at word boundaries and in between words are not embedded in the written scheme.
If we take the top 20 words in English, there are a bunch of articles, prepositions defining the noun contexts, verbs, pronouns and conjunctions. This defines the simplicity of the language structure. The noun contexts are not embedded in the words. The verb contexts are few. There is no gender specific grammar rules. All the above reasons make the English language and its written coding scheme simple and expose a lot of redundancy.
I was reading about the famous poet Kalidasa. His dictionary consisted of about 26K words. According to Herden, Entropy of German is 1.65, Russian is between 2.07 and 2.11 and Kalidasa alone is 2.05.
What made English so mainstream?
This topic is not directly related to Shannon’s compression scheme. Neverthless, the reasons above gave this thought. Perhaps there is a level of ease in English which makes it so mainstream. A small character set of letters, a few rules to start with.. And then keep on icreasing the rule base and dictionary for newer vocabulary.
It was British Colonies that started the English expansion. What if the Japanese, Chinese or Russians made it big? Would their languages been adopted with such ease? Some rumblings and musings on ‘what if’ and ‘why’
Into the Information Bandwagaon
Back to the topic of Shannon. Prior to his Information theory, he had worked with Eugenics on ‘algebra for genetics’. A notation for representing 5 generations before and after. He had also modelled biological systems under his channel theory. The transmission and reception of signals between nerves, the gene copy between DNA and mRNA and possible mutation of signal etc. He had seen the information coding in this space even before the actual DNA model was rolled out. And it is Eugenics which run the SAT tests in the US.
Applied Mathematicians vs Philosophers
In the words of Plato, Applied Mathematicians are like merchants and military.. keeping count of their stock and ammunition and have a way with numbers. The purists are like philosophers.. Exploring prime numbers.. patterns.. correlation with the world..
And when Shannon’s theory became so huge.. the purists were perhaps digesting the success of an applied mathematician..
On Einsteins and Newtons
It is the talent, training, stimulation, restlessness and joy in exploration that makes these people a class apart. When Shannon had to speak about why every person was not a genius, he said.. Given an idea, there are people like Einstein and Newton who come out with 2 ideas.. And there others come out with half an idea. Keep changing the perspective when stuck. Some times those who are green and new comers have the solution as they have no bias. Break the problem into smaller dimensions.
The AI thoughtworks
During the war times, it is interesting that Shannon and Turing worked in Bell Labs and discussed on future of machines though each were not allowed to discuss much on professional topics.
Shannon build his contraption of the mice that could escape from the maze. The mouse in question was actually a block of wood outfitted with a magnet, three wheels, and a set of copper whiskers. Its nickname was Theseus, and it had one skill: it could solve a maze.
Communication with the World
At a fundamental level, when humans beings communicate.. be it in person or in social media.. The signal is sent by the sender and is bound to be wrapped with some noise before it is decoded by the receiver.
From a philosophical lens, if were to examine the relation.. The sender modulates his thoughts and transmits the signal. Through the information highway of the sense organs, the receiver assembles the message. Sometimes, the receiver’s bandwidth is full or busy. Sometimes the receiver is not able to comprehend or feel the need to listen to the message. Some time the receiver is not able to demodulate the message in the same scheme as the sender. All of the above scenarios could lead to a lossy signal. When this idea of noise is implanted in the sender, he is not deluded by the response.
It is amazing to model our interaction with the Universe.. What can be white noise for one could be signal to another.. Perhaps our bandwith availability, our focus and interest determine the interaction.
An interesting narration from Scriptures by Swami Sarva Priyananda. Story of 10 kids and the problem of the missing one. And the teacher comes and shows the missing one. And then they do it for themselves and fix the error.
At timestamp 38.22 The 10th man parable
https://youtu.be/BMRbh3M4AGw
Entropy in the world
With communication, AI and recommendation engines, it seems that the Entropy of the world is reducing. One is able to keep in touch and look forward for the kind of things they are interested in. May be the energy and disruption that goes into making these systems increases the entropy of the world in unseen ways.
