Philosophical fate of AI and Humans

Alan Turing was the very first person in the world to formally ask- “Can machines think?” The ideas he presented in his famous paper has laid the pathways leading to the creation of modern computer science and today and tomorrow of artificial intelligence. There is no doubt that there will be a time when machines would be able to think just like humans do, but that should not be a negative aspect. There will be practical limitations to a human-like thinking machine too. So, the game would never be single sided. This should push humanity on a completely new path of evolution. That is also how we have become the humans we are today from the primitive apes.

Alan Turing’s world famous paper on future of human-like thinking ability in machines

The holy doubt – “Can machines think?”

We all know how modern machines/ computers have great abilities to make systematic thinking and take decisions accordingly; this is obviously attributed to the very programming embedded into them by us human beings. Many breakthroughs in storage capacities of computers, size of computers, efficiency of these machines, computation capabilities, evolution of programing languages, intersection of neuroscience and computer science, accessibility of these highly powerful machines to masses have shown world that such machines can do amazing marvels.

You know where I am going with this. Not mentioning Artificial Intelligence in these breakthroughs would be a straight crime. AI has unlocked a totally different capability in computing for which some are optimistic and some are fearful. In a crude sense, how AI stands out from other concepts of computing is its ability to change it programming to achieve given goal. This concept is very normal even for today’s child.

But, would you be open to such self-programming machine 75 years ago? A time when there were only mechanical calculators, electronic computers were in their infancy and were created only for certain restricted problem solving and number crunching. Even the experts of those times found this idea foolish because of the practical limitations of those times. How could a machine think like a human being when for doing some mechanical number crunching it takes such many resources, doesn’t have its own consciousness, its own soul, has no emotions to react to given stimuli? In simple words “thinking” is somehow associated as a special ability humans got because of the soul they have, the conscience they have (granted by nature, the Creator, the Almighty, the God or whatever but some higher power)

It is our tendency as human beings to have this notion of being superior species amongst all which brings in the confidence that machines cannot think. That is why this idea seemed foolish, but now we are comfortable (to some extent but not completely) with the idea of thinking machines.     

Alan Turing – a British mathematician, the code breaker of Enigma, the man who made Britain remain strategically resilient in World War 2, the Father of theoretical computer science wrote a paper which laid down the blueprint of what the future with AI would look like. For the times when this paper was published all the ideas were seemingly imaginary, impractical, and totally impossible to bring into the reality. But as the times changed, Alan’s ideas have become more and more important for the times in which we are living in and the coming future of Artificial Intelligence.

Weirdly enough, this paper which laid the foundations of artificial intelligence – thinking machines was published in journal of psychology and philosophy called “Mind”.

The world-famous concept of ‘Turing Test’ is explained by Alan in this very paper. He called this test as a game – an “Imitation Game”.

The paper reflects the genius of Alan Turing and how he had the foresight of the future – the future with thinking machine. After reading this paper you will appreciate why and how Alan was able to exactly point out every problem that would rise in future and their solutions. He was only limited by the advancements not happened in his time.

The Imitation Game   

Alan posed a simple question in this paper –

Can machines think?

The answer today (even after 75 years) is of course a straight “NO”. (Deep down we are realizing that even though machines can’t think they are way closer to copying the actions involved in thinking or “imitating” a thinking living thing)

The genius of Alan Turing was to pose practicality to find the answers to this question. He created very logical arguments in this paper where he used the technique of proof by contradiction to prove the feasibility of creating such ‘thinking machine’. The AI which has evolved today is the very result of following Alan’s blueprint for making thinking machines.

The famous Turing Test – the Imitation game is a game where an interrogator has to tell the difference between a machine and a human being by the responses they give to his/her questions.

The machine is not expected to think like humans but at least imitate them. The responses may feel completely human but it is not a condition or compulsion that machine should exactly think like a person. This practicality introduced by Alan and his arguments built upon this idea shows what are our limitations when we are actually thinking or making any decisions. This paper will change and also challenge the way we think or do anything. This paper might humble you if you think that we are superior beings because we can think and have/ express emotions. (Trust me you would also question ‘What is love?’ if love was your next answer to justify our superiority after reading this paper but that is not what Alan was focusing when he wrote this paper.)

The idea is not about creating an artificial replica of human, it is to create a machine which would respond just like humans do, the goal is to make their responses indistinguishable from ‘real’ human beings.

There are hundreds of simplified explanations on Turing test (ask Chat GPT if you want) which Alan has discussed in this paper but that is not my interest of discussion hereon.

I will be focusing only on the arguments made by Alan to prove why it is completely practical to create human-like thinking machines. My intent in doing so is that to show how we as humans can also be challenged by our practical limitations. These arguments also show a way to humans where they will get overpowered/ surpassed by AI. This does not mean that AI will eradicate humanity, rather it shows new pathways in which humanity would evolve. So, for me the arguments end on an optimistic note. Surely AI will take over the things which make us who we are but it will also push us into some completely unconventional pathways of rediscovery as the smart species.

The way in which Alan intended the Imitation game was the mode of question-answers – an interview. You would question why didn’t he think of a challenge where exact human like machine need to be created – that would be more challenging for the machines. I think, the idea behind rejecting the necessity for a machine to be in human form is like this-

The creation of human body is very similar to cloning a human body or augmenting the human parts to a mechanical skeleton. What is more difficult is to impart the consciousness and the awareness which is (supposedly) responsible to impart thinking in humans. So, even if a fully developed machine exactly looking like human being is in front of you and you are unable to tell that it is a machine, the moment that human-like machine would start expressing its thoughts everything would be easily given away.

In simple words, Alan was confident that the biological marvels, genetic engineering, cell engineering would easily take us to the physical replication of human form. What would be difficult is to create a set of logics (or self-thinking mechanism) which would demonstrate human like (thinking) capabilities. And such abilities can easily be checked by mere one on one conversation. Such was the genius of Alan Turing to bring such complexities using this simple experiment of Imitation Game.

We as human beings have certain insights, intuitions (I don’t want to use this word but don’t have any alternative word) which gives away if it is a machine or a human.

What Alan did masterfully and why he deserves full credit is that he pointed out the factors which can make machines respond and ‘think’ more like humans. While creating the confusions about the nature of human mind, consciousness, awareness, thoughts and their limitations and ambiguity, Alan also gave the possible arguments to solve these confusions.

Alan proves that human-like thinking machines can be created and he proves this by contradiction of the objections raised against this idea. I am diving deep into these objections hereon:

  1. The theological objection

The rigor that Alan used to prove his point deserves appreciation. Despite being a logical thinker and mathematician, he cared to answer the religious point of view, he wanted no stone left unturned while making an argument.

Alan aggressively (verbally) hammered the idea of God’s exclusivity to grant the immortal soul to only humans, the soul responsible to make humans think. Alan says that if soul is the reason, then animals have souls too. The true comparison then should be between living and nonliving things to support the point that machines cannot think. It is because they are nonliving things they have no soul so they cannot think.

But if the great almighty can give soul to an animal, then why this omnipotent God decided to not give same souls to the machines? Alan knew that any blind theologian would find a contrived argument to prove this idea but he clarifies his point by presenting the historical mistakes religious institutes committed because the truth was hard to swallow. Alan gives the examples of Galileo who presented that earth was not the center of the universe, against the ideas of Church. Later church was proved wrong.

So, even if the religious arguments may seem easy to understand, easy to ‘swallow’ but if they are not fitting in the logic, it makes no sense to take them forward. The theological inconsistency ‘As machines have no soul granted by the God, they cannot think like humans’ which Alan pointed out  was totally false. He justifies this point using the logic of God remaining the ultimate creator.

Alan explained that if we are stealing the powers of God to create a human-like thinking ‘thing’ its not a crime or a blasphemy. Does procreating and making children “to whom also God grants the soul for thinking” mean crime? In similar spirit ‘machines’ – thinking machines are our children whom to God should bless with his powers.    

“In attempting to construct such machines we should not be irreverently usurping His power of creating souls, any more than we are in the procreation of children: rather we are, in either case, instruments of His will providing mansions for the souls that He creates.”

No doubt he would also have been a great priest if he had thought of changing his career to theology.

  1. The ‘Heads in the Sand’ objection

Alan gives worst case scenarios on the superiority of human species out of all species. What if we are “the superior” species? If that is true then there is no reason to worry about thinking machines, they won’t surpass us.

But what if what we know is wrong? We have been proven wrong many times in history. What if we are not the superior species? Then there is no sense in blindly believing that we are superior. Rather this illusion of superiority steels us from the chances to fight the battle of superiority.

So, in either case, we cannot run out of the fate of thinking machines Vs humans. We may fake it, run from it, hide it from rest of the population but it is not in our favor if we do so.

“We like to believe that Man is in some subtle way superior to the rest of creation. It is best if he can be shown to be necessarily superior, for then there is no danger of him losing his commanding position.”
  1. The Mathematical Objection

Very beautifully Alan brought the Gödel’s incompleteness theorem to prove his mathematical argument. According to Gödel’s incompleteness theorem, if we start to prove every mathematical argument there exists in the universe, we end up into some arguments for which there exists no proof. In order to ensure that the whole mathematical system remains stable, consistent on logic one has to accept those arguments true. So, once such logically unprovable but true in existent reality statements are found in nature they create a new system of mathematical understanding.

In simple words, every mathematically logical system is inconsistent in the end, in order to remove that inconsistency a new rule must be accepted which create a new system of mathematics. (Which again would be inconsistent)

Further oversimplification goes like this,

A farmer wouldn’t know how to make a shoe. So, he would need knowledge of a cobbler. A cobbler wouldn’t know how to make metal tools, so he would need help of blacksmiths. Even if they have each other’s knowledge, skills they must accept certain thumb rules passed down from their ancestors (which are always true but unprovable) to master each other’s skills.  

So, even if you are creating a thinking machine based on purely mathematical system the mere limitation of mathematics will stop it from overpowering, surpassing humans.

This also does not mean that thinking machines are defeat-able. A machine with one mathematical system in totally different domain could support this logically inconsistent system just like the villagers with different professions.

Alan Turing’s doctoral thesis contains the ideas of Gödel’s Incompleteness theorem so it is a joy to read these arguments in this paper. They are well formed and super-intelligent.

(If you are really interested what this argument means, you can research the efforts that went into proving Fermat’s last theorem. A new field of mathematics had to be created to prove this simple to explain but difficult to prove mathematical theorem.)

There will always be something cleverer than the existing one – for humans and for thinking machines too.

“There would be no question of triumphing simultaneously over all machines. In, short, then, there might be men cleverer than any given machine, but then again there might be other machines cleverer again, and so on.”
  1. The argument from Consciousness

Even if the machine is feeling and thinking exactly like a human being, how could the “real humans” know that it does so? – Alan’s new argument.

“The only way to know that a man thinks, is to be that particular man. It is in fact the solipsistic point of view. It may be the most logical view to hold but it makes communication of ideas difficult.”

Communication between machines and the humans and its quality would be key proof to understand whether the machine thinks like human beings or not. Even if the machine is really thinking exactly like humans, it is futile if it cannot communicate so to humans.

(That is exactly why The Turing test with mere typed communication is more than enough to check the thinking ability of machines.)

It is the great philosophical mind of Alan to use the limitations of Solipsism to justify his point. According to solipsism all the world exists in the mind of the person because if the person dies then it doesn’t matter if world is there or not.

The key limitation of solipsism is that your survival is not directly connected to your mere thinking. If I think ‘I am dead’ that does not immediately kill me. If I think that I have eaten a lot without actually eating anything, that doesn’t end my hunger in ‘reality’. So, reality is not only your mind.

Also, solipsism fails to answer the common experiences we have in a group. If my mind is my world, I can create any rules for my world and things would always go as I desire. But that doesn’t happen in reality. There are certain ways, truths which are common to all of us that is why our world is not just our mind, rather it may be a shared world. You alone are not the representation of whole reality.

So, even if we accept that the machine ‘inwardly’ thinks like human being, it has to share some common truths to the interrogator to prove its humanly ways of thinking.

“I do not wish to give the impression that I think there is no mystery about consciousness. There is for instance, something of a paradox connected with any attempt to localize it. But I do not think these mysteries necessarily need to be solved before we can answer the question…. (the question – can machines think? Can they at least imitate humans? – the Imitation Game)”
  1. Argument from various disabilities

Alan is challenging the idea that even if machines are successful in thinking exactly like humans, they won’t be able to do certain things which humans can do better.

It’s like a human saying to a thinking machine –

“You machines can think like us but can you enjoy literature and poetry like we humans do, can you have sex just like humans do, enjoy it and procreate just like we (human) do? This is exactly why your thinking is not a human thinking.”

The key point Alan is trying to prove is that people always need a justification of given machine’s ability (through its ways of working, maybe its architecture, its technology, its components, its sensors) to prove that certain capability of the machine. When we are showing these justifications, we are also telling people indirectly what it cannot do thereby its disabilities. One ability would point to other disability.

People do not accept black box models in order to justify ability of the machine.

“Possibly a machine might be made to enjoy this delicious dish, but any attempt to make one do so would be idiotic. What is important about this disability is that it contributes to some of the other disabilities.”

In same fashion one argument is that even if machine could think like humans, it is difficult to have its own opinion. Alan strikes that too.

“The claim that a machine cannot be the subject if its own thought can of course only be answered if it can be shown that the machine has some thought with some subject matter.”

The key disability which was preventing Alan from creating a working thinking machine was the enormous storage space. You will appreciate this point today because you know how drastically storage capabilities have improved over the time. These improvements in storage created the AI we see today, although processing power is also on factor and there are other factors too but it boils down to the ability to simultaneously handle lot and lots of data.

Alan had this mathematical insight that once the storage ability is expanded enough the thinking machines is a practical reality. (Now researchers are not only working on to further improve storage capability but special efforts are also taken to effectively compress data. Ask Chat GPT about the Hutter Prize)

So, Alan makes a point that having variety of opinions in order to ‘think for itself’ machines don’t need logic, they need enough storage space just to process them simultaneously to create a new thought. In terms of humans, the more information and logic you can handle the crisper your understanding are. Same would be the case for thinking machines.

“The criticism that a machine cannot have much diversity of behavior is just a way of saying that it cannot have much storage capacity. “
  1. Lady Lovelace’s Objection

Charles Babbage was the first person to technically create calculator with memory – a programmable computer which they called Analytical Engine. Even though he knew how the Analytical Engine works Ada Lovelace created programs and published them to the masses to prove the effectiveness of the Analytical Engine. She was the first programmer of computer.   

Lady Lovelace’s key argument is based on the idea that the computer thereby a thinking machine cannot think for itself because it can only use what we have provided it. As we have provided whatever we know and have it cannot think outside of that information and generate new understandings, The machines cannot think “originally”.

Alan strikes down this argument easily using the idea of enough storage space. If the machine can store large enough data and instructions then it can create new inferences, original inference.

“Who can be certain that ‘original work’ that he has done was not simply growth of the seed planted in him by teaching, or the effect of following well-known general principles.”

Alan questioned the very nature of originality. Only a genius can do this in my opinion. Alan showed the world that the things which we call original are inspired, copied from something already existent. It is just matter of how unknown we are to this new thing.

He builds further upon that saying that if machines can think originally then they should surprise us. That is reality. Machines do surprise us by using unconventional approaches to our daily tasks. 

Alan links new argument for further justification, if machines can think originally then they can surprise us. In order for us to not get surprised we must get immediate understanding of what machine presents which never happens when such events happen. So, machines can think originally and can surprise us.

“The view that machines cannot give rise to surprises is due, I believe, to a fallacy to which philosophers and mathematicians are particularly subject, This is the assumption that as soon as a fact is presented to a mind all consequences of that fact spring into the mind simultaneously with it.”

What a brilliant argument!

  1. Argument from Continuity

“The nervous system is certainly not a discrete-state machine. A small error in the information about the size of a nervous impulse impinging on a neuron, may make a large difference to the size of the outgoing impulse. It may be argued that, this being so, one cannot expect to be able to mimic the behavior of the nervous system with a discrete-state system.”

Alan talks about an attempt to create thinking machines by mimicking nervous system which is a continuous system. A system which works in wave, signals (analog) and not in ones and zeros (discrete).

Alan says that even if we use such analog system in Turing test, the outputs it would give would be probabilistic instead of definite. This will actually make the interrogator difficult to distinguish human response from the machine one. Humans would be more frequently unsure and will give such probabilistic answers more frequently.  

  1. The argument from Informality of Behavior
“If each man had a definite set of rules of conduct by which be regulated his life, he would be no better than a machine. But there are not such rules, so men cannot be machines.”

The idea that machines work on certain defined rule even if they can alter their own program by themselves in order to think like humans, it feels obvious that they will be more formal and stuck to their rules while responding. This formality would give away their non-human nature.

Alan questions the very nature of what is means to have laws in a logical setup. Taking support from the Gödel’s Incompleteness theorem, not even single system – single logical system can confidently remain purely on its laws. It would assume some arbitrary point to make some sense out of given data even if it is using some mathematical frameworks. (Remember the simulations where you put garbage in and the simulations runs perfectly giving garbage out. But you know its garbage because you have certain test to judge the output with reality which are objective.)

There is no such objectivity to judge informality of a system – the word and logic itself says it all. Our search for formal laws would never end and this will always keep on creating new laws and new inconsistencies and informalities. There is no end.    

“We cannot so easily convince ourselves of the absence of complete laws of behavior as of complete rules of conduct. The only way we know of for finding such laws is scientific observation, and we certainly know of no circumstances under which we could say, ‘We have searched enough. There are no such laws.’”
  1. The Argument from Extra-sensory Perception
“The idea that our bodies move simply according to the known laws of physics, together with some others not yet discovered but somewhat similar, would be one of the first to go. This argument is to my mind quite a strong one. One can say in reply that many scientific theories seem to remain in practice, in spite of clashing with ESP; that in fact once can get along very nicely if one forgets about it.”

Again, Alan left no stone unturned. He made sure that even the pseudo-science fails to support the idea that machines cannot think like humans.

He explains that even if the human competing against the machine mimicking humans has telepathic abilities to know states of the machine or even the interrogator, it would actually confuse the interrogator. The only thing such telepathic person can do differently is to under-perform intentionally which again would confuse the interrogator.

The idea is that even when we are not sure of how such supernatural things works our current understanding of things and their workings are just fine. The supernatural things are not interfering in our formal understanding of nature and reality.

The implications of Alan Turing’s Paper on Computing Machinery and Intelligence

All the ideas explained by Alan in this paper are responsible for the modern technologies like efficient data storage, data compression, artificial neural networks, self-programming machines, black box models, machine learning algorithms, iterative learning, data storage, manipulation thereby data science, analog computing, self-learning, supervised learning algorithms, Generative Pre-Trained Transformers (GPTs) and what not.

This paper is holy grail for not only modern computer science but also for the literature and popular culture. Once you appreciate the ideas in this paper you will be able to see the traces of these ideas across all the modern science fiction we are consuming all the time.

Alan created practical ideas which were possible to implement in future based on the coming technological revolutions he foresaw. He logically knew that it is possible but the genius of him was to lay the practical foundation of what and how it needs to be done which is guiding our and will guide future generations.

Conclusion

What is there for humans if machines start thinking like humans?

For this, I will address each argument posed by Alan

  1.  The theological objection

God will actually bless us because we extended his (or her I don’t know) powers to create something like his own creation through thinking machines.

  1. The ‘Heads in the Sand’ objection

Even if thinking machines surpass us, we have to live with it and create our new ecosystem to ensure our survival. Even though for given times we are superior species, other species are existing with us in the same time with their special abilities. There is no running away from any possible outcome of this scenario.

  1. The Mathematical Objection

The mathematics itself restricts a single machine from knowing everything. So even if multiple machines come together to create superior understandings same would happen for humans. There will always be this race of superiority, sometimes machines will lead sometimes humans will lead. There is no conclusion to this race as far as the inherent flaw of mathematics goes.

  1. The argument from Consciousness

A machine has to be the communicator of its human thinking, it cannot remain in the dark abyss of self-cognizance and remain away from humans. If a machine starts thinking like humans, we all would definitely know about it. A machine has to communicate its ability of awareness to, it will a surprise but a very short lived one.

  1. Argument from various disabilities

If we don’t know how machines think like human that would not prevent them from thinking like humans. We have to accept the black boxes through which machines would think like humans. That is the only sane way out. We humans too are filled with disabilities but they are not directly linked to the ways we are able to think.

  1.  Lady Lovelace’s Objection

Machines will surprise us, they can also create original ideas, because what we call original is something that lies out of the limits of our current thinking. Rather it is an optimistic idea that if machines could think like humans do then they may give us totally new ideas for new discoveries, breakthroughs.

  1. Argument from Continuity

Continuous thinking machine or discrete thinking machine both can confuse humans if they achieve their thinking potentials. So, there is no point in creating an analogue thinker to beat digital thinker. We ourselves are an analogue thinker.

  1. The argument from Informality of Behavior

No system will have all laws already established, the system has to keep on creating new laws to justify new events, outliers. The process is never-ending. So even if machines surpass in human thinking we too have the advantage of informality to make the next move.

  1. The Argument from Extra-sensory Perception

Even if the supernatural abilities are proven be existent, they will have less to no contribution in the thinking abilities of machines. So, if you are a telepathic reader a human like thinking machine can fool you without exposing its real machine identity.

Going through all this you will appreciate how limited our human thinking is. There is no doubt that there will be a time when machines would be able to think just like humans do but that should not be a negative aspect. There will be practical limitations to a human-like thinking machine too. So, the game would never be single sided. This should push humanity on a completely new path of evolution. That is also how we have become the humans we are today.

Further references for reading:

  1. A. M. TURING, I.—COMPUTING MACHINERY AND INTELLIGENCE, Mind, Volume LIX, Issue 236, October 1950, Pages 433–460, https://doi.org/10.1093/mind/LIX.236.433
  2. Understanding the true nature of Mathematics- Gödel’s Incompleteness Theorem
  3. Questioning Our Consciousness – Solipsism

Logarithmic Harmony in Natural Chaos

Mathematics is one powerful tool to make sense out of randomness but bear in mind that not every randomness could be handled effectively with the mathematical tools we have at our disposal today. One of such tools called Benford’s Law proves that nature works in logarithmic growth and not in linear growth. The Benford’s law helps us to make sense of the natural randomness generated around us all the time. This is also one of the first-hand tools used by forensic accountants to detect possible financial frauds. It is one phenomenal part of mathematics which finds patterns in sheer chaos of the randomness of our existence.

Benford’s Law for natural datasets and financial fraud detection

People can find patterns in all kinds of random events. It is called apophenia. It is the tendency we humans have to find meaning in disconnected information.

Dan Chaon, American Novelist

Is There Any Meaning in Randomness?

We all understand that life without numbers is meaningless. Every single moment gazillions and gazillions of numbers are getting generated. Even when I am typing this and when you are reading this – some mathematical processing is happening in bits of the computer to make it happen. If we try to grasp/understand the quantity of numbers that are getting generated continuously, even the lifetime equivalent to the age of our Universe (13.7 billion) will fall short.

Mathematics can be attributed to an art of finding patterns based on certain set of reasoning. You have certain observations which are always true and you use these truths to establish the bigger truths. Psychologically we humans are tuned to pattern recognition, patterns bring in that predictability, predictability brings in safety because one has knowledge of future to certain extent which guarantees the higher chances of survival. So, larger understanding of mathematics in a way ensures better chances of survival per say. This is oversimplification, but you get the point.

Right from understanding the patterns in the cycles of days and nights, summers, and winters till the patterns in movements of the celestial bodies, the vibration of atoms, we have had many breakthroughs in the “pattern recognition”. If one is successful enough to develop a structured and objective reasoning behind such patterns, then predicting the fate of any process happening (and would be happening) which follows that pattern is a piece of cake. Thus, the power to see the patterns in the randomness is kind of a superpower that we humans possess. It’s like a crude version of mini-time machine.

Randomness inherently means that it is difficult to make any sense of the given condition, we cannot predict it effectively. Mathematics is one powerful tool to make sense out of randomness but bear in mind that not every randomness could be handled effectively with the mathematical tools we have at our disposal today. Mathematics is still evolving and will continue to evolve and there is not end to this evolution – we will never know everything that is there to know. (it’s not a feeling rather it is proved by Gödel’s incompleteness theorem.)

You must also appreciate that to see the patterns in any given randomness, one needs to create a totally different perspective. Once this perspective is developed then it no longer remains random. So, every randomness is random until we don’t have a different perspective about it.

So, is there any way to have a perspective on the gazillions of the numbers getting generated around us during transactions, interactions, transformations?

The answer is Yes! Definitely, there is a pattern in this randomness!!

Today we will be seeing that pattern in detail.

Natural Series – Real Life Data       

Take your account statement for an example. You will see all your transactions, debit amount, credit amount, current balance in the account. There is no way to make sense out of how the numbers that are generated, the only logic behind those numbers in account statement is that you paid someone certain amount and someone paid you certain amount. It is just net balance of those transactions. You had certain urgency someday that is why you spent certain amount on that day, you once had craving for that cake hence you bought that cake, you were rooting for that concert ticket hence you paid for that ticket, on one bad day you faced certain emergency and had to pay the bills to sort things out. Similarly, you did your job/ work hence you got compensated for those tasks – someone paid you for that, you saved some funds in deposits and hence that interest was paid to you, you sold some stocks hence that value was paid to you.

The reason to explain this example to such details is to clarify that even though you have control over your funds, you actually cannot control every penny in your account to that exact number that you desire. This is an example of natural data series. Even though you have full control over your transactions, how you account will turn out is driven by certain fundamental rules of debit/ credit and interest. The interactions of these accounting phenomenon are so intertwined that ultimately it becomes difficult to predict down to every last penny.

Rainfall all around the Earth is very difficult to predict to its highest precision due to many intermingling and unpredictable events in nature. So, by default finding trend in the average rainfall happened in given set of places is difficult. But we deep down know that if we know certain things about rainfall in given regions we can make better predictions about other regions in a better way, because there are certain fundamental predictable laws which govern the rainfall.  

The GDP of the nations (if reported transparently) is also very difficult to pin down to exact number, we always have an estimate, because there are many factors which affect that final number, same goes for the population, we can only predict how it would grow but it is difficult to pin point the number.

These are all examples of real life data points which are generated randomly during natural activities, natural transactions. We know the reason for these numbers but as the factors involved are so many it is very difficult to find the pattern in this randomness.

I Lied – There is A Pattern in The Natural Randomness!

What if I told you that there is certain trend and reference to the randomness of the numbers generated “naturally”? Be cautious – I am not saying that I can predict the market trend of certain stocks; I am saying that the numbers generated in any natural processes have preference – the pattern is not predictive rather it only reveals when you have certain bunch of data already at hand – it is retrospective.

Even though it is retrospective, it can help us to identify what was manipulated, whether someone tried to tamper with the natural flow of the process, whether there was a mechanical/ instrument bias in data generation, whether there was any human bias in the data generation?

Logarithm and Newcomb

Simon Newcomb (1835-1909) a Canadian-American astronomer once realized that his colleagues are using the initial pages of log table more than the other pages. The starting pages of log tables were more soiled, used than the later pages.

Simon Newcomb

Log tables were instrumental in number crunching before the invention of any type of calculators. The log tables start with 10 and end in 99.

Newcomb felt that the people using log tables for their calculations have more 1’s in their datasets repetitively in early digits that is why the initial pages where the numbers start with 1 are used more. He also knew that the numbers used in such astronomical calculations are the numbers available naturally. These numbers are not generated out randomly, they signify certain quantities attributed to the things available in nature (like diameter of a planet, distance between stars, intensity of light, radius of curvature of certain planet’s orbit). These were not some “cooked up” numbers, even though they were random but they had natural reason to exist in a way.

He published an article about this but it went unnoticed as there was no way to justify this in a mathematical way. His publication lacked that mathematical rigor to justify his intuition.

Newcomb wrote:

“That the ten digits do not occur with equal frequency must be evident to anyone making much use of logarithmic tables, and noticing how much faster the first one wears out than the last ones.”   

On superficial inquiry, anyone would feel that this observation is biased. It seemed counterintuitive, also Newcomb just reported the observation and did not explain in detail why it would happen. So, this observation went underground with the flow of time.

Frank Benford and The Law of Anomalous Numbers

Question – for a big enough dataset, how frequently any number would appear in first place? What is the probability of numbers from 1 to 9 to be the leading digit in given dataset?

Intuitively, one would think that any number can happen to be in the leading place for given dataset. If the dataset becomes large enough, all nine numbers will have equal chance to be in first place.

Frank Benford during his tenure in General Electric as a physicist made same observation about the log table as did Newcomb before him. But this time Frank traced back the experiments and hence the datasets from these experiments for which the log table was used and also some other data sets from magazines. He compiled some 20,000 data points from completely unrelated experiments and found one unique pattern!

Frank Benford

He realized that even though our intuition says that any number from 1 to 9 could appear as the leading digit with equal chance, “natural data” does not accept that equal chance. The term “Natural data” refers to the data representing any quantifiable attribution of real phenomenon, object around us, it is not a random number created purposefully or mechanically; it has some origin in nature however random it may seem.

Frank Benford thus discovered an anomaly in natural datasets that their leading digit is more 1 or two than the remaining ones (3,4,5,6,7,8,9). In simple words, you will see 1 as leading digit more often in the natural datasets than the rest of the numbers. As we go on with other numbers the chances that other numbers will be frequent in leading position are very less.

In simple words, any naturally occurring entity will have more frequent 1’s in its leading digits that the rest numbers.

Here is the sample of the datasets Frank Benford used to find this pattern:

Dataset used by Frank Benford in his 1938 paper “The Law of Anomalous Numbers”

So, according to Benford’s observations for any given “natural dataset” the chance of 1 being the leading digit (the first digit of the number) is almost 30%. 30% of the digits in given natural dataset will start with 1 and as we go on the chances of other numbers to appear frequent drop drastically. Meaning that very few number in given natural data set will start with 7,8,9.

Thus, the statement of Benford’s law is given as:

The frequency of the first digit in a populations’ numbers decreases with the increasing value of the number in the first digit.

Simply explained, as we go on from 1 to 9 as first digit in given dataset, the possibility of their reappearance goes on reducing.

1 will be the most repeated as the first number then 2 will be frequent but not more than 1 and the frequency of reappearance will reduce and flatten out till 9. 9 will rarely be seen as the leading digit.

The reason why this behavior is called as Benford’s Law (and not Newcomb’s Law) is due to the mathematical equation that Benford established.

Where, P(d) is the probability that a number starts with digit d. Digit d could be anything 1,2,3,4,5,6,8 or 9.

If we see the real-life examples, you will instantly realize how counterintuitive this law is and still nature chooses to follow it.

Here are some examples:

I have also attached an excel sheet for complete datasets and to demonstrate how simply one can calculate and verify Benford’s law.

Population of countries in the world –

The dataset contains population of 234 regions in the world. And you will see that 1 appears the most as first digit in this dataset. Most of the population numbers start with 1 (70 times out of 234) and rarely with 9 (9 times out of 234)

Country-wise average precipitation –

The dataset contains average rainfall from 146 countries in the world. Again, same pattern emerges.

Country wise Gross Domestic Product –

The dataset contains 177 countries’ GDP in USD. See the probability yourself:

Country-wise CO2 emissions:

The data contains 177 entries

Country wise Covid cases:

Here is one more interesting example:

The quarterly revenue of Microsoft since its listing also shows pattern of Benford’s Law!

To generalize we can find the trend of all these data points by averaging as follows:

This is exactly how Benford avearaged his data points to establish a generalized equation.

Theoretical Benford fit is calculated using the Benford equation expressed earlier.

So here is the relationship graphically:

Now, you will appreciate the beauty of Benford’s law and despite seeming counterintuitive, it proves how seemingly random natural dataset has preferences.

Benford’s Law in Fraud Detection

In his 1938 paper “The Law of Anomalous Numbers” Frank Benford beautifully showed the pattern that natural datasets prefer but he did not identify any uses of this phenomena.

1970 – Hal Varian, a Professor in University of California Berkely School of Information explained that this law could be used to detect possible fraud in any presented socioeconomic information.

Hal Varian

1988 – Ted Hill, an American mathematician found out that people cannot cook up some numbers and still stick to the Benford’s Law.

Ted Hill

When people try to cook up some numbers in big data sets, they reflect certain biases to certain numbers, however random number they may put in the entries there is a reflection of their preference to certain numbers. Forensic accountants are well aware of this fact.    

The scene where Christian pinpoints the finance fraud [Warner Bros. – The Accountant (2016)]

1992 – Mark Nigrini, a South African chartered accountant published how Benford’s law could be used for fraud detection in his thesis.

Mark Nigrini

Benford’s Law is allowed as a proof to demonstrate accounts fraud in US courts at all levels and is also used internationally to prove finance frauds.

It is very important to point the human factor, psychological factor of a person who is committing such numbers fraud. People do not naturally assume that some digits occur more frequently while cooking up numbers. Even when we would start generating random numbers in our mind, our subconscious preference to certain numbers gives a pattern. Larger the data size more it will lean to Benford’s behavior and easier will be the fraud detection.

Now, I pose one question here!

If the fraudster understands that there is such thing like Benford’s Law, then wouldn’t he cook up numbers which seem to follow the Benford’s Law? (Don’t doubt my intentions, I am just like a cop thinking like thieves to anticipate their next move!!!)

So, the answer to this doubt is hopeful!

The data generated in account statements is so huge and has multiple magnitudes that it is very difficult for a human mind to cook up numbers artificially and evade from detection.

Also, forensic accountants have showed that Benford’s Law is a partially negative rule; this means that if the law is not followed then it is possible that the dataset was tampered/ manipulated but conversely if the data set fits exactly / snuggly with the Benford’s law then also there is a chance that the data was tampered. Someone made sure that the cooked-up data would fit the Benford’s Law to avoid doubts!

Limitations of Benford’s Law

You must appreciate that nature has its ways to prefer certain digits in its creations. Random numbers generated by computer do not follow Benford’s Law thereby showing their artificiality.

Wherever there is natural dataset, the Benford’ Law will hold true.

1961 – Roger Pinkham established one important observation for any natural dataset thereby Benford’s Law. Pinkham said that for any law to demonstrate the behavior of natural dataset, it must be independent of scale. Meaning that any law showing nature’s pattern must be scale invariant.

In really simple words, if I change the units of given natural dataset, the Benford law will still hold true. If given account transactions in US Dollars for which Benford’s Law is holding true, the same money expressed in Indian Rupees will still abide to the Benford’s Law. Converting Dollars to Rupees is scaling the dataset. That is exactly why Benford’s Law is really robust!

After understanding all these features of Benford’s Law, one must think it like a weapon which holds enormous power! So, let us have some clarity on where it fails.

  1. Benford’s Law is reflected in large datasets. Few entries in a data series will rarely show Benford’s Law. Not just large dataset but the bigger order of magnitude must also be there to be able to apply Benford’s Law effectively.
  2. The data must describe same object. Meaning that the dataset should be of one feature like debit only dataset, credit only dataset, number of unemployed people per 1000 people in population. Mixture of datapoints will not reflect fit to Benford’s Law.
  3. There should not be inherently defined upper and lower bound to the dataset. For example, 1 million datapoints of height of people will not follow Benford’s Law, because human heights do not vary drastically, very few people are exceptionally tall or short. This, also means that any dataset which follows Normal Distribution (Bell Curve behavior) will not follow Benford’s Law.
  4. The numbers should not be defined with certain conscious rules like mobile numbers which compulsorily start with 7,8, or 9; like number plates restricted 4, 8,12 digits only.
  5. Benford’s Law will never pinpoint where exactly fraud has happened. There will always be need for in depth investigation to locate the event and location of the fraud. Benford’s Law only ensures that the big picture is holding true.

Hence, the examples I presented earlier to show the beauty of Benford’s Law are purposefully selected to not have these limitations. These datasets have not bounds, the order of magnitude of data is big, range is really wide compared to the number of observations.     

Now, if I try to implement the Benford’s Law to the yearly revenue of Microsoft it reflects something like this:

Don’t freak out as the data does not fully stick to the Benford’s Law, rather notice that for the same time window if my number of datapoints are reduced, the dataset tends to deviate from Benford’ Law theoretically. Please also note that 1 is still appearing as the leading digit very frequently, so good news for MICROSOFT stock holders!!!

In same way, if you see the data points for global average temperatures (in Kelvin) country-wise it will not fit the Benford’s Law; because there is no drastic variation in average temperatures in any given region.

See there are 205 datapoints – big enough, but the temperatures are bound to a narrow range. Order of magnitude is small. Notice that it doesn’t matter if I express temperature in degree Celsius of in Kelvins as Benford’s Law is independent of scale.

Nature Builds Through Compounded Growth, Not Through Linear Growth!

Once you get the hold of Benford’s law, you will appreciate how nature decides its ways of working and creating. The Logarithmic law given by Frank Benford is a special case of compounded growth (formula of compound interest). Even though we are taught growth of numbers in a periodic and linear ways we are masked from the logarithmic nature of the reality. Frank Benford in the conclusion of his 1937 paper mentions that our perception of light, sound is always in logarithmic scale. (any sound engineer or any lighting engineer know this by default) The growth of human population, growth of bacteria, spread of Covid follow this exponential growth. The Fibonacci sequence is an exponential growth series which is observed to be at the heart of nature’s creation. That is why any artificial data set won’t fully stick to logarithmic growth behavior. (You can use this against machine warfare in future!) This also strengthens the belief that nature thinks in mathematics. Despite seemingly random chaos, it holds certain predictive pattern in its heart. Benford’s Law thus is an epitome of nature’s artistic ability to hold harmony in chaos!  

You can download this excel file to understand how Benford’s law can be validated in simple excel sheet:

References and further reading:

  1. Cover image – Wassily Kandinsky’s Yellow Point 1924
  2. The Law of Anomalous Numbers, Frank Benford, (1938), Proceedings of the American Philosophical Society
  3. On the Distribution of First Significant Digits, RS Pinkham (1961), The Annals of Mathematical Statistics
  4. What Is Benford’s Law? Why This Unexpected Pattern of Numbers Is Everywhere, Jack Murtagh, Scientific American
  5. Using Excel and Benford’s Law to detect fraud, J. Carlton Collins, CPA, Journal of Accountancy
  6. Benford’s Law, Adrian Jamain, DJ Hand, Maryse Bйeguin, (2001), Imperial College London
  7. data source – Microsoft revenue – stockanalysis.com
  8. data source – Population – worldometers.info
  9. data source – Covid cases – tradingeconomics.com
  10. data source – GDP- worldometers.info
  11. data source – CO2 emissions – worldometers.info
  12. data source – unemployment – tradingeconomics.com
  13. data source – temperature – tradingeconomics.com
  14. data source – precipitation – tradingeconomics.com