Tag: Jake Coulter

Can we extend our experiences beyond the limitations of our senses?

A picture is worth thousand words, they say. You try to explain something to your friends and they get confused then you show them the photo/ video of the same thing and the need to explain things disappears. Visual media is literally ruling over all other types of media. This also highlights our dominant way of consuming information, any information. Hence, we can definitely say that vision is the most important of all the senses we use to interact with our surroundings. Important thing to understand is that we can perceive only certain range of light obviously known as the visible spectrum. What if we could extend this sense of vision?

Extending human vision

There are already good attempts for us to see in ultraviolet and infrared region which lie on the both sides of visible spectrum. Special optical filters and thermal sensors can create visuals in ultraviolet and infrared which normal human vision cannot see. In a way, we have achieved UV, IR vision which can increase our understanding about surrounding. This story goes good for people with normal vision. But how does a blind person can understand this highly visual world? Many of us would have experienced and will agree that the lack of a sense for a long time elevates the other senses in such differently-abled persons which helps them to perceive same object with different attributes of senses. We will see one such interesting story of a person who is not completely blind but the way in which he solved the problems in his vision have created a totally different understanding about our ideas on vision and our senses.  

Neil Harbisson – the cyborg with antenna

Neil Harbisson has a condition called ‘Achromatopsia’ also known as ‘Total Color Blindness’ where he cannot perceive the color of objects. The whole world is like black and white television for him. There are some types of spectacles available in market for partially color-blind people which effectively help them to distinguish different colors but there is no such solution for people with total color blindness. Neil solved this problem in a different way.

The problem of perception of different colors of objects was solved by converting the light into the sound. This simply means that certain sound frequency is assigned to certain color. When a digital camera scans such color, it will create auditory feedback which can be heard by the ears of the person. Now it becomes a learning for the color-blind person to differentiate between different sounds to understand the respective color. With the help of Adam Montadon, a computer scientist, Neil developed an antenna which is physically connected to his head – the rear part of skull – occipital bone. This antenna was further upgraded by Peter Kese and Matias Lizana. The early model of this eyeborg developed for Neil was like a headset with camera and later on it evolved to an antenna, reduced computer weight to computer integrated to a chip. Now, Neil has one tentacle-like antenna over his head.

The “eyeborg” and its feedback

Not only the design and evolution of eyeborg is fascinating but the feedback it provides is very interesting. As we know already that the different frequencies of color are converted to different sound frequencies. Whenever the camera in the antenna is pointed to any object, it gives sound feedback to Neil which enables him to distinguish and experience the color in a different way.

In the early version of eyeborg the sound was fed to Neil through the headphones to his ears. This blocked the ambient sounds for Neil making him unaware of the surroundings. It was like he sacrificed the ambient sound to understand the colors of surrounding objects. The smart update Neil did was to integrate the auditory feedback of eyeborg directly through his skull bones. This eliminated the over the ear feedback of eyeborg thereby freeing his ears for ambient sound. Now Neil hears ambient sound through his ears and the sound for color feedback is heard through bone conduction! Everyone can understand the difference between bone conducted sound and the ambient sound actually. When you hear your voice through a recording you always hate it (maybe this doesn’t happen with good singers!) because it sounds so weird and creepy. This happens because you have always heard your voice through bone conduction and others hear your voice as an ambient sound.

This doesn’t stop here. Neil was unable to understand the saturation in the color in the early versions so he assigned loudness of the frequency to the saturation of the color. The antenna containing camera serves as a Bluetooth/ Internet connection so that he can receive and transfer data to other devices.

Hearing music from a painting

Neuroplasticity of our brain has allowed Neil to understand the world in a different manner. The sonochromatic scale developed to understand the colors through the sound has elevated Neil’s understanding about colors to next level. First obvious thing is that his brain perceives the surrounding colors in terms of different sounds; this allows his brain to associate certain sound to certain objects. For normal humans when we say bus, we generally picture a red bus, when one says an airplane, we generally picture a white airplane, when someone says an ocean, we generally picture a blue ocean. Interestingly, Neil’s brain has developed itself to associate the objects to certain sounds which creates different synaptic feedback in his mind. His perception for general objects is totally different than all the people who can experience color.

We should understand how deeply colors are rooted into our fundamental understanding of nature. And when this understanding is changed to feedback of different sense which is a sound in Neil’s case the experiences are totally different and otherworldly. Even his dreams are filled of sounds for different colors, this is how deeply the ‘hybrid sense’ has been developed in Neil.

NASA released some images from James Webb Space Telescope where they assigned certain sounds to certain colors and tones to stars in the image. When tracked in a pattern you can experience the image in terms of sound which is on crude way of understanding the image through sound (crude way as in it is nowhere closer to the actual sound of the location, it is just conversion of image data to sound date with one to one replacement)

The colors from a song      

Turns out, that the reverse of feedback from the eyeborg is equally interesting. The process of association of sound to a color has gifted Neil to understand the sounds in different way. Though the initial purpose was to comprehend the color through different sensorial feedback, it has given Neil a different type of intellect. Now, when he listens to a song, sound or music he can picture some series of colors. In simple words, he can create a visual and colorful output of an audio array. Though we visualize the sounds or music through complicated waveform, they were never associated to colors to the scale that Neil has experienced before. According to Neil’s experiences now he has a color scheme for certain music pieces, world famous speeches, musical symphonies.

Neil’s Eyeborg and Synesthesia

Actually, there is one human condition which is much closer to the experiences of Neil. There is one condition where a person associates the feedback of a sense to a totally different sensorial experience. The feedback from a sense stimulates some different type of experience. For example, for people with a synesthesia, they may associate certain color to certain number, some people associate the sound to certain color. This stimulation of totally different cognitive pathway by the experience of a cognitive pathway is called as synesthesia. Synesthesia is a natural and rare condition. Neil has artificially developed this condition in him to heighten his awareness of the colors. There are different types of synesthesia based on the association of cognitive stimuli.  

In Grapheme–color synesthesia person associated a color for a letter as in A will stimulate the color red in his mind, in Spatial sequence synesthesia people associate events to certain stimuli so that they have exact memory of events as in with exact date and time, in  Auditory–tactile synesthesia people experience touch or sensation in certain body parts when they hear certain sounds, in Ordinal linguistic personification synesthesia the person associates some objects, sequences like number sequences, calendar months to certain people or genders,  in Misophonia synesthesia the person invokes certain emotions like anger, happiness, fear when certain sounds are heard, in Mirror-touch synesthesia the person are said to have heightened empathy so heightened that they experience the touches to the person in front of them as if someone has touched them actually, in Lexical–gustatory synesthesia the person experiences certain taste upon hearing certain words.

Chromesthesia is the synesthesia which deserves separate explanation for our discussion. In Chromesthesia, the person assigns a color to the sound they hear. Which exactly what is happening in the case of Neil but artificially. According to art historians the famous artist Vincent Van Gogh has   Chromesthesia which made his paintings so vibrant. According to one such comment Vincent tried to learn Piano but left it midway because the notes of piano invoked different colors in his mind thereby overwhelming him.

Jack Coulter – one of the artists of our generation has Chromesthesia. He is known for his abstract art pieces which reflect the way he interprets the sounds and music. In simple words, he paints the songs. Jack’s paintings of are such an experience. You can find his paintings on some famous songs on his official Instagram page which includes ‘Love of My Life’ by Harry Styles, ‘The Best Day’ by Taylor Swift, ‘Running Up That Hill’ by Kate Bush, ‘Don’t Tell Me’ by Avril Lavigne, ‘Take me Home Country Roads’ by John Denver, ‘As It Was’ by Harry Styles. He even has created an art piece for the world famous ‘Cornfield Chase’ soundtrack from Christopher Nolan’s Interstellar.     

A mix-tape of senses and the hidden reality – philosophical implications

This crisscross of our senses and the stimulus actually questions the nature of our reality. If we question the nature of evolution by question like why can only eyes see color and why can only ears hear the sound which are very fundamental philosophical questions (so fundamental that the person who poses them may be called a mental person), then we will understand that it is the nature of light to impart the color to the object, it is the nature of sound to impart the vibration of the object which developed such organs in a way.

But if we dig deeper and question the limitations of our perception of sound and light then we will realize that what we perceive or feel through our senses is just a sample or small piece of all the experiences presented by the reality. For simple example, though sound (mechanical) and light (electromagnetic) are all but vibrations, the extent of their experience or the ranges in which they exist or the things they are made up of cause them to collapse into two distinct sensorial experiences.

And when we are trying to resolve such questions about the senses, the conditions like Synesthesia or the cyborg movement created by people like Neil try to modify our understanding of the reality that we live in. If these people can interpret the colors into sounds and sounds into colors, are our senses limiting us from a different type of awareness? Are we missing some extra information from reality due the limitation of our senses? In simple words, even though there are many colors around us we cannot see in dark but most of the animals have awareness of infra-red spectrum or night vision which gives them extra information of the same surrounding where we all coexist. Bees, butterflies, insects can detect ultraviolet light to understand the pollen part of the flowers which we cannot see by or normal vision. This questions the reality and our experiences from it

For a no person with complete awareness of his/her senses, the cognitive stimulus and feedback is almost hardwired. This stimulus and feedback are what create reality for us. We cannot experience even a common thing without experiencing the interactions of our senses with the surrounding. And when we realize that there are other ways to experience the same reality by connecting it to totally different sense and stimulus, this opens a new chapter in our understanding of the nature. It’s like being aware of the ultrasonic sounds as same as owls experience it. This opens a totally different universe for all of us – the hidden one which was already there. Maybe this is the sixth sense many people talk about. Actually, our lives and the experiences that we have, are always being dictated by the five senses that we experience.

Bertrand Russel had made some attempts to crack such question in his world-famous book called ‘The Problems of Philosophy’. In the very first chapter called “Appearance and Reality” of this book Russel concludes that this can be a question which may remain unsolved forever. He uses a concept called ‘sense-data’ as in the information which we immediately know from our senses as in smell, color, sound. Russel clarifies based on a structured thought process that what we perceive as a reality is mere the what it appears to us based on the senses we experience (also known as sense-data). If our sense-data gets limited our understanding of reality will be limited. As the reality we experienced through our limited senses will be a reality for us, it will not be the complete reality as in “the reality” – “the ultimate/absolute reality”. (I know this gets confusing from hereon) And if the reality through our senses is not complete reality that how could we transcend to the realization of “the ultimate/absolute reality” by extending our senses? Or is it a completely different pathway? Russel leaves that idea to the power, need and importance of philosophy.

There is definitely much more information that we are missing only because of the limitation of our senses. So, the reality is not we experience through our senses only, it is more than that which maybe is impossible for us to grasp for now. Maybe in near or distant future we will be able to understand reality in different way like Neil, Jack and many people like them.

P.S. – Most of the artists have mastered this transcendence of senses in their own ways, which is what makes art so special for humanity. Maybe it is the only way we have right now to extend our senses.

References:

1. “A picture is worth a thousand words” – image by ElizabethHudy from flickr
2. Neil Harbisson: I listen to color – TED
3. Neil Harbisson – Image by Don Walton
4. Jack Coulter’s ‘Mr. Bad Guy’ painting -Photograph by Richard Gray for Freddie Mercury Estate
5. jackcoulter.com/
6. cyborgfoundation.com/