I recently wrote a post suggesting why Excel users should learn Python. In a nutshell, Excel was originally conceived with a business world in mind which slowly ceases to exist. Ramesh Gopal posted a very elaborate and thoughtful comment raising some concerns. I will respond to Ramesh's critique below and thus open the stage for further discussions on this subject.
Ramesh: "Many consultants, IT experts and even risk managers have pointed out for decades about how fragile spreadsheets are, and the risks posed to businesses who rely on Excel (or spreadsheets in general), without adopting proper safeguards.
However, Excel is the ultimate business software tool. Its very weaknesses, which you clearly point out in your article, are also its enduring strengths. It will be near impossible to dislodge Excel from its current position as it fills an important niche or gap in enterprises as THE information tool which runs business workgroups, departments, and divisions, especially when corporate IT takes ages to deliver functionality end-users want."
I personally can't think of a business world without Excel and I admittedly am an avid user myself. Over the last few years, major enhancements have been made, and Excel today is a whole different tool compared to what it was a decade ago. However, we need to redefine: What is an appropriate use case for Excel? And what is not?
As the volumes of data are growing at a rapid pace, and data science is becoming an integral part of large business operations, we need to rethink how we collaborate on data, how we store, process it and ultimately, how we automate mundane, error-prone processes which are well known to Excel users.
Ramesh: "And trying to teach programming to existing workers isn't likely to succeed in a big way for several reasons."
I couldn't agree more if we were living in a more static, stable world. Given how entire industries are being reshaped at warp speed though, we have to readjust our priorities. Businesses start firing workers by the thousands who lack sufficient technology and programming skills. We might face mass extinction of white collar jobs over the next 10-15 years, which is an alarming outlook both for businesses as well as for their employees. We need to move, and we need to do so fast.
Ramesh: "One, not everyone has a brain wired to "get" code. Studies have shown that you need a certain type of abstract thinking capabilities to understand programming."
This is true but not an inevitable fate. I have been a right (creative) brain biased person for almost my entire professional life, and coding was a very uncomfortable exercise in the beginning. My subjective observation is that my brain has rewired itself over the last years due to huge learning efforts. I do not have any research to support this observation, but it seems to be probable that our brains have the capacity to rewire themselves.
Ramesh: "Two, those in operations who are taught to code will still find coding time-consuming and frustrating if the skill is not exercised day-in, day-out. Developers do this as part of their jobs, but operations people have real work to do, in addition to using Excel as a data and model repository."
No-code platforms such as Excel promise to give power users without any programming skills. This is certainly true if the use cases are simple in nature. However, as they become more complex, using Excel becomes extremely time-consuming, tedious and complicated. Excel comes with four different languages: regular function language, M-Language for data cleansing, DAX as the Pivot Table calculation engine and VBA. Most of the work performed by knowledge workers with Excel cannot be reproduced, shared, and automated thus causing enormous labor costs due constant efforts to "reinvent the wheel".
Apart from that, I find business users to have an irrational fear of programming. I started providing Python crash courses which aim to tame the fear of programming. Since my offering is relatively new, I have not enough data points to generalize my observations. However, the vast majority of my course participants are willing to explore Python and consider it a viable alternative to Excel.
Ramesh: "Three, daily work pressures and deadlines also mean that enterprises aren't likely to send their personnel for programming training en masse."
Good point. This certainly needs to be addressed at the c-level, as a lot of these work pressures stem from constantly reinventing the wheels - in part, probably, for job security reasons. The harsh truth might likely be that by transitioning from Excel to Python a company could potentially get more done with less people.
Ramesh: "However, the future is bright for the coming generation. They're likely learning programming in school and such fundamental skills and concepts, learnt at a young age, will typically persist when they enter the workforce."
I am hoping that workers in their 30s or 40s (just like myself) get a fair chance to prepare themselves for future job market requirements. Millennials (aka coming generation) lack the experience of the aforementioned workers, and I am hoping for both to coexist. However, a great level of flexibility will be required primarily from the senior staff.
Ramesh: "So what is called for today is pragmatism, and figuring out how to co-exist with spreadsheets, while trying to minimize the damage done by spreadsheets gone bad."
Agreed. We need to redefine what Excel and other spreadsheet tools are good for and for which scenarios we have to replace them by Python in conjunction with Data Science PaaS solutions.
Thanks for taking the discussion further.
Some disclosure. I have used spreadsheets in a work context for over 27 years (all the major spreadsheet programs), and I have used Python for some 18 years. I am proficient with both spreadsheets and Python, but am by no means an expert in either. However, I continue to use both technologies for fun and for profit.
As for requiring a brain wired for code, I am heartened by your personal persistence and success, and I would also like to share the source of my earlier assertion.
Anyone who has the time can study the work of Dr Saeed Dehnadi from Middlesex University London here.
Dr Dehnadi studies what separates programmers from non-programmers, and if the ability to code can be tested. He asserts that learning to code is difficult, and the ability can be reliably tested for. As with all academic studies, I am sure there will be supporters and detractors, so anyone reading the work has to draw their own conclusions.
So for Python to succeed in becoming ubiquitous in the workplace, coding will need to become second nature, just as elementary school arithmetic is for most working adults. I would love to see more adults trained in coding, and with determination, anyone can do so today with the abundance of free materials on the Internet.
You also pondered about the appropriate use cases for Excel.
I'd wager that for most people, Excel offers end-users something developers love: rapid prototyping. It enables ordinary people to quickly put some data into cells, and link those cells together into a model with formulae. The practical ability to store small amounts of data, solve business computations (without having to call in IT), and reach decisions, coupled with the satisfaction that comes from the end result, likely results in a rush which perpetuates Excel's place in business.
I know this as I often create models in Excel, and then move to code once I get the model working.
However, your critique of Excel when used for more complicated tasks is correct. Having to master different languages and paradigms would mean greater complexity for the spreadsheet user. In this respect, Excel is a victim of its own success. Newer technologies and features have been constantly bolted on, all while retaining backward compatibility with existing skills and spreadsheets.
Time permitting in future, I will share some methods in which some spreadsheet problems can be tamed using non-Excel-specific data wrangling and coding skills.
Anyone who has the time can study the work of Dr Saeed Dehnadi from Middlesex University London here.
Thank you for sharing. I took the test, failed, and out of sheer frustration, I took a close look into every single document provided on the website. I noticed some severe inconsistencies across those documents. As you see in my overview with screenshots taken from various documents on the website: if you take the test (marked with the number 1 in the green circle) and you intuitively apply the correct mental model (marked with 2) you are either correct (3a) or wrong (3b) depending what document you look into.
The values in each of those documents don't match. I am the type of person who "codes in anger" when running into a seemingly insurmountable computational problem, and then I keep pushing until I solve the problem or - worst case - I ultimately surrender to it. So I felt when taking the test (and then later taking it apart). I had this roller-coaster of: "Am I wrong? Stupid?" Followed by: "Maybe I am not that stupid, but there's something wrong with the test?" As it turned out, my answers were correct in a sense that I consistently applied a mental model adequate for programming tasks, but the materials on the website seemingly had not been revised prior to release.
Which gets me to the point I am trying to make: "There are no poor students, only bad teachers."
First, I can empathize with students who would even fail to understand what they are required to do in this test. In my view, the instructions are not outlined explicitly enough, and thus not clear to everyone. Secondly, the assumption that coding is per se hard and difficult is simply wrong in my view. Most programming experts fail to explain the secrets of their trade in a way that is understandable to novices. Being a subject matter expert and having the capacity to communicate with non-experts are two entirely different skills. Many novices to programming (including myself back five years ago) apply the wrong mental model to programming, which is why they fail in the first place. They might apply mental models that are suitable for entirely different tasks such as: communicating with people, negotiating or selling. But they don't work in conjunction with coding.
Garry Kasparov once said something along the lines of: "You not only need to know how to solve a given problem but more importantly what kind of thinking you apply to do so."
As a kid, I was poor at math because (as I discovered decades later as an adult) I was never taught the underlying mental models. It was assumed by my teachers that I somehow bring the right models to the table, my school put no efforts whatsoever into teaching those. Same goes with chemistry, biology, and physics. I am guessing that a lot of people made similar experiences. I fell in love with coding and math in my 40s, but only once I realized I have to focus on teaching myself mental models in the first place. My affair with biology, chemistry, and physics hasn't even fully started yet.
To wrap things up, I am hoping for the educational system to change in such a way that it would become more inclusive. Instead of just assuming that someone's brain is not "wired" for coding, math or science, everyone should get a fair chance to acquire the proper underlying mental models. This would spare kids lots of pain and suffering.
I firmly believe that the hopes are not lost for professionals in their 30s and 40s either who need to ramp up their skills for the future job market requiring profound technical skills across entire enterprises. We need to rethink how we teach.
I can't comment on the test as Saeed does now point people to Richard Bornat's page for an updated version of the test. I took the test a long time ago, and didn't check the answers.
You mentioned about "Most programming experts fail to explain the secrets of their trade in a way that is understandable to novices."
I don't know about other programmers, but for me, programming has always just "clicked" after some study and practice, and I've little ability to teach how I do what I do to someone else.
But you're right about mental models. Industrial-age education systems expect that students are already equipped with certain mental models, and very often it has to be just one "right" model. Such an education delivery system doesn't cater for diversity in learning styles.
Maybe this is where AI can truly shine in future - with an ability to take standard academic content and deliver it in a way that suits the student's learning style, coupled with the appropriate mental model. One-to-many teaching systems today which are either instructor-led or via self-study courses can't provide this.
I came across this piece on Quora recently, and thought it might answer a bit about programming experts not sharing their secrets in a way that is understandable to novices.