Soft wear, tear, repair and resurgence: A troika of perspectives on the changing face of software

Software is invariably accorded a sanctified place of pride by the community at large. Professionals of this sphere are often granted demi god status irrespective of the depth and significance of the roles they perform as part of their daily grind.  For all you know, a person may be handling spooling jobs or testing an inconsequential part of some peripheral code, but he or she is invariably revered as an integral part of an enlightened species. Back home in India, it is common to encounter outlandish newspaper headlines like “Techie among seven killed in bus accident” or “Techie leads protests against dowry deaths” which ludicrously place this tribe above the ‘lesser mortals’ from other walks of life. No wonder, this haloed status has translated into soaring stock prices of most eligible tech brides and grooms in the marriage market, a fallout of a predictable chain of events: education and career in software, onsite visits, coveted green card, relocation to the West, and a bagful of “East coast to West coast”  tales smacking of  vanity and aimed at posterity.

 

Given the distended environs, meaningful, matter of fact public domain conversations on any aspect of software, more so its essence and significance, are always in short supply. Precisely why the delight of discovering and interacting with unassuming mavericks, whether at home and abroad, is beyond words. This blogpost is dedicated to all of them wherever they are across the globe. Here is a fictional ‘love story tribute’ to their selfless exploits as ace developers, solution architects, database experts, or networking wizards.  

Moving over to the men of the moment, three thought leader-practitioners, each carrying loads of insights, devoid of any claims and prescriptions to that effect. Do note that the questions are freewheeling in nature, and so are the answers. There is nothing black, white or grey about them. 

 

Dmitry Vostokov (DV): well known software diagnostics expert, educator, scientist, inventor, speaker, and author. More about him here. Dmitry cherishes the theoretical foundations of neural networks he studied at the Izhevsk State Technical University, as also the diploma evaluated as MSc in Software Engineering in Ireland. 

 

Jose Crespo (JC): seasoned software and hardware Engineer specializing in neurocomputation, compilers, interpreters, OS, networking protocols, cybersecurity, binary analysis, and embedded systems. Currently the tech lead at Cryptofisher, Jose is focused on new financial digital markets and cryptography. Jose not only knows his subject well, he is also a prolific writer and a non-conformist of sharp intellect and intuition.  

 

Luca Bellonda (LB): this self-taught professional hailing from the auto and biomedical hub of Modena, Italy got a first-hand experience of the mechanical, hydraulic, and electronic insights that go into the making of medical devices in his first corporate engagement. This was followed by a stint with a startup into diagnostic offerings for the healthcare world. Currently, Luca is a developer with a digital setup handling customer communication across different formats. An admirer of towering physicists Margherita Hack and Tullio Regge, Luca unassumingly believes that he is a software artisan at best, not the thought leader that he is!

Here goes!

 

 

 

In your reckoning, which are the programming languages that are still a timeless influence on modern-day software principles and practice? 

 

 

DV: 

 

C in procedural programming, and C++ in object-oriented programming despite some lag.  It is definitely Haskell in functional programming. I haven't heard much from logic programming in recent times. 

 

JC:    

 

In my view, a proficient software engineer who aims to transcend the ephemeral trends and the often destructive, frivolous noise of the current IT private sector should possess a deep understanding of assembly language. This perspective is shared by prominent figures like Donald Knuth, who emphasize the importance of connecting your algorithms and data structures directly with the underlying architecture, be it based on the Von Neumann or Harvard model, without any third-party abstractions. This approach offers pure freedom and power in terms of both computer science and software engineering. Conversely, and perhaps paradoxically, functional programming, particularly Haskell, represents another marvel of computer science and software engineering. Contrary to the widespread practices in the IT industry, Haskell teaches us that a software program is, as Donald Knuth suggests, subject to the same mathematical principles as proving a mathematical statement. The advantage of Haskell is that its compiler can catch many errors at compile time, outperforming many other languages, especially those from the Object-Oriented Programming (OOP) realm. These languages often have flawed type systems and mutable state management, leading to systematic bugs, security vulnerabilities, and the chaotic state of the current IT industry.

 

LB:

I started with assembly, Basic and LISP. Assembly and LISP have never really left us. C, Rust, Java, C#, Fortran, Pascal or imperative languages are derived from Assembler, all functional languages (Scala, Clojure, Haskell…) owe something to LISP. They have never really left us, only transformed and adapted to today's world. It is indeed a paradox that mathematically inclined paradigms such as functional programming force people to think like computers, but make complex operations easier to handle.

Conversely, which languages and paradigms do you feel are now well and truly redundant? Or there is nothing called redundancy in software progression? 

 

DV: 

 

In my opinion, it is the BASIC language. But no paradigms are redundant so far. 

 

JC: 

 

Object-Oriented Programming (OOP) has lost significant ground to functional and even declarative programming, both of which are grounded in solid mathematical principles such as abstract algebra, group and category theories, and set theory with relational and lambda calculus. OOP lacks a comparable foundation; it relies on design patterns, which can easily turn into anti-patterns with slight modifications. It's somewhat ludicrous that some patterns once considered important are now seen as anti-patterns or unnecessarily convoluted, such as singletons, visitors, or double dispatching. In this context, Java, despite incorporating some functional features from version 8, remains a problematic language. It's verbose, rigid with its flawed OOP paradigm, and confusingly mixed with declarative and functional features while maintaining its broken type system and mutable nature. Even JavaScript, with the addition of TypeScript, is a better language than Java. C++ is another behemoth that has become ludicrously bloated and even metastasized with a convoluted mess of unnecessary nuances, which often contradict other features of the language. For instance, smart pointers can easily turn into dumb pointers when dealing with exceptions in memory allocation, temporary object construction, or programmer-defined deleters. Like Java, the C++ committee has attempted to modernize the language by expanding the generic template stuff and introducing functional features, all while maintaining backward compatibility and the mutable OOP beast. It's akin to squaring the circle.

 

LB: 

Each language or paradigm moves from an idea, context, and a will to make an impact, so the question should be what context is no more valid today? What is fundamental is the context. Basic was born to start programming in a world of low resource computing, and this context is now lost on us. Fortran, instead, was a professional programming language focused on efficiency, and this context is still valid. It may be redundant, but it is useful, and useful programs never die. Generally, all paradigms are valid today, you need only to find the right context.

 

From a personal point of view, which stint of yours was most fruitful from the software development perspective? 

 

DV: 

 

Honestly, all my rewrite work from scratch has been most rewarding. I have done it twice. The first time I was hired to work on an editor/word processor, which was behind deadlines, and when I saw the procedural code, I sensed the need to scrap it and started using object-oriented principles. I intuitively used similar design patterns as in the GoF book, but I only learned about it 5 years later. The second rewrite happened when I was hired to extend a Visual Basic program and improve its performance. I rewrote in modern C++. Both projects were a success. 

 

JC: 

 

My recent adoption of Rust while working with the Zephyr Real-Time Operating System (RTOS) for embedded software development has been a truly rewarding discovery. Although Rust is still evolving, its Algebraic Data Types (ADTs) and functional nature allow it to approach the realm of rich mapping morphisms, which can eliminate convoluted exceptions and null pointer vulnerabilities inherent in the system's type.

 

LB: 

I have had the opportunity to work for very large and very small companies. In small companies you can experiment and learn. I have the opportunity to participate in a period in which the road was not traced and many roads could be traveled. In big companies, you help others do their job and you learn a lot by doing it. I really don't know what was the most fruitful period.

 

 Hype and noise come ahead of purpose and passion when it comes to the software development market. Do you agree? If yes, why is it so? 

 

DV: 

 

Agree and do not agree - all depends on the company, project, team, personalities - there's a lot of variance. I personally haven't experienced hype and noise yet; however, social media wants me to believe that it is everywhere. 

 

JC:    

 

We are living in a post-capitalist world where cronyism between large corporations, central banks, and governments plays a significant role in determining the stock value of these very corporations. Consequently, a paradigmatic IT company today is driven primarily by short-term profit. However, this value is not determined by innovative engineering achievements in a market eagerly seeking them, but by the value assigned in the stock market, which is influenced by policymakers' decisions on factors like interest rates, government debt, new regulations, and the greed of stock investors.

 

LB:  

If you want to get someone to adopt a specialized tool like a programming language, you have to win their heart first, so hype is inevitable, in the same way that vanity moves human actions. Software is about change and in creating new technologies that are supposed to solve all problems, hype is king. Being one of the underdogs using "old-fashioned ways" of development isn't cool, and in big companies nobody wants to be the only one using a certain language or framework. That said, when the context changes, you know you need to change. Quoting Bruce Sterling: “If you want to know the market, you have to know the hearts of the people, and for that you need a good poet”.

How do you look at the evolution of programming paradigms over the years? Which paradigms do you feel were the most path-breaking – procedural, structured, functional, logic-driven, oops, concurrent, and event-based? 

 

DV: 

 

I think it all depends on personal history and environment. For me, it was Message/Event based, then OOP, and much later FP. 

 

JC:    

 

The real question here is whether we have a way to create predictable, failure-free, and secure software. The answer, while not original, is an old one that has been affirmed by Donald Knuth, Edsger Dijkstra, and others: a software program can be proven correct by treating it as a mathematical object. This can be done either by viewing it as a series of orderly steps in an algorithm, as Knuth suggests, or as part of functional mapping and morphism following the laws and rules of abstract algebra and category theory, as Dijkstra proposes. These form the solid mathematical foundations for imperative and functional programming, respectively.

 

 

LB: 

Now distributed and networked computing is the norm. When software sizes are scaled many times over, languages face this problem. If you work for very low resource embedded systems, you need to have very fine control over the system. It's like being a mountain farmer or a farmer on the plains, the environment dictates your work. The context has today expanded to distributed computing, microservices, serverless functions and so on; it is like an organism that goes from being unicellular to multicellular.

Ideally, given that all major programming languages are Turing complete, the same program can be shown to different people as translated into different paradigms, or each person can use his or her own custom language, but this invariably leads to incommunicability with other developers, like it happened to mathematicians when they coined their own symbology.

Is a multiparadigm approach the best way to make software actionable and fulfilling? 

 

DV: 

 

Yes, I think it is. Anything goes, as philosopher Paul Feyerabend said. 

 

JC: 

 

The most effective paradigm, if we can call it that, is one that allows us to treat our program as a mathematical object whose correctness can be proven at compile time. From this perspective, imperative, declarative, and functional paradigms are indeed valid and complementary approaches to software development, and they can be effectively embodied in programming languages. On the other hand, Object-Oriented Programming (OOP) falls outside this category. It may be useful for initial architectural communication (such as UML), initial stages of a project (defining layers, interfaces between layers, and program flow), and for implementation only in the realm of GUIs and some types of exotic mutable data structures. For the rest, it would be best to avoid it entirely.

LB: 

Programming is like interacting with people, so multiparadigm is the best approach, but if you use it you lose your soul, you no longer have a defined path to follow. You have to decide at all times which paradigm to use, and other developers have a hard time reading your sources. You actually have to use a paradigm in the exact context. Each paradigm exists in its own context. Languages that have ambitions to be "general purpose" tend to include multi-paradigm constructs, like Java and C++ lately. Every paradigm is born with precise needs, but by changing the context, the paradigm’s utility undergoes a change.

Are Microservices a refined version of Service Oriented Architecture? Has the API economy proved a game changer or is it a security nightmare? 

 

DV: 

 

I'm a newcomer to microservices, but I recall SOA and XML from the 2010s, so there is some continuity here from component-based development. Also, for me, API is mostly Windows or Linux API - other API is what Visual Basic is for real programmers :-) But I understand the challenges there. Even Visual Basic required a book called Hardcore Visual Basic, which I read and absorbed. I programmed and debugged a lot in VB and even made small corrections to PHP code when I needed. 

 

JC: 

 

Microservices architecture, while currently in vogue, is not a one-size-fits-all solution and should not be adopted without careful consideration. It is a complex engineering approach that is only suitable for a limited set of use cases and is typically affordable only for large-scale companies. The adoption of microservices should not be driven by trends or the practices of industry giants like Netflix, but by a thorough analysis of the specific needs and resources of the organization.

 

Unfortunately, the current state of the IT market is characterized by a tension between the burden of legacy systems and the desire for innovation. However, this desire for change should be guided by scientifically-based evidence and a deep understanding of the potential limitations and challenges of new approaches, rather than by marketing strategies or industry trends.

 

The alternative to microservices is quite straightforward: a systems approach that divides the software into independent functional components. These components communicate through low-level interfaces for multiplexing and demultiplexing code and data. While this approach may not have the 'cool' factor that appeals to marketing and HR departments, it is both affordable and suitable for the majority of companies. It offers a practical and cost-effective solution that aligns with the needs and resources of most organizations, rather than following industry trends that may not be applicable or beneficial.

 

Adopting a systems approach, where software is divided into independent functional components that communicate through low-level interfaces for multiplexing and demultiplexing code and data, can offer significant cybersecurity advantages. One of the key advantages is the ease of interfacing a cybersecurity component with the rest of the system. Since the communication between components is limited to multiplexing and demultiplexing operations, the cybersecurity component can monitor and control these operations more effectively. This can lead to more robust security, as it can detect and prevent unauthorized or malicious activities at the interface level.

 

In contrast, in a microservices architecture, each service is typically exposed over a network, increasing the attack surface and making it more challenging to secure. Each microservice may require its own security measures, leading to increased complexity and potential inconsistencies in security implementation.

 

Furthermore, the systems approach can lead to better encapsulation and isolation of components, reducing the potential impact of a security breach. If one component is compromised, the impact can be contained within that component, preventing it from spreading to the rest of the system.

 

In summary, while microservices offer certain advantages in terms of scalability and flexibility, a systems approach can provide superior cybersecurity by simplifying security implementation, reducing the attack surface, and better containing the impact of security breaches.

 

LB: 

Microservices enable teams to work in parallel in a complex environment. It is one of the tools to allow the architecture to follow the desired evolutionary path of the system, I quote Mr. Tornhill here. The user is obviously not interested in the internal architecture of a system. Usually when you build an API system you can count on security components and a lot of automation, but you always need to manage them. APIs are a necessary step for services, but the complexity is inevitable.

In this era of AI and ML, which programming languages rule the roost – is it a Python Vs. R battle, or do you feel the market is wide open? 

 

DV: 

 

I feel Python is the leader. And I recently started liking Python after going thoroughly over the language and its internals. When everything is first-class, then you have a "democratic" language. 

 

JC: 

 

Machine Learning and Artificial Intelligence, particularly Artificial Neural Networks, are essentially applications of statistics and differential calculus to fit data and find local minima and maxima within a given solution space. In this context, a solid understanding of statistical analysis, linear algebra, and vector calculus is crucial. The choice of tool, whether it's R, Python, Matlab, or even Perl, is secondary and depends on the availability of libraries and personal preferences.

 

LB: 

Python is the language of choice when experimenting, you can use it for everything, if you need raw performance, just use the right libraries. I just studied R some time ago, but I never used it seriously enough to be able to endorse it. In AI, I am an amateur. R seems more like a niche product to me, but I could be wrong.

Your thoughts on the Rust vs Clojure debate? 

 

DV: 

 

I like Rust from FP and type-oriented perspective. From the memory safety perspective, I can use modern C++ safely, and I coded tens of thousands of lines never running a debugger. I learn Clojure as well, but mainly from having some interesting books that use it or use LISP/Scheme. They are so different languages, and their main environments are so different. 

 

JC: 

 

I have mixed opinions about Clojure due to its halfway design of the functional paradigm. For instance, it employs a dynamic type system, which can be seen as a drawback when compared to languages with strong and complete type systems. These systems, found in languages like Haskell and PureScript, provide a robust foundation for implementing functional programming mappings and morphisms such as monads and functors. As a result, Clojure is a muddled functional language with a parody of functional constructs. In the context of web development, PureScript, for example, offers a better alternative to Clojure. 

 

It is a purely functional language that transpiles to JavaScript, allowing developers to write high-level code while targeting the ubiquitous JavaScript runtime. Another better alternative is Scala. Like Clojure, it is compatible with the JVM and offers a blend of object-oriented and functional programming paradigms. It also features a strong type system, which can be beneficial for large codebases and complex applications. 

 

F#, a functional-first language on the .NET platform, is another viable alternative, offering a blend of functional and object-oriented programming features. Rust, while primarily a systems programming language, also allows for a functional programming style. It emphasizes speed, memory safety, and parallelism, making it an excellent choice for performance-critical applications. 

 

Rust's support for WebAssembly is noteworthy, enabling more performant web applications than what might be achievable with Clojure. Although Rust operates in a different domain, its ability to express functional constructs like monads and functors makes it clearly better than the half-witted Clojure. 

 

In conclusion, Clojure has been a good attempt to escape the verbose nightmare of Java dominance with concepts from the functional world, but most of them are wrongly taken. Other languages like PureScript, Scala, F#, and Rust offer much more compelling features and capabilities that make them worthy of consideration, depending on the specific requirements of the project.   

One question on Hadoop. Is it dead, as most voices claim, or is it still relevant? 

 

DV: 

 

I only have theoretical knowledge of Hadoop. Every book on big data I started reading in the past mentioned it. I never worked with it, though. 

 

JC: 

 

I will talk overall for Java; it is becoming legacy code for the enterprise realm, much like COBOL was for banks. In the realm of distributed systems, we have more cost-effective, performant, and suitable solutions such as container technology based on Go and cloud data platforms. Some of these platforms, like Azure, support Java technologies like Spark or Databricks, but remain under the control of a different cloud distributed technology. Java still has its niches, such as Software as a Service (SaaS) and Continuous Integration/Continuous Deployment (CI/CD), as exemplified by Jenkins. However, its original domain under the motto 'write once, run anywhere' has lost its relevance with the advent of containerization technology.

 

LB: 

I'm not an expert in this field, so I can't answer. Some of its components like Zookeeper are shining their own light in the new era.

What are your thoughts on NoSQL and polyglot persistence – the pros and cons for Big data analysis? 

 

DV: 

 

This is the same as the multiparadigm programming approach. Anything goes as soon as it fits the purpose. NoSQL fits naturally for diagnostic analysis. This is why I chose it for one of my books for a case of diagnostic analysis document management. 

 

JC: 

 

NoSQL became a trend when Facebook published its use cases, primarily for handling image and video persistence and searching. Similar to the microservices hype driven by Netflix, many companies jumped on the NoSQL bandwagon without fully understanding its limitations. NoSQL databases do not provide ACID guarantees and lack the solid mathematical foundation of set theory and relational calculus that underpin SQL databases. As a result, many companies found that NoSQL only offered a slight advantage in a limited set of use cases, leading to widespread disappointment.

 

Interestingly, Oracle learned from this trend and incorporated similar features into its distributed SQL database. This situation exemplifies a recurring issue in the IT industry: the influence of hype and buzzwords, often driven by marketing and HR departments, which can lead to misguided business decisions. This trend-centric approach has repeatedly led to crises and financial losses since the early 2000s, as companies invest heavily in trends that do not deliver the expected benefits.

 

This statement emphasizes the need for careful consideration and evidence-based decision making in the adoption of new technologies, such as NoSQL databases.

 

LB: 

The relational database was created for the exact representation and optimization of data and their transactions to the detriment of the horizontal scalability of the system. Scaling and distributing work is what we're doing now.

Efforts to put all data into relational databases make them a bottleneck for the entire application. In fact, there is no single type of NoSql database, each database is chosen for its function in a given context (a word that appears often, isn't?).

------

We end the thought piece with tributes to two pioneers: Barbara Liskov and  Mario Veronesi. 

Summing up the former’s influence as also her pathbreaking Liskov Substitution Principle, Dmitry says, “I was advised in earlier 1990s to read her book Abstraction and Specification in Program Development (which also introduced CLU language) as a way to break from the procedural style. I think it paved the way for me to understand OOD/P and abstraction in UML later. Thinking abstractly when doing software design is the main influence, I believe! LSP is a cornerstone of OOP and TOP type-oriented programming. Here's my definition (the usual one is about substitution): you can treat a reference (or a pointer) to an object of superclass (type) as a reference to an object of subclass (subtype) provided you have a sound class hierarchy: objects of subclasses behave the same as objects of superclasses.”

About the latter, Luca provides an enlightening snapshot. Late Mario Nino Veronesi is fondly remembered as the father of biomedical district of Mirandola in Modena province. A pharmacist and innovator par excellence, he was working on the development of home dialysis during his last entrepreneurial stint. A dialysis patient for the last seven years of his life, he fought a 15 year long battle with nephroangiosclerosis, a progressive disorder of chronic renal insufficiency.  

 

 

"Take away understanding of the problem, and the context in which it operates, and you are left with a cutting edge system that does nothing!"

Usually we find two tribes of thought leaders circulating in the market; one category has experts with great proficiency in the sphere of their choice but great believers of the self-defeating concept of power distance; this tribe typically has a machinery of pseudo marketing and social media agencies working overtime on their behalf to grow their fanbase and followers. The other category has leaders who are highly accessible and incredibly humble, but cursed with a superficial hold on the subject matter, which is what they try and compensate through polite and proactive communication with the community at large.

 

Hearteningly, there’s a rare category of mavericks who do not crave market attention, who are extremely democratic in their communication with the world at large, and who have achieved great length, width and depth in their disciplines and vocations. No wonder, this tribe is a rare and endangered species but hearteningly, the number is more than what we would instinctively like to believe. We have the pleasure and privilege to meet one of them in a Q & A session: Bhaskar Roy, renowned expert in AI, data science, and sustainability and revered Startup Mentor and Advisor.

 

Talking to Roy, I am instinctively reminded of legendary filmmaker Satyajit Ray’s summary of his first interaction with French Avant Garde director Jean Renoir when Ray was yet to become the Ray we all know and rave about: “Renoir was not only approachable, but so embarrassingly polite and modest that I felt if I were not too careful I would find myself discoursing upon the Future of the Cinema for his benefit.” Just replace cinema with Data Science and the same sentiment applies in toto!

 

 Excepts of my conversation with him, presented as a Q&A interaction for ease of readability and comprehension.    

 

Broadly speaking, in your reckoning, where are data-driven insights taking us, and what are they fetching us across different spheres. 

 

That’s a great question to start with, Sudhir! At its most basic level, data-driven insights are taking us into a new era of decision-making, where businesses and organizations can make more informed choices based on real-world data; no wonder, the world beyond is unraveling at a very fast pace with new tools and technologies surfacing on the shores with astounding regularity. This pace of development is something that’s not been seen for quite some time, most certainly an inflexion point for envisioning how and where the world of AI can lead us. By having access to data and insights, businesses can make better decisions about everything from marketing campaigns to product development. 

 

Data-driven decisioning brings us several concrete benefits across different aspects of businesses. Few examples worth a mention include:

 

Efficiencies: Data-driven insights help businesses identify areas where they can improve efficiency, such as by streamlining processes or identifying areas for cost savings.

 

Customer satisfaction: By understanding customer behavior and preferences, businesses can tailor their products and services to meet exact needs and even surpass them, which in turn boosts customer satisfaction and loyalty.

 

New product development: actionable data can be used to identify new product opportunities, as also to test and refine new products before they are launched.

 

Innovation: Data can be used to identify new ways of doing things and to develop new products and services.

 

Reduce risk: By understanding potential risks, businesses can take steps to mitigate them.

 

Better resource utilization: By understanding how resources are being used, businesses can make more efficient use of them.

 

Stay ahead of the competition: By using data to identify trends and opportunities, businesses can stay ahead of their competitors.

 

These benefits are seen across industries like Healthcare (improving patient care), Finance (better investment decisions), Retail (personalized shopping experience), etc., even though the maturity of each industry, and organization within it, differs!

 

As AI moves from general intelligence to super intelligence, will human intervention eventually become a disposable commodity? 

 

Let’s take a step back to frame the question better. The maturity/sophistication of AI systems is typically classified into 3 broad parts – 

 

Artificial Narrow Intelligence – learning algorithm is designed to perform narrowly defined tasks. Specialization is in one area, solving one problem. Most typical uses cases today fall here, for e.g. image and facial recognition systems, voice assistants, machine’s playing different games like Chess, Go, etc.

            

Artificial General Intelligence – here, machine can be made to think and function as a human mind, especially when faced with an unknown task. AGI is typically assessed on parameters like Common Sense, Context awareness, Transfer Learning ability, and causality, but we are still some time away from it.

            

Artificial Super Intelligence – this is where machine consciousness comes into play, and the machine becomes more like an intellect that is much smarter than the best human brains!

 

Having defined the different phases of AI sophistication, let’s go back to the intent of your original question – where is this all leading us, and what role do humans have in it!

 

As we look at the current state, we have done reasonably well on ANI, with some more years to go before we get to AGI (although the whole GPT revolution is helping give it a significant push). In the coming times I expect a lot of the mundane to get automated, to start freeing up more and more of the human brain to focus on creative tasks.

 

With that said, even for the automated tasks, human-in-the-loop is going to be the right way to go about things. This is because a key part missing for machines is the context and feelings in which decisions are being made (getting towards sentience, if I may call it that!), and ability to take learnings from one situation to another. This is going to continue for some time to come, and I fully believe that the current wave of AI is only going to take human creativity and ingenuity further.

 

At what level of maturity is the industry in countering typical ML challenges In your experience, how is the quality of entry level data science professionals in India?

 

Interesting question and pertinent too! In my limited understanding, the art of data science (Machine Learning being one of the arrows in the data scientists quiver) has lost its way a little bit.

 

We have a situation where far too many people calling themselves ‘data scientists’ have no clue of what they are doing. They might know the right libraries in python (or their programming language of choice), know how to import and execute them, but have no clue of how to interpret results that the system spits out. Many a times they don’t even want to massage the data first to understand what it’s all about, clearly a situation far from ideal and moving towards being unsustainable.

 

For a young data scientist starting out, it’s very important to understand the nuances of what they are doing. They should be able to understand the problem/ issue they are trying to address, and the context in which their final output will get used by end stakeholders (essentially elements of problem solving). It is only when you comprehend this, that the data comes in. It’s important to massage the data to understand what its all about, before starting to do anything else with it.

 

This is what then enables one to answer all the questions you had around understanding reliability of input data, sanctity of the test and control data considered, and over-fitting.

 

 

 

Will it be a good idea of institutionalizing a robust mechanism across all domains where probabilistic AI results are systemically validated in the light of intuition and discretion, to counter the perils of ‘over-obliging’ algorithms.

 

A short answer to the question is – Yes, it’s certainly a good idea as it would help to ensure that AI systems are not making decisions that are harmful or unfair, thereby reducing and/ or eliminating inherent bias.

 

This is typically done by establishing a Responsible/ Ethical AI framework, driven by multi-skilled team of experts in AI, Ethics and Law, anchored on basic principles of Transparency (no black box solutions), Fairness (equitable and inclusive treatment), Explainability (tracing back of decision factors) and Human-centricity (understand and solving for biases).

 

Of course, there are also some challenges to institutionalizing a robust mechanism for validating probabilistic AI results. One challenge is that it can be difficult to define what constitutes the different tenets of Responsible/ Ethical AI. Additionally, it can be time-consuming and expensive to validate AI results in this way. However, I believe that the benefits of institutionalizing a robust mechanism for validating probabilistic AI results outweigh the challenges.

 

Is it right to say that predictive and prescriptive analytics are likely to draw maximum mileage from ML models in terms of making intelligence accessible and actionable compared to the descriptive and diagnostic analytics?

 

Yes, I think it would be right to say that about predictive and prescriptive analytics, but a lot depends on maturity of the organization and industry where the solutions are being deployed. 

 

At a high level, ML models can be used to identify patterns in data that would be difficult or impossible for humans to find, and they can be used to generate predictions and recommendations that can help businesses make better decisions; for instance, ML models can be used to predict customer churn, identify fraud, or optimize supply chains.

 

Some specific examples of how ML models are being used in predictive and prescriptive analytics:

 

Customer churn prediction: ML models can be used to predict which customers are most likely to churn, so that businesses can take steps to prevent them from doing so. For example, Netflix uses ML models to predict which customers are likely to cancel their subscriptions, and then sends them targeted offers to try to keep them on board.

 

Fraud detection: ML models can be used to detect fraudulent activity, such as credit card fraud or insurance fraud. For example, banks use ML models to identify suspicious transactions, and then investigate them further to see if they are fraudulent.

 

Supply chain optimization: ML models can be used to optimize supply chains, so that businesses can reduce costs and improve efficiency. For example, Amazon uses ML models to predict demand for products, so that they can order the right amount of inventory and avoid stockouts.

 

One thing to note here is the level of maturity of the organization where these solutions are being deployed. For maximum efficacy, and impact, such organizations already have majority of their data issues (like data availability, quality, etc.) sorted, they have already done some level of Descriptive and Diagnostic Analytics (essentially understanding what has already transpired, and why it went the way it did!) for some period with decent adoption, and are now trying to get ahead of the game.

 

Currently, every other AI company seems to be recruiting prompt engineers. Notwithstanding the criticality of right prompts in helping AI models generate accurate content, is there an element of fad in the current deluge of recruitment?

 

An excellent question, Sudhir. There indeed is a lot of interest in Prompt engineering as a field. A lot of it is driven from the fact that mass market Generative AI technologies are still nascent (for e.g. ChatGPT), and still need to evolve to a stage where features that let you co-pilot with them become more widespread. They have begun, but will need more time to mature.

 

This basically leads to a situation where, in the current context, to drive optimal value from these tools, a whole deal of manual effort needs to be put in to get the desired output from these systems, and that’s what Prompt Engineers help us do.

 

Having said that, I feel it will be a short-lived phenomenon. On the other hand, the time-tested value of people with the right combination of business understanding and ability to solve problems will continue to hold good for many years to come.

 

Which sectors are likely to lead the AI evolution and progression in your reckoning, the first among equals so to say? 

 

It’s a little tricky to answer this question. AI has been a buzz word for many years now, but it has impacted different industries in myriad different ways. If we look at the traditional B2C industries like Banking, Telecom or Retail, AI has typically been used across the board to drive customer marketing efforts, optimization of logistics, supply chain, customer targeting, etc. This, in my view, has happened because data is available in humungous volumes and velocity in these industries.

 

Next in my mind has been the development over the last 5-7 years where significant amount of data has become available in the manufacturing related industries, emanating from sensors embedded in all kinds of machines (from the biggest of the biggest to the tiniest ones we carry in our hands/ pockets. This first led to a unique problem where removing noise from the data was crucial before you could let the magic of AI-ML unfold and drive business value. This value is going to continue accruing for some time to come and will manifest itself it much smarter machines.

 

Next, I expect industries like Pharma and Healthcare to also start seeing significant value realization from AI. These have typically been industries where consolidation across different types of data, namely structured (e.g. patient details), semi-structured/ unstructured (e.g. doctors’ notes combined with diagnostic reports OR different types of image data) has been a key issue. Some of these are challenges are getting resolved and the infusion of AI are making processes more efficient and even amazing for everyone involved!

 

Do you see some scope for reviving Mathematics as a stream of choice towards creating a pool of potential AI resources in the time to come? 

 

Great question again, and something that’s very close to my heart!

 

I have a strong bias (!) towards Mathematics, and related fields in the furtherance of a pool of people that are trained on AI. These are the core building blocks of what enables people to have the skills to create not just new kinds of AI, but also to enhance human intelligence and ingenuity. The right understanding of mathematics and related fields provides people the faculties to not only create new AI systems, but also understand what’s happening within AI systems (whether created by them or others) and where changes might be needed to move towards desired outcomes.

 

Knowledge of AI alone won’t suffice. Along with it needs to be inculcated the ability to problem solve. This is extremely crucial as even the best of AI systems exist to solve for specific problems, within the specific context of the industry, vertical, business or government, where it’s being implemented. Take away understanding of the problem, and the context in which it operates, and you are left with a cutting edge system that does nothing!

 

Is it right to say that Python is the reigning language of choice for data science? How are R, Scala, Matlab, Hive, Pig and the like faring? How would the language front evolve going forward?

 

Sudhir, while I’m no expert on this, I can say for sure that there is no one horse for all the races. Python has served the purposes well so far, given its flexibility and ability to adapt to changing tech environments as it lends itself well to integration efforts.

       

Having said that, tools like R, Matlab and their ilk also have very specific application areas in which they work well. For instance, R, before the advent of Python, was the programming language of choice in academia for those looking to have flexibility in data manipulation and analysis, but it has decreased in popularity over time.

 

Other languages like Scala, Hive, and Pig have evolved to address gaps in existing systems and languages for specific purposes. I have a feeling that these will also continue to co-exist with other languages, albeit for very specific purposes.

 

As an aside, what are your views on the open source software movement? Do you see enough mavericks around you who are keen to look at the holistic picture, who are not overtly obsessed with serving the proprietary tribes of coveted brands - both in product and service spaces.

 

Again a tough one to foretell, especially in the backdrop of the OpenAI movement being bought into by large corporates, who are now trying to monetize their investments into it! I have a feeling that there are enough and more ‘mavericks’ who want to do this, but end of the day it all needs either philanthropists to sustain, or for the value to be monetized. Each has their own interest groups looking to make a buck!

 

Could you briefly share the ‘what’, ‘when’, and ‘how’ of the patents you hold – viz. Booster centric resource allocation and Resource allocation based on retail incident information?

 

Sure Sudhir! Let me paint a scenario for you – think about visiting a store. Browsing through the aisles, we have all seen folks shoplifting in different forms – people outright stealing from the store, or those that switch labels from low priced items onto those that are higher priced, and many others. Consequent losses for retailers come to anywhere in the range of 1-2% of revenue!

 

What we don’t realize is that a large proportion of this activity actually is pretty organized in large global economies and is formally termed as Organized Retail Crime. These group go stealing from store to store, and have pre-identified target stores, timings, etc.

       

Now, let’s introduce an element of Karamchand Jasoos in the mix, with the carrot in hand, or Mr. Sherlock Holmes for the natively uninitiated. They start by looking at commonalities, and seeing things play out on maps, and patterns start to emerge. Now, once you have caught onto the trends, you can actually be one step ahead of these shoplifting groups and can predict to a large certainty where and when they are expected to be seen next, enabling their eventual exposé and arrest.

 

This at a high level is what I and my team had done for a large Retail Chain!

 

Could you share few notes on your personal voyage : education, career, and passion pursuits.

 

My grounding in left-brain dominated thinking started off with a Bachelor’s in Statistics, before being balanced out with a MA in Economics, and providing me with a perfectly created, but coincidentally arrived at, recipe of getting started in the nascent field of Analytics/ Data Science (it wasn’t even called DS then!).

 

I’ve worked with a few organizations through my career, both on the service provider side, and with in-house teams at large global corporations, with lots of learnings along the way from great bosses and mentors, that I’m eternally thankful for!

 

I truly want to have an impact on the world around me, in small but significant ways. Besides my day job, I’m associated with multiple educational institutions in different capacities as advisor to students, mentoring student start-up ideas, technical assessments and the like. These give me a great deal of satisfaction and reinforce the feeling that the true extent of human ingenuity continues to develop and will unfold in ways we can’t even begin to fathom!

 

Any parting thoughts?

 

I’ve covered most aspects. Only thing I will re-emphasize – human creativity and ingenuity is foremost. It’s us, Humans, who have created AI and have the power to continue moulding it to serve our purposes. We must stay in the loop and have to stay ahead of the game to lead the world forward with our unique creativity and ingenuity!