Author: Yuxiang Lin, Senior Solution Architect at Midships, yuxiang.lin@midships.io
This article is for anyone looking to leverage the power of AI and ML to solve problems within their organisations.
In recent years, Artificial Intelligence (AI) and Machine Learning (ML) have emerged as prominent forces within the dynamic technology landscape. Organisations across various industries are embracing this transformative wave, aiming to leverage its power to unlock unprecedented opportunities and innovations. However, the path to effectively adopt AI and ML remains unclear to some, with the risk of reducing these terms to mere buzzwords. This article aims to provide insights on how organisations can translate the immense potential of AI and ML into tangible value. It's important to note that AI and ML are not the solution to everything. To fully utilise their benefits, organisations need to embark on a carefully planned journey to reach their desired outcomes.
It Is Not Magic
Artificial Intelligence (AI) is a broad field of study that focuses on making machines intelligent. Within AI, Machine Learning (ML) is a subfield that's been the focus of academic and industrial interest for over 80 years. Hence, AI and ML are not new concepts. Within ML, the subfield of deep learning, characterised by neural network architecture, started in 1986 and this is where GenAI and Large Language Models (LLM) fall within.
Fundamentally, ML has introduced to the world a new way to build software systems. Instead of creating logic and rules to handle data, ML uses data to establish these rules and logic. However, the logic and rules created by ML models through training differ from those created by human intelligence. A trained ML model is a product of intelligence derived from large volumes of data and therefore the rules and logic it entails are statistical in nature.
It’s not the kind of intelligence you are thinking
It's essential for one to understand that at its root, ML are algorithms designed for making statistical predictions based on given inputs, also known as features. The parameters regulating these algorithms are computed through iterative training to reduce the model’s loss/cost function using available data.
Understanding the nature of machine learning (ML) helps us to see its limitations and challenges. Firstly, mainstream ML is best suited for problems that can be modelled statistically or where feedback loops can be easily simulated. This means that the belief in AI solutionism, where it assumes that given enough data, ML algorithms can solve any problem, is far from reality.
It's important to recognise that causality-based logic, typically used by human intelligence to solve problems, is often lacking in ML models due to their statistical nature. This implies that when making decisions using ML outcomes, one must be careful when justifying those decisions based on causality.
Moreover, advanced learning models often lack transparency and interpretability. This makes it challenging to identify specific modifications for desired behaviours and raises concerns when it's important to explain how a system produces results in a given context.
It can take a while
Secondly, the effort and time required to solve a problem using Machine Learning (ML) are typically harder to determine compared to other software development. This is primarily due to the non-deterministic nature of ML. Training ML models to achieve satisfactory performance involves experimentation and intuition.
When organization faces challenges in addressing ML issues such as over-fitting (where the model cannot make generalised predictions with new inputs) and under-fitting (where the model is highly biased with training data), they will have to go through cycles of feature engineering, data collection, model training, and evaluation, this could easily lead the organization down a rabbit hole.
It‘s all about the data
Thirdly, the performance of a machine learning (ML) model in solving a problem largely depends on the data it was trained on. Essentially, a ML model is only as good as the data it receives. Data availability and quality can often become significant obstacles for many organisations trying to harness the power of AI and ML. Issues such as data privacy, potential bias, and discrimination in the collected data are also crucial considerations. Therefore, before embracing AI and ML, one must first embrace data.
It takes much more to go production
Finally, machine learning (ML) and artificial intelligence (AI) do not operate independently. These systems, like any other software, require thoughtful architecture and engineering considerations. Topics such as scaling, resiliency, monitoring, continuous integration and continuous deployment (CICD), API security, and infrastructure must be addressed before any proof-of-concept can progress towards production. Given the complexity and the nature of ML, a mature and robust DevOps pipeline tailored to the needs of a ML pipeline is a key success factor in an organization's journey to embrace AI and ML.
It Is A Tool Not A Solution
In the previous section, we discussed machine learning (ML), a subfield of artificial intelligence (AI). This section will outline our recommendations for how organisations should approach AI and ML when addressing real business challenges.
It’s about the problem
Some may say that ML and AI have refined the way we approach problems. This is true to some extent, they have given us tools to tackle problems previously deemed technically too challenging to address. However, real business problems are often more complex than a single technical hurdle.
While designing a solution for a complex problem, it's crucial to let the problem drive the solution, not the other way around. AI and ML are tools, not solutions. To effectively address a problem, one needs to understand the problem and its context, define requirements, break down the problem into smaller sub-problems and finding appropriate approaches to address these sub-problems. So, organisations must not rush into using AI and ML to solve a problem before carrying out the essential solutioning processes. This avoids creating a solution that merely serves as buzzwords and fails to address the problem effectively.
It’s like cooking
Crafting a solution is akin to cooking; there are various approaches and tools that can be used as ingredients. However, there's no one-size-fits-all recipe. A proficient solution engineer must consider all aspects, including the timeline, budget, contextual limitations and constraints. The key is to choose the right ingredient, not necessarily the most extravagant one. This is a decision that the solution engineer, acting as the chef, must make. Sometimes AI and ML will be the main ingredients, other times they'll serve as seasonings, and sometimes they simply won't be the right ingredients for the dish.
It’s not a safe haven
Organisations and solution engineers should not view AI and ML as a safe haven. While AI and ML are powerful tools, they are not a panacea for avoiding complexity in the solutioning process. It can be tempting to turn to AI and ML when faced with a complicated problem where AI and ML use cases can be applied, thereby forgoing the more traditional human intelligence-based logic and rule approach. However, AI and ML also have their own challenges and limitations. Sometimes, stepping back may reveal new possibilities that could prove to be better options.
It Has Potential But How To Create Value
In the final section of this article, we will provide recommendations on how organisations can transform AI and ML's potential into actual business value. While AI and ML are not the solution to everything, they are crucial tools that can unlock innovations, providing organisations with a competitive edge. In this section we will use 3 examples to illustrate how organisations can use AI and ML in different problem contexts.
It can be used for optimisation
In manufacturing, machinery parameters such as vibration and temperature are continuously monitored. A common challenge can be scheduling maintenance to prevent production impact while minimising maintenance costs.
A simple solution might be rotating equipment on a fixed schedule to minimise operation faults. However, this approach does not necessarily minimise costs.
Given that data are collected that can predict machinery health, we can improve this simple solution by shifting from a fixed to a dynamic schedule. We can implement a maintenance scheduling service that uses a Machine Learning (ML) model trained with historical data. This model predicts when maintenance is likely needed based on the current data of each piece of equipment.
The maintenance scheduling service will be designed with logic that uses these predicted timings and the planned working capacity to create optimised schedules, thus reducing costs.
It can compliment other solutions
For financial institutions, detecting fraud is essential for customer protection. While deep learning models might seem attractive for detecting abnormalities, they pose challenges such as real-time evaluation performance and flexibility in adjusting behaviours. An alternative solution is to create a rule and logic-based engine that identifies defined anomalies. Although this alternative resolves many issues, it lacks the ability to find new patterns as fraud techniques evolve. Therefore, we can improve this solution by using machine learning techniques to do offline clustering and classification of tracked data. This will uncover new rules and patterns that can be added to the fraud detection engine, allowing its rule and logic profile to evolve.
It can help to save time
Organisations frequently maintain various internal documents. Searching for relevant documents and extracting necessary information for specific tasks can be a tedious and time-consuming process. With the current technology stacks around LLM, the creation of an internal knowledge base has become much simpler. This enables the extraction of organization-specific information in a precise and relevant manner. Such an approach can be an efficient solution to alleviate some of the daily pain points within the organization.
It does not need to revolutionary
As a final note of this article, the journey for an organization to embrace AI and ML is not an overnight one. Rather than viewing AI and ML as all-encompassing solutions, start by building a strong foundation. Begin small with data exploration and visualisation to gain insights. Then, apply ML models to address specific problems. Over time, the value and benefits of AI and ML will naturally become apparent. Take the first step into this exciting field, and see what unfolds.
I hope you enjoyed my first blog with Midships. Please reach out to me at yuxiang.lin@midships.io if you would like to learn more
Yuxiang