Select and integrate the relevant data from various data sources for the ML task. Catalogue and organise the data in a way that allows for simple and fast analysis.

• Ad-hoc data ingestion: Ingest data manually when there is an update on the original data source
• Data pipeline: A set of actions have been created to ingest raw data from disparate source and move the data to destination
• Realtime ingestion: Data collection process fully automated and raw data gets ingested whenever there’s an update

Validate the input data fed into the pipeline. Check descriptive statistics, inferred schema and data anomalies to reduce errors in data.

• Manual checks: Manually implement different checks to validate the input data
• Data validation tests: A structured and defined process to check the input data
• Realtime data triaging: Input data is validated automatically using a pre-defined method when it is put into the system.

Add and construct additional variables or features to the dataset to improve machine learning model performance and accuracy.

• Ad-hoc: Add/create feature data to the machine learning model from time to time as required
• Feature store: An ML-specific data system that runs data pipelines that transform raw data into feature values, and stores/manages the feature data itself
• Automated: New features automatically created from the dataset

Multiple experiments on model training can be executed before making the decision what model will be promoted to production.

• Improvised trials: Implement one-off or randomised experimentation
• Catalogued trials: Implement different sets of experimentation
• Reproduceable pipelines: A pipeline that can be executed repeatedly to experiment on model training using structured and pre-defined method

Implement different algorithms with prepared data to train various ML models.

• Siloed training: An unstructured and one-off way of training the model
• Scalable pipelines: A sets of method have been defined in the pipeline to train the model
• Continuous training: Automatically and continuously retrain the model to adapt to changes that might occur in the data
  

Confirm the model is adequate for deployment andthat its predictive performance is better than a certain baseline.

• Offline validation: Model validated manually offline
• Model validation suite: A set of model validation tests have been defined to perform the task
• Automated validation tests: An automated process to perform the validation test before the model being deployed

The pipeline and its components are built, tested, and packaged when
new code is committed or pushed to the source code repository.

• Manual build and test: Manually build and test the pipeline when new code being committed and pushed
• Automated build and test: The pipeline gets updated automatically when new code is committed and pushed
• Continuous integration: Merging all developers’ working copies into a shared repository several times a day

Deploy ML models to Cloud infrastructure and expose the API, which
enables other workers to consume and use the model. It is the
engineering task of exposing an ML model to real use.

• Manual deployment: Manually deploy the model when required
• Automated deployment: Deployment process gets automated so we can execute promptly when required
• Continuous deployment: Every change that passes the automated tests
  is deployed to production automatically, making changes that are visible to the software users

Monitor the model’s predictive performance metrics (throughput,
uptime, etc.) to potentially prompt a new iteration in the ML process.

• Manual inventory: Manually collect the model predictive performance metrics
• Model registry: A centralised model stores that stores model lineage, versioning and other configuration information.
• Model monitoring suite: A centralised monitoring system that tracks
  the performance of ML models in production and collects/stores
  performance metrics to identify potential issues

Collect feedback information to the model on how well it is performing.

• Ad-hoc collection: Collect feedback when required
• Feedback data stored: Feedback gets collected and stored
• Feedback loop to pipeline: Feedback gets collected systematically and looped into pipeline for further improvements

Rate of data refresh depends on how quickly the objective/observations change, where more frequent intervention needs to be applied to the data collection layer and the validation process when raw data gets refreshed often.

Materiality refers to the impact a potential AI application would have on the business. Higher materiality leaves less room for mistakes and the need for a quicker response time to address the issue.

Model governance refers to the business processes that govern model deployment and how heavily regulated an industry is. For example, a Financial Services organisation may need more rigorous model governance structures in place to adhere to local, national and international financial authority regulations.

Feedback latency is the delay between the input, the model prediction and the outcome feedback where one needs to consider how costly a delay in this communication is to the business or decision process. More efficient operation processes are needed if low latency is required.