¶ FAST - Dynamic Classes
FAST is the custom development framework inside the KubeFlow Pipelines for RENergetic. It consists of the implementation of dynamic processes within the KubeFlow Pipelines, so that custom processes can be implemented without having to deal with managing the pipelines themselves.
This is a functionality for AI Experts, it is not recommended for all people working within the RENergetic system.
¶ Example
As an example we showcase the need for custom processes in the following time series:
This data is clearly unfit to train a forecasting models in its current state, so a custom processing process is necessary to transform it into:
This can now be used to train a model.
¶ Description
FAST is an acronym for the names of the clases:
- Forge: Class dedicated to process the time series.
- Augur: Class dedicated to train time series models.
- Surveil: Class to implement new metrics.
- Translate: Class to implement explainability features into the pipelines.
Within the pipelines there are steps that are executed if the name of one of the custom classes is added to it, otherwise no custom process will be executed. Within the README files there is documentation on the development of these classes and a how-to guide.
The classes are saved in the MinIO server, and they are stored within folders for each measurement-asset, allowing for different processes and models for the different measurements.
¶ How it works
How does the FAST System work within the pipelines? First, not all the classes are used in all three pipelines:
| Training | Forecasting | Monitoring | |
|---|---|---|---|
| Forge | Yes | Yes | No |
| Augur | Yes | Yes | No |
| Surveil | Yes | No | Yes |
| Translate | Yes | No | No |
¶ Forge
Forge allows custom processing of time series. In both training and forecasting pipelines is used in the process task and make forecast task.
If no forge processing is indicated in the arguments of the pipeline (list_forges) only the regular processing will be done, however if any number of forge classes are named, they will be downloaded for the MinIO Server, and try to execute the process code, if however an error happens in the execution, the run will continue as if no forge was added.
¶ Augur
Augur allows new algorithms to be used in the training and forecasting pipelines. For each measurement-asset, there are at most 5 different models been trained as shown in the image below (screen capture of the training pipeline):
There of those models are the ones that come standard within the RENergetic Core, Prophet, LSTM and Transformers. However, there are two more algorithms available to use for training. If there is an issue in training a model using one of the custom algorithms the procedure will continue as if no augur class was included.
In the forecasting pipeline the augur is use to make the predictions based on the model saved in the MinIO Server.
¶ Surveil
The surveil class allows to include new ways to measure how accurate a model is. In the following step after training (named in the previous picture as CheckSetModels), several metrics are used to compare different models. If any surveil classes are added to the arguments on the pipeline, then those metrics will be used as well.
Similar to this the surveil will be used in the monitoring pipeline while comparing the models.
¶ Translate
The translator class only appear in the training pipeline to add any explainability or any custom figures that may want to be created after training. It will happen in the Forecast-Model task.