cybench.evaluation package

Submodules

cybench.evaluation.eval module

cybench.evaluation.eval.evaluate_model(model: BaseModel, dataset: Dataset, metrics=('normalized_rmse', 'mape', 'r2'))

Evaluate the performance of a model using specified metrics.

Parameters:

  • model – The trained model to be evaluated.

  • dataset – Dataset.

  • metrics – List of metrics to calculate.

Returns:

A dictionary containing the calculated metrics.

cybench.evaluation.eval.evaluate_predictions(y_true: ndarray, y_pred: ndarray, metrics=('normalized_rmse', 'mape', 'r2'))

Evaluate predictions using specified metrics.

Parameters:

  • y_true (numpy.ndarray) – True labels for evaluation.

  • y_pred (numpy.ndarray) – Predicted values.

  • metrics – List of metrics to calculate.

Returns:

A dictionary containing the calculated metrics.

cybench.evaluation.eval.get_default_metrics()
cybench.evaluation.eval.mape(y_true: ndarray, y_pred: ndarray)

Calculate Mean Absolute Percentage Error (MAPE). Note that in the provided implementation using scikit-learn, there is an absence of multiplication by 100

Args: - y_true (numpy.ndarray): True values. - y_pred (numpy.ndarray): Predicted values.

Returns: - float: Mean Absolute Percentage Error.

cybench.evaluation.eval.metric(func)

Decorator to mark functions as metrics

cybench.evaluation.eval.normalized_rmse(y_true: ndarray, y_pred: ndarray)

Calculate the normalized Root Mean Squared Error (RMSE) between true and predicted values.

Parameters:

  • y_true (numpy.ndarray) – True values.

  • y_pred (numpy.ndarray) – Predicted values.

Returns:

Normalized RMSE value as a percentage.

Return type:

float

cybench.evaluation.eval.r2(y_true: ndarray, y_pred: ndarray)

Calculate coefficient of determination (R2).

Args: - y_true (numpy.ndarray): True values. - y_pred (numpy.ndarray): Predicted values.

Returns: - float: r2.

cybench.evaluation.example_log_experiment module

cybench.evaluation.log_experiments module

cybench.evaluation.log_experiments.comet_wrapper(model: BaseModel, comet_experiment: Experiment | None = None, comet_api_key: str | None = None) → Experiment

Wrap model before training with a Comet experiment instance

Returns:

Comet_ml experiment instance

cybench.evaluation.log_experiments.existing_comet(comet_experiment: Experiment, comet_api_key: str | None = None, project_name: str | None = None, workspace: str | None = None) → Experiment
If api_key is not defined, comet_ml will start in anonymous mode and the logs can be accessed

through the URL provided in the console

Returns:

comet Experiment object

cybench.evaluation.log_experiments.get_comet_api_key(file=None) → str

Function that returns api_key.

Store the api key under evaluation/api_keys/…

Param:

file: file name that contains one line of the comet api key.

cybench.evaluation.log_experiments.log_benchmark_to_comet(results_dict: dict, model_name: str, run_name: str, params: dict | None = None, comet_experiment: Experiment | None = None, comet_api_key: str | None = None, end: bool = True) → None

Function to log benchmark results to comet

cybench.evaluation.log_experiments.log_benchmark_to_comet_post_hoc(run_name: str, model_name: str, params: dict | None = None, comet_experiment: Experiment | None = None, comet_api_key: str | None = None, end: bool = True) → None

Log benchmark results to comet after the training run from saved files

cybench.evaluation.log_experiments.log_to_comet_post_hoc(metrics: dict, params: dict, comet_experiment: Experiment | None = None, comet_api_key: str | None = None, name: str | None = None, model: BaseModel | None = None, asset_path: str | None = None, end: bool = False) → None

Log metrics, params, asset and model to Comet_ml

Parameters:

  • comet_experiment – comet experiment object

  • metrics – dict of metrics

  • params – dict of params/hyperparams

  • model – name of the saved model

  • name – name of the comet experiment

  • asset_path – path to asset. To log a custom asset (e.g., config file etc.) to Comet.

Returns:

None

Module contents