The telemetry dataset contains synthetic event logs for a variety of product types, with the following attributes:
| Attribute | Description |
|---|---|
| Product Type | A categorical attribute indicating the type of product. Possible values include A, B, C, D, E, F, and Others. |
| Event Type | The type of event logged. Possible values are: open, close, save, reset, and error. |
| Time of Event | A timestamp indicating when the event occurred, with values ranging from May 1, 2024, to July 31, 2024. |
| User ID | Anonymized ID of users who produced these events while using each product. |
The goal of the analysis is to identify product types with error rates significantly higher than the average error rate across all product types. Here's how this is done:
Error Count per Product Type:
For each product type $ P $, the total number of error events is computed:
$$
\text{ErrorCount}_P = \sum_{i=1}^n \mathbf{1}\{ \text{EventType}_i = \text{error} \text{ and } \text{ProductType}_i = P \}
$$
where $ \mathbf{1} $ is the indicator function that equals 1 if the condition is true and 0 otherwise.
Total Event Count per Product Type: The total number of events logged for each product type $ P $ is given by: $$ \text{TotalCount}_P = \sum_{i=1}^n \mathbf{1}\{\text{ProductType}_i = P \} $$
Error Rate per Product Type: The error rate for each product type $ P $ is computed as: $$ \text{ErrorRate}_P = \frac{\text{ErrorCount}_P}{\text{TotalCount}_P} $$
Average Error Rate Across All Product Types: The mean error rate across all product types is calculated as: $$ \text{MeanErrorRate} = \frac{1}{m} \sum_{P=1}^m \text{ErrorRate}_P $$ where $ m $ is the total number of product types.
Z-Score of Error Rates: To measure how far a product's error rate deviates from the mean, the z-score is calculated as: $$ Z_P = \frac{\text{ErrorRate}_P - \text{MeanErrorRate}}{\text{StandardDeviation}_\text{ErrorRate}} $$ where $ \text{StandardDeviation}_\text{ErrorRate} $ is the standard deviation of error rates across all product types.
The primary goal is to **identify product types with error rates large than the average error rate. These are defined as product types with z-scores greater than $0$.
Differentially privately release the (1) Z-scores of all Product Typess and (2) the Set of product types with Z-score $>0$. The privacy guarantee should be $(\varepsilon, \delta)$-differential privacy with $$ \varepsilon = 2.0, \delta = 1e-6. $$
For the DP-definition, we will use Add-Remove neighbor relationship by adding or removing one user (NOT one event).
You may use any differentially private methods. You may use any off-the-shelf DP-packages but please explain how you are using them. Please include your proof for differential privacy for your method ( you may use autodp to describe the DP-view of your method and to calibrate its parameters whenever applicable). For the DP proofs, feel free to use any results that we covered in the course or any other results in published papers.
Your goal is to report these numbers as accuately as possible under the following metricss.:
To properly report the uncertainty in the DP-release, you should run your algorithm for 100 times with different random seeds and report the max, min, 5%, 50% and 95% quantile of your results for each of the two accuracy metrics.
You may find the code for conducting the data analysis without privacy below as an example, which includes python function to load the data file into python as a pandas data frame.
You may also find python functions that compute performance metrics for your DP methods.
import pandas as pd
from scipy.stats import zscore
# Load data from the CSV file
def load_data(filename):
return pd.read_csv(filename, parse_dates=["Time of Event"])
# Compute error rates by product type and their z-scores
def compute_error_rates_and_zscores(data):
# Calculate the total number of events and errors for each product type
error_counts = (
data[data["Event Type"] == "error"]
.groupby("Product Type")
.size()
.rename("Error Count")
)
total_counts = (
data.groupby("Product Type")
.size()
.rename("Total Count")
)
# Merge and compute the error rate
error_rates = pd.concat([error_counts, total_counts], axis=1)
error_rates["Error Rate"] = error_rates["Error Count"] / error_rates["Total Count"]
# Fill missing values with 0 for products with no errors
error_rates.fillna(0, inplace=True)
# Compute z-scores for the error rates
error_rates["Error Rate Z-Score"] = zscore(error_rates["Error Rate"])
return error_rates
# Load the synthetic data
csv_file = "synthetic_telemetry_data.csv"
telemetry_data = load_data(csv_file)
# Compute error rates and their z-scores
error_rates_and_zscores = compute_error_rates_and_zscores(telemetry_data)
# Display the result
print("Error Rates and Z-Scores by Product Type:")
print(error_rates_and_zscores)
Error Rates and Z-Scores by Product Type:
Error Count Total Count Error Rate Error Rate Z-Score
Product Type
A 125 23096 0.005412 -0.907574
B 767 38325 0.020013 -0.389819
C 39 28470 0.001370 -1.050918
D 2718 30531 0.089024 2.057361
E 30 1581 0.018975 -0.426617
F 769 15064 0.051049 0.710730
Others 477 15289 0.031199 0.006838
import numpy as np
import pandas as pd
def compute_l_inf_error(true_z_scores, private_z_scores):
"""
Compute the L_inf error between true and privately released z-scores.
L_inf error is the maximum absolute difference.
"""
return np.max(np.abs(true_z_scores - private_z_scores))
def compute_iou(true_set, private_set):
"""
Compute the Intersection over Union (IOU) between the true set of selected products
and the privately released set of selected products.
"""
intersection = len(true_set.intersection(private_set))
union = len(true_set.union(private_set))
return intersection / union if union > 0 else 0
def summarize_metrics(metrics):
"""
Summarize the metrics (L_inf error and IOU) over multiple runs.
Report max, min, 5%, 50%, and 95% quantiles for the metrics.
"""
summary = {
"max": np.max(metrics),
"min": np.min(metrics),
"5%": np.percentile(metrics, 5),
"50%": np.percentile(metrics, 50),
"95%": np.percentile(metrics, 95),
}
return summary