Evaluating Privacy–Utility Trade-offs in Differentially Private Federated Learning: A Comparative Benchmarking Study

Abstract: The rapid proliferation of mobile phones, wearables, IoT sensors, and other edge computing devices has generated vast quantities of decentralized, privacy-sensitive data, intensifying interest in Federated Learning (FL) as a distributed machine learning paradigm. By keeping raw data on local devices and sharing only model updates, FL offers an appealing alternative to centralized training. However, model updates are not inherently private: prior work has shown that shared gradients and parameters can leak sensitive information through membership inference, model inversion, and gradient reconstruction attacks. Differential Privacy (DP) has therefore emerged as the de facto standard for providing formal, quantifiable privacy guarantees in FL. In a typical Differentially Private Federated Learning (DPFL) pipeline, local model updates are clipped to bound each client’s sensitivity, and calibrated Gaussian noise is injected before (or during) server aggregation, with the cumulative privacy loss tracked by a privacy accountant.

Despite the growing body of DPFL research, the literature lacks a systematic and fair comparison of existing approaches. Individual methods are typically evaluated under different datasets, model architectures, client populations, data partitioning schemes, and privacy accounting conventions, which makes reported results difficult to compare and obscures which design choices actually drive performance. This inconsistency hinders both researchers seeking to identify promising directions and practitioners seeking to select a method for deployment.

To address this gap, we present a comprehensive comparative study of fifteen state-of-the-art DPFL methods, all evaluated on the CIFAR-10 benchmark under unified and tightly controlled experimental conditions. The selected methods span the principal design dimensions of DPFL: (i) trust models (e.g., central versus local/distributed DP), (ii) noise mechanisms, (iii) gradient and update clipping strategies, (iv) privacy accounting methods, and (v) utility- enhancement techniques aimed at mitigating the accuracy loss caused by clipping and noise. All experiments use a ResNet-18 backbone with 500 federated clients and consider both IID and Non-IID data distributions, the latter simulated via Dirichlet partitioning with concentration parameters α ∈ {0.3, 0.6} to capture varying degrees of statistical heterogeneity. Each method is evaluated across privacy budgets ε ∈ {4, 6, 8, 10}, covering a practical range from relatively strong to moderate privacy protection.

Our results show that DP-FedSAM and its sparsified variant DP-FedSAM-topk consistently outperform all other baselines across the evaluated settings. Their advantage is most pronounced under heterogeneous (Non-IID) data, where sharpness-aware optimization yields flatter local minima and updates that are more robust to clipping and noise. For example, at ε = 8 under Non-IID conditions, DP-FedSAM-topk reaches a test accuracy of 53.64%, compared with 46.74% for the DP-FedAvg baseline, an absolute improvement of approximately 7 percentage points. Beyond these headline numbers, our analysis examines how the privacy–utility trade-off varies with the privacy budget, the degree of data heterogeneity, and the choice of clipping, noise, and accounting strategies.

This study makes four main contributions: (1) a unified, reproducible evaluation framework for benchmarking DPFL methods under identical conditions; (2) a consolidated comparison table covering all fifteen methods across every data distribution and privacy budget; (3) four comparative figures that visualize the privacy–utility trade-off and robustness to data heterogeneity; and (4) actionable insights into the strengths, limitations, and open challenges of existing DPFL approaches, intended to guide both future research and practical deployment.

Keywords: Federated Learning, Differential Privacy, Privacy–Utility Trade-off, Non-IID Data, Gradient Clipping, Sharpness-Aware Minimization, Comparative Study