Skip to content

Commit

Permalink
Add files via upload
Browse files Browse the repository at this point in the history
  • Loading branch information
@tnurbek
tnurbek authored May 14, 2024
1 parent 5aef50f commit 6d807c9
Showing 1 changed file with 245 additions and 0 deletions.
245 changes: 245 additions & 0 deletions index.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,245 @@

<br>

<p align="center">
<iframe width="560" height="315" src="https://www.youtube.com/embed/<videoid>" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</p>

<br>


![main figure](docs/ShapFed.png)
<p align="justify"> **Overview of our proposed ShapFed algorithm:** Each participant $i$ transmits their locally computed iterates $w_i$ to the server. The server then, <span style="color: blue">(i)</span> computes class-specific Shapley values (CSSVs) using the last layer parameters (gradients) $\hat{w}$ <span style="color: blue">(ii)</span> aggregates the weights by employing normalized contribution assessment values $\tilde{\gamma}_i$ for each participant $i$, and <span style="color: blue">(iii)</span> broadcasts the personalized weights $\bar{w}_i$ to each participant, using their individual, not-normalized contribution values $\gamma_i$.
</p>


## Abstract
<p align="justify">
Federated learning (FL) has emerged as a pivotal approach in machine learning, enabling multiple participants to collaboratively train a global model without sharing raw data. While FL finds applications in various domains such as healthcare and finance, it is challenging to ensure global model convergence when participants do not contribute equally and/or honestly. To overcome this challenge, principled mechanisms are required to evaluate the contributions made by individual participants in the FL setting. Existing solutions for contribution assessment rely on general accuracy evaluation, often failing to capture nuanced dynamics and class-specific influences. This paper proposes a novel contribution assessment method called ShapFed for fine-grained evaluation of participant contributions in FL. Our approach uses Shapley values from cooperative game theory to provide a granular understanding of class-specific influences. Based on ShapFed, we introduce a weighted aggregation method called ShapFed-WA, which outperforms conventional federated averaging, especially in class-imbalanced scenarios. Personalizing participant updates based on their contributions further enhances collaborative fairness by delivering differentiated models commensurate with the participant contributions. Experiments on CIFAR-10, Chest X-Ray, and Fed-ISIC2019 datasets demonstrate the effectiveness of our approach in improving utility, efficiency, and fairness in FL systems. The code can be found at https://github.com/tnurbek/shapfed.</p>


## ShapFed Main Algorithm
![Pseudocode](docs/pseudo.png)

<p align="justify">**Weighted Aggregation:** The optimal weights $w_s^{\star}$ are derived using Equation 7 from the paper, while $w_s$ represents the result of applying equal weights (FedAvg). **Personalization:** Rather than distributing a uniform global model to all users, we provide personalized weights $\bar{w}_i$, which are $\gamma_i$ combinations of individual user weights $w_i$ and the optimally aggregated weight $w_s^{\star}$.</p>

![Weighted ShapFed and Personalized ShapFed](docs/alignment.png)

## Results: Contribution Assessment of Our Approach
<p align="justify"> 1. Comparison of our proposed contribution assessment algorithm (CSSV) with CGSV and true Shapley value computations using ResNet-34 architecture on Chest X-Ray dataset. </p>

![CSSV vs CGSV](docs/contribution_comparison2.png)

<p align="justify"> 2. Heatmap visualization of class-specific Shapley values for heterogeneous setting (explained in Section 5.2 in the paper) evaluated on CIFAR-10 dataset. </p>

![Heatmap](docs/cifar10-skew.png)

## Results: Weighted Aggregation
<p align="justify">1. Comparing FedAvg and ShapFed-WA on CIFAR10 under an imbalanced split scenario: insights into the balanced accuracy of four individual participants.</p>

![ShapFed vs FedAvg - CIFAR10](docs/imbalanced_cifar.png)

<p align="justify">2. (**Left**) The balanced accuracy of our methods (ShapFed-WA & ShapFed) vs FedAvg. (**Right**) Per-participant accuracy using all methods evaluated on Fed-ISIC2019 dataset.</p>

![ShapFed vs FedAvg - FEDISIC](docs/fedisic-plot.png)


## Results: Personalization
<table border="1" style="width:100%; border-collapse: collapse;">
<caption>1. Performance and fairness comparison with our method and FedAvg. We use Pearson's correlation (↑) as a fairness metric on CIFAR-10. The red highlight indicates a negative gain from collaboration.</caption>
<thead>
<tr style="background-color: lightblue;">
<th colspan="2">Dataset / Partition</th>
<th>Setting</th>
<th>P1</th>
<th>P2</th>
<th>P3</th>
<th>P4</th>
<th>P5</th>
<th>Corr.</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="3">ChestXRay</td>
<td rowspan="3">Hetero.</td>
<td>Individual</td>
<td>50.0</td>
<td>64.7</td>
<td>62.0</td>
<td>53.7</td>
<td>50.0</td>
<td>---</td>
</tr>
<tr>
<td>FedAvg</td>
<td>50.0</td>
<td>55.8</td>
<td>61.9</td>
<td>54.2</td>
<td>50.0</td>
<td>0.82</td>
</tr>
<tr style="background-color: lightgreen;">
<td>ShapFed</td>
<td style="background-color: lightgreen;">50.0</td>
<td style="background-color: lightgreen;">65.2</td>
<td style="background-color: lightgreen;">69.5</td>
<td style="background-color: lightgreen;">58.5</td>
<td style="background-color: lightgreen;">50.0</td>
<td style="background-color: lightgreen; font-weight: bold;">0.93</td>
</tr>
<tr>
<td rowspan="8">CIFAR-10</td>
<td rowspan="4">Imb.</td>
<td>Individual</td>
<td>75.8</td>
<td>45.4</td>
<td>48.6</td>
<td>31.6</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>FedAvg</td>
<td style="background-color: lightred;">56.6</td>
<td>56.8</td>
<td>63.8</td>
<td>64.2</td>
<td>---</td>
<td>-0.60</td>
</tr>
<tr>
<td>CGSV</td>
<td style="background-color: lightred;">57.2</td>
<td>59.0</td>
<td>58.8</td>
<td>60.4</td>
<td>---</td>
<td>-0.98</td>
</tr>
<tr style="background-color: lightgreen;">
<td>ShapFed</td>
<td style="background-color: lightgreen;">81.4</td>
<td style="background-color: lightgreen;">78.2</td>
<td style="background-color: lightgreen;">71.8</td>
<td style="background-color: lightgreen;">73.6</td>
<td style="background-color: lightgreen;">---</td>
<td style="background-color: lightgreen; font-weight: bold;">0.74</td>
</tr>
<tr>
<td rowspan="4">Hetero.</td>
<td>Individual</td>
<td>75.2</td>
<td>68.8</td>
<td>66.8</td>
<td>69.0</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>FedAvg</td>
<td>74.6</td>
<td>70.2</td>
<td>70.2</td>
<td>76.0</td>
<td>---</td>
<td>0.53</td>
</tr>
<tr>
<td>CGSV</td>
<td style="background-color: lightred;">55.0</td>
<td style="background-color: lightred;">55.8</td>
<td style="background-color: lightred;">57.2</td>
<td style="background-color: lightred;">52.6</td>
<td>---</td>
<td>-0.26</td>
</tr>
<tr style="background-color: lightgreen;">
<td>ShapFed</td>
<td style="background-color: lightgreen;">79.8</td>
<td style="background-color: lightgreen;">75.4</td>
<td style="background-color: lightgreen;">69.0</td>
<td style="background-color: lightgreen;">75.0</td>
<td style="background-color: lightgreen;">---</td>
<td style="background-color: lightgreen; font-weight: bold;">0.90</td>
</tr>
</tbody>
</table>


<table border="1" style="width:87.5%; border-collapse: collapse; margin: auto;">
<caption>2. Performance and fairness comparison using Pearson's correlation (↑) as a fairness metric on Fed-ISIC2019. The red highlight indicates a negative gain from collaboration.</caption>
<thead>
<tr style="background-color: lightblue;">
<th>Setting</th>
<th>P1</th>
<th>P2</th>
<th>P3</th>
<th>P4</th>
<th>P5</th>
<th>P6</th>
<th>Corr.</th>
</tr>
</thead>
<tbody>
<tr>
<td>Individual</td>
<td>67.2</td>
<td>25.7</td>
<td>42.3</td>
<td>31.0</td>
<td>18.5</td>
<td>15.6</td>
<td>---</td>
</tr>
<tr>
<td>FedAvg</td>
<td style="background-color: lightred;">65.4</td>
<td>40.9</td>
<td>57.2</td>
<td>59.3</td>
<td>51.5</td>
<td>56.2</td>
<td>0.63</td>
</tr>
<tr style="background-color: lightgreen;">
<td>ShapFed-WA</td>
<td>69.3</td>
<td>44.3</td>
<td>65.0</td>
<td>63.1</td>
<td>54.8</td>
<td>61.2</td>
<td>0.62</td>
</tr>
<tr style="background-color: lightgreen;">
<td>ShapFed</td>
<td>68.5</td>
<td>44.4</td>
<td>61.9</td>
<td>60.4</td>
<td>40.6</td>
<td>53.2</td>
<td style="font-weight: bold;">0.84</td>
</tr>
</tbody>
</table>


## Summary

<p align="justify">This work proposes Class-Specific Shapley Values (CSSVs) to quantify participant contributions at a granular level. The contributions of this work include a novel method to deepen the understanding of participant impact and improve fairness analysis. Evaluation against FedAvg shows superior performance and additional experiments reveal enhanced fairness by personalizing client updates based on contributions. Overall, the approach aims to achieve a more equitable distribution of benefits in FL. In future, we plan to conduct an in-depth theoretical analysis aimed at identifying the specific characteristics that contribute to an effective estimation of Shapley values. This analysis will enhance our understanding of the factors that influence the accuracy and reliability of Shapley value approximations. Furthermore, an investigation into what makes our approximation of cosine similarity from the last layer a robust indicator of contributions will be explored. </p>



## BibTeX
If you like our work, please consider citing us.
```
@article{Tastan2024ShapFed,
title={Redefining Contributions: Shapley-Driven Federated Learning},
author={Tastan, Nurbek and Fares, Samar and Aremu, Toluwani and Horvarth, Samuel and Nandakumar, Karthik},
journal={https://arxiv.org/abs/<id>},
year={2024}
}
```

0 comments on commit 6d807c9

Please sign in to comment.