What should I do if I only have a GPUNode with 8 graphics card to run the distributed algorithm?

[iuserea]

Could anyone please tell me what should I do if I only have a GPUNode functioning as a login node and a compute node at the same time to run the distributed algorithm, please?
It works pretty well when running fedavg algorithm under fedml's distributed architecture based on our hardware architecture saying above.
But when I try to run fednas,fedgkt,fedavg_robust algorithm,they'll fail in the same reason in the end as showing in the screenshot below.

FedNAS:

FedGKT:

[chaoyanghe]

@iuserea I just run our code using 4 and 8 compute nodes, it works well. The issue you mentioned happens when you have only 1 compute node but do not change the compute topology. To address this issue, you can try to modify the computing topology at init_training_device() function, in which you can force all workers/clients run in the same GPU device ID (since many clients/workers share the same GPU device, you may also need to make the batch size smaller to fit the memory constraints, which may degrades the accuracy a little bit and leads to a relatively slow training speed). Besides, you should also change the client_number/worker_number at run_xxx.sh

[iuserea]

@chaoyanghe Could fedgkt algorithm be running with only 2 clients or even one client?I tried but I failed.

[iuserea]

run_fedgkt.sh:

[iuserea]

main_fedgkt.py:

[iuserea]

CMD for 10 clients:
sh run_FedGKT.sh 8 cifar10 homo 10 10 1 Adam 0.001 1 0 resnet56 fedml_resnet56_homo_cifar10 "./../../../data/cifar10" 64 10

[iuserea]

The success flag I found is that b_all_received = True.
When the process failed, it either failed in b_all_received = false or 'b_all_received' variable even didn't appear after all the clients training has finished.

However it's just the surface of the real problem.

[iuserea]

When I set client/worker's number to 2,fedgkt algorithm will also create 8 processes which may result in the failure of itself.

[chaoyanghe]

Change your sh script: -n is still 9

[iuserea]

@chaoyanghe
The -n option is not essential for training of two clients.

The quesion is when training the two clients,the message below didn't appear.
handle_message_receive_feature_and_logits_from_client
add_model. index = 7

[chaoyanghe]

Hi @iuserea We have supported GPU mapping, please have a look at this:

FedML/fedml_experiments/distributed/fedavg/main_fedavg.py

Line 308 in 966a36d

device = mapping_processes_to_gpu_device_from_yaml_file(process_id, worker_number, args.gpu_mapping_file, args.gpu_mapping_key)

[rambo-coder]

Hi @iuserea , Could you please share your configuration of both software and hardware?

[rambo-coder]

I only have two gpus in one server, how can I train fedgdk?

[rambo-coder]

@iuserea Hi, How do you set your mpi_host_file?

[rambo-coder]

[rambo-coder]