Please help me how to use more in cluster mode

[Krout0n]

I'm a newbie in this great framework, Ray, MPC, and distributed computing. So, sorry if my question is off the mark to Secretflow and so on...

What I would like to accomplish

To send the sharized-CSVs that Alice and Bob each have to the head-node that does the PSI, join them, and then send the result to Carol.

Suppose we have a cluster of Secretflow clusters as shown in the figure below.

Question 1: How should I define a cluster in this case?

In the head-node, $ ray status displays the following result.

(secretflow) bash-4.4# ray status
======== Autoscaler status: 2022-11-15 10:40:01.877003 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_121dda96aa66771165706b0a5e5fc6b7c0b9b6fb3685ce4844df46a0
 1 node_b72a6338e6a71f79375c5e252abec71ae9fe7e8b03531ed1e51cf53f
 1 node_e3c8f1dcca31619ee132e41bdf61ef71ead34747dcc53ab4c9e56365
 1 node_d5f4302e52968f476e694b439fe49c7315de23c5bf71b7ed44d3c065
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/32.0 CPU
 0.0/8.0 alice
 0.0/8.0 bob
 0.0/8.0 carol
 0.00/84.307 GiB memory
 0.00/37.253 GiB object_store_memory

The connections from all compute-nodes are confirmed. So, I try to define our cluster into the code, and get stuck on totally...

import spu
import secretflow as sf

# Use ray head adress
sf.init(address='172.0.0.2:3000')

cluster_def={
    'nodes': [
        {
            'party': 'alice',
            'id': '0',
            # Which port number to specify?
            'address': '172.0.0.3:????',
        },
        {
            'party': 'bob',
            'id': '1',
            # Which port number to specify?
            'address': '172.0.0.4:????',
        },
        # Carol should be outside of SPU cluster, right?
    ],
    'runtime_config': {
        'protocol': spu.spu_pb2.SEMI2K,
        'field': spu.spu_pb2.FM128,
        'sigmoid_mode': spu.spu_pb2.RuntimeConfig.SIGMOID_REAL,
    }
}

spu = sf.SPU(cluster_def=cluster_def)

The above code is expected to be executed in head-node, right?

Question 2: What provided-method should I call for let Alice and Bob send sharized CSV to the head-node?

After defining SPU cluster, then let Alice and Bob send sharized CSV. I have a question. What should I do to let them send? How can I order Alice and Bob to send a local CSV from the head node? I read this tutorial, but this situation is that both of their plain CSV are been in the head-node from the beginning. I read 问题7, 问题8 in #176 through DeepL, but need more information🙏

Question 3: How does Carol confirm the plain result in her context?

After I manage to get through joining CSVs, let the head-node send the result to Carol. But how? In secretflow, do you assume that all operations(sending, joining, confirming) are performed at the head node?

Source

binary

Secretflow Version

0.7.7b1

OS Platform and Distribution

docker

Python version

3.8

[zhouaihui]

Q 1:
Just choose an unused port is ok.

[Krout0n]

Sorry to bother you again, unused port of Alice and Bob's host machine, right?

[Krout0n]

All of your answer sound correct and I confirmed! Thank you🙏

[zhouaihui]

Q 2:
The tutorial shows how to read csv and it's suitable for your situation also.
E.g. The code below means alice reads '.data/alice.csv' on alice node.

input_path = {alice: '.data/alice.csv', bob: '.data/bob.csv'}
output_path = {alice: '.data/alice_psi.csv', bob: '.data/bob_psi.csv'}
spu.psi_csv('uid', input_path, output_path, 'alice')

[zhouaihui]

Q 3:
spu.psi_csv gives you an option whether to broadcast result to alice/bob both with argument broadcast_result=True, or only one party can receive the result with argument receiver='alice', broadcast_result=False (alice gets the result only)

If alice has the result, and then you can send it to carol via alice with code as following and head node does not touch the result

import pandas as pd

alice_result_df = alice(pd.read_csv)(''.data/alice_psi.csv'')
# use `to` here
carol_result_df = alice_result.to(carol)
carol(lambda df : df.to_csv('path to save result for carol'))(carol_result_df )