Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jsonrpc-feature returned #417

Merged
merged 42 commits into from
Nov 29, 2023
Merged

jsonrpc-feature returned #417

merged 42 commits into from
Nov 29, 2023

Conversation

Copy link

codecov bot commented Nov 7, 2023

Codecov Report

Merging #417 (f794950) into main (47238ae) will decrease coverage by 0.18%.
The diff coverage is 61.85%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #417      +/-   ##
==========================================
- Coverage   70.12%   69.94%   -0.18%     
==========================================
  Files          72       80       +8     
  Lines        7190     8995    +1805     
==========================================
+ Hits         5042     6292    +1250     
- Misses       2148     2703     +555     
Files Coverage Δ
homestar-core/src/workflow/input/parse.rs 86.36% <ø> (ø)
homestar-core/src/workflow/instruction.rs 69.23% <100.00%> (+4.36%) ⬆️
homestar-core/src/workflow/pointer.rs 76.72% <100.00%> (-0.59%) ⬇️
homestar-core/src/workflow/task.rs 91.30% <100.00%> (ø)
homestar-runtime/src/cli.rs 62.90% <100.00%> (-0.59%) ⬇️
homestar-runtime/src/cli/show.rs 84.61% <ø> (ø)
homestar-runtime/src/db/utils.rs 0.00% <ø> (ø)
homestar-runtime/src/event_handler/cache.rs 22.50% <100.00%> (ø)
homestar-runtime/src/libp2p/multiaddr.rs 100.00% <ø> (ø)
homestar-runtime/src/logger.rs 92.10% <100.00%> (+0.10%) ⬆️
... and 44 more

... and 1 file with indirect coverage changes

zeeshanlakhani and others added 20 commits November 29, 2023 15:36
Includes:
  - re-purposing of feature flags
    * metrics is always a thing (on)
    * monitoring is the gated feature
    * websocket-server flag is gone, we only gate push notifications
  - jsonrpc setup and rpc method register
  - prometheus exposition format to json parser
  - Added todo around split networking config

Notes:
  - will not merge this in until example app has been restored with the client changes
# Description

This pull request implements the following changes:

- [x] Add event notifications
- [x] Add `emit_event` notification utility function
- [x] Add `emit_receipt` notification utility function
- [x] Move receipt notifications to event handler notifications
- [x] Add `ConnnectionEstablished`, `ConnnectionClosed`, `ListeningOn`,
`OutgoingConnectionError`, and `IncomingConnectionError` network
notifications
- [x] Test JSON event notification bytes roundtrip
- [x] Test JSON event notification string roundtrip
- [x] Integration test connection notifications with two Homestar nodes

## Link to issue

Closes #407

## Type of change

- [x] New feature (non-breaking change that adds functionality)

## Test plan (required)

We have included unit tests to check roundtrip conversions between JSON
bytes and strings. In addition, we have included an integration test
that subscribes and listens for connection messages between Homestar
nodes.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
zeeshanlakhani and others added 22 commits November 29, 2023 15:36
# Description

This PR implements the following changes:

- [x] Add receipt published and received notifications
- [x] Update receipt sharing log messages
- [x] Add receipt sharing integration test
- [x] Add `remove_db` and `wait_for_socket_connection` test utilities
- [x] Update JSON `peer_id` key to `peerId`

## Link to issue

Implements #131 

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

This PR includes an integration test that checks for gossiped receipts
in websocket notifications and logs.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
# Description

- [x] Fix the connection notification integration test
- [x] Avoid event hooks to IPFS in tests 

## Type of change

- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

Improved connection notification test included to move away from
arbitrary waits to event-driven notification tests.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
# Description

This PR includes the following changes:

- [x] Set gossip `max_transmit_size`
- [x] Handle Kademlia `InboundRequest` events

## Type of change

- [x] Refactor (non-breaking change that updates existing functionality)
Includes:

- also, add starter functions for default .pem/settings initialization
- error handling around retry, wasms
- ups the transport timeout
# Description

This PR implements the following changes:

- [x] Add gosssipsub message wrapper
- [x] Rename `store_and_notify` to `publish_and_notify`

The message wrapper includes a header with a nonce to force the gossip
of duplicate receipts. We will likely expand on the header in future
work and make the nonce optional.

## Link to issue

Implements #421.

## Type of change

- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

We've added a unit test to roundtrip a gossiped message to bytes and
back again. We also have a gossip notifications integration test to
confirm messages are still sent.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
Fix: remove unnecessary decode of already utf8-ed string from wasmval.
# Description

This PR implements the following changes:

- [x] Remove temporary databases after test runs

## Type of change

- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

The tests should pass and leave no databases behind.
Includes:

- a rename of `asyncboundedchannel*` to `asyncchannel` to accomodate
unbounded channels
- remove receipt logging with too much output
# Description

This PR implements the following changes:

- [x] Break up `[node.network]` settings into smaller groups
- [x] Move `[monitoring]` settings to `[node.monitoring]`
- [x] Add `defaults.toml` with complete settings matching the defaults
- [x] Reduce the settings passed to consumers to the minimum necessary
- [x] Update test fixtures using the new settings
- [x] Update and add comments

A complete version of the updated settings will look something like:

```toml
[node]
gc_interval = 1800
shutdown_timeout = 20

[node.database]
url = "homestar.db"
max_pool_size = 100

[node.monitoring]
process_collector_interval = 5000
console_subscriber_port = 6669

[node.network]
events_buffer_len = 1024
poll_cache_interval = 1000

[node.network.ipfs]
host = "127.0.0.1"
port = 5001

[node.network.libp2p]
listen_address = "/ip4/0.0.0.0/tcp/0"
node_addresses = []
announce_addresses = []
transport_connection_timeout = 60
max_connected_peers = 32
max_announce_addresses = 10

[node.network.libp2p.mdns]
enable = true
enable_ipv6 = false
query_interval = 300
ttl = 540

[node.network.libp2p.rendezvous]
enable_client = true
enable_server = false
registration_ttl = 7200
discovery_interval = 600

[node.network.libp2p.pubsub]
enable = false
duplication_cache_time = 1
heartbeat = 60
idle_timeout = 86400
max_transmit_size = 10485760
mesh_n_low = 1
mesh_n_high = 10
mesh_n = 2
mesh_outbound_min = 1

[node.network.libp2p.dht]
p2p_provider_timeout = 30
receipt_quorum = 2
workflow_quorum = 3

[node.network.keypair_config]
random = { }

[node.network.metrics]
port = 4000

[node.network.rpc]
host = "::1"
port = 3030
max_connections = 10
server_timeout = 120

[node.network.webserver]
host = "127.0.0.1"
port = 1337
timeout = 120
websocket_capacity = 2048
websocket_receiver_timeout = 30000

```

## Link to issue

Closes #442 

## Type of change

- [x] Refactor (non-breaking change that updates existing functionality)
- [ ] This change requires a documentation update. (It does. Will do.)
- [x] Comments have been added/updated.

## Test plan (required)

All existing tests should pass with the updated fixtures.
This PR implements the following changes:

- [x] Update example to use Homestar client:
https://www.npmjs.com/package/@fission-codes/homestar
- [x] Remove emulation mode

Implements #390

- [x] Refactor (non-breaking change that updates existing functionality)

The example should be tested manually to verify it works.

---------

Signed-off-by: Brian Ginsburg <[email protected]>
Co-authored-by: Hugo Dias <[email protected]>
- adds npm packages to homestar-runtime.
- adds CI/CD to build and publish wrapper package and os specific packages.
# Description

Remove many TODOs, add comments on *pub* (*pub* crate), and work through
logs.

- Includes a few other minor fixins and additions. 

## Type of change

- [X] Refactor (non-breaking change that updates existing functionality)

Closes #262.

---------

Signed-off-by: Zeeshan Lakhani <[email protected]>
Co-authored-by: Brian Ginsburg <[email protected]>
@bgins bgins merged commit ac9167c into main Nov 29, 2023
27 checks passed
@bgins bgins deleted the zl/jsonrpc branch November 29, 2023 21:14
bgins added a commit that referenced this pull request Nov 29, 2023
Includes:
  - re-purposing of feature flags
    * metrics is always a thing (on)
    * monitoring is the gated feature
    * The websocket-server flag is gone, we only gate push notifications
  - JSON-RPC setup and RPC method register
  - Prometheus exposition format to JSON parser

Other features and other fixes:

- [x] e2e testing of run workflow
- [x] #407
- [x] #410
- [x] #418
- [x] #424
- [x] #354
- [x] #409
- [x] #425
- [x] #426
- [x] #429
- [x] #433
- [x] #435
- [x] #421
- [x] #436
- [x] #437
- [x] #444
- [x] #438
- [x] #390
- [x] #451
- [x] #456

---------

Signed-off-by: Brian Ginsburg <[email protected]>
Signed-off-by: Zeeshan Lakhani <[email protected]>
Co-authored-by: Brian Ginsburg <[email protected]>
Co-authored-by: Hugo Dias <[email protected]>
This was referenced Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants