Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ReadWriteMany accessMode #69

Open
jbanety opened this issue Apr 30, 2020 · 21 comments
Open

ReadWriteMany accessMode #69

jbanety opened this issue Apr 30, 2020 · 21 comments

Comments

@jbanety
Copy link

jbanety commented Apr 30, 2020

Hi,
I'm trying to create a StatefulSet with a ReadWriteMany accessMode.
It doesn't work. Is it possible to do ReadWriteMany access mode with Linstor and Lisntor CSI ?

Events on test-0 :

Events:
  Type     Reason                  Age                    From                                  Message
  ----     ------                  ----                   ----                                  -------
  Warning  FailedScheduling        <unknown>              default-scheduler                     persistentvolumeclaim "data" not found
  Normal   Scheduled               <unknown>              default-scheduler                     Successfully assigned default/test-0 to node2
  Normal   SuccessfulAttachVolume  8m18s                  attachdetach-controller               AttachVolume.Attach succeeded for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196"
  Warning  FailedMount             7m31s (x3 over 8m3s)   kubelet, node2  MountVolume.SetUp failed for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: resourcegroup does not belong to paramKey values
  Warning  FailedMount             3m58s (x2 over 6m16s)  kubelet, node2  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[default-token-m9cnr data]: timed out waiting for the condition
  Warning  FailedMount             111s (x8 over 8m3s)    kubelet, node2  MountVolume.SetUp failed for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: drbdoptions/net/allow-two-primaries does not belong to paramKey values
  Warning  FailedMount             104s                   kubelet, node2  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-m9cnr]: timed out waiting for the condition

kubectl logs linstor-csi-node-pvdjb -n linstor -c linstor-csi-plugin :

time="2020-04-30T06:13:33Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=driver method=/csi.v1.Node/NodeGetCapabilities nodeID=node2 provisioner=linstor.csi.linbit.com req= resp="capabilities:<rpc:<type:GET_VOLUME_STATS > > " version=v0.7.4-9-g7383849
time="2020-04-30T06:13:33Z" level=debug msg="looking up resource by CSI volume id" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" csiVolumeID=pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 linstorCSIComponent=client
time="2020-04-30T06:13:33Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-controller:3371/v1/resource-definitions/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196'"
time="2020-04-30T06:13:33Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ExternalName:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 Props:map[Aux/csi-volume-annotations:{\"name\":\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\",\"id\":\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-30T06:03:00.12667645Z\",\"sizeBytes\":1073741824,\"readonly\":false,\"parameters\":{\"DrbdOptions/Net/allow-two-primaries\":\"yes\",\"resourceGroup\":\"linstor_group\",\"storagePool\":\"pool_linstor\"},\"snapshots\":[]} DrbdOptions/Net/allow-two-primaries:yes DrbdOptions/Resource/quorum:off DrbdPrimarySetOn:NODE1 StorPoolName:pool_linstor] Flags:[] LayerData:[{Type:DRBD Data:0xc0003d2e10} {Type:STORAGE Data:<nil>}] Uuid:d4b77817-bf44-405a-93c0-c40035fca705}" volume="&{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ID:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-30 06:03:00.12667645 +0000 UTC SizeBytes:1073741824 Readonly:false Parameters:map[DrbdOptions/Net/allow-two-primaries:yes resourceGroup:linstor_group storagePool:pool_linstor] Snapshots:[]}"
time="2020-04-30T06:13:33Z" level=debug msg="getting assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client targetNode=node2 volume="&{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ID:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-30 06:03:00.12667645 +0000 UTC SizeBytes:1073741824 Readonly:false Parameters:map[DrbdOptions/Net/allow-two-primaries:yes resourceGroup:linstor_group storagePool:pool_linstor] Snapshots:[]}"
time="2020-04-30T06:13:33Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-controller:3371/v1/resource-definitions/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196/resources/node2/volumes/0'"
time="2020-04-30T06:13:33Z" level=debug msg="found assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client volumeAssignment="&{Vol:0xc000208800 Node:node2 Path:/dev/drbd1000}"
time="2020-04-30T06:13:33Z" level=error msg="method failed" func="github.com/sirupsen/logrus.(*Entry).Error" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:297" error="rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: drbdoptions/net/allow-two-primaries does not belong to paramKey values" linstorCSIComponent=driver method=/csi.v1.Node/NodePublishVolume nodeID=node2 provisioner=linstor.csi.linbit.com req="volume_id:\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\" target_path:\"/var/lib/kubelet/pods/eecc1e3b-df42-4012-a2d9-24fadf59ef18/volumes/kubernetes.io~csi/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"csi.storage.k8s.io/ephemeral\" value:\"false\" > volume_context:<key:\"csi.storage.k8s.io/pod.name\" value:\"test-0\" > volume_context:<key:\"csi.storage.k8s.io/pod.namespace\" value:\"default\" > volume_context:<key:\"csi.storage.k8s.io/pod.uid\" value:\"eecc1e3b-df42-4012-a2d9-24fadf59ef18\" > volume_context:<key:\"csi.storage.k8s.io/serviceAccount.name\" value:\"default\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1588184499057-8081-linstor.csi.linbit.com\" > " resp="<nil>" version=v0.7.4-9-g7383849

linstor controller

$ linstor storage-pool list
+----------------------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node  | Driver   | PoolName   | FreeCapacity | TotalCapacity | CanSnapshots | State |
|==================================================================================================================================|
| DfltDisklessStorPool | node1 | DISKLESS |            |              |               | False        | Ok    |
| DfltDisklessStorPool | node2 | DISKLESS |            |              |               | False        | Ok    |
| pool_linstor         | node1 | LVM      | vg_linstor |   334.34 GiB |    335.35 GiB | False        | Ok    |
| pool_linstor         | node2 | LVM      | vg_linstor |   335.35 GiB |    335.35 GiB | False        | Ok    |
+----------------------------------------------------------------------------------------------------------------------------------+

$ 
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node  | Resource                                 | StoragePool          | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|=================================================================================================================================================================|
| node1 | pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 | pool_linstor         |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | Unused | UpToDate |
| node2 | pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 | DfltDisklessStorPool |     0 |    1000 | /dev/drbd1000 |           | Unused | Diskless |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ linstor resource-group list
+------------------------------------------------------------------------+
| ResourceGroup | SelectFilter                    | VlmNrs | Description |
|========================================================================|
| DfltRscGrp    | PlaceCount: 2                   |        |             |
|------------------------------------------------------------------------|
| linstor_group | PlaceCount: 1                   | 0      |             |
|               | StoragePool: pool_linstor       |        |             |
|               | LayerStack: ['DRBD', 'STORAGE'] |        |             |
+------------------------------------------------------------------------+

$ linstor resource-group list-properties linstor_group
+----------------------------------------------------+
| Key                                 | Value        |
|====================================================|
| DrbdOptions/Net/allow-two-primaries | yes          |
| StorPoolName                        | pool_linstor |
+----------------------------------------------------+

drbdadm status on node2

pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 role:Secondary
  disk:Diskless
  node1 role:Secondary
    peer-disk:UpToDate

drbdsetup events on node2

exists resource name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 role:Secondary suspended:no
exists connection name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 peer-node-id:0 conn-name:node1 connection:Connected role:Secondary
exists device name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 volume:0 minor:1000 disk:Diskless client:yes quorum:yes
exists peer-device name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 peer-node-id:0 conn-name:node1 volume:0 replication:Established peer-disk:UpToDate peer-client:no resync-suspended:no
exists -

Definition

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
  storagePool: "pool_linstor"
  resourceGroup: "linstor_group"
  DrbdOptions/Net/allow-two-primaries: "yes"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  serviceName: test
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: test
    spec:
      securityContext:
        fsGroup: 1001
        runAsUser: 1001
      containers:
        - name: debug
          image: alpine:3.11
          command: ["/bin/sh"]
          stdin: true
          tty: true
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data
            readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: linstor-basic-storage-class
---
@rck
Copy link
Member

rck commented Apr 30, 2020

This is being worked on. By default you get a mounted directory with a file system on a DRBD device. The DRBD device/the resource (per default) is not allowed to be dual primary, and even if so, having multiple writers on the file system would break anyways.

The only thing supported right now is raw block mode, with setting DRBD two-primaries via the storage class and having them dual primary while live migration (e.g., kubevirt).

@jbanety
Copy link
Author

jbanety commented Apr 30, 2020

OK. Thanks for the reply.
I gone further with :

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
  storagePool: "pool_linstor"
  DrbdOptions/Net/allow-two-primaries: "yes"
  placementCount: "2"
  placementPolicy: "AutoPlace"

Here my volumes are OK on the two nodes with the resource created correctly (no Diskless).
But still in error.

Can I help somehow on this ?

@rck
Copy link
Member

rck commented Apr 30, 2020

Exactly, but now you mount the FS on two nodes, which really really is not a good idea. Contributions are always welcome, but AFAIK there is a team working on it, so just waiting might be the best for now. @alexzhc , how is the progress on that?

@jbanety
Copy link
Author

jbanety commented Apr 30, 2020

OK. I was able to do ReadWriteMany access with Rook and Ceph but performances are better with Linstor.
My goal is to create load balancing on web app with access to the same filesystem to store and read files.

@roosemberth
Copy link

Hi!

I have a very similar use-case as @jbanety commented above: I have many applications, which I want to horizontally scale and need to have access to a shared FS.

@rck: You mentioned work was being done on it. Is it open source? If so where can I find more information or the ongoing development?

@rck
Copy link
Member

rck commented Jul 6, 2020

I don't know, I'm not working on this part, @alexzhc is still the one that could give updates on it.

@Jordanhf-dc
Copy link

Jordanhf-dc commented Sep 24, 2020

Hi,
Would also like to combine Linstor's performance and ReadWriteMany access for our usecase.
An update to have a rough idea of when / if this might happen would be very helpful @alexzhc , thanks.

@alexzhc
Copy link
Member

alexzhc commented Oct 10, 2020

Integrating an NFS export by using a NFS-Ganesha POD on top of a drbd RWO volume is not hard as demonstrated by Rook-NFS. However, by doing so, the NFS-Ganesha POD with the drbd RWO volume needs to failover in a reasonable RTO in case of node failure. However, besides 5-minutes k8s node timeout, currently in CSI, RWO volume attachment failover has a hard-coded timeout of 6 minutes. Such a long failover time will definitely cause NFS mount to go stale.

We need to solve the RWO failover RTO issue before implementing NFS export.

@kvaps
Copy link
Member

kvaps commented Oct 10, 2020

Hi, take a look at nfs-server-provisioner project, we're using it together with linstor for having ReadWriteMany volumes.

Failover is ensured by kube-fencing.

@jbanety
Copy link
Author

jbanety commented Oct 11, 2020

Hi @kvaps, it sounds interesting.
Are NFS performances good enough for a heavy loaded web app?
Do you have a working example with linstor, NFS provisionnel and kube-fencing?

@kvaps
Copy link
Member

kvaps commented Oct 11, 2020

Are NFS performances good enough for a heavy loaded web app?

Performance is fine, you can see my benchmarks for comparison:
https://gist.github.com/kvaps/8c2831ca15cf161a11e62cc4276793f7

Do you have a working example with linstor, NFS provisionnel and kube-fencing

For the nfs-server-provisioner, we're using the helm chart:
https://github.com/kvaps/nfs-server-provisioner-chart

here is example values:

  persistence:
    enabled: true
    size: 50Gi
    storageClass: linstor-1
  storageClass:
    name: example-nfs

After deployment, you can request new volume using example-nfs storageclass

Fencing is very specific to your environment, here is an example for HP ilo
https://raw.githubusercontent.com/kvaps/kube-fencing/master/deploy/examples/hp-ilo.yaml

Meanwhile I would advise to avoid using ReadWriteMany in case if you don't really needed it, since it is adding extra abstraction layers, which might made whole system less stable.

Consider using S3 (minio) in your application instead

@jbanety
Copy link
Author

jbanety commented Oct 11, 2020

Meanwhile I would advise to avoid using ReadWriteMany in case if you don't really needed it, since it is adding extra abstraction layers, which might made whole system less stable.

Consider using S3 (minio) in your application instead

Yes. I wanted to have a deployment with two pods on two different nodes to serve the same Prestashop app for load balancing. It must sharing a file system for image uploads,... I can't use S3 by design.
As a workaround, I created a StatefulSet with only 1 replica to fail over on the other node but no load balancing.

@kvaps
Copy link
Member

kvaps commented Oct 11, 2020

Well we're using same approach to host Nextcloud itself, but the user data is stored on another server using S3 backend

@Rid
Copy link

Rid commented Jun 7, 2022

@kvaps We have tried using https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner per your suggestion with a pireaus StorageClass, however when the NFS pod is rescheduled on to another node due to node failure, the mount on the new node via the NFS service just hangs.

[pid 1729737] openat(AT_FDCWD, "/etc/protocols", O_RDONLY|O_CLOEXEC) = 3
[pid 1729737] lseek(3, 0, SEEK_CUR)     = 0
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=2932, ...}) = 0
[pid 1729737] read(3, "# Internet (IP) protocols\n#\n# Up"..., 3072) = 2932
[pid 1729737] close(3)                  = 0
[pid 1729737] openat(AT_FDCWD, "/etc/netconfig", O_RDONLY) = 3
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=767, ...}) = 0
[pid 1729737] read(3, "#\n# The network configuration fi"..., 1024) = 767
[pid 1729737] close(3)                  = 0
[pid 1729737] openat(AT_FDCWD, "/etc/protocols", O_RDONLY|O_CLOEXEC) = 3
[pid 1729737] lseek(3, 0, SEEK_CUR)     = 0
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=2932, ...}) = 0
[pid 1729737] read(3, "# Internet (IP) protocols\n#\n# Up"..., 3072) = 2932
[pid 1729737] close(3)                  = 0
[pid 1729737] openat(AT_FDCWD, "/etc/netconfig", O_RDONLY) = 3
[pid 1729737] fstat(3, {st_mode=S_IFREG|0644, st_size=767, ...}) = 0
[pid 1729737] read(3, "#\n# The network configuration fi"..., 1024) = 767
[pid 1729737] close(3)                  = 0
[pid 1729737] mount("10.71.214.117:/export/pvc-aad0a19d-8cfa-4344-b58c-59f95acfccf7", "/nfs", "nfs", 0, "addr=10.71.214.117,vers=3,proto="...

We've tried nfsver 3 & 4 and are having the same issue.

Weirdly you can still connect via NFS on the same service if you change the nfsver parameter, but the original nfsver setting does not work.

Have you experienced this issue, do you have any advice which may help?

@kvaps
Copy link
Member

kvaps commented Jun 8, 2022

Sounds like network problem
May I get know, what CNI and which mode of kube-proxy you're using?

@Rid
Copy link

Rid commented Jun 8, 2022

@kvaps thanks for the reply, I agree. Currently trying to debug the issue with tcpdump/conntrack & cilium. The CNI is cilium, unfortunately tcpdump on the service IP doesn't show anything as the routing is done in BPF.

I can see the packets getting through to the correct endpoint (10.0.0.252 below), but after switching to UDP the packets stop (they don't seem to be dropped).

15:02:25.202882 IP (tos 0x0, ttl 64, id 50089, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.4.242.34278 > 10.0.0.252.111: Flags [S], cksum 0x1a1c (incorrect -> 0xb9bc), seq 800816836, win 64860, options [mss 1410,sackOK,TS val 1768478956 ecr 0,nop,wscale 8], length 0
	0x0000:  4500 003c c3a9 4000 4006 5d25 0a00 04f2  E..<..@.@.]%....
	0x0010:  0a00 00fc 85e6 006f 2fbb 7ec4 0000 0000  .......o/.~.....
	0x0020:  a002 fd5c 1a1c 0000 0204 0582 0402 080a  ...\............
	0x0030:  6968 d8ec 0000 0000 0103 0308            ih..........
15:02:25.203303 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [S.], cksum 0xc601 (correct), seq 2873377936, ack 800816837, win 64308, options [mss 1410,sackOK,TS val 1826134308 ecr 1768478956,nop,wscale 8], length 0
	0x0000:  4500 003c 0000 4000 4006 20cf 0a00 00fc  E..<..@.@.......
	0x0010:  0a00 04f2 006f 85e6 ab44 4490 2fbb 7ec5  .....o...DD./.~.
	0x0020:  a012 fb34 c601 0000 0204 0582 0402 080a  ...4............
	0x0030:  6cd8 9924 6968 d8ec 0103 0308            l..$ih......
15:02:25.203446 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [S.], cksum 0xc601 (correct), seq 2873377936, ack 800816837, win 64308, options [mss 1410,sackOK,TS val 1826134308 ecr 1768478956,nop,wscale 8], length 0
	0x0000:  4500 003c 0000 4000 3f06 21cf 0a00 00fc  E..<..@.?.!.....
	0x0010:  0a00 04f2 006f 85e6 ab44 4490 2fbb 7ec5  .....o...DD./.~.
	0x0020:  a012 fb34 c601 0000 0204 0582 0402 080a  ...4............
	0x0030:  6cd8 9924 6968 d8ec 0103 0308            l..$ih......
15:02:25.203449 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [S.], cksum 0xc601 (correct), seq 2873377936, ack 800816837, win 64308, options [mss 1410,sackOK,TS val 1826134308 ecr 1768478956,nop,wscale 8], length 0
	0x0000:  4500 003c 0000 4000 3f06 21cf 0a00 00fc  E..<..@.?.!.....
	0x0010:  0a00 04f2 006f 85e6 ab44 4490 2fbb 7ec5  .....o...DD./.~.
	0x0020:  a012 fb34 c601 0000 0204 0582 0402 080a  ...4............
	0x0030:  6cd8 9924 6968 d8ec 0103 0308            l..$ih......
15:02:25.203638 IP (tos 0x0, ttl 64, id 50090, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.34278 > 10.0.0.252.111: Flags [.], cksum 0x1a14 (incorrect -> 0xeed2), seq 1, ack 1, win 254, options [nop,nop,TS val 1768478957 ecr 1826134308], length 0
	0x0000:  4500 0034 c3aa 4000 4006 5d2c 0a00 04f2  E..4..@.@.],....
	0x0010:  0a00 00fc 85e6 006f 2fbb 7ec5 ab44 4491  .......o/.~..DD.
	0x0020:  8010 00fe 1a14 0000 0101 080a 6968 d8ed  ............ih..
	0x0030:  6cd8 9924                                l..$
15:02:25.231179 IP (tos 0x0, ttl 64, id 50091, offset 0, flags [DF], proto TCP (6), length 112)
    10.0.4.242.34278 > 10.0.0.252.111: Flags [P.], cksum 0x1a50 (incorrect -> 0x4c09), seq 1:61, ack 1, win 254, options [nop,nop,TS val 1768478985 ecr 1826134308], length 60
	0x0000:  4500 0070 c3ab 4000 4006 5cef 0a00 04f2  E..p..@.@.\.....
	0x0010:  0a00 00fc 85e6 006f 2fbb 7ec5 ab44 4491  .......o/.~..DD.
	0x0020:  8018 00fe 1a50 0000 0101 080a 6968 d909  .....P......ih..
	0x0030:  6cd8 9924 8000 0038 d2ad 422d 0000 0000  l..$...8..B-....
	0x0040:  0000 0002 0001 86a0 0000 0002 0000 0003  ................
	0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0060:  0001 86a3 0000 0003 0000 0006 0000 0000  ................
15:02:25.231567 IP (tos 0x0, ttl 64, id 45867, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [.], cksum 0xee61 (correct), seq 1, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 0
	0x0000:  4500 0034 b32b 4000 4006 6dab 0a00 00fc  E..4.+@[email protected].....
	0x0010:  0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01  .....o...DD./...
	0x0020:  8010 00fb ee61 0000 0101 080a 6cd8 9940  .....a......l..@
	0x0030:  6968 d909                                ih..
15:02:25.231667 IP (tos 0x0, ttl 63, id 45867, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [.], cksum 0xee61 (correct), seq 1, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 0
	0x0000:  4500 0034 b32b 4000 3f06 6eab 0a00 00fc  E..4.+@.?.n.....
	0x0010:  0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01  .....o...DD./...
	0x0020:  8010 00fb ee61 0000 0101 080a 6cd8 9940  .....a......l..@
	0x0030:  6968 d909                                ih..
15:02:25.231567 IP (tos 0x0, ttl 64, id 45868, offset 0, flags [DF], proto TCP (6), length 84)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [P.], cksum 0x5140 (correct), seq 1:33, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 32
	0x0000:  4500 0054 b32c 4000 4006 6d8a 0a00 00fc  E..T.,@[email protected].....
	0x0010:  0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01  .....o...DD./...
	0x0020:  8018 00fb 5140 0000 0101 080a 6cd8 9940  [email protected]..@
	0x0030:  6968 d909 8000 001c d2ad 422d 0000 0001  ih........B-....
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0050:  0000 0801                                ....
15:02:25.231675 IP (tos 0x0, ttl 63, id 45868, offset 0, flags [DF], proto TCP (6), length 84)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [P.], cksum 0x5140 (correct), seq 1:33, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 32
	0x0000:  4500 0054 b32c 4000 3f06 6e8a 0a00 00fc  E..T.,@.?.n.....
	0x0010:  0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01  .....o...DD./...
	0x0020:  8018 00fb 5140 0000 0101 080a 6cd8 9940  [email protected]..@
	0x0030:  6968 d909 8000 001c d2ad 422d 0000 0001  ih........B-....
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0050:  0000 0801                                ....
15:02:25.231671 IP (tos 0x0, ttl 63, id 45867, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [.], cksum 0xee61 (correct), seq 1, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 0
	0x0000:  4500 0034 b32b 4000 3f06 6eab 0a00 00fc  E..4.+@.?.n.....
	0x0010:  0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01  .....o...DD./...
	0x0020:  8010 00fb ee61 0000 0101 080a 6cd8 9940  .....a......l..@
	0x0030:  6968 d909                                ih..

    
15:02:25.231676 IP (tos 0x0, ttl 63, id 45868, offset 0, flags [DF], proto TCP (6), length 84)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [P.], cksum 0x5140 (correct), seq 1:33, ack 61, win 251, options [nop,nop,TS val 1826134336 ecr 1768478985], length 32
	0x0000:  4500 0054 b32c 4000 3f06 6e8a 0a00 00fc  E..T.,@.?.n.....
	0x0010:  0a00 04f2 006f 85e6 ab44 4491 2fbb 7f01  .....o...DD./...
	0x0020:  8018 00fb 5140 0000 0101 080a 6cd8 9940  [email protected]..@
	0x0030:  6968 d909 8000 001c d2ad 422d 0000 0001  ih........B-....
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0050:  0000 0801                                ....
15:02:25.231857 IP (tos 0x0, ttl 64, id 50092, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.34278 > 10.0.0.252.111: Flags [.], cksum 0x1a14 (incorrect -> 0xee3e), seq 61, ack 33, win 254, options [nop,nop,TS val 1768478985 ecr 1826134336], length 0
	0x0000:  4500 0034 c3ac 4000 4006 5d2a 0a00 04f2  E..4..@.@.]*....
	0x0010:  0a00 00fc 85e6 006f 2fbb 7f01 ab44 44b1  .......o/....DD.
	0x0020:  8010 00fe 1a14 0000 0101 080a 6968 d909  ............ih..
	0x0030:  6cd8 9940                                l..@
15:02:25.232044 IP (tos 0x0, ttl 64, id 50093, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.34278 > 10.0.0.252.111: Flags [F.], cksum 0x1a14 (incorrect -> 0xee3c), seq 61, ack 33, win 254, options [nop,nop,TS val 1768478986 ecr 1826134336], length 0
	0x0000:  4500 0034 c3ad 4000 4006 5d29 0a00 04f2  E..4..@.@.])....
	0x0010:  0a00 00fc 85e6 006f 2fbb 7f01 ab44 44b1  .......o/....DD.
	0x0020:  8011 00fe 1a14 0000 0101 080a 6968 d90a  ............ih..
	0x0030:  6cd8 9940                                l..@
15:02:25.232293 IP (tos 0x0, ttl 64, id 36895, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.4.242.49762 > 10.0.0.252.2049: Flags [S], cksum 0x1a1c (incorrect -> 0x081a), seq 3636937518, win 64860, options [mss 1410,sackOK,TS val 1768478986 ecr 0,nop,wscale 8], length 0
	0x0000:  4500 003c 901f 4000 4006 90af 0a00 04f2  E..<..@.@.......
	0x0010:  0a00 00fc c262 0801 d8c7 432e 0000 0000  .....b....C.....
	0x0020:  a002 fd5c 1a1c 0000 0204 0582 0402 080a  ...\............
	0x0030:  6968 d90a 0000 0000 0103 0308            ih..........
15:02:25.232407 IP (tos 0x0, ttl 64, id 45869, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [F.], cksum 0xee3d (correct), seq 33, ack 62, win 251, options [nop,nop,TS val 1826134337 ecr 1768478986], length 0
	0x0000:  4500 0034 b32d 4000 4006 6da9 0a00 00fc  E..4.-@[email protected].....
	0x0010:  0a00 04f2 006f 85e6 ab44 44b1 2fbb 7f02  .....o...DD./...
	0x0020:  8011 00fb ee3d 0000 0101 080a 6cd8 9941  .....=......l..A
	0x0030:  6968 d90a                                ih..
15:02:25.232461 IP (tos 0x0, ttl 63, id 45869, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [F.], cksum 0xee3d (correct), seq 33, ack 62, win 251, options [nop,nop,TS val 1826134337 ecr 1768478986], length 0
	0x0000:  4500 0034 b32d 4000 3f06 6ea9 0a00 00fc  E..4.-@.?.n.....
	0x0010:  0a00 04f2 006f 85e6 ab44 44b1 2fbb 7f02  .....o...DD./...
	0x0020:  8011 00fb ee3d 0000 0101 080a 6cd8 9941  .....=......l..A
	0x0030:  6968 d90a                                ih..
15:02:25.232463 IP (tos 0x0, ttl 63, id 45869, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.111 > 10.0.4.242.34278: Flags [F.], cksum 0xee3d (correct), seq 33, ack 62, win 251, options [nop,nop,TS val 1826134337 ecr 1768478986], length 0
	0x0000:  4500 0034 b32d 4000 3f06 6ea9 0a00 00fc  E..4.-@.?.n.....
	0x0010:  0a00 04f2 006f 85e6 ab44 44b1 2fbb 7f02  .....o...DD./...
	0x0020:  8011 00fb ee3d 0000 0101 080a 6cd8 9941  .....=......l..A
	0x0030:  6968 d90a                                ih..
15:02:25.232581 IP (tos 0x0, ttl 64, id 50094, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.34278 > 10.0.0.252.111: Flags [.], cksum 0x1a14 (incorrect -> 0xee3a), seq 62, ack 34, win 254, options [nop,nop,TS val 1768478986 ecr 1826134337], length 0
	0x0000:  4500 0034 c3ae 4000 4006 5d28 0a00 04f2  E..4..@.@.](....
	0x0010:  0a00 00fc 85e6 006f 2fbb 7f02 ab44 44b2  .......o/....DD.
	0x0020:  8010 00fe 1a14 0000 0101 080a 6968 d90a  ............ih..
	0x0030:  6cd8 9941                                l..A
15:02:25.232637 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [S.], cksum 0xc65f (correct), seq 312290074, ack 3636937519, win 64308, options [mss 1410,sackOK,TS val 1826134337 ecr 1768478986,nop,wscale 8], length 0
	0x0000:  4500 003c 0000 4000 4006 20cf 0a00 00fc  E..<..@.@.......
	0x0010:  0a00 04f2 0801 c262 129d 2b1a d8c7 432f  .......b..+...C/
	0x0020:  a012 fb34 c65f 0000 0204 0582 0402 080a  ...4._..........
	0x0030:  6cd8 9941 6968 d90a 0103 0308            l..Aih......
15:02:25.232677 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [S.], cksum 0xc65f (correct), seq 312290074, ack 3636937519, win 64308, options [mss 1410,sackOK,TS val 1826134337 ecr 1768478986,nop,wscale 8], length 0
	0x0000:  4500 003c 0000 4000 3f06 21cf 0a00 00fc  E..<..@.?.!.....
	0x0010:  0a00 04f2 0801 c262 129d 2b1a d8c7 432f  .......b..+...C/
	0x0020:  a012 fb34 c65f 0000 0204 0582 0402 080a  ...4._..........
	0x0030:  6cd8 9941 6968 d90a 0103 0308            l..Aih......
15:02:25.232681 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [S.], cksum 0xc65f (correct), seq 312290074, ack 3636937519, win 64308, options [mss 1410,sackOK,TS val 1826134337 ecr 1768478986,nop,wscale 8], length 0
	0x0000:  4500 003c 0000 4000 3f06 21cf 0a00 00fc  E..<..@.?.!.....
	0x0010:  0a00 04f2 0801 c262 129d 2b1a d8c7 432f  .......b..+...C/
	0x0020:  a012 fb34 c65f 0000 0204 0582 0402 080a  ...4._..........
	0x0030:  6cd8 9941 6968 d90a 0103 0308            l..Aih......
15:02:25.232814 IP (tos 0x0, ttl 64, id 36896, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.49762 > 10.0.0.252.2049: Flags [.], cksum 0x1a14 (incorrect -> 0xef31), seq 1, ack 1, win 254, options [nop,nop,TS val 1768478986 ecr 1826134337], length 0
	0x0000:  4500 0034 9020 4000 4006 90b6 0a00 04f2  E..4..@.@.......
	0x0010:  0a00 00fc c262 0801 d8c7 432f 129d 2b1b  .....b....C/..+.
	0x0020:  8010 00fe 1a14 0000 0101 080a 6968 d90a  ............ih..
	0x0030:  6cd8 9941                                l..A
15:02:25.233022 IP (tos 0x0, ttl 64, id 36897, offset 0, flags [DF], proto TCP (6), length 96)
    10.0.4.242.49762 > 10.0.0.252.2049: Flags [P.], cksum 0x1a40 (incorrect -> 0xcbcd), seq 1:45, ack 1, win 254, options [nop,nop,TS val 1768478987 ecr 1826134337], length 44: NFS request xid 3534571951 40 null
	0x0000:  4500 0060 9021 4000 4006 9089 0a00 04f2  E..`.!@.@.......
	0x0010:  0a00 00fc c262 0801 d8c7 432f 129d 2b1b  .....b....C/..+.
	0x0020:  8018 00fe 1a40 0000 0101 080a 6968 d90b  [email protected]..
	0x0030:  6cd8 9941 8000 0028 d2ad 49af 0000 0000  l..A...(..I.....
	0x0040:  0000 0002 0001 86a3 0000 0003 0000 0000  ................
	0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
15:02:25.233357 IP (tos 0x0, ttl 64, id 41130, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [.], cksum 0xef05 (correct), seq 1, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 0
	0x0000:  4500 0034 a0aa 4000 4006 802c 0a00 00fc  E..4..@.@..,....
	0x0010:  0a00 04f2 0801 c262 129d 2b1b d8c7 435b  .......b..+...C[
	0x0020:  8010 00fc ef05 0000 0101 080a 6cd8 9942  ............l..B
	0x0030:  6968 d90b                                ih..
15:02:25.233407 IP (tos 0x0, ttl 63, id 41130, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [.], cksum 0xef05 (correct), seq 1, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 0
	0x0000:  4500 0034 a0aa 4000 3f06 812c 0a00 00fc  E..4..@.?..,....
	0x0010:  0a00 04f2 0801 c262 129d 2b1b d8c7 435b  .......b..+...C[
	0x0020:  8010 00fc ef05 0000 0101 080a 6cd8 9942  ............l..B
	0x0030:  6968 d90b                                ih..
15:02:25.233411 IP (tos 0x0, ttl 63, id 41130, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [.], cksum 0xef05 (correct), seq 1, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 0
	0x0000:  4500 0034 a0aa 4000 3f06 812c 0a00 00fc  E..4..@.?..,....
	0x0010:  0a00 04f2 0801 c262 129d 2b1b d8c7 435b  .......b..+...C[
	0x0020:  8010 00fc ef05 0000 0101 080a 6cd8 9942  ............l..B
	0x0030:  6968 d90b                                ih..
15:02:25.233619 IP (tos 0x0, ttl 64, id 41131, offset 0, flags [DF], proto TCP (6), length 80)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [P.], cksum 0x526b (correct), seq 1:29, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 28: NFS reply xid 3534571951 reply ok 24 null
	0x0000:  4500 0050 a0ab 4000 4006 800f 0a00 00fc  E..P..@.@.......
	0x0010:  0a00 04f2 0801 c262 129d 2b1b d8c7 435b  .......b..+...C[
	0x0020:  8018 00fc 526b 0000 0101 080a 6cd8 9942  ....Rk......l..B
	0x0030:  6968 d90b 8000 0018 d2ad 49af 0000 0001  ih........I.....
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
15:02:25.233658 IP (tos 0x0, ttl 63, id 41131, offset 0, flags [DF], proto TCP (6), length 80)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [P.], cksum 0x526b (correct), seq 1:29, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 28: NFS reply xid 3534571951 reply ok 24 null
	0x0000:  4500 0050 a0ab 4000 3f06 810f 0a00 00fc  E..P..@.?.......
	0x0010:  0a00 04f2 0801 c262 129d 2b1b d8c7 435b  .......b..+...C[
	0x0020:  8018 00fc 526b 0000 0101 080a 6cd8 9942  ....Rk......l..B
	0x0030:  6968 d90b 8000 0018 d2ad 49af 0000 0001  ih........I.....
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
15:02:25.233662 IP (tos 0x0, ttl 63, id 41131, offset 0, flags [DF], proto TCP (6), length 80)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [P.], cksum 0x526b (correct), seq 1:29, ack 45, win 252, options [nop,nop,TS val 1826134338 ecr 1768478987], length 28: NFS reply xid 3534571951 reply ok 24 null
	0x0000:  4500 0050 a0ab 4000 3f06 810f 0a00 00fc  E..P..@.?.......
	0x0010:  0a00 04f2 0801 c262 129d 2b1b d8c7 435b  .......b..+...C[
	0x0020:  8018 00fc 526b 0000 0101 080a 6cd8 9942  ....Rk......l..B
	0x0030:  6968 d90b 8000 0018 d2ad 49af 0000 0001  ih........I.....
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
15:02:25.233760 IP (tos 0x0, ttl 64, id 36898, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.49762 > 10.0.0.252.2049: Flags [.], cksum 0x1a14 (incorrect -> 0xeee7), seq 45, ack 29, win 254, options [nop,nop,TS val 1768478987 ecr 1826134338], length 0
	0x0000:  4500 0034 9022 4000 4006 90b4 0a00 04f2  E..4."@.@.......
	0x0010:  0a00 00fc c262 0801 d8c7 435b 129d 2b37  .....b....C[..+7
	0x0020:  8010 00fe 1a14 0000 0101 080a 6968 d90b  ............ih..
	0x0030:  6cd8 9942                                l..B
15:02:25.233910 IP (tos 0x0, ttl 64, id 36899, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.49762 > 10.0.0.252.2049: Flags [F.], cksum 0x1a14 (incorrect -> 0xeee6), seq 45, ack 29, win 254, options [nop,nop,TS val 1768478987 ecr 1826134338], length 0
	0x0000:  4500 0034 9023 4000 4006 90b3 0a00 04f2  E..4.#@.@.......
	0x0010:  0a00 00fc c262 0801 d8c7 435b 129d 2b37  .....b....C[..+7
	0x0020:  8011 00fe 1a14 0000 0101 080a 6968 d90b  ............ih..
	0x0030:  6cd8 9942                                l..B
15:02:25.234214 IP (tos 0x0, ttl 64, id 41132, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [F.], cksum 0xeee6 (correct), seq 29, ack 46, win 252, options [nop,nop,TS val 1826134339 ecr 1768478987], length 0
	0x0000:  4500 0034 a0ac 4000 4006 802a 0a00 00fc  E..4..@.@..*....
	0x0010:  0a00 04f2 0801 c262 129d 2b37 d8c7 435c  .......b..+7..C\
	0x0020:  8011 00fc eee6 0000 0101 080a 6cd8 9943  ............l..C
	0x0030:  6968 d90b                                ih..
15:02:25.234261 IP (tos 0x0, ttl 63, id 41132, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [F.], cksum 0xeee6 (correct), seq 29, ack 46, win 252, options [nop,nop,TS val 1826134339 ecr 1768478987], length 0
	0x0000:  4500 0034 a0ac 4000 3f06 812a 0a00 00fc  E..4..@.?..*....
	0x0010:  0a00 04f2 0801 c262 129d 2b37 d8c7 435c  .......b..+7..C\
	0x0020:  8011 00fc eee6 0000 0101 080a 6cd8 9943  ............l..C
	0x0030:  6968 d90b                                ih..
15:02:25.234262 IP (tos 0x0, ttl 63, id 41132, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.0.252.2049 > 10.0.4.242.49762: Flags [F.], cksum 0xeee6 (correct), seq 29, ack 46, win 252, options [nop,nop,TS val 1826134339 ecr 1768478987], length 0
	0x0000:  4500 0034 a0ac 4000 3f06 812a 0a00 00fc  E..4..@.?..*....
	0x0010:  0a00 04f2 0801 c262 129d 2b37 d8c7 435c  .......b..+7..C\
	0x0020:  8011 00fc eee6 0000 0101 080a 6cd8 9943  ............l..C
	0x0030:  6968 d90b                                ih..
15:02:25.234357 IP (tos 0x0, ttl 64, id 36900, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.4.242.49762 > 10.0.0.252.2049: Flags [.], cksum 0x1a14 (incorrect -> 0xeee3), seq 46, ack 30, win 254, options [nop,nop,TS val 1768478988 ecr 1826134339], length 0
	0x0000:  4500 0034 9024 4000 4006 90b2 0a00 04f2  E..4.$@.@.......
	0x0010:  0a00 00fc c262 0801 d8c7 435c 129d 2b38  .....b....C\..+8
	0x0020:  8010 00fe 1a14 0000 0101 080a 6968 d90c  ............ih..
	0x0030:  6cd8 9943                                l..C
15:02:25.234533 IP (tos 0x0, ttl 64, id 28657, offset 0, flags [DF], proto UDP (17), length 68)
    10.0.4.242.1004 > 10.0.0.252.20048: [bad udp cksum 0x1a2f -> 0x23cf!] UDP, length 40
	0x0000:  4500 0044 6ff1 4000 4011 b0ca 0a00 04f2  E..Do.@.@.......
	0x0010:  0a00 00fc 03ec 4e50 0030 1a2f 9ab6 4e33  ......NP.0./..N3
	0x0020:  0000 0000 0000 0002 0001 86a5 0000 0003  ................
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0040:  0000 0000                                ....
15:02:25.234978 IP (tos 0x0, ttl 64, id 8996, offset 0, flags [DF], proto UDP (17), length 52)
    10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 24
	0x0000:  4500 0034 2324 4000 4011 fda7 0a00 00fc  E..4#$@.@.......
	0x0010:  0a00 04f2 4e50 03ec 0020 aa99 9ab6 4e33  ....NP........N3
	0x0020:  0000 0001 0000 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0000                                ....
15:02:25.235033 IP (tos 0x0, ttl 63, id 8996, offset 0, flags [DF], proto UDP (17), length 52)
    10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 24
	0x0000:  4500 0034 2324 4000 3f11 fea7 0a00 00fc  E..4#$@.?.......
	0x0010:  0a00 04f2 4e50 03ec 0020 aa99 9ab6 4e33  ....NP........N3
	0x0020:  0000 0001 0000 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0000                                ....
15:02:25.235036 IP (tos 0x0, ttl 63, id 8996, offset 0, flags [DF], proto UDP (17), length 52)
    10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 24
	0x0000:  4500 0034 2324 4000 3f11 fea7 0a00 00fc  E..4#$@.?.......
	0x0010:  0a00 04f2 4e50 03ec 0020 aa99 9ab6 4e33  ....NP........N3
	0x0020:  0000 0001 0000 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0000                                ....
15:02:25.235253 IP (tos 0x0, ttl 64, id 28658, offset 0, flags [DF], proto UDP (17), length 192)
    10.0.4.242.1004 > 10.0.0.252.20048: [bad udp cksum 0x1aab -> 0x17b9!] UDP, length 164
	0x0000:  4500 00c0 6ff2 4000 4011 b04d 0a00 04f2  E...o.@[email protected]....
	0x0010:  0a00 00fc 03ec 4e50 00ac 1aab 9bb6 4e33  ......NP......N3
	0x0020:  0000 0000 0000 0002 0001 86a5 0000 0003  ................
	0x0030:  0000 0001 0000 0001 0000 0048 0000 0000  ...........H....
	0x0040:  0000 002d 766d 392d 6e6f 6465 322e 3233  ...-vm9-node2.23
	0x0050:  2d31 3036 2d36 312d 3139 332e 6c6f 6e2d  -106-61-193.lon-
	0x0060:  3031 2e75 6b2e 6170 7073 6f6c 6f2e 636f  01.uk.appsolo.co
	0x0070:  6d00 0000 0000 0000 0000 0000 0000 0001  m...............
	0x0080:  0000 0000 0000 0000 0000 0000 0000 0030  ...............0
	0x0090:  2f65 7870 6f72 742f 7076 632d 6161 6430  /export/pvc-aad0
	0x00a0:  6131 3964 2d38 6366 612d 3433 3434 2d62  a19d-8cfa-4344-b
	0x00b0:  3538 632d 3539 6639 3561 6366 6363 6637  58c-59f95acfccf7
15:02:25.235818 IP (tos 0x0, ttl 64, id 8997, offset 0, flags [DF], proto UDP (17), length 96)
    10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 68
	0x0000:  4500 0060 2325 4000 4011 fd7a 0a00 00fc  E..`#%@[email protected]....
	0x0010:  0a00 04f2 4e50 03ec 004c 759b 9bb6 4e33  ....NP...Lu...N3
	0x0020:  0000 0001 0000 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0000 0000 0000 0000 001c 4300 0001  ............C...
	0x0040:  1645 9300 0000 e803 0000 8187 0000 0000  .E..............
	0x0050:  0000 00c4 2cf1 b000 0000 0001 0000 0001  ....,...........
15:02:25.235864 IP (tos 0x0, ttl 63, id 8997, offset 0, flags [DF], proto UDP (17), length 96)
    10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 68
	0x0000:  4500 0060 2325 4000 3f11 fe7a 0a00 00fc  E..`#%@.?..z....
	0x0010:  0a00 04f2 4e50 03ec 004c 759b 9bb6 4e33  ....NP...Lu...N3
	0x0020:  0000 0001 0000 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0000 0000 0000 0000 001c 4300 0001  ............C...
	0x0040:  1645 9300 0000 e803 0000 8187 0000 0000  .E..............
	0x0050:  0000 00c4 2cf1 b000 0000 0001 0000 0001  ....,...........
15:02:25.235865 IP (tos 0x0, ttl 63, id 8997, offset 0, flags [DF], proto UDP (17), length 96)
    10.0.0.252.20048 > 10.0.4.242.1004: [udp sum ok] UDP, length 68
	0x0000:  4500 0060 2325 4000 3f11 fe7a 0a00 00fc  E..`#%@.?..z....
	0x0010:  0a00 04f2 4e50 03ec 004c 759b 9bb6 4e33  ....NP...Lu...N3
	0x0020:  0000 0001 0000 0000 0000 0000 0000 0000  ................
	0x0030:  0000 0000 0000 0000 0000 001c 4300 0001  ............C...
	0x0040:  1645 9300 0000 e803 0000 8187 0000 0000  .E..............
	0x0050:  0000 00c4 2cf1 b000 0000 0001 0000 0001  ....,...........

@kvaps
Copy link
Member

kvaps commented Jun 8, 2022

I was testing it with cilium in kube-proxy free mode, it was working fine in my cluster.
Do you use kube-proxy?

@Rid
Copy link

Rid commented Jun 8, 2022

@kvaps I'm not using kube-proxy, however I've found if I force delete the pods using the pvs and force unmount the NFS mounts, it works.

Were you deleting the pods consuming the pvs? I was trying to avoid this.

@kvaps
Copy link
Member

kvaps commented Jun 8, 2022

No it was working to me without any additional removings

@Rid
Copy link

Rid commented Jun 8, 2022

@kvaps Could you share your mount options and any other server options? It looks like for some reason the mounts are not timing out, they're just stuck.

@kvaps
Copy link
Member

kvaps commented Jun 8, 2022

Unfortunately I have no this setup anymore.
The behavior reminds me of my old problem, but you don't use kube-proxy and IPVS though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants