Second Job ML job failed after successfully signing a transaction.
Summary
I have got a step or two further with testing and have successfully run one job using the SPD then started a second job that got as far as signing the transaction (a step further than yesterday) I get a job is submitted message from the SPD but the job never seems to complete looking as DMS logs it seems there a connection wrror while trying to create a new request. I see this error in the log:
WARN libp2p/messaging.go:109 Connection Error: invalid character 'O' looking for beginning of value
Steps to reproduce
Run a job successfully close the browser reopen the browser and open SPD to run another job. Sign transaction get job is submitted message
What is the current bug behavior?
Deployment request is not accepted by peer
What is the expected correct behavior?
deployment request should be accepted and job should run
Relevant logs and/or screenshots
cleaned log from both jobs below.
Filtered with the following command
sudo journalctl -f -n 10000 -u nunet-dms | grep -v "traces export" | grep -v "dial backoff" | grep -v "UpdateDHT Create Stream error:" | grep -v "Attempting to Send DHT Update to:" | grep -v "Sending DHT update to" | grep -v "dht update from:" | grep -v "Obtained peers for DHT update:"
May 17 13:41:23 sams-asus systemd[1]: Started NuNet Device Management Service..
May 17 13:41:28 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:41:28 | 200 | 18.1784ms | 127.0.0.1 | GET "/api/v1/onboarding/provisioned"
May 17 13:41:28 sams-asus nunet-dms[16863]: WARNING: failed to determine nodes: open /sys/devices/system/node: no such file or directory
May 17 13:41:28 sams-asus nunet-dms[16863]: WARNING: failed to read int from file: open /sys/class/drm/card0/device/numa_node: no such file or directory
May 17 13:41:28 sams-asus nunet-dms[16863]: WARNING: failed to determine nodes: open /sys/devices/system/node: no such file or directory
May 17 13:41:28 sams-asus nunet-dms[16863]: WARNING: error parsing the pci address "vgem"
May 17 13:41:28 sams-asus nunet-dms[16863]: 2023/05/17 13:41:28 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details.
May 17 13:41:28 sams-asus nunet-dms[16863]: 2023-05-17T13:41:28.665+0700 INFO libp2p/host.go:371 Self Peer Info QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P -> [/ip4/127.0.0.1/tcp/9000 /ip4/127.0.0.1/udp/9000/quic /ip4/172.20.39.150/tcp/9000 /ip4/172.20.39.150/udp/9000/quic]
May 17 13:41:28 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:29 sams-asus nunet-dms[16863]: 2023-05-17T13:41:29.571+0700 INFO libp2p/dht.go:35 Connected to Bootstrap Node QmQ2irHa8aFTLRhkbkQCRrounE4MbttNp8ki7Nmys4F9NP
May 17 13:41:29 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:29 sams-asus nunet-dms[16863]: 2023-05-17T13:41:29.626+0700 INFO libp2p/host.go:371 Self Peer Info QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P -> [/ip4/127.0.0.1/tcp/9000 /ip4/172.20.39.150/tcp/9000]
May 17 13:41:29 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:30 sams-asus nunet-dms[16863]: 2023-05-17T13:41:30.504+0700 INFO libp2p/dht.go:35 Connected to Bootstrap Node QmQ2irHa8aFTLRhkbkQCRrounE4MbttNp8ki7Nmys4F9NP
May 17 13:41:30 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:30 sams-asus nunet-dms[16863]: 2023-05-17T13:41:30.685+0700 INFO libp2p/dht.go:35 Connected to Bootstrap Node Qmf16N2ecJVWufa29XKLNyiBxKWqVPNZXjbL3JisPcGqTw
May 17 13:41:30 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:31 sams-asus nunet-dms[16863]: 2023-05-17T13:41:31.580+0700 INFO libp2p/dht.go:35 Connected to Bootstrap Node Qmf16N2ecJVWufa29XKLNyiBxKWqVPNZXjbL3JisPcGqTw
May 17 13:41:31 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:32 sams-asus nunet-dms[16863]: 2023-05-17T13:41:32.091+0700 INFO libp2p/dht.go:35 Connected to Bootstrap Node QmTkWP72uECwCsiiYDpCFeTrVeUM9huGTPsg3m6bHxYQFZ
May 17 13:41:32 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:32 sams-asus nunet-dms[16863]: 2023-05-17T13:41:32.091+0700 INFO libp2p/dht.go:39 Done Bootstrapping {"package": "utils"}
May 17 13:41:32 sams-asus nunet-dms[16863]: 2023-05-17T13:41:32.091+0700 ERROR libp2p/dht.go:113 UpdateDHT error: unable to read self peer_info: item not found {"package": "utils"}
May 17 13:41:32 sams-asus nunet-dms[16863]: gitlab.com/nunet/device-management-service/libp2p.UpdateDHT
May 17 13:41:32 sams-asus nunet-dms[16863]: /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/dht.go:113
May 17 13:41:32 sams-asus nunet-dms[16863]: 2023-05-17T13:41:32.853+0700 INFO statsdb/stats_db.go:116 Responding: QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P {"package": "utils"}
May 17 13:41:32 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:41:32 | 201 | 4.444572732s | 127.0.0.1 | POST "/api/v1/onboarding/onboard"
May 17 13:41:32 sams-asus nunet-dms[16863]: 2023-05-17T13:41:32.999+0700 INFO libp2p/dht.go:35 Connected to Bootstrap Node QmTkWP72uECwCsiiYDpCFeTrVeUM9huGTPsg3m6bHxYQFZ
May 17 13:41:32 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:41:32 sams-asus nunet-dms[16863]: 2023-05-17T13:41:32.999+0700 INFO libp2p/dht.go:39 Done Bootstrapping {"package": "utils"}
May 17 13:41:50 sams-asus nunet-dms[16863]: * [/ip4/213.227.68.204/tcp/9000/p2p/Qmf3qWBPJF7LoSgXVAUxxJEuj7uUi9DZmMpZ6dGjcQbpQj/p2p-circuit] error opening hop stream to relay: context deadline exceeded
May 17 13:41:50 sams-asus nunet-dms[16863]: * [/ip4/213.227.68.204/udp/9000/quic/p2p/Qmf3qWBPJF7LoSgXVAUxxJEuj7uUi9DZmMpZ6dGjcQbpQj/p2p-circuit] error opening hop stream to relay: context deadline exceeded --- peer: QmPnRU8GcqAtyzBQPsBkpKEknCjC3a9V4RyrC3zQqpMNL5 {"package": "utils"}
May 17 13:41:55 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmRdBDsNZN6VW4vYKts8r7ChvYah9KXsLFmw8vQjjhtKrG, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmRdBDsNZN6VW4vYKts8r7ChvYah9KXsLFmw8vQjjhtKrG {"package": "utils"}
May 17 13:42:00 sams-asus nunet-dms[16863]: * [/ip4/172.20.39.150/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmSZP7NZKD4m1Phz2XcwbCKGfABgy7oKZ6DMNj8e4YvWMY, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 13:42:00 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmSZP7NZKD4m1Phz2XcwbCKGfABgy7oKZ6DMNj8e4YvWMY, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmSZP7NZKD4m1Phz2XcwbCKGfABgy7oKZ6DMNj8e4YvWMY {"package": "utils"}
May 17 13:42:10 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:10 | 200 | 1.4977ms | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:42:10 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:10 | 200 | 2.5418ms | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:42:14 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:14 | 200 | 813.1µs | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:42:14 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:14 | 200 | 278.2µs | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:42:15 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:15 | 200 | 549.9µs | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:42:15 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:15 | 200 | 380.3µs | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:42:35 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:35 | 200 | 613.2µs | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:42:35 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:42:35 | 200 | 292.6µs | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:42:55 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmYMYgmGFA3t5iuMZF8QPFqeaTSXazpxNyDYLKiumCub5X, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmYMYgmGFA3t5iuMZF8QPFqeaTSXazpxNyDYLKiumCub5X {"package": "utils"}
May 17 13:43:20 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:43:20 | 200 | 641.3µs | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:43:20 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:43:20 | 200 | 474.7µs | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:43:20 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmcbsRysE3vtR68bFTD8DJNZw6uXTJ5wwpqbdi4X1hsDNN, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 13:43:20 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.90/tcp/9000] dial tcp4 0.0.0.0:9000->192.168.1.90:9000: i/o timeout
May 17 13:43:20 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.90/udp/9000/quic] context deadline exceeded --- peer: QmcbsRysE3vtR68bFTD8DJNZw6uXTJ5wwpqbdi4X1hsDNN {"package": "utils"}
May 17 13:43:21 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:43:21 | 200 | 785.1µs | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:43:21 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:43:21 | 200 | 302.5µs | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:43:51 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmcbsRysE3vtR68bFTD8DJNZw6uXTJ5wwpqbdi4X1hsDNN, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 13:43:51 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.90/tcp/9000] dial tcp4 0.0.0.0:9000->192.168.1.90:9000: i/o timeout
May 17 13:43:51 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.90/udp/9000/quic] context deadline exceeded --- peer: QmcbsRysE3vtR68bFTD8DJNZw6uXTJ5wwpqbdi4X1hsDNN {"package": "utils"}
May 17 13:44:19 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmYAYEoBtVWfsrPXLhY6JcYxoCWuKAeezLHCzoasZuxeQY, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 13:44:19 sams-asus nunet-dms[16863]: * [/ip4/172.17.19.26/tcp/9000] dial tcp4 172.17.19.26:9000: i/o timeout
May 17 13:44:19 sams-asus nunet-dms[16863]: * [/ip4/172.17.19.26/udp/9000/quic] context deadline exceeded --- peer: QmYAYEoBtVWfsrPXLhY6JcYxoCWuKAeezLHCzoasZuxeQY {"package": "utils"}
May 17 13:44:21 sams-asus nunet-dms[16863]: * [/ip4/183.88.106.192/tcp/9000] conn-23534: system: cannot reserve connection: resource limit exceeded
May 17 13:44:21 sams-asus nunet-dms[16863]: * [/ip4/10.0.0.232/tcp/9000] conn-23535: system: cannot reserve connection: resource limit exceeded
May 17 13:44:21 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 13:44:21 sams-asus nunet-dms[16863]: * [/ip4/10.0.0.232/udp/9000/quic] timeout: no recent network activity
May 17 13:44:21 sams-asus nunet-dms[16863]: * [/ip4/183.88.106.192/udp/9000/quic] timeout: no recent network activity --- peer: Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi {"package": "utils"}
May 17 13:47:52 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:47:52 | 200 | 3.787399ms | 127.0.0.1 | GET "/api/v1/peers"
May 17 13:47:52 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:47:52 | 200 | 5.266798ms | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 13:48:07 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:48:07 | 200 | 6.3968ms | ::1 | GET "/api/v1/run/deploy"
May 17 13:48:49 sams-asus nunet-dms[16863]: 2023-05-17T13:48:49.351+0700 INFO machines/handlers.go:70 estimated ntx price 0.27280000000000004 {"package": "utils"}
May 17 13:48:54 sams-asus nunet-dms[16863]: 2023-05-17T13:48:54.361+0700 DEBUG machines/handlers.go:111 compute provider: %!(EXTRA models.PeerData={QmXgE14gtcMikdhsSTiKNEZhciDsE9ek6F1wpYv1zPybaZ false false [] addr_test1qrfcsw0g8fl9efwja89ds7rrvjjfneeda2apv5e309u5j5crw5mt799xefvtychnmry86nrmea5j6anqufc03s0l80ssmd04tt {1 35000 0 5000 0 9 0 0} [] 1684178569}) {"package": "utils"}
May 17 13:48:54 sams-asus nunet-dms[16863]: 2023-05-17T13:48:54.361+0700 ERROR machines/handlers.go:121 unable to extract public key from peer id: public key is not embedded in peer ID {"package": "utils"}
May 17 13:48:54 sams-asus nunet-dms[16863]: gitlab.com/nunet/device-management-service/libp2p/machines.HandleRequestService
May 17 13:48:54 sams-asus nunet-dms[16863]: /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/machines/handlers.go:121
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 13:48:54 sams-asus nunet-dms[16863]: go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin.Middleware.func1
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin@v0.39.0/gintrace.go:90
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/recovery.go:101
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.LoggerWithConfig.func1
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/logger.go:240
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/gin.go:616
May 17 13:48:54 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Engine).ServeHTTP
May 17 13:48:54 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/gin.go:572
May 17 13:48:54 sams-asus nunet-dms[16863]: net/http.serverHandler.ServeHTTP
May 17 13:48:54 sams-asus nunet-dms[16863]: /usr/local/go/src/net/http/server.go:2947
May 17 13:48:54 sams-asus nunet-dms[16863]: net/http.(*conn).serve
May 17 13:48:54 sams-asus nunet-dms[16863]: /usr/local/go/src/net/http/server.go:1991
May 17 13:48:54 sams-asus nunet-dms[16863]: 2023-05-17T13:48:54.364+0700 INFO machines/handlers.go:128 sending fund contract request to oracle {"package": "utils"}
May 17 13:48:54 sams-asus nunet-dms[16863]: 2023-05-17T13:48:54.370+0700 INFO oracle/oracle.go:93 sending funding request to oracle {"package": "oracle"}
May 17 13:48:55 sams-asus nunet-dms[16863]: 2023-05-17T13:48:55.347+0700 INFO oracle/oracle.go:105 funding response from oracle: {{{} [] [] <nil>} 0 [] 4cb02f736430017dee9434f6cb19b8ffe187b2be455783bfa84cd1039b6ba7c4b5614d4984ff844e59e916d22f6af83b1a4bccd6b3acd4fec2dcd53bc5890600 funding-58205135f1a62f46971ec831f46d87582de0b9079f5805c01bbcb0ebeeabc871dd94} {"package": "oracle"}
May 17 13:48:55 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:48:55 | 200 | 6.015541592s | ::1 | POST "/api/v1/run/request-service"
May 17 13:49:06 sams-asus nunet-dms[16863]: 2023-05-17T13:49:06.103+0700 DEBUG machines/handlers.go:299 deployment request: {addr_test1qqp4py9gw6u0tv3dajpav5t6h2zcpfxdmcnrfdrrcvkckhua8pf6hwnw9nztqndtnl497n8766382qhpyu4hrvsp5xdsk276r8 10 Cardano ml-training-cpu 2023-05-17 13:49:06.103695509 +0700 +07 m=+462.239952875 {registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu https://gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/-/raw/develop/examples/cpu-ml-test-scikit-learn.py [matplotlib] QmXgE14gtcMikdhsSTiKNEZhciDsE9ek6F1wpYv1zPybaZ QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P RSA } {Low 500 2000 2000 170 10} {0000000000000000 00000000000000000000000000000000 00 }} {"package": "utils"}
May 17 13:49:06 sams-asus nunet-dms[16863]: 2023-05-17T13:49:06.105+0700 INFO libp2p/messaging.go:242 Creating a new depReq! {"package": "utils"}
May 17 13:49:06 sams-asus nunet-dms[16863]: 2023-05-17T13:49:06.107+0700 DEBUG otelzap@v0.1.21/otelzap.go:399 deployment request: {"address_user":"addr_test1qqp4py9gw6u0tv3dajpav5t6h2zcpfxdmcnrfdrrcvkckhua8pf6hwnw9nztqndtnl497n8766382qhpyu4hrvsp5xdsk276r8","max_ntx":10,"blockchain":"Cardano","service_type":"ml-training-cpu","timestamp":"2023-05-17T13:49:06.103695509+07:00","params":{"image_id":"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu","model_url":"https://gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/-/raw/develop/examples/cpu-ml-test-scikit-learn.py","packages":["matplotlib"],"node_id":"QmXgE14gtcMikdhsSTiKNEZhciDsE9ek6F1wpYv1zPybaZ","public_key":"","local_node_id":"QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P","local_public_key":"RSA","machine_type":""},"constraints":{"complexity":"Low","cpu":500,"ram":2000,"vram":2000,"power":170,"time":10},"traceinfo":{"trace_id":"0000000000000000","span_id":"00000000000000000000000000000000","trace_flags":"00","trace_state":""}} {"package": "utils"}
May 17 13:49:36 sams-asus nunet-dms[16863]: 2023-05-17T13:49:36.238+0700 DEBUG libp2p/messaging.go:334 received raw data from stream: {"msg_type":"JobStatus","msg":"{\"ID\":1,\"CreatedAt\":\"2023-05-16T23:49:38.740449153-07:00\",\"UpdatedAt\":\"2023-05-16T23:49:38.740449153-07:00\",\"DeletedAt\":null,\"JobStatus\":\"running\",\"JobDuration\":5,\"EstimatedJobDuration\":10,\"ServiceName\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"ContainerID\":\"\",\"ResourceRequirements\":0,\"ImageID\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"LogURL\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 13:49:36 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:49:36 sams-asus nunet-dms[16863]: 2023-05-17T13:49:36.239+0700 DEBUG libp2p/messaging.go:131 received deployment update: {"msg_type":"JobStatus","msg":"{\"ID\":1,\"CreatedAt\":\"2023-05-16T23:49:38.740449153-07:00\",\"UpdatedAt\":\"2023-05-16T23:49:38.740449153-07:00\",\"DeletedAt\":null,\"JobStatus\":\"running\",\"JobDuration\":5,\"EstimatedJobDuration\":10,\"ServiceName\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"ContainerID\":\"\",\"ResourceRequirements\":0,\"ImageID\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"LogURL\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 13:49:36 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:51:21 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 13:51:21 sams-asus nunet-dms[16863]: * [/ip4/183.88.106.192/tcp/9000] failed to negotiate security protocol: read tcp4 172.20.39.150:9000->183.88.106.192:9000: read: connection reset by peer
May 17 13:51:21 sams-asus nunet-dms[16863]: * [/ip4/183.88.106.192/udp/9000/quic] timeout: no recent network activity
May 17 13:51:21 sams-asus nunet-dms[16863]: * [/ip4/10.0.0.232/udp/9000/quic] context deadline exceeded
May 17 13:51:21 sams-asus nunet-dms[16863]: * [/ip4/10.0.0.232/tcp/9000] dial tcp4 0.0.0.0:9000->10.0.0.232:9000: i/o timeout --- peer: Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi {"package": "utils"}
May 17 13:54:44 sams-asus nunet-dms[16863]: 2023-05-17T13:54:44.450+0700 DEBUG libp2p/messaging.go:334 received raw data from stream: {"msg_type":"DepResp","msg":"{\"success\":true,\"content\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 13:54:44 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:54:44 sams-asus nunet-dms[16863]: 2023-05-17T13:54:44.450+0700 DEBUG libp2p/messaging.go:131 received deployment update: {"msg_type":"DepResp","msg":"{\"success\":true,\"content\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 13:54:44 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:54:44 sams-asus nunet-dms[16863]: 2023-05-17T13:54:44.450+0700 DEBUG libp2p/messaging.go:166 received deployment response: {"msg_type":"DepResp","msg":"{\"success\":true,\"content\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 13:54:44 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 13:54:44 sams-asus nunet-dms[16863]: 2023-05-17T13:54:44.451+0700 DEBUG libp2p/messaging.go:179 deployment update message model: {true https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7} {"package": "utils"}
May 17 13:54:44 sams-asus nunet-dms[16863]: 2023-05-17T13:54:44.451+0700 INFO machines/handlers.go:219 Deployment response received. Sending it to connected websocket client {"package": "utils"}
May 17 13:54:44 sams-asus nunet-dms[16863]: 2023-05-17T13:54:44.453+0700 DEBUG machines/handlers.go:234 deployment response to websock: {"action":"deployment-response","message":{"success":true,"content":"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7"}} {"package": "utils"}
May 17 13:54:48 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:54:48 | 200 | 4.1632ms | 127.0.0.1 | GET "/api/v1/peers/self"
May 17 13:56:44 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 13:56:44 | 200 | 1.5292ms | 127.0.0.1 | GET "/api/v1/peers/self"
May 17 13:57:20 sams-asus nunet-dms[16863]: * [/ip4/172.17.19.26/udp/9000/quic] context deadline exceeded
May 17 13:57:20 sams-asus nunet-dms[16863]: * [/ip4/172.17.19.26/tcp/9000] dial tcp4 172.17.19.26:9000: i/o timeout --- peer: QmYAYEoBtVWfsrPXLhY6JcYxoCWuKAeezLHCzoasZuxeQY {"package": "utils"}
May 17 13:57:21 sams-asus nunet-dms[16863]: * [/ip4/172.17.19.26/udp/9000/quic] context deadline exceeded
May 17 13:57:21 sams-asus nunet-dms[16863]: * [/ip4/172.17.19.26/tcp/9000] dial tcp4 172.17.19.26:9000: i/o timeout --- peer: QmYAYEoBtVWfsrPXLhY6JcYxoCWuKAeezLHCzoasZuxeQY {"package": "utils"}
May 17 13:59:21 sams-asus nunet-dms[16863]: * [/ip4/213.227.68.204/udp/9000/quic/p2p/Qmf3qWBPJF7LoSgXVAUxxJEuj7uUi9DZmMpZ6dGjcQbpQj/p2p-circuit] error opening hop stream to relay: context deadline exceeded --- peer: QmPnRU8GcqAtyzBQPsBkpKEknCjC3a9V4RyrC3zQqpMNL5 {"package": "utils"}
May 17 14:01:20 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmYMYgmGFA3t5iuMZF8QPFqeaTSXazpxNyDYLKiumCub5X, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmYMYgmGFA3t5iuMZF8QPFqeaTSXazpxNyDYLKiumCub5X {"package": "utils"}
May 17 14:01:49 sams-asus nunet-dms[16863]: 2023-05-17T14:01:49.799+0700 DEBUG libp2p/messaging.go:334 received raw data from stream: {"msg_type":"JobStatus","msg":"{\"ID\":1,\"CreatedAt\":\"2023-05-16T23:49:38.740449153-07:00\",\"UpdatedAt\":\"2023-05-17T00:01:52.316066883-07:00\",\"DeletedAt\":null,\"JobStatus\":\"finished without errors\",\"JobDuration\":5,\"EstimatedJobDuration\":10,\"ServiceName\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"ContainerID\":\"9b5ba8395e10bb60d3ec32c10d027f145c321a94c5400faf57fb4e6238f35b96\",\"ResourceRequirements\":0,\"ImageID\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"LogURL\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 14:01:49 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 14:01:49 sams-asus nunet-dms[16863]: 2023-05-17T14:01:49.799+0700 DEBUG libp2p/messaging.go:131 received deployment update: {"msg_type":"JobStatus","msg":"{\"ID\":1,\"CreatedAt\":\"2023-05-16T23:49:38.740449153-07:00\",\"UpdatedAt\":\"2023-05-17T00:01:52.316066883-07:00\",\"DeletedAt\":null,\"JobStatus\":\"finished without errors\",\"JobDuration\":5,\"EstimatedJobDuration\":10,\"ServiceName\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"ContainerID\":\"9b5ba8395e10bb60d3ec32c10d027f145c321a94c5400faf57fb4e6238f35b96\",\"ResourceRequirements\":0,\"ImageID\":\"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu\",\"LogURL\":\"https://gist.github.com/nunet-dev/f8bc1a1519c6369413ceea79f824c1d7\"}"}
May 17 14:01:49 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 14:01:49 sams-asus nunet-dms[16863]: 2023-05-17T14:01:49.810+0700 DEBUG libp2p/messaging.go:124 Stream closed with EOF, ending read loop {"package": "utils"}
May 17 14:01:51 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmeUgD1PKSfBYFQhFAwG7o7B5B25LdfXjAX71X7cP42LeE, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmeUgD1PKSfBYFQhFAwG7o7B5B25LdfXjAX71X7cP42LeE {"package": "utils"}
May 17 14:05:21 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmeUgD1PKSfBYFQhFAwG7o7B5B25LdfXjAX71X7cP42LeE, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmeUgD1PKSfBYFQhFAwG7o7B5B25LdfXjAX71X7cP42LeE {"package": "utils"}
May 17 14:05:52 sams-asus nunet-dms[16863]: 2023-05-17T14:05:52.936+0700 ERROR machines/handlers.go:202 websocket: close 1001 (going away) {"package": "utils"}
May 17 14:05:52 sams-asus nunet-dms[16863]: gitlab.com/nunet/device-management-service/libp2p/machines.listenToOutgoingMessage
May 17 14:05:52 sams-asus nunet-dms[16863]: /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/machines/handlers.go:202
May 17 14:06:05 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 14:06:05 | 200 | 3.7698ms | ::1 | GET "/api/v1/run/deploy"
May 17 14:06:20 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmX4kUvhaqQeqBeEGVgTas1p16MEJwnSc344qBtn6J2bde, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P --- peer: QmX4kUvhaqQeqBeEGVgTas1p16MEJwnSc344qBtn6J2bde {"package": "utils"}
May 17 14:06:33 sams-asus nunet-dms[16863]:
May 17 14:06:33 sams-asus nunet-dms[16863]: 2023/05/17 14:06:33 /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/network.go:38 SLOW SQL >= 200ms
May 17 14:06:33 sams-asus nunet-dms[16863]: [348.192ms] [rows:1] UPDATE `connections` SET `updated_at`="2023-05-17 14:06:33.184" WHERE peer_id = "QmUJnxnQhG6s8TdfZABr6VRnsHkQkW7CRE6X94JcCwiRk4" AND `connections`.`deleted_at` IS NULL AND `id` = 27
May 17 14:06:59 sams-asus nunet-dms[16863]: 2023-05-17T14:06:59.838+0700 INFO machines/handlers.go:70 estimated ntx price 0.27280000000000004 {"package": "utils"}
May 17 14:07:01 sams-asus nunet-dms[16863]: 2023-05-17T14:07:01.847+0700 DEBUG machines/handlers.go:111 compute provider: %!(EXTRA models.PeerData={Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi true false [{GA106 [GeForce RTX 3060 Lite Hash Rate] 0 0}] addr_test1qzcvk8agtjxqxhv3cehawn5fuj6lgw06d64m0787vzy3s0cnwg64s22csgql6tnd7ya9jr005mrqesj5scrjftwfte6qzlds0y {1 8000 0 8000 0 2 0 0} [] 1684307176}) {"package": "utils"}
May 17 14:07:01 sams-asus nunet-dms[16863]: 2023-05-17T14:07:01.850+0700 ERROR machines/handlers.go:121 unable to extract public key from peer id: public key is not embedded in peer ID {"package": "utils"}
May 17 14:07:01 sams-asus nunet-dms[16863]: gitlab.com/nunet/device-management-service/libp2p/machines.HandleRequestService
May 17 14:07:01 sams-asus nunet-dms[16863]: /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/machines/handlers.go:121
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 14:07:01 sams-asus nunet-dms[16863]: go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin.Middleware.func1
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin@v0.39.0/gintrace.go:90
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/recovery.go:101
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.LoggerWithConfig.func1
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/logger.go:240
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Context).Next
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/gin.go:616
May 17 14:07:01 sams-asus nunet-dms[16863]: github.com/gin-gonic/gin.(*Engine).ServeHTTP
May 17 14:07:01 sams-asus nunet-dms[16863]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/gin.go:572
May 17 14:07:01 sams-asus nunet-dms[16863]: net/http.serverHandler.ServeHTTP
May 17 14:07:01 sams-asus nunet-dms[16863]: /usr/local/go/src/net/http/server.go:2947
May 17 14:07:01 sams-asus nunet-dms[16863]: net/http.(*conn).serve
May 17 14:07:01 sams-asus nunet-dms[16863]: /usr/local/go/src/net/http/server.go:1991
May 17 14:07:01 sams-asus nunet-dms[16863]: 2023-05-17T14:07:01.852+0700 INFO machines/handlers.go:128 sending fund contract request to oracle {"package": "utils"}
May 17 14:07:01 sams-asus nunet-dms[16863]: 2023-05-17T14:07:01.857+0700 INFO oracle/oracle.go:93 sending funding request to oracle {"package": "oracle"}
May 17 14:07:02 sams-asus nunet-dms[16863]: 2023-05-17T14:07:02.764+0700 INFO oracle/oracle.go:105 funding response from oracle: {{{} [] [] <nil>} 0 [] 36190f6bebe0c9f6429345828afaef184b87eadbcf3b83153c841394f44d739aabdae2ca1c5b5c1a182c595ea4c4563673b9536885c8d669d64a8744b3502d03 funding-582007734628c25e93e203dac2fc140aa9f6947be8e6240f033e3b34ab543e0487b3} {"package": "oracle"}
May 17 14:07:02 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 14:07:02 | 200 | 2.945153746s | ::1 | POST "/api/v1/run/request-service"
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.168+0700 DEBUG machines/handlers.go:299 deployment request: {addr_test1qqp4py9gw6u0tv3dajpav5t6h2zcpfxdmcnrfdrrcvkckhua8pf6hwnw9nztqndtnl497n8766382qhpyu4hrvsp5xdsk276r8 1 Cardano ml-training-cpu 2023-05-17 14:07:19.168417068 +0700 +07 m=+1555.304674434 {registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu https://gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/-/raw/develop/examples/cpu-ml-test-scikit-learn.py [matplotlib] Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P RSA } {Low 500 2000 2000 170 10} {0000000000000000 00000000000000000000000000000000 00 }} {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.170+0700 INFO libp2p/messaging.go:242 Creating a new depReq! {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.171+0700 DEBUG otelzap@v0.1.21/otelzap.go:399 deployment request: {"address_user":"addr_test1qqp4py9gw6u0tv3dajpav5t6h2zcpfxdmcnrfdrrcvkckhua8pf6hwnw9nztqndtnl497n8766382qhpyu4hrvsp5xdsk276r8","max_ntx":1,"blockchain":"Cardano","service_type":"ml-training-cpu","timestamp":"2023-05-17T14:07:19.168417068+07:00","params":{"image_id":"registry.gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/develop/ml-on-cpu","model_url":"https://gitlab.com/nunet/ml-on-gpu/ml-on-cpu-service/-/raw/develop/examples/cpu-ml-test-scikit-learn.py","packages":["matplotlib"],"node_id":"Qmd4k1wT5VPaxhihogbnRsbzS2gmbxY1ZXPHaSmB4Ygyzi","public_key":"","local_node_id":"QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P","local_public_key":"RSA","machine_type":""},"constraints":{"complexity":"Low","cpu":500,"ram":2000,"vram":2000,"power":170,"time":10},"traceinfo":{"trace_id":"0000000000000000","span_id":"00000000000000000000000000000000","trace_flags":"00","trace_state":""}} {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.179+0700 DEBUG libp2p/messaging.go:334 received raw data from stream: Open Stream Length Exceeded. Closing Stream.
May 17 14:07:19 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.179+0700 DEBUG libp2p/messaging.go:131 received deployment update: Open Stream Length Exceeded. Closing Stream.
May 17 14:07:19 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.180+0700 INFO libp2p/messaging.go:136 cannot unmarshal json {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: 2023-05-17T14:07:19.181+0700 WARN libp2p/messaging.go:109 Connection Error: invalid character 'O' looking for beginning of value
May 17 14:07:19 sams-asus nunet-dms[16863]: {"package": "utils"}
May 17 14:07:19 sams-asus nunet-dms[16863]: gitlab.com/nunet/device-management-service/libp2p.DeploymentUpdateListener.func1
May 17 14:07:19 sams-asus nunet-dms[16863]: /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/messaging.go:109
May 17 14:07:19 sams-asus nunet-dms[16863]: runtime.gopanic
May 17 14:07:19 sams-asus nunet-dms[16863]: /usr/local/go/src/runtime/panic.go:884
May 17 14:07:19 sams-asus nunet-dms[16863]: gitlab.com/nunet/device-management-service/libp2p.DeploymentUpdateListener
May 17 14:07:19 sams-asus nunet-dms[16863]: /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/messaging.go:137
May 17 14:09:20 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.222/udp/9000/quic] context deadline exceeded
May 17 14:09:20 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.222/tcp/9000] dial tcp4 0.0.0.0:9000->192.168.1.222:9000: i/o timeout
May 17 14:09:20 sams-asus nunet-dms[16863]: * [/ip4/81.28.127.80/tcp/64166/p2p/QmQyGQT1vQAAwGEXocp58gkqD6Tbp2FH4qDEPgbpA9YWL4/p2p-circuit] error opening hop stream to relay: context deadline exceeded --- peer: QmeUgD1PKSfBYFQhFAwG7o7B5B25LdfXjAX71X7cP42LeE {"package": "utils"}
May 17 14:11:17 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 14:11:17 | 200 | 1.6748ms | 127.0.0.1 | GET "/api/v1/peers/self"
May 17 14:11:20 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 14:11:20 | 200 | 2.7365ms | 127.0.0.1 | GET "/api/v1/peers"
May 17 14:11:20 sams-asus nunet-dms[16863]: [GIN] 2023/05/17 - 14:11:20 | 200 | 392.6µs | 127.0.0.1 | GET "/api/v1/peers/dht"
May 17 14:11:33 sams-asus nunet-dms[16863]:
May 17 14:11:33 sams-asus nunet-dms[16863]: 2023/05/17 14:11:33 /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/network.go:38 SLOW SQL >= 200ms
May 17 14:11:33 sams-asus nunet-dms[16863]: [437.904ms] [rows:1] INSERT INTO `connections` (`created_at`,`updated_at`,`deleted_at`,`peer_id`,`multiaddrs`) VALUES ("2023-05-17 14:11:33.009","2023-05-17 14:11:33.009",NULL,"12D3KooWSjcuZm8BqdpZGPQKwR2aZCU9b931SZznF4d3ai9bzZBn","/ip4/76.9.186.156/udp/4001/quic") RETURNING `id`
May 17 14:15:51 sams-asus nunet-dms[16863]: * [/ip4/127.0.0.1/udp/9000/quic] CRYPTO_ERROR 0x12a (local): peer IDs don't match: expected QmcbsRysE3vtR68bFTD8DJNZw6uXTJ5wwpqbdi4X1hsDNN, got QmcJtkCAR7bDbAJ9UWPFhCtVxHXahqJXwW6RQ6PMnmZ75P
May 17 14:15:51 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.90/tcp/9000] dial tcp4 0.0.0.0:9000->192.168.1.90:9000: i/o timeout
May 17 14:15:51 sams-asus nunet-dms[16863]: * [/ip4/192.168.1.90/udp/9000/quic] context deadline exceeded --- peer: QmcbsRysE3vtR68bFTD8DJNZw6uXTJ5wwpqbdi4X1hsDNN {"package": "utils"}
May 17 14:16:33 sams-asus nunet-dms[16863]:
May 17 14:16:33 sams-asus nunet-dms[16863]: 2023/05/17 14:16:33 /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/network.go:38 SLOW SQL >= 200ms
May 17 14:16:33 sams-asus nunet-dms[16863]: [339.126ms] [rows:1] INSERT INTO `connections` (`created_at`,`updated_at`,`deleted_at`,`peer_id`,`multiaddrs`) VALUES ("2023-05-17 14:16:33.027","2023-05-17 14:16:33.027",NULL,"12D3KooWBWFG1uQmv8aB8RYKh9LDz2SDdaQZUxZWe4ZXsPQDjMJK","/ip4/185.51.184.93/udp/8001/quic-v1") RETURNING `id`
May 17 14:16:33 sams-asus nunet-dms[16863]:
May 17 14:16:33 sams-asus nunet-dms[16863]: 2023/05/17 14:16:33 /tmp/builds/uUvV2XC1/2/nunet/device-management-service/libp2p/network.go:62 SLOW SQL >= 200ms
May 17 14:16:33 sams-asus nunet-dms[16863]: [435.416ms] [rows:1] UPDATE `connections` SET `deleted_at`="2023-05-17 14:16:33.357" WHERE `connections`.`id` = 1263 AND `connec
### Version number of components and SO (if applicable)
<!-- Inform the SO version and components version if applicable. -->
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->Edited by Samuel Lake