DMS cannot connect to oracle
<!---
Please read this!
Before opening a new bug, make sure to search for keywords in the issues
filtered by "bug" label:
- https://gitlab.com/groups/nunet/-/issues/?label_name=type::bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
DMS fails to initiate an ML on GPU job when triggered by the Service provider dashboard webapp.
Webconsole has a 503 error cannot connect to oracle
### Steps to reproduce
From a machine running DMS 0.4.69 and the SPD 0.1.1
Ensure the machine has DHT peers that have a GPU that has been onboarded.
open the service provider dashboard and run a GPU ML job
select the default cpu job (https://gitlab.com/nunet/ml-on-gpu/ml-on-gpu-service/-/raw/develop/examples/pytorch/cifar-10_checkpointed.py)
specify a low job with a 120 minute runtime
connect wallet and click next.
specify max tokens amount as 10
check the recaptcha
click submit
NOTE it doesn't ask to sign transaction, get a 503 error in the webconsole
### What is the current *bug* behavior?
When running the compute provider webapp to initiate a GPU ML job the compute provider fails to execute the job.
The webapp just stays busy (shows its busy animation on the submit button) with no error message.
### What is the expected *correct* behavior?
Job should be triggered and web app should confirm receipt of job / status
### Relevant logs and/or screenshots
please see screen shots of job config and dev console output from the browser
ran the process twice both logs attached.
``` May 03 15:27:58 sams-asus nunet-dms[3446]: * [/ip4/220.78.153.167/udp/51878/quic] dial backoff
May 03 15:27:58 sams-asus nunet-dms[3446]: * [/ip4/220.78.153.167/tcp/65306] dial backoff --- peer: QmSFgAXu7vsdhCihZGv5hVbUmJtsSQPu9jJ1LCi9TywxBK {"package": "internal"}
May 03 15:28:54 sams-asus nunet-dms[3446]: 2023-05-03T15:28:54.033+0700 ERROR machines/handlers.go:196 websocket: close 1001 (going away) {"package": "internal"}
May 03 15:28:54 sams-asus nunet-dms[3446]: gitlab.com/nunet/device-management-service/libp2p/machines.listenForDeploymentStatus
May 03 15:28:54 sams-asus nunet-dms[3446]: /tmp/builds/uUvV2XC1/0/nunet/device-management-service/libp2p/machines/handlers.go:196
May 03 15:28:55 sams-asus nunet-dms[3446]: [GIN] 2023/05/03 - 15:28:55 | 200 | 18.018599ms | ::1 | GET "/api/v1/run/deploy"
May 03 15:29:17 sams-asus nunet-dms[3446]: * [/ip4/91.112.68.174/udp/9000/quic] dial backoff
May 03 15:29:17 sams-asus nunet-dms[3446]: * [/ip4/91.112.68.174/udp/1024/quic] dial backoff
May 03 15:29:17 sams-asus nunet-dms[3446]: * [/ip4/91.112.68.174/tcp/1024] dial backoff --- peer: QmVoV75iKrJtuU5LujbKQUdbF91HQEGnQZDqnjzt5gSyBp {"package": "internal"}
May 03 15:29:51 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/udp/9000/quic/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] dial backoff
May 03 15:29:51 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/tcp/9000/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] dial backoff --- peer: QmWrzsqxjqrxJ65zZ93hMFFhkjro9v2Fqpj1dovhRBUkwS {"package": "internal"}
May 03 15:30:10 sams-asus nunet-dms[3446]: 2023-05-03T15:30:10.517+0700 ERROR machines/handlers.go:196 websocket: close 1001 (going away) {"package": "internal"}
May 03 15:30:10 sams-asus nunet-dms[3446]: gitlab.com/nunet/device-management-service/libp2p/machines.listenForDeploymentStatus
May 03 15:30:10 sams-asus nunet-dms[3446]: /tmp/builds/uUvV2XC1/0/nunet/device-management-service/libp2p/machines/handlers.go:196
May 03 15:30:11 sams-asus nunet-dms[3446]: [GIN] 2023/05/03 - 15:30:11 | 200 | 202.2µs | ::1 | GET "/api/v1/run/deploy"
May 03 15:31:02 sams-asus nunet-dms[3446]: 2023-05-03T15:31:02.636+0700 INFO machines/handlers.go:65 estimated ntx price 3.2736 {"package": "internal"}
May 03 15:31:18 sams-asus nunet-dms[3446]: 2023-05-03T15:31:18.946+0700 ERROR libp2p/network.go:132 failed to read pong message: %!w(*errors.errorString=&{stream reset}) {"package": "internal"}
May 03 15:31:18 sams-asus nunet-dms[3446]: gitlab.com/nunet/device-management-service/libp2p.PingPeer
May 03 15:31:18 sams-asus nunet-dms[3446]: /tmp/builds/uUvV2XC1/0/nunet/device-management-service/libp2p/network.go:132
May 03 15:31:18 sams-asus nunet-dms[3446]: gitlab.com/nunet/device-management-service/libp2p/machines.HandleRequestService
May 03 15:31:18 sams-asus nunet-dms[3446]: /tmp/builds/uUvV2XC1/0/nunet/device-management-service/libp2p/machines/handlers.go:82
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.(*Context).Next
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 03 15:31:18 sams-asus nunet-dms[3446]: go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin.Middleware.func1
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin@v0.39.0/gintrace.go:90
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.(*Context).Next
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/recovery.go:101
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.(*Context).Next
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.LoggerWithConfig.func1
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/logger.go:240
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.(*Context).Next
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/context.go:173
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/gin.go:616
May 03 15:31:18 sams-asus nunet-dms[3446]: github.com/gin-gonic/gin.(*Engine).ServeHTTP
May 03 15:31:18 sams-asus nunet-dms[3446]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/gin.go:572
May 03 15:31:18 sams-asus nunet-dms[3446]: net/http.serverHandler.ServeHTTP
May 03 15:31:18 sams-asus nunet-dms[3446]: /usr/local/go/src/net/http/server.go:2947
May 03 15:31:18 sams-asus nunet-dms[3446]: net/http.(*conn).serve
May 03 15:31:18 sams-asus nunet-dms[3446]: /usr/local/go/src/net/http/server.go:1991
May 03 15:31:20 sams-asus nunet-dms[3446]: 2023-05-03T15:31:20.950+0700 DEBUG machines/handlers.go:106 compute provider: %!(EXTRA models.PeerData={Qmehd5L8gQDRjZp4EnxLzJ1m9uDRGCUk7ac1Z24UQMsN9K true false [{unknown 0 0}] addr_test1qp2fs6lrtqrmsdvuyvqkktyk75n4k0v0g9fvptr4rswy26lpvvsd4j77vfl0m8qge95d0fc6ajc2wmfctf3rtg7y8t9splatvm {1 17000 0 15000 0 3 0 0} [] 1683102495}) {"package": "internal"}
May 03 15:31:20 sams-asus nunet-dms[3446]: 2023-05-03T15:31:20.950+0700 INFO machines/handlers.go:109 sending fund contract request to oracle {"package": "internal"}
May 03 15:31:20 sams-asus nunet-dms[3446]: 2023-05-03T15:31:20.955+0700 INFO oracle/oracle.go:69 sending funding request to oracle {"package": "oracle"}
May 03 15:31:21 sams-asus nunet-dms[3446]: 2023-05-03T15:31:21.955+0700 INFO oracle/oracle.go:72 funding request failed %vrpc error: code = DeadlineExceeded desc = context deadline exceeded {"package": "oracle"}
May 03 15:31:21 sams-asus nunet-dms[3446]: 2023-05-03T15:31:21.955+0700 INFO machines/handlers.go:112 sending fund contract request to oracle failed {"package": "internal"}
May 03 15:31:21 sams-asus nunet-dms[3446]: [GIN] 2023/05/03 - 15:31:21 | 503 | 19.347991966s | ::1 | POST "/api/v1/run/request-service"
May 03 15:31:57 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/tcp/9000/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] error opening relay circuit: NO_RESERVATION (204)
May 03 15:31:57 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/udp/9000/quic/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] concurrent active dial through the same relay failed with a protocol error --- peer: QmQyGQT1vQAAwGEXocp58gkqD6Tbp2FH4qDEPgbpA9YWL4 {"package": "internal"}
May 03 15:32:02 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/udp/9000/quic-v1/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] dial backoff
May 03 15:32:02 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/tcp/9000/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] dial backoff --- peer: QmRFJEtatFH8H3UeLD9d5ps9S3PsEuk14icCp8VkDtYN74 {"package": "internal"}
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/200.155.164.246/tcp/23864] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/200.170.240.174/tcp/18393] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/200.170.240.174/tcp/23864] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/200.170.240.174/tcp/61486] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/200.155.164.246/tcp/61486] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/200.155.164.246/tcp/18393] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/udp/9000/quic-v1/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] dial backoff
May 03 15:32:21 sams-asus nunet-dms[3446]: * [/ip4/65.109.11.100/tcp/9000/p2p/QmTuuV1DG2F94LS1qTg47oVRhU7ptLAfnfzsjvuE3xNhif/p2p-circuit] dial backoff --- peer: QmRLmeBzhaHhP6DtwsYz1bwA9ikBeMjGmu66y9wA6yZ5Z1 {"package": "internal"}
May 03 15:32:26 sams-asus nunet-dms[3446]: * [/ip4/220.78.153.167/udp/51878/quic] dial backoff
May 03 15:32:26 sams-asus nunet-dms[3446]: * [/ip4/220.78.153.167/tcp/65306] dial backoff --- peer: QmSFgAXu7vsdhCihZGv5hVbUmJtsSQPu9jJ1LCi9TywxBK {"package": "internal"}
### Version number of components and SO (if applicable)
DMS 0.4.69
SPD 0.1.1
### Possible fixes
Unsure
issue