Hi guys ! I'm trying to experiment with the kubeflowpipelinesorchestration and therefore i have to deploy kubeflow via the recipe k3d-modular. I am hitting several issues in the process. i have installed Helm, terraform, k3d and kubectl. I was getting the following error:
Error: Post "": dial tcp 127.0.0.1:80: connect: connection refused with kubernetes_namespace.k8s-workloads, on line 33, in resource "kubernetes_namespace" "k8s-workloads": 33: resource "kubernetes_namespace" "k8s-workloads" {
I checked the k3d docker and realized that another port was being used.
I attempted to update the configuration in order to change this port as:
k3d_kube_api = { host = "0.0.0.0" host_port = 46107 }
Since this was not taking into account, i therefore tried another workarround by changing the API port of my cluster using:
k3d cluster create --api-port 0.0.0.0:80 minimal-zenml-cluster
the error became:
```Error: the server rejected our request for an unknown reason (post namespaces)

with kubernetesnamespace.k8s-workloads, on line 33, in resource "kubernetesnamespace" "k8s-workloads":
33: resource "kubernetes_namespace" "k8s-workloads" {```
Does anyone have any idea of what is going on ?

I wanna add that i have tried the fix proposed on the github using terraform destroy and terraform apply, which didn't change anything. Thank you a lot for your help !

@jayesh.ext

Hey @nicolas, which operation system are you running this one? and is this the port that was used 6445 when you checked k3d-docker?

I am running :
``` - Docker version 20.10.17, build 100c701

• Ubuntu 20.04
• k3d version v5.4.8
  k3s version v1.25.6-k3s1 (default)
• kubectl:
  • Client Version: v1.26.2
  • Kustomize Version: v4.5.7
  • Server Version: v1.25.6+k3s1
• zenML version 0.35.1`` the port is changing for every time i create a new cluster. i also tried to fix it using arguments without any luck: currently i am using: "/bin/sh -c nginx-pr…" 80/tcp, 0.0.0.0:35905->6443/tcp k3d-minimal-zenml-cluster-serverlb`
  Thank you a lot !

I am sorry, i am still confused what is the exact problem here, can you please try destroy and delete any running container based on the recipe and run it again without any change, if it fails for the first time just try terrafrom apply for a second time, i will try to re-run the recipe and see if it’s still working as intended.

I am not sure what you mean by deleting any container. the container displayed in my previous message comes from the command k3d cluster create minimal-zenml-cluster

I meant any container that was created by the recipe itself if destroy command didn’t already delete them all. Now typically after getting connection refused error the best thing is to try to re-apply and let terraform try to match the desire state if the error was just a random thing that’s why i asked to redeploy again if you haven’t already.

i already did redo the procedure stated at which stated to do a terraform destroy followed by terraform destroy Though this didn't make any difference in my case

```Error: Post "": dial tcp 127.0.0.1:80: connect: connection refused

  with kubernetes_namespace.k8s-workloads,
  on  line 33, in resource "kubernetes_namespace" "k8s-workloads":
  33: resource "kubernetes_namespace" "k8s-workloads" {

WARNING:python_terraform:error: None
Error: Error running recipe k3d-modular: None
Please look at the error message to figure out why the command failed. If the error is due some wrong configuration, please consider checking the  file to verify if the inputs are correct. Most commonly, the command can fail due to a timeout error. In that case, please run zenml stack recipe deploy k3d-modular again.```

Further investigation lead me to see that it was actually failing at # kubernetes_namespace.k8s-workloads will be created

From there as stated on the documentation of zenml stack, i did

```cd k3d-modular
terraform destroy
terraform apply```

which lead to the same error:

```│ Error: Post "": dial tcp 127.0.0.1:80: connect: connection refused
│
│   with kubernetes_namespace.k8s-workloads,
│   on  line 33, in resource "kubernetes_namespace" "k8s-workloads":
│   33: resource "kubernetes_namespace" "k8s-workloads" {
│```

Hey @nicolas, can you please confirm if the command you are using is the following one
zenml stack recipe deploy k3d-modular without any additional arguments?

Hello ! Totally, i didn't change a thing

Ah i think i found a potential problem when running that, can you please try to run the command as the following zenml stack recipe deploy k3d-modular -i kubeflow -i minio --no-server

Thank you a lot for your time ! I have tried the command you proposed. The script is indeed running with it. Are you planning to fix this issue soon ? the hostname i am getting for kubeflow is the following : . Since i work behind a firewall with DNS rebind protection, and the stack is deployed on a server in my network, i can't connect through this reverse proxy. Would there be a way to disable the reverse proxy and access it using ports instead ?

@ansarialtaf23.aa

@safoine.ext Did you manage to effectively run the example based on the stack recipe provided ?

Hey @nicolas i haven’t tried to run it, you facing some issue with it?

Hello @safoine.ext, Indeed i am running into some issues. As i am just starting on kubernetes and terraform, it is a bit difficult for me to investigate. I did my best to fill up this . Would it be possible for you to try it out ?