Logo-amall

Hello, I've got a question regarding Seldon model deployer step, that comes with seldon integration. I'm trying to set resources for pods created by Seldon, but the docs seem to be outdates (). The docs tell to set requests and limits with `resources` property of `SeldonDeploymentConfig` ; but this property does not exist in the config. What's the proper way to request and limit resources of Seldon Deployment pods in ZenML now?

Last active 9 days ago

19 replies

12 views

  • MI

    Hello, I've got a question regarding Seldon model deployer step, that comes with seldon integration. I'm trying to set resources for pods created by Seldon, but the docs seem to be outdates (). The docs tell to set requests and limits with resources property of SeldonDeploymentConfig ; but this property does not exist in the config. What's the proper way to request and limit resources of Seldon Deployment pods in ZenML now?

  • SA

    Hey @Misha Iakovlev I apologize for the inconvenience. Currently, the resource configuration is only available for use with the KServe Model Deployer and can’t be configured from within zenml pipeline at the moment. We are looking into ways to include support for this and other features in the Seldon client and will make every effort to make it available as soon as possible.

  • HA

    @Misha Iakovlev happy to receive a contribution if youre up for it!

  • MI

    @hamza I'll have a look to see if it's something I can have a quick stab at

  • MI

    I've spun up an additional node for now. However, I'm hitting another problem with Seldon Deployment.

    Seldon Deployment pod tries to use gs://<redacted>-artifact-store/seldon_model_deployer_step/output/<redacted>/seldon path to get the model from, but seldon_model_deployer does not exist in that bucket (every other step is present there). What might have happened here?

    The pipeline looks like this:

    def analytics_pipeline(
        wrap_metric,
        model_deployer
    ):
        ...
    
        # Wrap metrics into a dummy model for serving
        decision, model = wrap_metric(group_averages)
    
        # Deploy the model
        model_deployer(decision, model)```
    and run like this:
    

    pipeline = analyticspipeline( wrapmetric(),
    seldonmodeldeployerstep( params=SeldonDeployerStepParameters( serviceconfig=SeldonDeploymentConfig(
    modelname="my-model", replicas=1, implementation="SKLEARNSERVER"
    ),
    timeout=120
    )
    )
    )

    pipeline.writerunconfigurationtemplate(path='myconfig.yaml') pipeline.run(configpath="pipelines/config.yaml")`` I can see the artifact written afterwrapmetric(including the model) step in that bucket, but notseldonmodeldeployerstep` which the container is trying to fetch

  • HA

    @Safoine El Khabich do you have an idea about the above?

  • SA

    Hey @Misha Iakovlev may i know which ZenML version you are using? we had a PR that just got released with ZenML 0.32.0 which fixes a very similar issue.

  • MI

    0.31.0 on both the client and the server

  • SA

    Is there a possibility to update the client to 0.32.0? since that what would be required to fix the copy function that’s failing on the 0.31.0

  • MI

    Sure, I'll give it a shot now. Any potential side-effects of mismatch between the client and the server versions?

  • SA

    Ah sorry, you are right you will have to update the server too, since the latest version have a new feature that would require the same version.

  • MI

    Yep. Updating to 0.32.0 for both client and server helped, the joblib file gets uploaded to GCS. Although classifier-model-initializer hits 403 error now. We're using gcp-minimal stack recipe, I guess there's some issue with k8s service account permissions

  • SA

    Great to know that it solved the copy issue. Regarding the 403 error, have you made any changes on the gcp-minimal stack since i think the GKE Service-account is configured to get access to the artifact-store by default in the recipe.

  • MI

    No changes there, service account exists, it's default to the nodes, and has correct roles. However, we had the same problem before with running the pipelines.

    Still unresolved though. My suspicion is that it might be because workload identities are enabled (a different way of managing permissions in GCP) . Not sure how to properly confirm that

  • MI

    Well, if I go through workload identity route, it works now. I suppose the recipe assumes that workload identity/metadata server is disabled on GKE (which is enabled by default now).

    I've hit another thing now though. How do I use sklearn regressors? I'm providing seldonmodeldeployer_step with DummyRegressor model, which does not have a predict_proba method. Is there a way to specify which method to use (I do not see such property in the deployment config)?

  • MI

    I'd specifically like to define it like this

  • MI

    I can see extra_args property in SeldonDeploymentConfig, but as far as I can see it is not used anywhere in the code

  • SA

    I think you can actually define all custom properties following the custom deployment example which allows more flexibility in what can be done .
    The easy built-in deployment is actually quite limited in this case due to not having a sdk that can handle it. For that we have tried to implement a which only handles most standard use-cases. I would love to collaborate on adding more features to the client, if you are open to contribute.

  • MI

    Going down the expanding seldon_client route sounds better to me so far. I'll see if I can contribute here. (At least for the first glance it seems that edits shouldn't take too long).

Last active 9 days ago

19 replies

12 views