Hey everyone. A couple questions that have come up as I’ve started to test out ZenML. Would love any feedback. 1. We have some pipelines that need to be run at least partially on MacOS, because we use CoreML models. I know there isn’t a containerized way to do this, but I was just wondering if you had any thoughts on best practices there with ZenML. Would it be best to just use the local orchestrator and what not, and potentially have it write to a deployed ZenML server? We currently use Gitlab pipelines to kick stuff off, so we may just have to continue using that and then use ZenML to organize it and get the step logic out of it. 2. Is there any way to dynamically add steps to a pipeline? For example, if you wanted to iterate over a list of strings, and run some logic on each one, where parallelization would be useful, is there a way to do that inside of a pipeline, so that each string can have its own step, and potentially run in its own environment (depending on the orchestrator)? I saw a comment mentioning doing this and kicking off a pipeline for each string, but it would be nice to be able to contain it within one pipeline. Thanks!
Last active 10 hours ago
Another question. Is it correct that the zenml integrated label-studio needs cloud artifact stores and secret managers to work? I'm working on a highly secured on-prem system and I'll be running label-studio as a local service on an ubuntu VM within a corporate VPN, I would like to set it up using zenml, and make use of the additional functionalities of mlflow, orchestration, and possibly evidently, and seldon. Can label-studio work without cloud integrations?
Last active a day ago
Hi everyone, I've been trying out zenml today, but quickly ran into issues. I'm using python 3.9.14, and using poetry 1.3 as a package manager. Installing zenml works fine, but if I install label-studio, with `zenml integrations install label_studio` the zenml cli stops working for me. It throws a typing error `zenml TypeError: typing.ClassVar is not valid as a type argument` that is associated with importing the zenml.annotators module. Has anyone else experienced this?
Last active a day ago
Hey guys, I'm facing an issue when running a scheduled pipeline with the `run_name` parameter. I was able to reproduce the error using the taken from the ZenML Blog. When I run the pipeline without explicitly defining the `run_name` parameter (as exemplified below), everything works as expected: ```pipe.run( schedule=Schedule(cron_expression="*/5 * * * *") )``` In the meantime, when I set the a `run_name` , like this ```pipe.run( schedule=Schedule(cron_expression="*/5 * * * *"), run_name="example_scheduled_run" )``` the first run works, but the subsequent executions fail, with the following error: ```EntityExistsError: EntityExistsError: Unable to create step 'get_first_num': A step with this name already exists in the pipeline run with ID '78cfc53a-ac53-4a14-97fe-a60492bfae5b'``` It seems that the subsequent executions are using the same run ID as the first one, which is the cause of the error, according to my understanding Could someone help me with this issue? I'm using the version `0.30.0` of ZenML
Last active 2 days ago
Moving onwards with my pipeline and getting some technical hurdles. I am in the following situation: my first step `step1` returns a custom class `CustomClass` as an output. I have a materializer written for that and it seems to work. The, I need to run some additional steps that use _only some attributes_ of `CustomClass`. For instance I have `step2` which uses `CustomClass.attribute1` and `CustomClass.attribute2`, and then `step3` that uses `CustomClass.attribute1` and `CustomClass.attribute3`. When using the `step` decorator code breaks because the returned `CustomClass` gets wrapped in `BaseStep._OutputArtifact` that doesn’t have any of the required attributes. I can see several ways forward here, and I don’t particularly like any: • Change all my functions so that they receive the full class as input - forces me to rewrite a lot of code, will mess up caching of some steps down the line • Write custom “unpack” functions that will take the class as an input and extract subsets of attributes - less rewriting but it will still mess up the caching • Write an additional wrapper around `step1` to unpack the output - would work with caching but it is annoying as I will need to have several copies of the same things (for some steps I will need the entire class, including class methods, for some I will only need a few attributes!) Am I missing something here? What this the usual way to get around this issue?
Last active 2 days ago
Hi, I’m new and just trying out zenml. I have a basic project started and I’m checking out the run DAG in my local dashboard. I see that the inputs and outputs are shown as local directories, and that the data.json files are there. What is the normal workflow for this. If a step fails and I’m trying to inspect those inputs and outputs to see what went wrong, would you normally open those files locally and check there, or is there a way to see those values in the dashboard? Or is this something that a stack component would be better for?
Last active 2 days ago
[Kubeflow orchestrator][GCP Kubeflow KServe Stack Recipe] Hello hello again :smile: ! Did anyone already have an issue where kubeflow pipeline pods didn't seem to receive the permissions issued to the kubeflow service account ? I can't seem to access GCS buckets even though the service account actually as admin rights on GCS. I tried to play around with defining KubeflowOrchestratorSettings but it either didn't work or i didn't do it correctly. My pipeline currently doesn't specify any kubeflow settings.
Last active 5 days ago
*AMAZING OPPORTUNITY* *FOR THE ZENML COMMUNITY IN THE BAY AREA* Hey ! I have a great chance for those of us in the community: We applied for the and got accepted for both an in-person and virtual talk (the same talk twice): :tada::tada::tada: This is a big win for us here as its one of biggest dev conferences in the world and its a great way to showcase the methodology behind ZenML. Unfortunately, *no one in our team* can make it, so now we turn to our community (you guys!). Is there anyone here, based close to Oakland who can travel to the event and do the talk? It would be a great opportunity to visit the conference and meet with like-minded people. Here is a similar talk, for reference. *Here are the logistical details:* LIVE in-person Talk - Oakland Convention Center (Oakland, CA) Date: 2/15/2023 Time: 12:00 PM - 12:50 PM (PT) Stage: DeveloperWeek PRO STAGE A Length: 50 Minutes If you are interested, let me know in the thread here! Cheers and looking forward!
Last active 6 days ago