Contributing to the Task Library

Explore

Development
Development

Getting setup to test your task locally

Testing your task with a scratch flow or just in an interactive interpreter is the best way to get quick-and-dirty insights into how your development is going.

To get setup, follow the instructions at

https://docs.prefect.io/core/development/overview.html⁠

to install prefect in editable mode in your preferred Python environment manager, and use that environment to run an interpreter or execute python scripts that have scratch flows utilizing your new task in different ways.

For example, this is how mine is set up:

I cloned the prefect repo to /Users/laura/Development/prefect. Create a branch for yourself:

⁠

Screen Shot 2020-04-17 at 11.49.48 AM.png

⁠

I have a conda environment called prefect_dev that I installed prefect into with pip install -e ".[dev]", as instructed in the above linked instructions:

⁠

Screen Shot 2020-04-17 at 11.25.03 AM.png

⁠

Use an interactive python interpreter to create task instances, execute flows, or whatever other tests you want to do while you edit the code.

⁠

Screen Shot 2020-04-17 at 11.33.05 AM.png

⁠

For example in the above screenshot, on the left I have some scratch flow code that I %paste into my ipython terminal in the middle, and on the right I have the task code I'm editing open with a change in line 163 that just prints a statement out. When I run my flow from my ipython terminal ー with my flow variable name of f, that is with f.run() - I can quickly execute my flow with my task changes in it and see my friendly print statement.

Since I'm in an interactive interpreter, I can also easily introspect f afterwards, such as with f.visualize() or f.tasks so especially in the beginning of writing a new task, operating in the interactive interpreter can be a great way to get started:

⁠

Screen Shot 2020-04-17 at 11.38.17 AM.png

⁠

Once you have yourself set up comfortably with a groove, it will be quick and easy to iterate on changes to your task code, so pick a style that is comfortable for you. Whether that means following the above example and using an interactive interpreter for local testing, executing your flows as a script on the terminal, or using an IDE to run your scratch flow, just pick what is best for you (which also may change as your development goes on!)

Actually write the task code

Now it's time to actually write in the body of the run method! Depending on how big of a task you are adding, this might be a few simple lines calling the wrapped python client's method, or quite a bit more.

First off, if you are creating a task or tasks that don't fit neatly into one of our existing subpackages in prefect/src/prefect/tasks, create a new directory.

There are a couple patterns we usually see, and it is worth studying each of these from start to end to get a feel for the different levels of complexity and find where your task fits in.

Some task code styles we usually see are:

Small units of work in-process by a Python library, like these spacy tasks (

https://github.com/PrefectHQ/prefect/blob/a8d68c2784e884df9dbe76c5f88437cdb960d46d/src/prefect/tasks/spacy/spacy_tasks.py⁠

)

Tasks that need to initialize a client that authenticates to some third-party service, and then uses that client's API to get some data or do some action, like the DropboxDownload task (

https://github.com/PrefectHQ/prefect/blob/a8d68c2784e884df9dbe76c5f88437cdb960d46d/src/prefect/tasks/dropbox/dropbox.py⁠

)

More complicated versions of the previous, where the task initializes a client to a third-party service and then uses that client's API to get some data or do some action, but that modularize shared code into a base class and/or private methods (such as client initialization) and may need to call multiple client methods or do more error checking to achieve the work of the task, like the Google Cloud Storage tasks (

https://github.com/PrefectHQ/prefect/blob/a8d68c2784e884df9dbe76c5f88437cdb960d46d/src/prefect/tasks/gcp/storage.py⁠

)

While you're writing your task's run method, you will be faced with a few task-specific design decisions, which you can use your judgement ー or examples of what is in the library ー to make a call on. For example:

what logs should I emit at different steps of the task, and at what log level? Think about what is useful to an end user here

what exceptions should I catch, and which should I raise? Are there connections or other resources I need to close before raising them?

are there meaningful

Prefect signals⁠

, a built-in type of exception tied to task states, that I should use to communicate the state of this task based on what happens during the run method during conditional logical in the task? For example, does it make sense for you to raise a success signal early

like this example⁠

, raise a skip signal instead of the error itself under a specific caught exception, or raise a failure signal under a certain condition?

should I break out the run code into multiple private methods, or leave it all inline? What about creating a base class?

You may be faced with many other task-specific design decisions, depending on the uniqueness or complexity of the task you are writing. When in doubt, make a call based on your best judgement. In the end, this will be PR'd somewhere that other contributors can take a look and help you, so feel free to add a #todo comment in wherever you are unsure and circle back with the other contributors later!

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.

DevelopmentDevelopment

Getting setup to test your task locally

Actually write the task code

Development
Development