@parallel doesn't work with Python 3.8 #27
Comments
|
We have seen this kind of thing before ... the fabric-1 parallel mode forks separate processes, and this apparently requires pickling some objects into bytes to pass between processes, and in odd cases, some objects in the tree can't be pickled. This happened on windows before (possibly still). I'll take a look and see what can be done, maybe in a few days, unless you figure it out first :) |
|
It looks like this is going to be quite a pickle. I untangled some wrappers in master...ploxiln:python38_parallel and it seems that it's the task callables that are the real problem with Python-3.8: This seems related to https://stackoverflow.com/questions/9336646/python-decorator-with-multiprocessing-fails and https://stackoverflow.com/questions/1412787/picklingerror-cant-pickle-class-decimal-decimal-its-not-the-same-object I tried a few different hacks to get any kind of task working, as a proof of concept, with no luck. |
|
I came up with a hacky way, which might only work for top-level (not proper enough to merge though) |
|
I've had great success with ensuring the default processing call uses diff --git a/fabric/tasks.py b/fabric/tasks.py
index 47a2b49f..c56c35a2 100644
--- a/fabric/tasks.py
+++ b/fabric/tasks.py
@@ -329,6 +329,9 @@ def execute(task, *args, **kwargs):
# if it can't.
try:
import multiprocessing
+ ctx = multiprocessing.get_context('fork')
+ # Set up job queue for parallel cases
+ queue = ctx.Queue()
except ImportError:
import traceback
tb = traceback.format_exc()
@@ -338,12 +341,11 @@ def execute(task, *args, **kwargs):
traceback.) Please make sure the module is installed
or that the above ImportError is fixed.""")
else:
- multiprocessing = None
+ ctx = None
+ queue = None
# Get pool size for this task
pool_size = task.get_pool_size(my_env['all_hosts'], state.env.pool_size)
- # Set up job queue in case parallel is needed
- queue = multiprocessing.Queue() if parallel else None
jobs = JobQueue(pool_size, queue)
if state.output.debug:
jobs._debug = True
@@ -355,7 +357,7 @@ def execute(task, *args, **kwargs):
try:
results[host] = _execute(
task, host, my_env, args, new_kwargs, jobs, queue,
- multiprocessing
+ ctx
)
except NetworkError as e:
results[host] = e@ploxiln it might be worth making this change for non-Windows (or all if we don't maintain Windows support), until such a time as the multi-thread method can be rewritten as you suggest. |
Hi,
First of all, thanks for the fork and for the work done!
We're migrating from
Fabric3==1.14.post1tofab-classic.Both packages have the same problem:
paralleldoesn't work with Python 3.8.You can import it as:
a.
from fabric.api import parallelorb.
from fabric.decorators import parallelHere is an example:
Result:
No problems with Python 3.7.7.
The text was updated successfully, but these errors were encountered: