Pipeline import broken by identically-named nested directories

[Arkanosis]

The pipeline import code can handle both:

• a zip archive with the pipeline files directly in it;
• a zip archive with a directory in it that contains the pipeline files.

The latter case is handled by moving the files outside of the directory and then deleting the directory.

Unfortunately, the behavior of shutil.move when the destination already exists consists in moving the source within the destination (ie. as cp does on unices), meaning that if a zip archive contains name/name, then the subdirectory is “moved” into the directory (hence not actually moving). The result is 1) files left in a subdirectory 2) the removal failing because os.rmdir can't remove directories that are not empty.

[ikit]

I don't see what you are talking about. Can you cite where it is in the source code ?

[Arkanosis]

I've put links in my report above.

[ikit]

ok I understand.
Did you know a better way to manage the both "packaging" strategy ?

[Arkanosis]

To be honest, I'd rather see a single valid way of packaging pipelines, and that would be without a top directory as it doesn't provide any benefit (the first thing we do when it exists is to remove it). See jar files in Java: there's a single valid place where a manifest can be.

But if we really want to handle that, I guess a sound approach would be to start by renaming the top directory to some name that doesn't exist in it. Something like:

# remove intermediate folder
if zip_root != "":
    new_name = os.path.join(root_path, uuid.uuid4())
    shutil.move(os.path.join(root_path, zip_root), new_name)
    zip_root = new_name
    for filename in os.listdir(zip_root):
        shutil.move(os.path.join(zip_root, filename), os.path.join(root_path, filename))
    os.rmdir(zip_root)