I have now succeeded to get the standard cell generation to the point that it makes sense to run them massively in parallel. I would like to try generating 800 cells in parallel on the rad-lab-silicon.dev environment. I guess that we could try to run maximum 10 jobs in parallel per machine (since some jobs take a few GB of RAM, and some parts of the flow are also parallelized already (but some parts are sequential). Now my question is which scheduling system should I try to use on GCP to get that done? On a single machine I was used to make -j but that wouldn't distribute it on several machines. I think for the Google cloud I should package up everything necessary into a container, and then perhaps use

gcloud batch jobs submit

to submit the 800 individual jobs? I only have to collect all the generated files in the end, so I guess I have to add a bucket or NFS storage where the results are stored. (Should I use the notebooks bucket for that?)