Philipp Gühring

04/21/2023, 10:44 PM
I have now succeeded to get the standard cell generation to the point that it makes sense to run them massively in parallel. I would like to try generating 800 cells in parallel on the environment. I guess that we could try to run maximum 10 jobs in parallel per machine (since some jobs take a few GB of RAM, and some parts of the flow are also parallelized already (but some parts are sequential). Now my question is which scheduling system should I try to use on GCP to get that done? On a single machine I was used to make -j but that wouldn't distribute it on several machines. I think for the Google cloud I should package up everything necessary into a container, and then perhaps use
gcloud batch jobs submit
to submit the 800 individual jobs? I only have to collect all the generated files in the end, so I guess I have to add a bucket or NFS storage where the results are stored. (Should I use the notebooks bucket for that?)