Hi All, I have a failed tapeout job and I dont know exactly why is falling This is the tapeout job log:

[05/13/22 13:37:32 PDT] SUBMITTED
[05/13/22 13:41:27 PDT] STARTED
[05/13/22 13:41:37 PDT] EXTRACTING FILES
            Extracting compressed files
[05/13/22 13:41:42 PDT] PROJECT TYPE INFO
            digital
 
[05/13/22 13:41:43 PDT] PROJECT GDS INFO
            user_project_wrapper: 999f4051646e5ed6ff0cf3aecb2dd595af722dc5
 
[05/13/22 13:41:43 PDT] TOOLS INFO
            KLayout: v0.27.8 | Magic: v8.3.294
 
[05/13/22 13:41:43 PDT] PDKS INFO
            PDK: sky130A | Open PDKs: 05af1d05227419f0955cd98610351f4680575b95 | Skywater PDK: c094b6e83a4f9298e47f696ec5a7fd53535ec5eb
 
[05/13/22 13:41:43 PDT] START
            Tapeout Started, the full logs can be found in u5646_maurici/design/sonaronchip8/jobs/tapeout/5673b3d3-3950-4776-98fd-138ed570b146/logs
 
[05/13/22 13:41:43 PDT] STEP UPDATE
            Generating Final Layout: Step 1 of 7
[05/13/22 13:41:43 PDT] STEP UPDATE
            Generating Final Layout: Step 2 of 7
[05/13/22 13:41:44 PDT] STEP UPDATE
            Generating Final Layout: Step 3 of 7
[05/13/22 13:42:16 PDT] STEP UPDATE
            Generating Final Layout: Step 4 of 7
[05/13/22 14:03:52 PDT] STEP UPDATE
            Generating Final Layout: Step 5 of 7
[05/13/22 14:10:57 PDT] STEP UPDATE
            Generating Final Layout: Step 6 of 7
[05/13/22 14:27:29 PDT] STEP UPDATE
            Generating Final Layout: Step 7 of 7
[05/13/22 14:29:22 PDT] STEP UPDATE
            Converting Final Layout: GDSII -> OASIS
[05/13/22 14:30:08 PDT] STEP UPDATE
            Executing Check 1 of 5: Klayout Missing Cells on [caravel_0006699b.oas]
[05/13/22 14:30:25 PDT] STEP UPDATE
            Executing Check 2 of 5: Klayout Front End Of Line on [caravel_0006699b.oas]
[05/13/22 14:47:44 PDT] STEP UPDATE
            Executing Check 3 of 5: Klayout Back End Of Line on [caravel_0006699b.oas]
[05/13/22 15:47:44 PDT] STEP UPDATE
            Executing Check 4 of 5: Klayout Metal Minimum Clear Area Density on [caravel_0006699b.oas]
[05/13/22 15:55:17 PDT] STEP UPDATE
            Executing Check 5 of 5: Klayout Field Oxide Mask Density on [caravel_0006699b.oas]
[05/13/22 15:55:40 PDT] FINISH
            Executing Finished, the full logs can be found in u5646_maurici/design/sonaronchip8/jobs/tapeout/5673b3d3-3950-4776-98fd-138ed570b146/logs
[05/13/22 15:55:40 PDT] FAILED
 STDOUT: {{Extracting Files}} Extracting compressed files
 STDOUT: {{Project Type Info}} digital
 STDOUT: {{Project GDS Info}} user_project_wrapper: 999f4051646e5ed6ff0cf3aecb2dd595af722dc5
 STDOUT: {{Tools Info}} KLayout: v0.27.8 | Magic: v8.3.294
 STDOUT: {{PDKs Info}} PDK: sky130A | Open PDKs: 05af1d05227419f0955cd98610351f4680575b95 | Skywater PDK: c094b6e83a4f9298e47f696ec5a7fd53535ec5eb
 STDOUT: {{START}} Tapeout Started, the full logs can be found in u5646_maurici/design/sonaronchip8/jobs/tapeout/5673b3d3-3950-4776-98fd-138ed570b146/logs
 STDOUT: {{Step Update}} Generating Final Layout: Step 1 of 7
 STDOUT: {{Step Update}} Generating Final Layout: Step 2 of 7
 STDOUT: {{Step Update}} Generating Final Layout: Step 3 of 7
 STDOUT: {{Step Update}} Generating Final Layout: Step 4 of 7
 STDOUT: {{Step Update}} Generating Final Layout: Step 5 of 7
 STDOUT: {{Step Update}} Generating Final Layout: Step 6 of 7
 STDOUT: {{Step Update}} Generating Final Layout: Step 7 of 7
 STDOUT: GDSII File [caravel_0006699b.gds] sha1sum: b7416c50ea812509ebc33eb5603b1971d624f1d7
 STDOUT: {{Step Update}} Converting Final Layout: GDSII -> OASIS
 STDOUT: OASIS File [caravel_0006699b.oas] sha1sum: e74b0e0ef1e28c7021b499c83db442917c102ac8
 STDOUT: {{Step Update}} Executing Check 1 of 5: Klayout Missing Cells on [caravel_0006699b.oas]
 STDERR: [ WARN ] Missing Cells Check Result: 'caravel_0006699b.oas' has 0 missing cells.
 STDOUT: {{Step Update}} Executing Check 2 of 5: Klayout Front End Of Line on [caravel_0006699b.oas]
 STDERR: [ WARN ] FEOL Check Result: 'caravel_0006699b.oas' has 0 DRC violations.
 STDOUT: {{Step Update}} Executing Check 3 of 5: Klayout Back End Of Line on [caravel_0006699b.oas]
 STDOUT: {{Step Update}} Executing Check 4 of 5: Klayout Metal Minimum Clear Area Density on [caravel_0006699b.oas]
 STDERR: [ WARN ] MET Density Check Result: 'caravel_0006699b.oas' has 0 DRC violations.
 STDOUT: {{Step Update}} Executing Check 5 of 5: Klayout Field Oxide Mask Density on [caravel_0006699b.oas]
 STDERR: [ WARN ] FOM Density Check Result: 'caravel_0006699b.oas' has 0 DRC violations.
 STDOUT: {{FINISH}} Executing Finished, the full logs can be found in u5646_maurici/design/sonaronchip8/jobs/tapeout/5673b3d3-3950-4776-98fd-138ed570b146/logs

@User sorry you are facing this weird issue, we'll be looking into it soon

great, let me know when you have an update about this. Thank you for your help!

@Mauricio Montanares it seems that your BEOL Check failed at some point, I can't replicate this issue anywhere on my end, please try submitting a new tapeout, it worked successfully on my end. Reach out to me again, if the problem persists, thanks for understanding

Hi @omla, I launch a new precheck job and a new tapeout job. Precheck is done and tapeout still fails

[05/20/22 09:29:23 PDT] FAILED
            STDOUT: {{Extracting Files}} Extracting compressed files
STDOUT: {{Project Type Info}} digital
STDOUT: {{Project GDS Info}} user_project_wrapper: 999f4051646e5ed6ff0cf3aecb2dd595af722dc5
STDOUT: {{Tools Info}} KLayout: v0.27.8 | Magic: v8.3.294
STDOUT: {{PDKs Info}} PDK: sky130A | Open PDKs: 05af1d05227419f0955cd98610351f4680575b95 | Skywater PDK: c094b6e83a4f9298e47f696ec5a7fd53535ec5eb
STDOUT: {{START}} Tapeout Started, the full logs can be found in u5646_maurici/design/sonaronchip8/jobs/tapeout/39adf838-3132-4d32-9dec-0fd45c571269/logs
STDOUT: {{Step Update}} Generating Final Layout: Step 1 of 7
STDOUT: {{Step Update}} Generating Final Layout: Step 2 of 7
STDOUT: {{Step Update}} Generating Final Layout: Step 3 of 7
STDOUT: {{Step Update}} Generating Final Layout: Step 4 of 7
STDOUT: {{Step Update}} Generating Final Layout: Step 5 of 7
STDOUT: {{Step Update}} Generating Final Layout: Step 6 of 7
STDOUT: {{Step Update}} Generating Final Layout: Step 7 of 7
STDOUT: GDSII File [caravel_0006699b.gds] sha1sum: bc9d8fef8624aab67cd4f60c6931bef984ce0774
STDOUT: {{Step Update}} Converting Final Layout: GDSII -> OASIS
STDOUT: OASIS File [caravel_0006699b.oas] sha1sum: e74b0e0ef1e28c7021b499c83db442917c102ac8
STDOUT: {{Step Update}} Executing Check 1 of 5: Klayout Missing Cells on [caravel_0006699b.oas]
STDERR: [ WARN ] Missing Cells Check Result: 'caravel_0006699b.oas' has 0 missing cells.
STDOUT: {{Step Update}} Executing Check 2 of 5: Klayout Front End Of Line on [caravel_0006699b.oas]
STDERR: [ WARN ] FEOL Check Result: 'caravel_0006699b.oas' has 0 DRC violations.
STDOUT: {{Step Update}} Executing Check 3 of 5: Klayout Back End Of Line on [caravel_0006699b.oas]
STDOUT: {{Step Update}} Executing Check 4 of 5: Klayout Metal Minimum Clear Area Density on [caravel_0006699b.oas]
STDERR: [ WARN ] MET Density Check Result: 'caravel_0006699b.oas' has 0 DRC violations.
STDOUT: {{Step Update}} Executing Check 5 of 5: Klayout Field Oxide Mask Density on [caravel_0006699b.oas]
STDERR: [ WARN ] FOM Density Check Result: 'caravel_0006699b.oas' has 0 DRC violations.
STDOUT: {{FINISH}} Executing Finished, the full logs can be found in u5646_maurici/design/sonaronchip8/jobs/tapeout/39adf838-3132-4d32-9dec-0fd45c571269/logs

@Mauricio Montanares, I am looking into it again, it seems to be the KLayout BEOL check failing again, will update you soon

Thank you!

@Mauricio Montanares I am unable to replicate the issue on my end, please try submitting a tapeout again on your end, we are still investigating from our but it just an isolated few cases

Hi @omla, I will submit a new tapeout job I'll let you know when I finish this new attempt

Hi @omla, it seems that we are facing the same issue. We've tried to submit tapeout couple of times (cb4dc831-5c89-4a37-b90d-e7f6bfeaa554 and 3ca25017-8f45-4f57-b5ec-3f0e5545aafc), but both times BEOL check got a SIGTERM after writing report database without a clear reason. Unfortunately I do not see a way to copy logs out of Open Galaxy. Should we just resubmit tapeout job until it succeeds, or will you check something on your side?

@Mauricio Montanares, I am glad it finally got through, we are still looking at the underlying cause and will let you know if anything comes up, thanks 🙏

Thanks @omla! I've tried to reimplement the project in openlane to get a bit different GDS. But unfortunately even with new GDS tapeout still fails (39bdb514-a2f3-49ee-a9bc-80e7f9f4170d, 75b64a90-edc4-4c22-b1e0-bdf1e39abff7). I had an issue with "Git pulling" new revision, so I had to delete & recreate the workspace. I hope logs from older runs are still available to you for debug, but they looked absolutely the same as latest ones anyway.

@Anuj Dubey looks similar to your problem

@Egor Lukyanchenko Just a quick update, this seems to be an issue in KLayout related to some designs failing with a

SIGTERM

when exercising a specific function. I guess you can try a couple more times but we are still investigating how to deal with this. @Mauricio Montanares was able to get lucky in one of his runs and avoid the failure. Thanks for your patience so far and please let me know if your situation changes.

@omla May be there is a way to add some temporary exception to this check like treating SIGTERM as success if no DRC errors were reported? MPW6 deadline is near and we still can not make our design pass tapeout. We've tried 4 slightly different GDS files already and all of them get SIGTERM. Also it seems that our design is not the only one with such problem.

@Egor Lukyanchenko I don't believe we can do that as we never get the full report, thus never knowing if there are DRC errors or not. We are aware of the deadline and we will be doing our best to get this resolved before deadline

@omla @Egor Lukyanchenko @vlada @jeffdi @Matt Venn Did you perhaps manage to solve this problem? Our tapeout run also fails because of Klayout BEOL SIGTERM. Everything else seems to be in order.

AFAIK if the tapeout job fails with no 'real' errors then it's a problem with the tapeout job at efabless side - the SIGTERM issue has been cropping up for a few people

@Dejan Petkovic we believe we'll be able to get a fix in the next couple of hours, we'll keep you updated.

Alright, thanks a lot.

@Dejan Petkovic @Egor Lukyanchenko we have deployed a fix to deal with the

SIGTERM

issue you face in your

tapeout

. You can re submit your jobs and hopefully that gets you through to the end of the tapeout without problems from our end.

Thanks, @omla. Let's hope for the best. 🙏