<!channel>: As of today, magic supports reading and writing gzip-compressed GDS files (by making system calls to

gzip

and

gunzip

). Since klayout already supports reading and writing of gzip-compressed files, we should now be able to get rid of the

make compress

and

make uncompress

steps in the caravel project, which has always been a bit of a pain. To read or write a compressed GDS file, just add the

.gz

extension to the filename in the

gds read

or

gds write

command.

@User I love this feature!

I never even thought about gzipping GDS files until github limitations forced us to start doing it. Then when I found out that klayout would just handle compressed files transparently, I realized I needed to play catch-up. No, I do not have support for compressed

.mag

files because it calls slightly different routines for opening the file so I couldn't use the same code. It's easy enough to do, though. But one think I like about

.mag

files is that they are ASCII and human-readable and human-editable (can be very helpful in a pinch), and that gets removed by one step as soon as you compress the file. GDS is already binary and inscrutable without a GDS-to-ASCII translator, so compressing it doesn't change that.

@User (1) we use

.gds.gz

quite often, the files are too large otherwise on chip level. (2) Agree that gzipping

.mag

is not super urgent as here most often block-level work is done, but since chips will get larger over time, you know… (3) text editing

.mag

is super handy, this is my favourite way of adding ports :-)

I think vim has .gz editing support, I am not sure.

@User @User Yes, vim (as vi)

VIM - Vi IMproved 8.1 (2018 May 18, compiled Dec 25 2021 15:48:51)

can edit

.gz

files without explicit decompression beforehand. However, in the current docker version of vim (as vi)

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 13 2020 16:12:33)

, it is not enabled. May be possible with http://vimdoc.sourceforge.net/htmldoc/autocmd.html#gzip-example

@User Looks good! My GDS files are over 100MB gzip compressed, so we'll still need a way to split/recombine GDS files in caravel to get around github limitations. We currently do that in the make compress/uncompress stage. Other options are to compress with xz (would need to be added to both magic and klayout), or add oasis support (supported by klayout but I presume a bunch of work for magic):

620M    user_project_wrapper.gds
123M    user_project_wrapper.gds.gz
51M     user_project_wrapper.gds.xz

60M     user_project_wrapper.oas
24M     user_project_wrapper.oas.gz
18M     user_project_wrapper.oas.xz

I would suggest to use splitting the .gz files instead of just using compression, because with splitting we can scale to any size, by splitting it into multiple files

Oasis is another good choice, since the oasis format is based specifically on the knowledge of what makes mask data repetitive, so it can often achieve what seems to be absurdly high compression rates (although what that really means is that GDS is an absurdly bad format for representing mask data). Since magic's implementation is just to make an external call to

gzip/gunzip

, it can be extended easily to apply oasis compression/decompression, or bzip2, or 7zip, or whatever compression type has a standalone application.

I assume this isn't in the flow yet for openlane/caravel?