Git - support for exotic charset

Question

I'm working in the environment of abas-ERP, which uses - for historical reasons - an esoteric charset named abas-s3. We use Git with local repositories on the ERP-server for versioning our programs.

Naturally Git doesn't support abas-s3. On the erp-servers there is a program called s3_conv, which converts from abas-s3 to the outside world (e.g. UTF-8 and UTF-16) and the other way round.

Is there any way to use this program, to use UTF-8 for the repository and s3 for the worktree?

I appreciate any feedback! Thanks in advance!

Answer 1

If the problem is with file contents, you can use clean/smudge filters (see gitattributes) to have Git convert to UTF-8 when storing to repository, and to abas-s3 when checking out the contents into the working area.

You can find explanation also in Pro Git book.

---

Summary:

1. You need to define in the config file (per repository, per user, or system-wide) a filter consisting of two commands: clean that transforms from working area representation to repository representation, and smudge that works in reverse. Assuming that s3_conv works like iconv, it could look like this (this is a fragment of appropriate config file):

   [filter "s3conv"]
       clean  = s3_conv --from abas-s3 --to utf-8
       smudge = s3_conv --from utf-8 --to abas-s3
   

2. Declare with gitattributes file (inside repository, or per-repository, or per-user, or system-wide) which files should be transformed using this operation. Assuming that you want to transform files with SQL, using *.sql extension, appropriate fragment of a file could look like this (note that there cannot be whitespace around = here):

   *.sql filter=s3conv
   

Note that Git is encoding-agnostic, and you don't need to convert the contents of files... unless you need it for interop?

---

If you need to have git diff display changes correctly (assuming that you don't have pager that can display abas-s3 encoding, and could be used in core.pager), you might want to configure textconv filter INSTEAD:

1. In config file:

   [diff "s3conv"]
        textconv = s3_conv --from abas-s3 --to utf-8
   

   This assumes that terminal and pager is configured to display utf-8

2. In gitattributes file:

   *.sql diff=s3conv