MapGen: добавить поддержку сжатия gzip для farbfeld-изображений #13

Closed
opened 2026-01-12 20:09:33 +00:00 by ssdaniel24 · 3 comments
Owner

См. TODO в коде для подробностей.

Решить задачу хочу самостоятельно — оставляю за собой. 😈

См. TODO в коде для подробностей. Решить задачу хочу самостоятельно — оставляю за собой. 😈
ssdaniel24 self-assigned this 2026-01-12 20:09:34 +00:00
ssdaniel24 added the due date 2026-02-09 2026-01-12 20:10:22 +00:00
varttist added the
task
assigned
2
label 2026-01-13 17:52:57 +00:00
varttist added this to the Tasks Kanban project 2026-01-14 14:31:37 +00:00
varttist moved this to To Do in Tasks Kanban on 2026-01-14 14:35:02 +00:00
Author
Owner

Возникли некоторые осложнения, возможно перейдём на парсинг PNG (хотя мне хотелось бы farbfeld из-за того, что он suckless) из библиотеки modlib.

Переписка с автором modlib в канале #development дискорд-сервера Luanti.
TL;DR: надо дебагать...

ga — 10.01.2026, 18:25
Hello, is it possible to compress file with some utility outside of luanti and then decompress it with built-in core.decompress?
I saw that it support deflate method, so it is gzip(?). But it didn't worked with it...
luatic — 10.01.2026, 19:04
Yes, it is possible. But don't confuse the low level compression methods with the higher level "compressed file / archive" formats. gzip does use deflate, but it also has a small header:
A gzip file (described in the table below) contains a 10-byte header, optional extra header fields, a DEFLATE-compressed payload and an 8-byte trailer.
https://en.wikipedia.org/wiki/Gzip
ga — 10.01.2026, 19:28
So, i can just skip 10 bytes, read XLEN, which is size of extra field, skip extra field and decompress all data after it?
luatic — 10.01.2026, 19:33
Something like that, though you also need to cut off the trailer probably (or ideally, just don't read it in the first place), and deal with the details of the payload, like file names or CRCs. See the Wiki article and some references for those details.
Source

ga — 14.01.2026, 16:41
I wrote implementation, but encountered in fact that it can't decompress deflate from gzip. core.compress with same data provides different compressed data then gzip (levels are also same).
luatic — 14.01.2026, 17:17
well, what i can tell you is that core.compress and core.decompress can most certainly correctly compress and decompress deflate, they just defer to zlib. either the gzip is somehow weird and doesn't use standard deflate, or there's a bug in your implementation.
ga — 14.01.2026, 18:20

---Decompresses [GZIP](https://en.wikipedia.org/wiki/Gzip) format.
---@param  zip string   bytes
---@return     string   bytes
local function decompressGzip(zip)
  local flags = byte2bits(string.byte(zip, 4)) -- checking flags (FLG field)
  local cursor = 11
  -- XLEN + Extra Field skip
  if flags[3] then -- FEXTRA
    local xlen = uintb2num(zip:sub(cursor, cursor+1), true)
    cursor = cursor + 2 + xlen
  end
  -- file name skip
  if flags[4] then -- FNAME
    while string.byte(zip, cursor) ~= 0 and cursor < #zip do
      cursor = cursor + 1
    end
    cursor = cursor + 1 -- also skipping zero byte
  end
  -- file comment skip
  if flags[5] then -- FCOMMENT
    while string.byte(zip, cursor) ~= 0 and cursor < #zip do
      cursor = cursor + 1
    end
    cursor = cursor + 1 -- also skipping zero byte
  end
  --- CRC16 skip
  if flags[2] then -- FHCRC
    cursor = cursor + 2
  end

  local data = zip:sub(cursor, #zip-8) -- getting compressed data without trailing 8 bytes (CRC32, ISIZE)

  return core.decompress(data, 'deflate', 1)
end

Checked in ghex that it chooses bytes correctly.
argument "zip" is string from file:read('*a). First bytes of decompressed file are "farbfeld".
Тип вложенного файла: unknown. Имя: img_easy.ff.gz. Размер: 74 bytes.
ops, forgot byte2bits function, but everything ok with it: convert byte 1000101 into table { [1] = 1, [2] = 0, ... }
luatic — 14.01.2026, 19:43
did you make sure to open the file in "rb" mode?
ga — 14.01.2026, 19:44
yes, but there is no different as i saw
luatic — 14.01.2026, 19:45
on linux there is no difference, on windows "r" mode performs linefeed normalization which is why i asked you this
ga — 14.01.2026, 20:52
Well, searching discord server and interrogating deepseek didn't end as i except.
Nobody tried to decompress deflate data field of gzipped file with luanti core.decompress with deflate method...
They just... Incompatible?
Source

luatic — 20:57
i'm not sure what you want me to do here. i can pretty much assure you that core.[de]compress work as advertised for deflate. the reason for this is that these are relatively thin wrappers around the battle-tested zlib library (see [de]compressZlib for how we use it). furthermore, i have employed both successfully in my own PNG reader and writer in modlib (PNG also uses deflate), and then read PNGs written by different programs and written PNGs that were read by different programs, so i have reason to believe that the way we wrap them also works as expected.
i think it is most plausible that there might be an oversight in your code. unfortunately it's not something that can be seen at a glance, i'd have to dive into the exact gzip specification, and i'd need to figure out whether whatever thing you use to generate the files might be doing something weird that is not explicitly specified but de facto standard. if it's a simple error, it's likely easier to catch, but either way it needs some debugging tedium, and unfortunately that is not something i have time for at the moment. you could try churning stuff through AI. maybe it's just a waste of time, but maybe if you feed it with the specs and everything it can figure it out quickly.
if after a serious debugging session you have good reason to doubt the correctness of core.[de]compress, feel free to do your own testing, e.g. by comparing it against established pure lua implementations, or different zlib bindings to another programming language, or raw zlib, or an alternative zlib implementation, or trustworthy examples from the wides of the interwebs. and if something doesn't work as expected, please then file a proper bug report with a minimal reproducer ("this string is deflate-compressed, this is what it should decompress to, but it doesn't") and we'll look into it.
ga — 21:27
i'm not sure what you want me to do here
Sorry for possible pressure. Thank you for big answer.
I tried to write compression support for project farbfeld ("suckless" format: "farbfeld" magic number, 4 bytes for width, 4 bytes for height and data of image in RGBA - 2 bytes for every) image parser, but got what i got.
Your solution of PNG parser looks good, i think that i can use it in project. Thank you again.
Source

Возникли некоторые осложнения, возможно перейдём на парсинг PNG (хотя мне хотелось бы farbfeld из-за того, что он [suckless](https://suckless.org/)) из библиотеки [modlib](https://github.com/appgurueu/modlib/blob/master/minetest/png.lua). Переписка с автором modlib в канале #development дискорд-сервера Luanti. **TL;DR**: надо дебагать... > **ga — 10.01.2026, 18:25** > Hello, is it possible to compress file with some utility outside of luanti and then decompress it with built-in core.decompress? > I saw that it support deflate method, so it is gzip(?). But it didn't worked with it... > **luatic — 10.01.2026, 19:04** > Yes, it is possible. But don't confuse the low level compression methods with the higher level "compressed file / archive" formats. gzip does use deflate, but it also has a small header: > A gzip file (described in the table below) contains a 10-byte header, optional extra header fields, a DEFLATE-compressed payload and an 8-byte trailer. > https://en.wikipedia.org/wiki/Gzip > **ga — 10.01.2026, 19:28** > So, i can just skip 10 bytes, read XLEN, which is size of extra field, skip extra field and decompress all data after it? > **luatic — 10.01.2026, 19:33** > Something like that, though you also need to cut off the trailer probably (or ideally, just don't read it in the first place), and deal with the details of the payload, like file names or CRCs. See the Wiki article and some references for those details. [Source](https://discord.com/channels/369122544273588224/369123175583186964/1459568967663484969) > **ga — 14.01.2026, 16:41** > I wrote implementation, but encountered in fact that it can't decompress deflate from gzip. core.compress with same data provides different compressed data then gzip (levels are also same). > **luatic — 14.01.2026, 17:17** > well, what i can tell you is that core.compress and core.decompress can most certainly correctly compress and decompress deflate, they just defer to zlib. either the gzip is somehow weird and doesn't use standard deflate, or there's a bug in your implementation. > **ga — 14.01.2026, 18:20** ```lua ---Decompresses [GZIP](https://en.wikipedia.org/wiki/Gzip) format. ---@param zip string bytes ---@return string bytes local function decompressGzip(zip) local flags = byte2bits(string.byte(zip, 4)) -- checking flags (FLG field) local cursor = 11 -- XLEN + Extra Field skip if flags[3] then -- FEXTRA local xlen = uintb2num(zip:sub(cursor, cursor+1), true) cursor = cursor + 2 + xlen end -- file name skip if flags[4] then -- FNAME while string.byte(zip, cursor) ~= 0 and cursor < #zip do cursor = cursor + 1 end cursor = cursor + 1 -- also skipping zero byte end -- file comment skip if flags[5] then -- FCOMMENT while string.byte(zip, cursor) ~= 0 and cursor < #zip do cursor = cursor + 1 end cursor = cursor + 1 -- also skipping zero byte end --- CRC16 skip if flags[2] then -- FHCRC cursor = cursor + 2 end local data = zip:sub(cursor, #zip-8) -- getting compressed data without trailing 8 bytes (CRC32, ISIZE) return core.decompress(data, 'deflate', 1) end ``` > Checked in ghex that it chooses bytes correctly. > argument "zip" is string from file:read('*a). First bytes of decompressed file are "farbfeld". > Тип вложенного файла: unknown. Имя: img_easy.ff.gz. Размер: 74 bytes. > ops, forgot byte2bits function, but everything ok with it: convert byte 1000101 into table { [1] = 1, [2] = 0, ... } > **luatic — 14.01.2026, 19:43** > did you make sure to open the file in "rb" mode? > **ga — 14.01.2026, 19:44** > yes, but there is no different as i saw > **luatic — 14.01.2026, 19:45** > on linux there is no difference, on windows "r" mode performs linefeed normalization which is why i asked you this > **ga — 14.01.2026, 20:52** > Well, searching discord server and interrogating deepseek didn't end as i except. > Nobody tried to decompress deflate data field of gzipped file with luanti core.decompress with deflate method... > They just... Incompatible? [Source](https://discord.com/channels/369122544273588224/369123175583186964/1460992318227353642) > **luatic — 20:57** > i'm not sure what you want me to do here. i can pretty much assure you that core.[de]compress work as advertised for deflate. the reason for this is that these are relatively thin wrappers around the battle-tested zlib library (see [de]compressZlib for how we use it). furthermore, i have employed both successfully in my own PNG reader and writer in modlib (PNG also uses deflate), and then read PNGs written by different programs and written PNGs that were read by different programs, so i have reason to believe that the way we wrap them also works as expected. > i think it is most plausible that there might be an oversight in your code. unfortunately it's not something that can be seen at a glance, i'd have to dive into the exact gzip specification, and i'd need to figure out whether whatever thing you use to generate the files might be doing something weird that is not explicitly specified but de facto standard. if it's a simple error, it's likely easier to catch, but either way it needs some debugging tedium, and unfortunately that is not something i have time for at the moment. you could try churning stuff through AI. maybe it's just a waste of time, but maybe if you feed it with the specs and everything it can figure it out quickly. > if after a serious debugging session you have good reason to doubt the correctness of core.[de]compress, feel free to do your own testing, e.g. by comparing it against established pure lua implementations, or different zlib bindings to another programming language, or raw zlib, or an alternative zlib implementation, or trustworthy examples from the wides of the interwebs. and if something doesn't work as expected, please then file a proper bug report with a minimal reproducer ("this string is deflate-compressed, this is what it should decompress to, but it doesn't") and we'll look into it. > **ga — 21:27** > i'm not sure what you want me to do here > Sorry for possible pressure. Thank you for big answer. > I tried to write compression support for project farbfeld ("suckless" format: "farbfeld" magic number, 4 bytes for width, 4 bytes for height and data of image in RGBA - 2 bytes for every) image parser, but got what i got. > Your solution of PNG parser looks good, i think that i can use it in project. Thank you again. [Source](https://discord.com/channels/369122544273588224/369123175583186964/1462143695871807650)
varttist added the
priority
medium
2
label 2026-01-18 17:32:28 +00:00
Author
Owner

Просто так добавить поддержку GZIP не получится. Требуется изменение движка.

См. issue: core.(de)compress('deflate'): add support for raw deflate data · Issue #16894 · luanti-org/luanti

Просто так добавить поддержку GZIP не получится. Требуется изменение движка. См. issue: [`core.(de)compress('deflate')`: add support for raw deflate data · Issue #16894 · luanti-org/luanti](https://github.com/luanti-org/luanti/issues/16894)
ssdaniel24 removed the due date 2026-02-09 2026-01-29 12:52:17 +00:00
Owner

Неактуально в связи с #25

Неактуально в связи с #25
varttist added
task
canceled
3
and removed
priority
medium
2
task
assigned
2
labels 2026-03-05 17:18:21 +00:00
varttist moved this to Done in Tasks Kanban on 2026-03-05 17:19:43 +00:00
Sign in to join this conversation.
No Label
task
canceled
3
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Apancore.Studio/simple-makes-complex#13