You could handle your own vector graphics in the Lua without too much work. By using a combination of glScale, glTranslate, glRotate, and drawQuad, you can draw lines anywhere on the screen. I believe the C already takes care of blending Quads thinner than one pixel, not sure, but again, you could handle that in the Lua. You'd have to design an interpreter for the particular file format you were using, as well as a structure for keeping that format loaded in memory (don't want to load your vector files every frame). I have no idea what vector graphics file formats are like, but it might be easiest to design your own file format. It wouldn't be five minutes of work, but neither would it be five weeks of work. I don't know how much acceleration you'd be taking advantage of by doing things this way-- the coordinate shifts should be telling your graphics card to do the shift, but maybe they're not.
I haven't done sufficient investigation, but I don't believe the Lua gives full access to the full drawing of the map. In other words, it calls a drawMap() in C, rather than calling for x, y in iterate(entities.pos)) drawEntity(x,y) end. You can get around this by handling your own drawing of the map-- after all, you have access to fast C functions for drawing, you can comment out drawMap() or replace entities' images with blank textures, you can assign a display callback function, you can iterate over all entities in FOV, all maptiles on the map, etc. The Lua-to-C action is very fast; I've drawn a tremendous number of textures, very inefficiently, very lazily, without seeing any impact on game performance. Trig functions are all just calls to C functions, and again, I've never noticed any performance problem associated with them, even though, again, I've frequently used them very irresponsibly.
But I don't believe that implementing vector graphics is the right solution for tiles. Graphics memory is cheap these days, has been for years, and there is a maximum zoom to expect from ToME. I would instead rasterize my vector graphics to the highest resolution I might potentially need and use those bitmaps instead. The results would be indistinguishable from vector graphics, and easier, and faster. If you wanted to make an add-on where the player was expected to take advantage of infinite zoom, then vector graphics might start making more sense, but I can't imagine any add-on doing that.
Proud father of Fx4fx
add-ons; proud mother of Fated