11/21/12

Tribulations in porting software

Long time no update, time to fix that!

So, last night my friend Morg asked me to port a certain tool (namely, ttm_unpack) to Python from C, since he didn't want to introduce a C-compiler as a dependency for GOGonLinux project. A quick note about GOGonLinux - the project's main task is to make installing games from GOG on Linux as easy as possible. I'm an avid supporter and follower and this is my first major contribution to the project. Join us on #gogonlinux @ freenode and contribute to the codebase!

Back to the topic at hand, since we already had the C-code available I thought "why not?" and said yes, since I thought it would take an hour at maximum with testing.

How wrong can a man be?  Very wrong, I'll tell you that. But in the end, we succeeded, and you can download ttm_unpack_py from my Github-page.

So, what kind of problems did we have while porting the tool?



  • Even though Python has been built upon C, Python is very different from C. It should not be a surprise, but some of the inner workings of the way how Python works were quite surprising. For example, integers in Python don't really overflow, at least not like they do in C. We learned this the hard way when the original tool depended on C's uint32_t going over its maximum value and overflowing back to 0. What this meant was that we needed to implement the overflowing manually to get datafiles extracted correctly.
  • Python has been described a multitude of times as being "explicit language", where you need to declare self when dealing with object's inner workings and so forth. It's all good and dandy, but you can't actually have a short int-type, at least not without some hacky stuff. You want to know how to do it? You need to declare an array and have it store only short ints. So, yeah, no go.
  • So, if you need to convert an integer to binary value in Python, you'll use bin(). This, however, creates a string representing the binary integer, which may cause some problems (as we found). If you want to convert the binary back into integer, you need to explicitly pass the base of the binary to int() function, otherwise your code will fail.
  • Python has immutable strings. Immutable strings are a pet peeve of mine, and having grown mostly programming C and C++ I've never really understood why you'd want strings to be immutable without explicitly telling. Anyhow, it can be worked around, so this is nothing big.
  • When unpacking data from binary files with struct.unpack() into a string, for some reason you need to specify the amount of chars you need. For example, we needed to use struct.unpack(str(fnameLen) + 's', pfile.read((char_size * fnameLen))), since if we used only 's' as format paramater, the unpacker would be expecting only a single character as parameter (even if 's' is supposed to be string, as per documentation!). For a dynamic language this seems awfully non-dynamic and C-like.
  • General complaint: when you write software with magic numbers and operations, explain what are you doing and why! We spent too long wondering what the original author meant with the following line of code: char xorValue = ((char*)(&decryptState))[idx&3]; where decryptState was defined as int decryptState = 0xdeadcafe;. Also, please explain WHY you do things and not only what you are doing. I know I may not be perfect in doing that either, but still.
  • So, when you use Python and you make a long value, what happens that Python automatically appends 'L' or 'l' at the end of the representation of the value. You can quickly see where this is going - need to manually strip the character from string in order to have it do what we needed.


So, there we have it. Python is more than well suited for some things but for some other things I'd rather use better tools or even existing tools, but sometimes you simply don't have a choice. Until next time!

5 comments:

  1. Hi, do you mind if I use this python version in a playonlinux install script? I know your copyright notice should let me do this, but I figured it doesn't hurt to ask nicely first :))

    Also I would love to contribute to "GOGonlinux" project. But for wine games, have you guys considered just calling playonlinux binary? Because there are a LOT of GOG games in there already.

    You could link playonlinux in your program like you do with scummvm. I think you can do: playonlinux mygoggame-setup-file-1.0.exe and it will recognize the MD5 of the setup file and automatically use the correct POL install script. Then after installation you can launch the game from your GOGonlinux program directly with something like: playonlinux --play mygoggame. The advantage of POL is you can have specific wine version (even patched wine) per game.

    So it might be worth considering (up to you :)). Also if you think some playonlinux script is bad, you can submit new version there, so it becomes better for everyone.

    Best Regards, Kweepeer.

    ReplyDelete
    Replies
    1. Sure, you are more than welcome to use it, no problems with that.

      Actually, we haven't discussed about the possibility of using PlayOnLinux. I doubt it would be used though, since Morg (the main developer) of the project wants to keep the dependencies to a minimum. It is an interesting possibility, though.

      Anyhow, pop up in #gogonlinux at irc.freenode.net and leave a message for either me or Morgawr and we can discuss more. :)

      Regards,
      Kaira-

      Delete
  2. Hi!

    I'm the original developer of ttm_unpack, and it's amazing to see that you've put all this effort into porting it!

    I originally wrote it as a quick hack to get the game working, having spent a bit of time staring at disassembly. That's why there were lots of low-level c-style things. I certainly never expected it to see quite the response it's got.

    I've put up a webpage about it (at http://davidgow.net/hacks/ttm_unpack.html), and linked back here. I hope you don't mind: I found this post really interesting.

    Thanks,
    -- David

    ReplyDelete
  3. Hi Kaira! Wanted to leave a little note of my visit here. I spotted your reply on the review of our indie RPG Driftmoon at Rock Paper Shotgun, and found myself reading this nice blog of yours. Looks good! All the best in your endeavors as a starting game developer! :)

    ReplyDelete
  4. Hey its amazing to see your efforts in this blog and it is a nice one. The way you have expressed in this blog stuns good. All the best for your future endeavors to start as a Game Developer.

    ReplyDelete