Fish Scratch Fever


Zell as seen in FF8, probably enjoying some Fish Scratch Fever. It's been a while. I've been through a couple of contract projects, worked for Sony for a bit, did a bunch of stuff to Noesis, and now I'm on another contract project. I think that pretty much sums up the last 9 months.

My current project is rather cool in concept, but I can't talk about it yet. In fact, I can't go into detail about any of the professional work I've done recently. Everyone loves NDA's. That aside, however, I figured it might still be fun to go through and highlight some of the more noteworthy changes/additions that have been made to Noesis since my last update here.

Stereoscopic Rendering
Stereo image
A stereo rendering of a model from Rengoku II. (left eye on top, right eye on bottom)

I was briefly in possession of some 3D hardware for a CalTech contract, and during that time, I figured I might as well take the opportunity to add native stereoscopic rendering to Noesis via quadbuffering. I also added the option to do stereo ultra-shots, which still works on hardware that doesn't support actual stereo rendering. In addition to that, Noesis gained native import and export support for the MPO and JPS image formats, two of the main stereo image formats supported by a wide variety of stereo-capable hardware and software, including the Nintendo 3DS in the case of MPO.

Python
Bullet Witch
Rendering of a model from Bullet Witch, one of the first formats supported purely via Python script.

Noesis had for some time supported plugin modules in the form of native libraries. A friend and fellow reverser by the name of revelation made great use of them to add a number of new formats to Noesis. However, at the time, I had anticipated an impending need to use Python in my professional work, and I thought Noesis could benefit well from the extra accessibility in supporting a higher-level framework via an interpreted language. So, Python support was added. As I recall, the total implementation time was something short of a week of on and off work.

The rather cool thing about Python support was that I was able to add it completely within the existing native plugin framework. Not a single private hack or ugly API reacharound was necessary for the implementation to fit snugly and rather perfectly into the Noesis framework. The API has its very many shortcomings, but I was still very pleased by the ease and outcome of my Python implementation. It's definitive proof that anyone can add support for new languages and custom frameworks within the existing native plugin API.

FBX Support
FBX support was a welcome relief to some of the unpleasant issues which were accompanying the COLLADA support. However, it came with its own brand new set of difficulties and inconsistencies in implementations... despite all of these implementations being maintained by Autodesk! If you've ever worked with/for Autodesk, though, you probably don't find that the least bit surprising. Their difficulties in actual engineering are outmatched only by their difficulties in maintaining a sane infrastructure and reasonable flow of communication between products and departments. Ragging on Autodesk aside, I think that FBX support has become a new cornerstone of Noesis for reliably exporting and importing data across a wide variety of tools and software, although quite a few users still seem to favor COLLADA or even SMD/PSK. (and I'm often personally appalled by the latter)

Renderer Abstraction
If you could see the Noesis codebase, you would be horrified. Parts of it date back a good 10 years. Other parts have been licensed out to companies in the past (though I retain rights to tools/libraries in any contract I'm ever willing to sign, I've occasionally agreed to not distribute source code for bits and pieces which can be found throughout Noesis), and still others are legacy code from entirely separate tools I've written. Throughout all of this, an even more terrifying presence of OpenGL code had crept into the codebase. Some code was uploading textures directly, some code was using it to perform one-off offscreen rendering tasks, some code was hijacking the entire fucking application loop and doing its own rendering for debugging purposes. Things were not pretty.

However, it was only going to get worse if I kept leaving it alone, so I sat down and had at it. After a good solid 2-3 days of work, all of the rendering code had been nicely abstracted, a brand new graphics API established, and all rendering/GL-related code moved into its own nice little module accessible only through this new graphics API. This still means very little to the users, but paves the very convenient way for new rendering modules and solutions, from software-based to Direct3D-based. D3D is on my todo list, but it's still rather low-priority at the moment.

Auto-Update and Streamlined Build Process
There's now a convenient option under the Tools menu in Noesis to check for updates. There's even a persistent flag you can set in the data viewer to automatically check for updates on startup. If there's an update available, Noesis will ask if you want it, and if you say yes, it will automatically download it and apply it with no manual intervention. This is a pretty handy feature that I underestimated for far too long through the development of Noesis.

The other thing that I neglected for far too long was an actual build process. For a long time, I was manually zipping builds up, starting my FTP client up, and uploading builds directly. It was annoying, but I was too lazy to do anything about it. Well, wanting to also package up a self-extracting update package for the auto-update feature, the whole manual approach wasn't even an option. I wrote some custom tools and used a custom linker section (for dedicating a section of Noesis to a custom build stamp), and now the entire process is 100% automated, with dependencies in mind so that users are never exposed to an incomplete build if any step of the process fails. This includes pulling predesignated files in for the distribution, compressing them, auto-generating the self-updater, uploading all of the necessary files to a variety of servers, updating the current build version, and generating/updating the Xentax download page. This is now all done simply as a result of me double-clicking a .bat file. Very, very handy - though it sometimes results in me being a little bit trigger happy in putting new builds out.

Tons of Formats and Mini-Features
The list of formats supported by Noesis has continued to grow, albeit at a somewhat less staggering rate than it has in the past. This has been due to a few things, but most namely my focus on other framework-related aspects of the program, and my decision to primarily focus on older and/or more demanding formats. For one thing, enough people are writing plugins/scripts for Noesis now that I often don't even have to address new formats myself. For another, I'm not a big fan of figuring out the latest and greatest game's model format just so that some group of idiots can rip that game's models and distribute them all over the internet, and even though I tell people to not ever fucking do that, they do anyway. However, I believe it's usually for a greater cause to go after what's long gone and forgotten anyway. In most cases, no one else is going to, and it's often the only way that data can be fully preserved and appreciated by future generations.

Desert Strike
The state of my life, or 3 sprite frames from Desert Strike. I hope this is still covered under fair use.

Desert Strike was an interesting one. Tomaz (a friend mainly from the Quake community) had come to me asking if I could help him figure it out, because he had some interest in doing a remake and using the original assets to start. It turned out that some half-assed attempts at figuring out the game's compression format had been made, but nothing was correct or successful in decompressing all of the game's data, much less the entire compressed 32-bit LE at the start of its archive.

I opened the 16-bit stub executable in IDA and started tracing through it. The code was rather readable, but also kind of terrible, in that it tended to immediate-mov addresses into the bx and dx registers and use this to determine which operation(s)/sub-operation(s) to perform from a central decompression loop. This confused the hell out of IDA, and required me to actually hop around manually telling it which chunks of the binary were code. It didn't take long at all to translate it back to C, though, and I ended up with a flawless (well, not counting some stylistic monstrosities from my own half-assed translation job) decompression routine. The graphics formats were then simple, with the sprites dictating transparency with 8-bit "control" bytes sprinkled throughout the image. It'll be fun to see when Tomaz has something up and running in a new engine using all of these resources.

Nothing else had a process particularly worth mentioning, but there have nonetheless been some other formats/additions worth noting:
  • Multi-material support. Demonstrated by setting lightmaps up as actual materials in the Quake 2 BSP Python script.
  • New UV preview/flip options in the viewer.
  • Support for animated textures and flipping through multi-texture bundles directly in the preview pane.
  • FFXIII-2, as FFXIII was already supported. They made some extremely questionable decisions with this game, at least one of which no doubt resulted in either subtly increased load times or horrible cache misses while churning through vertex data.
  • GIF import and export. This is how I generated the bomber GIF above. It can be pretty damn convenient, if you aim to spread horrible animated GIF files all over the internet. And that's what CompuServe intended, isn't it?
  • FF Type-0 support.
  • Various Source MDL formats, including DOTA2, Portal, and maybe some others.
  • Finally went back and got those Zell/Kiros weapon models working for FF8.
  • Implemented LZX decompression with all the bells and whistles (presumably) of the latest MS implementation, which also meant going back and adding proper support for the Capcom ARC files using it. Prior to that, I'd also added big-endian/console platform support for the MOD format itself.
  • Some GMO fixes, various other image/model/compression formats that no one actually cares about, and so on.
The list goes on for a long while, but I consider those the highlights of my more recent-ish changes. I also recently added a PVRTC decoder function accessible through the various API's, but I don't expect it will find much use outside of iOS games. This is quite different from the VQTC used on the Dreamcast's PVR hardware, which while not quite visually up to par at similar data rates, also did not have nearly the same devastating impact on fetches that you'll usually see at 2bpp on an iOS device.

I was actually working on a Dreamcast port of Corporate Fury for a bit, before I got sidetracked by yet another contract (I hope to get back to it one day), so I happen to already have some textures from it laying around which have already been compressed using VQTC and PVRTC. Both use roughly the same storage space. (at ~2bpp, with the additional need for a codebook in the case of VQTC)

Steve
Original

Steve VQTCSteve PVRTC
VQTC on the left, PVRTC on the right.

This is also using the Noesis PVRTC implementation to decompress the data again, which actually happens to look quite noticeably better than on hardware or in the PVRTexTool's implementation. This is likely thanks to the high-precision interpolation of the modulation values combined with bicubic sampling.

Well, then, I think that covers everything I can actually talk about. I'm working on a next-gen renderer again as well, but it's all the same shit everyone else is doing so it's not even worth talking about. If I even could talk about it. I'll probably continue twattering crap every now and then, and mentioning on there when I do a noteworthy Noesis release.
Share:


Comments

24 comments in total.
Post a comment

IP: 68.0.165.94

Rich Whitehouse

April 2, 2012 at 11:07 pm (CST)
Shawn:
1. Glad it works.
2. You mean the visual size of joint nodes? That kinda varies depending on the importer. Not sure what kind of control I have over it in the FBX SDK, although I'd guess there's something there for it. Real joint scale is controlled by -scale and works in tandem with scaling geometry.
3. Did they include a nude model or something? I played through SO:TLH on 360, but I never did look at any of the actual game files. I was pretty surprised when I stumbled across naked Vanille too, particularly given that it was apparently in the Japanese version and they still didn't bother removing it or censoring it for the NA release. Could've caused them some serious issues if it'd turned into another "hot coffee" incident, but maybe people are finally over crap like that.
4. I got "engorged stinky" myself. Then I hit my own word filter and had to try again and got "donkey cock". That word list is probably my crowning achievement as an engineer.


IP: 71.37.142.2

Shawn

April 1, 2012 at 4:13 pm (CST)
1. Thanks for FBX support, models will load correctly into Softimage finally!
2. Is it possible to create an option for joint size on export?
3. Speaking of distributing game models, that's why I'm absolutely NOT telling anyone where a certain "version" of Reimi from Star Ocean: The Last Hope is located in the thousands of game files. I'm seriously afraid of what people would do with her...>-(
4. lemon jizz? what kind of captcha setup are you running? Ha!


IP: 68.157.228.254

LUBDAR

April 1, 2012 at 10:56 am (CST)
Ahhh yes, it is always good to see an update!


IP: 86.129.114.38

JRRRR

March 30, 2012 at 10:33 pm (CST)
Always nice to see an update. GL.


Comment Pages:
Previous ... 1 2 [3]


Post a comment

Name:


Enter the following (refresh if you can't read it):
Read image


Comment:





2555239 page hits since February 11, 2009.

Site design and contents (c) 2009 Rich Whitehouse. Except those contents which happen to be images or screenshots containing shit that is (c) someone/something else entirely. That shit isn't really mine. Fair use though! FAIR USE!
All works on this web site are the result of my own personal efforts, and are not in any way supported by any given company. You alone are responsible for any damages which you may incur as a result of this web site or files related to this web site.