March 29, 2012 at 6:45 am (CST)
It's been a while. I've been through a couple of contract projects, worked for Sony for a bit, did a bunch of stuff to Noesis, and now I'm on another contract project. I think that pretty much sums up the last 9 months.
My current project is rather cool in concept, but I can't talk about it yet. In fact, I can't go into detail about any of the professional work I've done recently. Everyone loves NDA's. That aside, however, I figured it might still be fun to go through and highlight some of the more noteworthy changes/additions that have been made to Noesis since my last update here.
A stereo rendering of a model from Rengoku II. (left eye on top, right eye on bottom)
I was briefly in possession of some 3D hardware for a CalTech contract, and during that time, I figured I might as well take the opportunity to add native stereoscopic rendering to Noesis via quadbuffering. I also added the option to do stereo ultra-shots, which still works on hardware that doesn't support actual stereo rendering. In addition to that, Noesis gained native import and export support for the MPO
image formats, two of the main stereo image formats supported by a wide variety of stereo-capable hardware and software, including the Nintendo 3DS in the case of MPO.
Rendering of a model from Bullet Witch, one of the first formats supported purely via Python script.
Noesis had for some time supported plugin modules in the form of native libraries. A friend and fellow reverser by the name of revelation made great use of them to add a number of new formats
to Noesis. However, at the time, I had anticipated an impending need to use Python in my professional work, and I thought Noesis could benefit well from the extra accessibility in supporting a higher-level framework via an interpreted language. So, Python support was added. As I recall, the total implementation time was something short of a week of on and off work.
The rather cool thing about Python support was that I was able to add it completely within the existing native plugin framework. Not a single private hack or ugly API reacharound was necessary for the implementation to fit snugly and rather perfectly into the Noesis framework. The API has its very many shortcomings, but I was still very pleased by the ease and outcome of my Python implementation. It's definitive proof that anyone can add support for new languages and custom frameworks within the existing native plugin API.
FBX support was a welcome relief to some of the unpleasant issues which were accompanying the COLLADA support. However, it came with its own brand new set of difficulties and inconsistencies in implementations... despite all of these implementations being maintained by Autodesk! If you've ever worked with/for Autodesk, though, you probably don't find that the least bit surprising. Their difficulties in actual engineering are outmatched only by their difficulties in maintaining a sane infrastructure and reasonable flow of communication between products and departments. Ragging on Autodesk aside, I think that FBX support has become a new cornerstone of Noesis for reliably exporting and importing data across a wide variety of tools and software, although quite a few users still seem to favor COLLADA or even SMD/PSK. (and I'm often personally appalled by the latter)
If you could see the Noesis codebase, you would be horrified. Parts of it date back a good 10 years. Other parts have been licensed out to companies in the past (though I retain rights to tools/libraries in any contract I'm ever willing to sign, I've occasionally agreed to not distribute source code for bits and pieces which can be found throughout Noesis), and still others are legacy code from entirely separate tools I've written. Throughout all of this, an even more terrifying presence of OpenGL code had crept into the codebase. Some code was uploading textures directly, some code was using it to perform one-off offscreen rendering tasks, some code was hijacking the entire fucking application loop and doing its own rendering for debugging purposes. Things were not pretty.
However, it was only going to get worse if I kept leaving it alone, so I sat down and had at it. After a good solid 2-3 days of work, all of the rendering code had been nicely abstracted, a brand new graphics API established, and all rendering/GL-related code moved into its own nice little module accessible only through this new graphics API. This still means very little to the users, but paves the very convenient way for new rendering modules and solutions, from software-based to Direct3D-based. D3D is on my todo list, but it's still rather low-priority at the moment.
Auto-Update and Streamlined Build Process
There's now a convenient option under the Tools menu in Noesis to check for updates. There's even a persistent flag you can set in the data viewer to automatically check for updates on startup. If there's an update available, Noesis will ask if you want it, and if you say yes, it will automatically download it and apply it with no manual intervention. This is a pretty handy feature that I underestimated for far too long through the development of Noesis.
The other thing that I neglected for far too long was an actual build process. For a long time, I was manually zipping builds up, starting my FTP client up, and uploading builds directly. It was annoying, but I was too lazy to do anything about it. Well, wanting to also package up a self-extracting update package for the auto-update feature, the whole manual approach wasn't even an option. I wrote some custom tools and used a custom linker section (for dedicating a section of Noesis to a custom build stamp), and now the entire process is 100% automated, with dependencies in mind so that users are never exposed to an incomplete build if any step of the process fails. This includes pulling predesignated files in for the distribution, compressing them, auto-generating the self-updater, uploading all of the necessary files to a variety of servers, updating the current build version, and generating/updating the Xentax download page. This is now all done simply as a result of me double-clicking a .bat file. Very, very handy - though it sometimes results in me being a little bit trigger happy in putting new builds out.
Tons of Formats and Mini-Features
The list of formats supported by Noesis has continued to grow, albeit at a somewhat less staggering rate than it has in the past. This has been due to a few things, but most namely my focus on other framework-related aspects of the program, and my decision to primarily focus on older and/or more demanding formats. For one thing, enough people are writing plugins/scripts for Noesis now that I often don't even have to address new formats myself. For another, I'm not a big fan of figuring out the latest and greatest game's model format just so that some group of idiots can rip that game's models and distribute them all over the internet, and even though I tell people to not ever fucking do that
, they do anyway. However, I believe it's usually for a greater cause to go after what's long gone and forgotten anyway. In most cases, no one else is going to, and it's often the only way that data can be fully preserved and appreciated by future generations.
The state of my life, or 3 sprite frames from Desert Strike. I hope this is still covered under fair use.
Desert Strike was an interesting one. Tomaz (a friend mainly from the Quake community) had come to me asking if I could help him figure it out, because he had some interest in doing a remake and using the original assets to start. It turned out that some half-assed attempts at figuring out the game's compression format had been made, but nothing was correct or successful in decompressing all of the game's data, much less the entire compressed 32-bit LE at the start of its archive.
I opened the 16-bit stub executable in IDA and started tracing through it. The code was rather readable, but also kind of terrible, in that it tended to immediate-mov addresses into the bx and dx registers and use this to determine which operation(s)/sub-operation(s) to perform from a central decompression loop. This confused the hell out of IDA, and required me to actually hop around manually telling it which chunks of the binary were code. It didn't take long at all to translate it back to C, though, and I ended up with a flawless (well, not counting some stylistic monstrosities from my own half-assed translation job) decompression routine. The graphics formats were then simple, with the sprites dictating transparency with 8-bit "control" bytes sprinkled throughout the image. It'll be fun to see when Tomaz has something up and running in a new engine using all of these resources.
Nothing else had a process particularly worth mentioning, but there have nonetheless been some other formats/additions worth noting:
- Multi-material support. Demonstrated by setting lightmaps up as actual materials in the Quake 2 BSP Python script.
- New UV preview/flip options in the viewer.
- Support for animated textures and flipping through multi-texture bundles directly in the preview pane.
- FFXIII-2, as FFXIII was already supported. They made some extremely questionable decisions with this game, at least one of which no doubt resulted in either subtly increased load times or horrible cache misses while churning through vertex data.
- GIF import and export. This is how I generated the bomber GIF above. It can be pretty damn convenient, if you aim to spread horrible animated GIF files all over the internet. And that's what CompuServe intended, isn't it?
- FF Type-0 support.
- Various Source MDL formats, including DOTA2, Portal, and maybe some others.
- Finally went back and got those Zell/Kiros weapon models working for FF8.
- Implemented LZX decompression with all the bells and whistles (presumably) of the latest MS implementation, which also meant going back and adding proper support for the Capcom ARC files using it. Prior to that, I'd also added big-endian/console platform support for the MOD format itself.
- Some GMO fixes, various other image/model/compression formats that no one actually cares about, and so on.
The list goes on for a long while, but I consider those the highlights of my more recent-ish changes. I also recently added a PVRTC decoder function accessible through the various API's, but I don't expect it will find much use outside of iOS games. This is quite different from the VQTC used on the Dreamcast's PVR hardware, which while not quite visually up to par at similar data rates, also did not have nearly the same devastating impact on fetches that you'll usually see at 2bpp on an iOS device.
I was actually working on a Dreamcast port of Corporate Fury for a bit, before I got sidetracked by yet another contract (I hope to get back to it one day), so I happen to already have some textures from it laying around which have already been compressed using VQTC and PVRTC. Both use roughly the same storage space. (at ~2bpp, with the additional need for a codebook in the case of VQTC)
VQTC on the left, PVRTC on the right.
This is also using the Noesis PVRTC implementation to decompress the data again, which actually happens to look quite noticeably better than on hardware or in the PVRTexTool's implementation. This is likely thanks to the high-precision interpolation of the modulation values combined with bicubic sampling.
Well, then, I think that covers everything I can actually talk about. I'm working on a next-gen renderer again as well, but it's all the same shit everyone else is doing so it's not even worth talking about. If I even could talk about it. I'll probably continue twattering crap every now and then, and mentioning on there when I do a noteworthy Noesis release.
24 comments in total.
Post a comment
August 28, 2013 at 2:19 pm (CST)
All those features sound like a real pain to work with. Especially the DX10 renderer. You might be better off using the same one from Avalanche (which is OpenGL, if I'm not mistaken).
Ever since I played Avalanche last year, for the first time, I've been enjoying the hell out of it. So, if and when you do release Corporate Fury for PC, I'd gladly get it without hesitation. Even if it means buying it. :)
August 26, 2013 at 7:43 pm (CST)
Yeah, I'd still like to do a Windows release of Corporate Fury. I've just been too busy and drained by work-related stuff to get back to it, and it's been getting juggled around with my other hobby projects as well. Getting a Windows port to a shippable state shouldn't be too much effort, but doing it justice with a better interface, controls, and asset touch-ups is going to take a little bit of time. I'd like to do a DX10 renderer for it as well, but that probably isn't worth the time investment.
In any case, I'm glad someone out there actually cares. :)
August 25, 2013 at 12:56 am (CST)
I was wondering, are there any plans for a port of Corporate Fury for PC? I think it was mentioned once. If it was mentioned before, is it still planned or have you moved on from that?
Because I don't have an iPhone and the only thing of interest on it, seems to be Corporate Fury. Call me selfish; but I'd rather not spend so much money on an iPhone just for one game.
July 4, 2013 at 3:27 pm (CST)
No Linux plans at the moment. Sometimes it runs on WINE and sometimes it doesn't, usually when it does they don't get the OpenGL context handling right. They seem to like going back and forth on breaking things. I'm always very careful not to use new API's or otherwise break WinXP compatibility, so it really shouldn't be hard to keep it running well in WINE. I'd like it if Noesis was on one of their compatibility goal lists or something.
July 1, 2013 at 6:49 am (CST)
Hello, this Noesis software looks amazing, it's exactly what I need. Is there a Linux version floating around, a repository I can hack on or anything planned to get it running on Linux? I tried with wine but it didn't seem to run quite well :(
May 6, 2013 at 6:25 am (CST)
If you're willing to crack another game model format, I've got one for you. Incoming's ".ian" format.
It'd be pretty damn awesome if you could crack the format so I can use Noesis to convert the models.
April 25, 2013 at 7:13 pm (CST)
That could be. If you want to open it up in a hex editor to check it for validity, it should start with a header that's 16 bytes (first 4 bytes is a 32 bit int specifying file count), followed by file entries that are a total of 16 bytes each, 8 bytes of each entry being the file name. The first 2 file names (in both DESERT.DAT and JUNGLE.DAT) should be "! " and "01CHARS ", the strings being located 16 and 32 bytes into the file respectively. Noesis rather hackishly uses the presence of those files to identify the format, because it doesn't have any other identifying characteristics, and .dat is a very generic extension.
If your archive isn't corrupted and doesn't match that spec, I'd be curious to hear about it, you may have a different archive file layout (maybe from a different region) and I'd be interested in adding code to make sure it gets recognized properly. I'm inclined to suspect you're right about it being corrupted if it no longer runs at all, though, since they seem to have kept that file ordering into Jungle Strike.
April 25, 2013 at 6:09 pm (CST)
..Thank you for checking and the response. I’m familiar enough with the program that I expected to see the export prompt if it couldn’t preview (like for .pak files), but it just tells me it’s an unknown file. I was at work when I read your old post about being able to extract the Desert Strike files and wanted to try it. It didn’t occur to me I might have a corrupted dat file in my game directory that I carry around with me on a usb stick. After it failed to be recognized in Noesis I attempted to run the game within dosbox and it wouldn’t so it’s not much of a stretch. When I get a chance I’ll dig out a backup of it and try again. Thanks again for your efforts.
April 25, 2013 at 7:56 am (CST)
Thanks. :) Yeah, DESERT.DAT and JUNGLE.DAT still seem to be extracting without issue in the latest Noesis. Support for those isn't in an external DLL, either, so it wouldn't be a dependency issue.
You should be able to browse to the file and double-click it. It will tell you it can't be previewed (since it's an archive), but it may be exportable, and ask you if you want to export it. Say yes, then click OK on the export dialog and it'll spit out all the files. Where are things going wrong for you in that process?
April 24, 2013 at 9:36 pm (CST)
Thanks for Noesis! I love being able to tear apart old data from the dos games(and newer). So many favorites from back then I still play via dosbox today, specially Desert Strike. Noesis 4.0785 sees the desert.dat container, but does not known what to do with it. I can't view/export anything from inside the container. Did Desert Strike archives support get broken I wonder? I don't have version 3.853 when it was 1st introduced to test against so I have no idea. Congrats on the marriage thing & welcome to the club! :)
 2 3 ... Next
Post a comment