Moving and rotating OARs [UPDATED]
[Update 20091216: ZOE v 0.1 has been released. You'll find a manual and the download link here.]
…and some filtering too
Opensim Archives (OARs) are nowadays the standard tool to share and interchange full Opensim regions. Justin Clark-Casey, the core Opensim developer in charge of OARs, has lately added merge functionality to the loading of OARs. This means that you can merge (i.e., add) the contents of an OAR file to an existing region, without deleting its contents. This opens a plethora of interesting possibilities, but it also is the source of a lot of new problems. What if I’m very interested in merging an OAR with one of my existing regions, but I’d prefer the new contents to be placed differently from their original position? For example, I can be given an airboxed level at 1000 meters, but I might want to put it at 2000 m (for example, because I already have contents at 1000 m). Or I might be interested in an OAR with (say) some nice palace and gardens, but it’s located in the NE corner of the sim and I’d like to install it in the SW part of the sim.
At present, the “load oar” command does not allow for this kind of transformations. Indeed, it’s debatable whether Opensim should include code for this kind of application at all, or whether such manipulations should better be handled offline.
In this article I present a very simple (and highly experimental) OAR manipulation tool called ZOE (for Zonja’s Oar Editor :-)) that implements some very basic forms of filtering, relocation and rotation for OARs as a whole.
I have been preparing (with the help of Ludmilla Writer) an exhibition of virtual worlds sculpture and photography (I’ll blog about it later). At first I was thinking of setting up a permanent art exhibition (i.e., an exhibition that would never close, and where the artists would rotate), but then I realized that Opensim technology allowed us to do something different and much more interesting: since there’s virtually no prim limit in Opensim, I could set up an “eternal” exhibition instead. By “eternal” I mean the following: the exhibitors would continue to rotate, like in a permanent exhibition, but before removing the old pictures and substitute them for the new artist’s work, I could take a snapshot of the exhibition and put it in an airbox. That way, visitors should be able to look at the “current” exhibition, but also to go “back in time” and look at all the previous exhibitions by visiting a set of airboxes.
Now I was confronted with a problem: how to make the task of taking a snapshot of the exhib and putting it in an airbox as automatic as possible? Linking the whole exhib into a mega linkset was out of question: objects have different creators, and the creators change when a new artist substitutes an old one, and besides, large linksets are difficult to manage. What would be ideal would be the following: a tool that allowed me to 1) take a snapshot of the whole of Condensation Land (the region where the exhibition is located) — this is currently doable using the “save oar” command of Opensim; 2) apply filters to the OAR so that (ideally) only the prims belonging to the exhibition are left; 3) relocate all the remaining prims to an airbox level by adding (say) 1000 meters to each linkset; and eventually 4) rotate the exhibition 90 (180, 270) degrees so that I could put four snapshots in each aixbox level (the exhibition occupies less than 1/4 of a sim).
Since to the best of my knowledge a publicly available tool that implemented points 2, 3 and 4 above did not exist, I decided to write one myself. I’ll describe first what I implemented, and then I’ll discuss briefly how I implemented it.
OARs are gzipped tarballs
Since OARs are gzipped tarballs, I needed a tool to extract the information from the OAR, to be able to manipulate it. Following the instructions in the Opensim wiki, I downloaded and installed 7-zip, a freely available program that is able to extract (and create) OARs. I started by looking at the structure of some OARs, by reading the (scarce) documentation in the wiki and by manually inspecting the xml files for the stored linksets. My first idea was to write a tool that worked directly on the directory tree of the extracted OAR, buy then I realized that I had to manually pack and unpack OARs many times to be able to test the tool, and I ended up by incorporating 7-Zip support to my program, so that now I could choose whether to work directly against an OAR or against its unpacked filesystem contents. I designed the program to be interactive, i.e., command driven, since I did not want to design a gui (who wants a gui when you can use the command line, anyway? :-)).
The first thing that I needed was some way to get rid of (most of) the prims unrelated to the exhibition. To make the tests, I downloaded release r11651 of the Diva Distribution of Opensim, which includes support for the merge option of the load oar command, and installed it locally. I then went to the production console of Condensation Land, took an OAR of the Condensation Land region, and loaded it into my local machine. The OAR loaded itself in the SW region of the standard 2×2 megaregion:
The exhibition is located in the SE part of the region (it’s the inverted “L” to the bottom right):
Clearly, I needed a way to get rid of the zeppelin, the rotating balloon, my home, Arrabal, etc. I logged in and saw that the leftmost prim in the exhib had an X greater than 129, so I programmed a little and implemented a “delete” command:
delete where x < 129
Here’s the result:
Now I saw that all objects that interested me had an Y lower than 127, so that I now used:
delete where y > 127
and got the following:
which looked pretty good! There were some remaining unwanted objects (including a carpet and some few plants), but one thing is to manually delete ten objects and another one is to delete almost a whole sim. Here’s another view of what was left after the two “delete” operations:
Now I needed a way to move all contents up, so that I implemented an “add” command that added a (positive or negative) number to any of the X, Y, Z coordinates to each and every linkset in the OAR. Then I wrote
add 30 to z
to test whether raising worked properly. Here’s the result:
Moving and megaregions
Since I didn’t implement any kind of control about the quantities you add to X, Y or Z, I asked myself what would happen if I added 256 to a whole OAR, and then load-merged the resulting OAR into a megaregion. I tried it, and here’s what I got:
That’s the result of using the following command sequence to CL.oar:
delete where x < 129 delete where y > 127 add 256 to x
then saving the resulting oar as “two.oar”, then loading CL.oar, then load-merging “two.oar”.
This was an unexpected application of ZOE that looked very promising! :-) It allows to take a normal OAR, and prepare it to be loaded at any sub-region of a megaregion. Unfortunately, the terrain does not load, but, although this is nuisance, the terrain can always be loaded by other means.
I then checked whether the insidious red error messages appeared by using this novel method of loading OARs, but unfortunately they still appear, so that resorting to the use of the “fix-phantoms” command is still mandatory.
I finally needed some form of rotation. Rotation is more difficult to implement than translation, so that I opted for a simple approach: I’d implement first a rotation of 90 degrees (clockwise), and learn from there. I had to learn how to use quaternions, but after a short while I had a “rotate” command working:
The above is the result of filtering CL.oar and loading it, then rotating the oar 90 degrees and moving it 256 meters to the right, then load-merging it.
The “rotate” command only rotates prims. It does nothing to the terrain files — you can always edit them with some other program and rotate them yourself.
As an aside, and quite interestingly, Second Life “does not have the ability” to rotate regions ;-) It took me less than three hours to program the 90 degrees rotation, and I had to learn quaternion arithmetic and investigate the (undocumented) structure of the xml files contained in an OAR. Makes me wonder… :-)
Now I’ll describe how I wrote the program. Unfortunately, the documentation for OARs is minimal. Particularly, there’s no documentation, to the best of my knowledge, for the structure of the xml files representing linksets: linksets seem to be stored in the XML2 format, but XML2 appears to be also undocumented. I inspected several of the XML files by hand (using the notepad application and also opening them with IE, which shows best the xml structure).
Linksets are stored as a long line containing the xml representation of each and every prim composing the linkset. Particularly, there’s a section called “GroupPosition” that stores the position of each prim in the linkset relative to the region, for example:
would apply to a prim at <120,118,22>. There’s a “GroupPosition” section for each prim in the linkset, but all the sections have the same values. This is so because the child prims store their position relative to the root prim and in a different section.
Therefore, to start with we don’t need to do any complicated xml parsing — we just have to load the xml file, parse the first “GroupPosition” block, and we’ll get the X,Y,Z coordinates of the linkset (Opensim-generated OARs contain a rounding of the linkset’ coordinates to the nearest integer coded in their filenames, but this information can’t be used reliably, because of two reasons: it’s not accurate enough, because of the rounding, and it can be changed without making the OAR invalid, i.e., you’re not guaranteed that the values stored in the filename will coincide with the real “GroupPosition” values).
Now implementing filtering is trivial: each linkset is inspected in turn, and the ones that don’t satisfy the filtering criteria are deleted (from memory and from the filesystem too). Similarly, implementing moving is easy: in this case, we do have to parse all the occurences of “GroupPosition” blocks and substitute the new coordinates for every prim in the linkset, then rewrite the xml file. Note that the xml files representing linksets can grow quite large when the linksets themselves are large: for example, the linkset for my 2889 prims Klein bottle is > 10 MB — and, unsurprisingly, moving it means that we have to perform 2889 coordinate substitutions in a 10 MB string, which is relatively expensive.
Rotating is a little more involved. We have to calculate the new <X,Y> coordinates for the root prim, and rotate the root prim. This is done using the “OffsetPosition” and “RotationOffset” blocks — these come immediately after each “GroupPosition” block, and are easy to parse. The “GroupPosition” and “OffsetPosition” values have to be recomputed using simple trigonometry, and “RotationOffset” is a quaternion, which has to be multiplied by another quaternion of the form <0,0,sin(alpha)/2,cos(alpha)/2> for a Z-rotation of “alpha” radians.
Finally, to implement support for automatic handling of files (in Windows, I don’t have access to a linux machine), one has to download and install 7-Zip, and modify the PATH environment variable so that the 7-Zip executable (7z.exe) is accesible from the command line.
The program is written in Open Object Rexx, a language I’m very familiar with and which allows me to write code very fast. Here’s a test run of ZOE:
Now I only need some time to write a short readme file, adjoin a license, and upload it all to some public place, in case somebody wants to play/experiment with it :-)
Some final remarks, and some things to do
I would have loved to implement some form of filtering by creator and/or by owner, but unfortunately the current version (0.2) of the OAR file format does not store the avatar names — only their UUIDs are stored. It would be really handy and convenient if a simple UUID <-> Avatar name mapping could be added to OARs, this would greatly enhance the possibilities of offline OAR editors like ZOE.
Linksets refer to assets, i.e., they refer to textures, scripts, notecards, etc., either used by the prims themselves (textures) or stored inside some of these prims (scripts, notecards, …). ZOE works by brute force, i.e., it leaves all the assets in the OAR intact. The right way to proceed would be to keep a reference count on the assets and delete those assets that are no longer needed.
I’m currently loading all the linksets in memory. Even for a moderately large OAR, the Condensation Land region, a 70MB oar file containing 10,000+ prims collected into 1000+ objects, the memory footprint is very small (50 MB on my Windows machine).
I’m amazed at how easy it has been to program ZOE. I’ve invested more time learning how to multiply quaternions and writing this blog post than in writing the whole program. I see a great future in tools like ZOE — there are some things that are better done offline, rather than doing them in-world. In this sense, it would be good to see the “load oar” and “save oar” Opensim commands directly supporting the filesystem, like some posts in the Opensim-dev lists have already suggested.