Zonja Capalini

Moving and rotating OARs [UPDATED]

[Update 20091216: ZOE v 0.1 has been released. You’ll find a manual and the download link here.]

…and some filtering too

Opensim Archives (OARs) are nowadays the standard tool to share and interchange full Opensim regions. Justin Clark-Casey, the core Opensim developer in charge of OARs, has lately added merge functionality to the loading of OARs. This means that you can merge (i.e., add) the contents of an OAR file to an existing region, without deleting its contents. This opens a plethora of interesting possibilities, but it also is the source of a lot of new problems. What if I’m very interested in merging an OAR with one of my existing regions, but I’d prefer the new contents to be placed differently from their original position? For example, I can be given an airboxed level at 1000 meters, but I might want to put it at 2000 m (for example, because I already have contents at 1000 m). Or I might be interested in an OAR with (say) some nice palace and gardens, but it’s located in the NE corner of the sim and I’d like to install it in the SW part of the sim.

At present, the “load oar” command does not allow for this kind of transformations. Indeed, it’s debatable whether Opensim should include code for this kind of application at all, or whether such manipulations should better be handled offline.

0705 - ZOE- A test run

In this article I present a very simple (and highly experimental) OAR manipulation tool called ZOE (for Zonja’s Oar Editor :-)) that implements some very basic forms of filtering, relocation and rotation for OARs as a whole.

Motivation

I have been preparing (with the help of Ludmilla Writer) an exhibition of virtual worlds sculpture and photography (I’ll blog about it later). At first I was thinking of setting up a permanent art exhibition (i.e., an exhibition that would never close, and where the artists would rotate), but then I realized that Opensim technology allowed us to do something different and much more interesting: since there’s virtually no prim limit in Opensim, I could set up an “eternal” exhibition instead. By “eternal” I mean the following: the exhibitors would continue to rotate, like in a permanent exhibition, but before removing the old pictures and substitute them for the new artist’s work, I could take a snapshot of the exhibition and put it in an airbox. That way, visitors should be able to look at the “current” exhibition, but also to go “back in time” and look at all the previous exhibitions by visiting a set of airboxes.

Now I was confronted with a problem: how to make the task of taking a snapshot of the exhib and putting it in an airbox as automatic as possible? Linking the whole exhib into a mega linkset was out of question: objects have different creators, and the creators change when a new artist substitutes an old one, and besides, large linksets are difficult to manage. What would be ideal would be the following: a tool that allowed me to 1) take a snapshot of the whole of Condensation Land (the region where the exhibition is located) — this is currently doable using the “save oar” command of Opensim; 2) apply filters to the OAR so that (ideally) only the prims belonging to the exhibition are left; 3) relocate all the remaining prims to an airbox level by adding (say) 1000 meters to each linkset; and eventually 4) rotate the exhibition 90 (180, 270) degrees so that I could put four snapshots in each aixbox level (the exhibition occupies less than 1/4 of a sim).

Since to the best of my knowledge a publicly available tool that implemented points 2, 3 and 4 above did not exist, I decided to write one myself. I’ll describe first what I implemented, and then I’ll discuss briefly how I implemented it.

OARs are gzipped tarballs

Since OARs are gzipped tarballs, I needed a tool to extract the information from the OAR, to be able to manipulate it. Following the instructions in the Opensim wiki, I downloaded and installed 7-zip, a freely available program that is able to extract (and create) OARs. I started by looking at the structure of some OARs, by reading the (scarce) documentation in the wiki and by manually inspecting the xml files for the stored linksets. My first idea was to write a tool that worked directly on the directory tree of the extracted OAR, buy then I realized that I had to manually pack and unpack OARs many times to be able to test the tool, and I ended up by incorporating 7-Zip support to my program, so that now I could choose whether to work directly against an OAR or against its unpacked filesystem contents. I designed the program to be interactive, i.e., command driven, since I did not want to design a gui (who wants a gui when you can use the command line, anyway? :-)).

Filtering

The first thing that I needed was some way to get rid of (most of) the prims unrelated to the exhibition. To make the tests, I downloaded release r11651 of the Diva Distribution of Opensim, which includes support for the merge option of the load oar command, and installed it locally. I then went to the production console of Condensation Land, took an OAR of the Condensation Land region, and loaded it into my local machine. The OAR loaded itself in the SW region of the standard 2×2 megaregion:

0697 - Condensation Land loaded in the SW part of a 2x2 megaregion

The exhibition is located in the SE part of the region (it’s the inverted “L” to the bottom right):

0698 - Condensation Land

Clearly, I needed a way to get rid of the zeppelin, the rotating balloon, my home, Arrabal, etc. I logged in and saw that the leftmost prim in the exhib had an X greater than 129, so I programmed a little and implemented a “delete” command:

    delete where x < 129

Here’s the result:

0699 - Condensation Land after deleting all objects with x less than 129

Now I saw that all objects that interested me had an Y lower than 127, so that I now used:

    delete where y > 127

and got the following:

0700 - Condensation Land after deleting all objects where x lt 129 and y gt 127

which looked pretty good! There were some remaining unwanted objects (including a carpet and some few plants), but one thing is to manually delete ten objects and another one is to delete almost a whole sim. Here’s another view of what was left after the two “delete” operations:

0701 - Another view of the same

Moving

Now I needed a way to move all contents up, so that I implemented an “add” command that added a (positive or negative) number to any of the X, Y, Z coordinates to each and every linkset in the OAR. Then I wrote

    add 30 to z

to test whether raising worked properly. Here’s the result:

0702 - After Add 30 to z

Moving and megaregions

Since I didn’t implement any kind of control about the quantities you add to X, Y or Z, I asked myself what would happen if I added 256 to a whole OAR, and then load-merged the resulting OAR into a megaregion. I tried it, and here’s what I got:

0703 - Condensation Land merged with the exhib + 256 m

That’s the result of using the following command sequence to CL.oar:

    delete where x < 129
    delete where y > 127
    add 256 to x

then saving the resulting oar as “two.oar”, then loading CL.oar, then load-merging “two.oar”.

This was an unexpected application of ZOE that looked very promising! :-) It allows to take a normal OAR, and prepare it to be loaded at any sub-region of a megaregion. Unfortunately, the terrain does not load, but, although this is nuisance, the terrain can always be loaded by other means.

I then checked whether the insidious red error messages appeared by using this novel method of loading OARs, but unfortunately they still appear, so that resorting to the use of the “fix-phantoms” command is still mandatory.

Rotating

I finally needed some form of rotation. Rotation is more difficult to implement than translation, so that I opted for a simple approach: I’d implement first a rotation of 90 degrees (clockwise), and learn from there. I had to learn how to use quaternions, but after a short while I had a “rotate” command working:

0704 - The exhib and its rotated clone

The above is the result of filtering CL.oar and loading it, then rotating the oar 90 degrees and moving it 256 meters to the right, then load-merging it.

The “rotate” command only rotates prims. It does nothing to the terrain files — you can always edit them with some other program and rotate them yourself.

As an aside, and quite interestingly, Second Life “does not have the ability” to rotate regions ;-) It took me less than three hours to program the 90 degrees rotation, and I had to learn quaternion arithmetic and investigate the (undocumented) structure of the xml files contained in an OAR. Makes me wonder… :-)

The program

Now I’ll describe how I wrote the program. Unfortunately, the documentation for OARs is minimal. Particularly, there’s no documentation, to the best of my knowledge, for the structure of the xml files representing linksets: linksets seem to be stored in the XML2 format, but XML2 appears to be also undocumented. I inspected several of the XML files by hand (using the notepad application and also opening them with IE, which shows best the xml structure).

Linksets are stored as a long line containing the xml representation of each and every prim composing the linkset. Particularly, there’s a section called “GroupPosition” that stores the position of each prim in the linkset relative to the region, for example:

    <GroupPosition><X>120</X><Y>118</Y><Z>22</Z></GroupPosition>

would apply to a prim at <120,118,22>. There’s a “GroupPosition” section for each prim in the linkset, but all the sections have the same values. This is so because the child prims store their position relative to the root prim and in a different section.

Therefore, to start with we don’t need to do any complicated xml parsing — we just have to load the xml file, parse the first “GroupPosition” block, and we’ll get the X,Y,Z coordinates of the linkset (Opensim-generated OARs contain a rounding of the linkset’ coordinates to the nearest integer coded in their filenames, but this information can’t be used reliably, because of two reasons: it’s not accurate enough, because of the rounding, and it can be changed without making the OAR invalid, i.e., you’re not guaranteed that the values stored in the filename will coincide with the real “GroupPosition” values).

Now implementing filtering is trivial: each linkset is inspected in turn, and the ones that don’t satisfy the filtering criteria are deleted (from memory and from the filesystem too). Similarly, implementing moving is easy: in this case, we do have to parse all the occurences of “GroupPosition” blocks and substitute the new coordinates for every prim in the linkset, then rewrite the xml file. Note that the xml files representing linksets can grow quite large when the linksets themselves are large: for example, the linkset for my 2889 prims Klein bottle is > 10 MB — and, unsurprisingly, moving it means that we have to perform 2889 coordinate substitutions in a 10 MB string, which is relatively expensive.

Rotating is a little more involved. We have to calculate the new <X,Y> coordinates for the root prim, and rotate the root prim. This is done using the “OffsetPosition” and “RotationOffset” blocks — these come immediately after each “GroupPosition” block, and are easy to parse. The “GroupPosition” and “OffsetPosition” values have to be recomputed using simple trigonometry, and “RotationOffset” is a quaternion, which has to be multiplied by another quaternion of the form <0,0,sin(alpha)/2,cos(alpha)/2> for a Z-rotation of “alpha” radians.

Finally, to implement support for automatic handling of files (in Windows, I don’t have access to a linux machine), one has to download and install 7-Zip, and modify the PATH environment variable so that the 7-Zip executable (7z.exe) is accesible from the command line.

The program is written in Open Object Rexx, a language I’m very familiar with and which allows me to write code very fast. Here’s a test run of ZOE:

Moving and rotating OARs

Now I only need some time to write a short readme file, adjoin a license, and upload it all to some public place, in case somebody wants to play/experiment with it :-)

Some final remarks, and some things to do

I would have loved to implement some form of filtering by creator and/or by owner, but unfortunately the current version (0.2) of the OAR file format does not store the avatar names — only their UUIDs are stored. It would be really handy and convenient if a simple UUID <-> Avatar name mapping could be added to OARs, this would greatly enhance the possibilities of offline OAR editors like ZOE.

Linksets refer to assets, i.e., they refer to textures, scripts, notecards, etc., either used by the prims themselves (textures) or stored inside some of these prims (scripts, notecards, …). ZOE works by brute force, i.e., it leaves all the assets in the OAR intact. The right way to proceed would be to keep a reference count on the assets and delete those assets that are no longer needed.

I’m currently loading all the linksets in memory. Even for a moderately large OAR, the Condensation Land region, a 70MB oar file containing 10,000+ prims collected into 1000+ objects, the memory footprint is very small (50 MB on my Windows machine).

I’m amazed at how easy it has been to program ZOE. I’ve invested more time learning how to multiply quaternions and writing this blog post than in writing the whole program. I see a great future in tools like ZOE — there are some things that are better done offline, rather than doing them in-world. In this sense, it would be good to see the “load oar” and “save oar” Opensim commands directly supporting the filesystem, like some posts in the Opensim-dev lists have already suggested.

December 14, 2009 - Posted by | OpenSim, Tech News, ZOE | , , , , ,

23 Comments »

  1. I just want to say this looks like fantastic stuff, Zonja – it’s just the kind of thing I was hoping would happen. I’m really gummed up with high priority tasks right now but I’ll definitely be coming back later for a leisurely read and much more considered comments. But I already agree with all the points I’ve skimmed (undocumented xml2, avatar names not saved, etc.) and maybe we can even do something about them, someday :)

    Also very much look forward to the code coming out ;)

    Comment by Justin Clark-Casey | December 14, 2009

  2. […] This post was mentioned on Twitter by Stefan Andersson, Justin Clark-Casey. Justin Clark-Casey said: RT @lbsa71: Simple Offline Editing of #OpenSim OAR: http://tinyurl.com/yejog9v (supercool from Zonja Capalini. Must get reading time… ngg) […]

    Pingback by Tweets that mention Moving and rotating OARs « Zonja Capalini -- Topsy.com | December 14, 2009

  3. Zonja —

    Fantastic work! I can see an OpenSim content management system starting to coalesce… :-)

    — Maria Korolov
    Editor, Hypergrid Business (http://www.hypergridbusiness.com)

    Comment by Maria Korolov | December 14, 2009

  4. Social comments and analytics for this post…

    This post was mentioned on Twitter by lbsa71: Simple Offline Editing of #OpenSim OAR: http://tinyurl.com/yejog9v

    Trackback by uberVU - social comments | December 14, 2009

  5. Querida Zonja, felicitaciones por esta fantástica herramienta que es realmente práctica y tiene un potencial tremendo. De hecho insisto en la creación del término “Zoeador” como sinónimo de aquél usuario que planifique y organice un sim usando tu herramienta.
    Un gran aplauso xiqueta ;-)

    Comment by Alberto Navarro | December 15, 2009

  6. Ah, managed to steal some time today to read your post properly :)

    * Parts of the OAR format are certainly not well documented which is ironic since I think that this is one of the better documented parts of OpenSim. Admittedly this isnt’ saying much :). The ‘xml2’ format for prim data certainly is a pita. I used it because it already existed when I was creating the OAR facility. But it’s not much more than the automatic .NET serialization of the internal SceneObjectGroup/SceneObjectPart classes and as such is completely undocumented.

    Hopefully this will change one day – I would certainly like to see something that is at least documented, even if the ‘xml2’ format itself is pretty trashy and contains redundant information. I would love to do that if I had time but unfortunately it’s way down on the priority list.

    * Regarding avatar names, I’ve actually implemented something like that for IARs, though only for creator names. I wrote information up on this at

    http://justincc.org/blog/2009/09/25/preserving-content-creator-credit-in-opensim-iar-transfers/

    which I suspect that you’re read. Essentially, instead of saving avatar uuids, avatar names are saved instead and resolved to user accounts when the IAR is loaded.

    This was to support a grander effort (now in suspension) to support persistent creator data even if a user had no local account on the OpenSim to which an IAR was loaded. But it also has the effect of inserting avatar names into the archives (though avatar uuids would be missing). Would this be sufficient? It’s on my long TODO list to put this into OARs but I might be able to move it up a bit…

    * I tend to agree with you that tools like ZOE are better off outside OpenSim. It might be nice to have some of the most simple transformations in OpenSim itself but anything more complicated should probably take place in an external tool.

    All in all, I really like the sound of ZOE, Zonja, and am looking forward to seeing the release. I’ve been asked various times about these kinds of changes and transformations and soon I’ll just be able to point people at ZOE :)

    Comment by Justin Clark-Casey | December 15, 2009

  7. @Maria: Thanks! :-) Anyway, I’m afraid it won’t be me who writes it — I’m more for KISS tools, like ZOE :-)

    @Alberto: Gracias, y también gracias por ser un beta-Zoeador ;-)

    @Justin: Many thanks for your nice comments, and for your very detailed reply! :-)

    A solution to the uuid <-> name problem which would be, as far as I understand, very easy to implement, and which would not break backwards compatibility, would be the following:

    * Define a 0.3 OAR format that includes a supplemental directory, maybe called “users”.

    * Include in it a single ASCII7 xml file containing a set of (uuid, name) pairs, after selecting suitable xml node names.

    * Dump all linkset creators and owners there. This could be done after dumping the linksets themselves, and so it appears to be trivial (in principle).

    The advantages of this proposal are:

    * This should be extremely easy to implement

    * It doesn’t break backwards compatibility (i.e., 0.3 OARs are legible and understandable by 0.2 installations)

    * It’s agnostic wrt the uuid/name debate, and therefore it allows for the implementation of different politics at the “load” end — or at an imagined previous pre-load point.

    The main disadvangate is that it’s not very compatible with your implementation of OSPAs in IARs.

    Btw I’d love to help in documenting the xml2 format — I consider myself to be a quite good technical writer (apart from the language barrier — I’m native catalan and spanish by birth). My main problem with it is recursive: I can’t document what’s not documented :-)

    Comment by Zonja Capalini | December 16, 2009

  8. The idea of a query language (OSQL? ;)) in-world (or outside) looks much more than interresting. Thanks for sharing this. It gives a lot of ideas.

    Comment by Laurent | December 16, 2009

  9. […] that I wrote recently. You can find a detailed description of its functionality in my recent post “Moving and rotating OARs”. In this post I’ll describe how to install and use Zoe. I’ve designed Zoe with […]

    Pingback by ZOE version 0.1 released « Zonja Capalini | December 16, 2009

  10. @Laurent: Thanks! :-) I love SQL, why invent something different when one can borrow from something that is well designed and works? :-)

    Comment by Zonja Capalini | December 16, 2009

  11. @Justin (and all): here’s the manual and the release: https://zonjacapalini.wordpress.com/2009/12/16/zoe-version-0-1-released/ :-)

    Comment by Zonja Capalini | December 16, 2009

  12. Zonja,

    once again, thank you for an excellent piece. Way to go!

    If you and/or Justin (and possibly Diva, since it’s really spot-on for hypergrid) would address the user ref bits, can I please just reiterate my suggestion that all user refs should be urls pointing to the full (public) xml user profile on that grid. (“http://mygrid.net/users/zonja/profile” f.ex) This way, there will be only ever one instance of a user profile floating around; proliferating and duplicating user profile data, will lead to major pita fast. This is really one of those things where a quick fix now could lead to major disadvantages later.

    Comment by Stefan Andersson | December 16, 2009

  13. @Stefan: Many thanks :-) It’s the same for me to parse an URL than a firstname lastname pair, except for first or last names with a period inside (I for example have a Zonja.Capalini Lives account to make some tests — don’t ask XD), so that maybe the urls should be in the form firstname/lastname instead of firstname.lastname.

    I see a problem in the fact that many grids run on ip addresses and not domain names — http://condensationland.com/users/zonja/capalini/profile is pretty cool, but http://87.216.55.54/users/zonja/capalini/profile is a major mess.

    Another problem is that many Opensim installations are running a wamp or xampp installation to handle the database, but not necessarily opensimwi, which is “not recommended” for standalones. I think if such a solution were adopted, the xml files should be served by Opensim itself, not by an external add on.

    But having said that, I think you’re completely right, of course.

    Comment by Zonja Capalini | December 16, 2009

  14. @Stefan: Thinking a little more about the subject. I’d still store the uuid and create a uuid <-> profile mapping. Assume that you have an installation at mygrid.net that saves an OAR, then for some reason is forced to change domains to mygrid.org. Having the uuids stored in the OAR allow for a correct restoring, while having only the profiles would force Opensim to erase all owners and creators. I think a mixed solution is in order here.

    Comment by Zonja Capalini | December 16, 2009

  15. Felicitaciones !!!

    Y gracias por tu trabajo para esta comunidad que va tomando cada vez más fuerza !!!

    Comment by Ludmilla | December 19, 2009

  16. Muchas gracias, Ludmilla! Y también por tu esfuerzo y dedicación en Condensation, concretamente en Condensatio Beach y colaborando en Mirror Worlds! :-)

    Comment by Zonja Capalini | December 19, 2009

  17. ¿Puedo ser una “Zoeadora” o es solamente para los entendidos ?

    Comment by Ludmilla | December 19, 2009

  18. Ja ja! :-) Es un poco hardcore, pero te puedo enseñar :-) Mándame un email y te lo explico :-)

    Comment by Zonja Capalini | December 19, 2009

  19. Un artículo chulísimo sobre Zoe escrito por Alberto (en castellano): http://exploradorvirtual.blogspot.com/2009/12/utilizando-zoe.html

    Comment by Zonja Capalini | December 21, 2009

  20. […] I noted in my previous post “Moving and rotating OARs“, offline editing of Opensim Archives (OARs) opens a lot of interesting possibilities. In […]

    Pingback by Using SQL to manipulate OARs « Zonja Capalini | January 1, 2010

  21. @Laurent: Your suggestion of using SQL to manipulate OARs inspired me: https://zonjacapalini.wordpress.com/2010/01/01/using-sql-to-manipulate-oars/ :-)

    Comment by Zonja Capalini | January 1, 2010

  22. […] Now since (if we got enough artists willing to participate) I envisaged having to perform the cloning operation several times, I decided to develop an automated tool to assist in the cloning, and I developed the (admittedly hacky) tool I called Zoe — you can find a detailed description of the technical details in my post “Moving and rotating OARs“. […]

    Pingback by Mirror Worlds: a hypergridded eternal art exhib « Zonja Capalini | January 17, 2010

  23. Where can I download the program. Zonja’s OAR Editor. I have lost mine.

    Comment by Leslie Kling | June 21, 2015


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: