Partial documentation for SceneObjectGroup serialization in XML2 format

Warning: This is a work-in progress. I’m publishing this article now to see whether it generates interest and which kind of feedback I get. The material published here is incomplete, and contains for sure errors, ommisions and inaccuracies. I plan on extensively modifying it during the following days. Please take that into account when reading it.

The XML2 format

The XML2 format is used in Opensim to serialize objects or linksets (called “SceneObjectGroups” in the Opensim parlance). XML2 dumps of objects can be obtained using the ’save xml2′ or ’save prims xml2′ console commands, and are used internally to serialize objects in Opensim Archive (OAR) and Inventory Archive (IAR) files. For example, OARs are gzipped tar files containing a directory called ‘objects’ with one XML2 file for each linkset in the region.

XML2 files are standard XML files using an ASCII encoding and no whitespace. This means that there’s no extra indenting or formatting between tags to make the file more pleasant to the eye. You can use any XML editor (including Internet Explorer) to display an XML2 file with some additional prettyprinting and formatting.

XML2 is practically undocumented, but the fact that it’s undocumented is itself (informally) documented :-), in comments scattered through the Opensim wiki and in the Opensim-dev and Opensim-users lists. Essentially, what this means is that the format is subject to change, but not much change: attributes (i.e., new tags) may be added to the format without breaking it — the developers program the classes taking this possibility into account and storing default values when an attribute is expected but not present (for example, if you’re loading an old OAR file which does not have the ‘AllowedDrop’ attribute into a newer installation which does have this attribute).

In this article I document partially the structure of XML2 files using a variant of the BNF notation. I include the productions for most of the format, interspersing comments wherever I have some idea of the meaning of the tag contents. I’ve obtained this information by manually looking at the XML2 files when developing the simple OAR editor I called ZOE and in recent modifications to that editor. I’ve also taken a look at the source code for Opensim and at the code for libopenmetaverse. There may be errors and there are for sure omissions; any kind of comment, addition, correction or clarification would be greatly appreciated.

The whole SceneObjectGroup/SceneObjectPart model is about to be heavily refactored to be able to include very cool things like hyerarchical objects or meshes, and this will most probably require a new format (XML3?) to serialize Opensim objects. I am writing this article as a way to teach myself, and I’m making it public in case somebody else can find its contents useful. Please keep in mind at all times that this is not official documentation :-) — that’s one of the reasons why this appears as a blog article instead as a page in the Opensim wiki, the other being that I’d prefer to get some feedback before writing such a complicated wiki page, in case it’s considered useful to have it.

Mirror Worlds: a hypergridded eternal art exhib

In this long overdue post, I introduce “Mirror Worlds”, an eternal (I will explain what I understand by ‘eternal’ in a minute) art exhibition located in the Opensim-based, hypergrid accesible, Condensation Land grid. The exhibition features pictures from Florence Babenco, Monika Finchy, Shoshisn Shilova, and Senna (SennaSpirit) Coronet, and from the Condensation Land residents Ludmilla Writer, Mikil Tiki and myself. Shoshisn also has contributed several of her wonderful sculptures for the exhibition.

Art exhibitions of all sorts are very common in Second Life, but unfortunately they are more unusual in the Opensim-based worlds — and, in case they exist at all, they get little or no publicity. Which is a real pity, because, contrary to the general lack of content for avatar clothing and embellishment in Opensim, for example, there is nothing which prevents Opensim-based worlds from hosting beautiful exhibitions. Nobody will ask where did you take a picture, after all — and texture uploads can’t be cheaper :-)

In this post I’ll present the exhibition and explain some of its more unusual features.  I’ll also describe some of the difficulties that I encountered in my work as an amateur curator — some of these difficulties may be due to my inexperience at the task, but I think that the other ones may help as a diagnosis of some problems the Opensim movement is facing.

Using SQL to manipulate OARs

Introduction

As I noted in my previous post “Moving and rotating OARs“, offline editing of Opensim Archives (OARs) opens a lot of interesting possibilities. In this article I present the results of several simple tests I have made by extending my “Zoe” OAR editor so that it supports (a very simplified sub-dialect of) SQL. Even in a simplified form, SQL has a lot of expressive power and allows to perform a number of pretty neat operation on OARs, when it comes to selecting and deleting parts of a region. Updating OARs, though, is much more complicated, because there is no “natural” semantics to apply when using the standard SQL UPDATE statement. Some simple cases can be singled out, however, where “natural” semantics actually make sense — and in some other “interesting” use cases, a custom pseudo-SQL statement seems to be sufficient.

Reverse-engineering OARs: Understanding parcel maps

Format 0.2 Opensim Archives (OARs) include, since Opensim version 3c271b, a “landdata” directory that stores parcel data. For each parcel in the region there is a file in XML format in the directory. Amost all objects in Opensim are associated with a unique uuid that identifies them, and parcels are not an exception to that rule. Files in the “landdata” directory use the uuid of the parcel as their filename and have a “.xml” extension.

What’s inside each one of these .xml files? Let’s take a peek at a sample one:

    <?xml version="1.0" encoding="utf-16"?>
    <LandData>
      <Area>6256</Area>
      <AuctionID>0</AuctionID>
      <AuthBuyerID>00000000-0000-0000-0000-000000000000</AuthBuyerID>
      <Category>-1</Category>
      <ClaimDate>1255170099</ClaimDate>
      <ClaimPrice>0</ClaimPrice>
      <GlobalID>bfbf7bd4-7d7b-4a35-aba0-c352201cddcf</GlobalID>
      <GroupID>00000000-0000-0000-0000-000000000000</GroupID>
      <IsGroupOwned>False</IsGroupOwned>
      <Bitmap>AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAP//fwAAAAAA//9/AAAAAAD//38AAAAAAP//fwAAAAAA//9/AAAAAAD//38AAAAAAP//f
       wAAAAAA//9/AAAAAAD//38AAAAAAP//fwAAAAAA//9/AAAAAAD//38AAAAAAP//fwAAAAAA/
       /9/AAAAAAD//38AAAAAAP//fwAAAAAA//9/AAAAAAA=</Bitmap>
      <Description />
      <Flags>505454667</Flags>
      <LandingType>0</LandingType>
      <Name>Arrabal Tango Club</Name>
      <Status>0</Status>
      <LocalID>4</LocalID>
      <MediaAutoScale>0</MediaAutoScale>
      <MediaID>00000000-0000-0000-0000-000000000000</MediaID>
      <MediaURL />
      <MusicURL>http://xx.xx.xx.xx:8000</MusicURL>
      <OwnerID>69af4428-cec0-48d8-a0d9-c05268facca5</OwnerID>
      <ParcelAccessList />
      <PassHours>0</PassHours>
      <PassPrice>0</PassPrice>
      <SalePrice>0</SalePrice>
      <SnapshotID>00000000-0000-0000-0000-000000000000</SnapshotID>
      <UserLocation>&lt;0, 0, 0&gt;</UserLocation>
      <UserLookAt>&lt;0, 0, 0&gt;</UserLookAt>
      <Dwell>0</Dwell>
      <OtherCleanTime>0</OtherCleanTime>
    </LandData>

That’s the xml file for Arrabal Tango Club in Condensation Land — I’ve only obscured the music URL and wrapped the part between the <Bitmap> and </Bitmap> tags, which originally used only one line, with no intervening blanks. This string encodes the shape and position of the parcel in the region. How do we know that? Well, the parcel map has to be stored somewhere, and it has to be quite large, so the only candidate is the “Bitmap” string :-) Most probably this long string is a text encoding of binary data.

Indeed this is the case: the string is using the Base64 encoding (you can find that quickly by browsing the Wikipedia). After decoding the string, we get a binary chunk of exactly 512 bytes. I checked that all the terrain files have strings of exactly the same length, so this number had to mean something.

ZOE version 0.1 released

Zoe

Zoe (Zonja’s OAR Editor) is a simple offline OAR manipulation tool that I wrote recently. You can find a detailed description of its functionality in my recent post “Moving and rotating OARs”. In this post I’ll describe how to install and use Zoe. I’ve designed Zoe with simplicity in mind, so that currently it only implements some very basic operations — by combining these operations, however, a number of pretty cool things can already be done. I’d love to get some feedback about the functionality of Zoe — at the end of the post I’ll list some of the things that I want to add to it, but I’m open to suggestions.

Moving and rotating OARs [UPDATED]

[Update 20091216: ZOE v 0.1 has been released. You'll find a manual and the download link here.]

…and some filtering too

Opensim Archives (OARs) are nowadays the standard tool to share and interchange full Opensim regions. Justin Clark-Casey, the core Opensim developer in charge of OARs, has lately added merge functionality to the loading of OARs. This means that you can merge (i.e., add) the contents of an OAR file to an existing region, without deleting its contents. This opens a plethora of interesting possibilities, but it also is the source of a lot of new problems. What if I’m very interested in merging an OAR with one of my existing regions, but I’d prefer the new contents to be placed differently from their original position? For example, I can be given an airboxed level at 1000 meters, but I might want to put it at 2000 m (for example, because I already have contents at 1000 m). Or I might be interested in an OAR with (say) some nice palace and gardens, but it’s located in the NE corner of the sim and I’d like to install it in the SW part of the sim.

At present, the “load oar” command does not allow for this kind of transformations. Indeed, it’s debatable whether Opensim should include code for this kind of application at all, or whether such manipulations should better be handled offline.

In this article I present a very simple (and highly experimental) OAR manipulation tool called ZOE (for Zonja’s Oar Editor :-)) that implements some very basic forms of filtering, relocation and rotation for OARs as a whole.

An interview by Alyne Dagger (with some final remarks)

At the end of June 2009 I was interviewed by Alyne Dagger for the online magazine Bad Girls Magazine (BG Magazine). The interview was planned around may, and I received a draft questionnaire by email at the end of June. I wrote a first working draft with my replies, and we interchanged some emails until we both were satisfied with the results. Working with Alyne was fantastically easy, and the interview, first appeared on the portuguese edition of BG Magazine, no. 19, and later in the english edition, no. 19, was very nicely presented; big thanks to Alyne for her wonderful, careful work :-) I reproduce the interview here in its entirety, with permission. Text in italics is from BG Magazine, while my replies use a normal font; image subtitles, when present, are also from BG magazine. The images are all mine; some of them where proposed by me, and some of them were chosen by the interviewer.

Most of the material covered in the interview can also be found here and here, but the presentation is different — being in an interview format, the reading is probably more agile. You’ll also find two arguments about why Second Life cannot scale; these were previously published, in a similar form, as comments on other people’s blogs.

When speaking about virtual worlds, two months are like two years in RL. That’s why I include, at the end of the post, a small section with some corrective remarks; these were not part of the original interview.

The interview

HERALD OF DIGITAL FREEDOM

Intense and passionate in every project she’s got herself involved, ZONJA CAPALINI was a mix between muse and investor of the Metaverse when the Openspaces crisis blew up in 2008.

Revolted, she’s began to try to revert the price policy through mobilization and protest, but seeing doesn’t worry with the investors and residents like her, she went to the fight and had searched for solutions for her business in other metaverses, before to begin her own grid, using the Opensim as a tool.

She tells us with exclusivity how was this painful process which can have opened horizons and frontiers for the age of the free metaverses.

BG Magazine: How did you come to SL and what you did in your first SL year?

Zonja Capalini: I was captured, as many other people were, by the hype about Second Life at the end of 2006 and the beginning of 2007. I first created an avatar in december of 2006, but I had some difficulties with it and did never log in. Zonja first rezzed on february the first, 2007. It was a pure adventure. Complete immersion from the first second.

At the time the process was more complicated than it is today, and it was almost not localized. I spent several days at Orientation Island, then Help Island, until I found a way to get to the mainland. I first teleported into a german-speaking infohub (all europeans seemed to be routed to that hub at the moment), and started to socialize.

I was feeling awkward, was very shy, and besides I didn’t control my avie properly, I was crashing against all the walls I found, flying unexpectedly, etc. — as everybody else. I had the impression to have landed in the recovery area of a hospital specialized in brain injuries :-)

One of my main customers is a company dedicated, amongst other things, to education.

I saw that LL was announcing voice in SL, and I thought that creating a campus for that company in SL could be a great way to allow them to have students from all over the world. I talked to the executives of the company and showed them SL, and they agreed that it looked as an interesting platform to evaluate. So that from the beginning my SL experience was dual: on the one hand I was living my own second life, experiencing the universe as a “resident”; on the other hand, I was learning to master SL as a technological platform.

2889: Working with very large linksets in Opensim

[Update 20090721:] Added links to source code and a reference, corrected some typos.

[Update 20090723:] You can download an OAR with the sample bottle and the generator at rexxed.com: http://www.rexxed.com/2009/07/klein-bottle/

[Update 20090725:] OSGrid users: the sample bottle and the generator are now available at my store in OSGrid.

Introduction

Most objects in Opensim and Second Life are made of prims. Indeed, if we exceptuate avies, wearables (i.e., non-attachable clothes and body parts), Linden trees and particles, everything you see is made of prims. Prims are small, often uncomplicated geometrical shapes, like a cube, a sphere or a cylinder. By combining (“linking”) prims, builders create more complex objects like houses, vehicles, shoes, hair, jewellery, etc. When prims are linked, the resulting object remembers the order in which they were linked (which is important, for example when an object is scripted) and some other properties; the set comprised of the linked prims and these additional properties is called a link set or linkset.

Linksets in Second Life have several limitations: two prims can’t be linked if they are too far apart, and an object cannot contain more than 256 prims. This is the cause all kind of problems. For example, if you buy a big house, i.e., a house that has more than 256 prims or that it is very large (and therefore its prims are very far apart), the house can’t be assembled into a single object, because of the above rules. Then the house is normally packaged with a rezzer, a scripted assistant that helps you to position the house; once the house is positioned to your liking, you normally ask the rezzer to freeze the house in place, and then delete the rezzer.

Opensim (when used with an appropriate viewer, like the Hippo viewer) supports large linksets, i.e., objects comprising more than 256 prims. In this post I’ll describe some experiments I made with large and very large linksets. In one of these experiments, I created a single object consisting of 2889 prims. Such extremely large linksets are difficult to manage and create, but once you get them built they work wonderfully.

Grider: Prototypying the next generation Hypergrid

Hypergrid

I’ve already blogged (here and here) about hypergrid, a virtual world architecture that allows the federation of Opensim grids (or “worlds”), and in particular teleports between grids. Using hypergrid, you can teleport, for example, from OSGrid to my grid, the Condensation Land Grid, or between OSGrid and Sciencesim, or between Sciencesim and the Cyberlandia grid, etc. Wagner James Au reported in the New World Notes blog about a demo run by Tom Murphy at the Metaverse University in which hypergrid teleports were shown between grids located in different parts of the US. Hypergrid is developing very fast, and once it’s stabilized and distributed widely, it will implement the vision of a world-wide network of connected grids, where big corporate or educational grids will coexist with small, home-based grids. And thanks to the hypergrid teleport, all these grids will appear as a single world like Second Life is today — hence the name “hypergrid” :-)

Here’s a sample video showing hypergrid teleports in action:

In this article I’ll discuss some details of the current hypergrid architecture, some of the security risks involved, and how a new, more secure hypergrid architecture (sometimes called “safe hypergrid” or “hypergrid 2″) is being developed.

The Openspace fiasco: six months later

The Openspace fiasco

Six months ago, I blogged about the Openspace fiasco and how it made business customers and creative residents equally angry. 

In case you’ve not read the post, let me make a quick summary: I work for a company which we will call “C”; C owned a standard sim C1 and three openspaces C2, C3 and C4; at the same time, I was the owner of the Condensation Land archipielago (the Condensation Land, Condensation North, Condensation Beach., Condensation South and Condensation Southwest sims); all the islands were Openspaces, except Condensation Land. When the unjustifiable increase of 66% in the price of Openspaces was announced, C’s executives asked me to start researching for an alternative to Second Life, because they had lost their confidence in Linden Lab. I was also forced to look elsewhere for my own islands, because most of my tenants were not able to afford the price increase, got fed up, and left Second Life.

In this post I’ll describe the alternatives that were considered, the decisions that were taken, and what we learned in the process. Although the experience has been bitter in many aspects (having to abandon land which you have carefully created and helped to make beautiful is awful; Second Life customer service is a disaster; etc), the final results are quite interesting. I’ve though to share them, in case somebody can find them of interest and profit from our experience.