Zonja Capalini

Splitting Megaregions

One of the most interesting developments in Opensim lately have been megaregions. Megaregions are non-standard “big” regions created by combining several standard, 256×256 meters regions. For example, you can combine four standard regions in a square to get a megaregion of 512×512 m. Megaregions are very convenient because of several reasons: for example the usual lag in region crossings disappears completely (since you are in the “same” region), and the same is true of vehicles. On the other hand, megaregion support is still experimental, and a number of things that work in normal regions don’t work or work only partially in megaregions. For example, you cannot define parcels outside of the SW region (this is due to the way parcel maps work), and therefore you can’t have independent music streams (for example) when x > 256 or y > 256.

Diva Canto’s distribution of Opensim directly supports megaregions. You can also migrate an existing non-megaregion installation to a megaregion configuration, if you follow certain precautions. Migrating from a megaregion configuration to a non-megaregion configuration is also possible — you change the setting that implements megaregions, “CombineContiguousRegions” to false, and it works as expected.

OAR support for megaregions is partial: since technically all the objects in the megaregion are stored in the SW region, when you take an OAR of the SW region you get all the objects of the megaregion but only the terrain for that same region — this means that you cannot fully rebuild a megaregion from an OAR, which is a pity (you will be missing the terrain files for all regions except the SW one).

Splitting megaregions

In this post I’ll describe some experiments I made with an OAR manipulation tool I developped following a suggestion of Stefan Andersson. The tool takes as its input a megaregion OAR and splits it into several non-megaregion OARs, creating one OAR for each region that constitutes the megaregion and has objects in it. This is so because megaregions may have holes, and when a possible region does not have objects there’s simply no information in the megaregion OAR about whether a it’s a region with no objects (for example, a sea-only region) or a hole in the megaregion.

Continue reading

February 13, 2010 Posted by | OpenSim, Tech News, ZOE | , , , | 10 Comments

Partial documentation for SceneObjectGroup serialization in XML2 format

Warning: This is a work-in progress. I’m publishing this article now to see whether it generates interest and which kind of feedback I get. The material published here is incomplete, and contains for sure errors, ommisions and inaccuracies. I plan on extensively modifying it during the following days. Please take that into account when reading it.

The XML2 format

The XML2 format is used in Opensim to serialize objects or linksets (called “SceneObjectGroups” in the Opensim parlance). XML2 dumps of objects can be obtained using the ‘save xml2’ or ‘save prims xml2’ console commands, and are used internally to serialize objects in Opensim Archive (OAR) and Inventory Archive (IAR) files. For example, OARs are gzipped tar files containing a directory called ‘objects’ with one XML2 file for each linkset in the region.

XML2 files are standard XML files using an ASCII encoding and no whitespace. This means that there’s no extra indenting or formatting between tags to make the file more pleasant to the eye. You can use any XML editor (including Internet Explorer) to display an XML2 file with some additional prettyprinting and formatting.

XML2 is practically undocumented, but the fact that it’s undocumented is itself (informally) documented :-), in comments scattered through the Opensim wiki and in the Opensim-dev and Opensim-users lists. Essentially, what this means is that the format is subject to change, but not much change: attributes (i.e., new tags) may be added to the format without breaking it — the developers program the classes taking this possibility into account and storing default values when an attribute is expected but not present (for example, if you’re loading an old OAR file which does not have the ‘AllowedDrop’ attribute into a newer installation which does have this attribute).

In this article I document partially the structure of XML2 files using a variant of the BNF notation. I include the productions for most of the format, interspersing comments wherever I have some idea of the meaning of the tag contents. I’ve obtained this information by manually looking at the XML2 files when developing the simple OAR editor I called ZOE and in recent modifications to that editor. I’ve also taken a look at the source code for Opensim and at the code for libopenmetaverse. There may be errors and there are for sure omissions; any kind of comment, addition, correction or clarification would be greatly appreciated.

The whole SceneObjectGroup/SceneObjectPart model is about to be heavily refactored to be able to include very cool things like hyerarchical objects or meshes, and this will most probably require a new format (XML3?) to serialize Opensim objects. I am writing this article as a way to teach myself, and I’m making it public in case somebody else can find its contents useful. Please keep in mind at all times that this is not official documentation :-) — that’s one of the reasons why this appears as a blog article instead as a page in the Opensim wiki, the other being that I’d prefer to get some feedback before writing such a complicated wiki page, in case it’s considered useful to have it.

Continue reading

January 25, 2010 Posted by | OpenSim, Tech News | , | 11 Comments

Using SQL to manipulate OARs

Introduction

As I noted in my previous post “Moving and rotating OARs“, offline editing of Opensim Archives (OARs) opens a lot of interesting possibilities. In this article I present the results of several simple tests I have made by extending my “Zoe” OAR editor so that it supports (a very simplified sub-dialect of) SQL. Even in a simplified form, SQL has a lot of expressive power and allows to perform a number of pretty neat operation on OARs, when it comes to selecting and deleting parts of a region. Updating OARs, though, is much more complicated, because there is no “natural” semantics to apply when using the standard SQL UPDATE statement. Some simple cases can be singled out, however, where “natural” semantics actually make sense — and in some other “interesting” use cases, a custom pseudo-SQL statement seems to be sufficient. Continue reading

January 1, 2010 Posted by | OpenSim, Tech News, ZOE | , , , , | 3 Comments

Reverse-engineering OARs: Understanding parcel maps

Format 0.2 Opensim Archives (OARs) include, since Opensim version 3c271b, a “landdata” directory that stores parcel data. For each parcel in the region there is a file in XML format in the directory. Amost all objects in Opensim are associated with a unique uuid that identifies them, and parcels are not an exception to that rule. Files in the “landdata” directory use the uuid of the parcel as their filename and have a “.xml” extension.

What’s inside each one of these .xml files? Let’s take a peek at a sample one:

    <?xml version="1.0" encoding="utf-16"?>
    <LandData>
      <Area>6256</Area>
      <AuctionID>0</AuctionID>
      <AuthBuyerID>00000000-0000-0000-0000-000000000000</AuthBuyerID>
      <Category>-1</Category>
      <ClaimDate>1255170099</ClaimDate>
      <ClaimPrice>0</ClaimPrice>
      <GlobalID>bfbf7bd4-7d7b-4a35-aba0-c352201cddcf</GlobalID>
      <GroupID>00000000-0000-0000-0000-000000000000</GroupID>
      <IsGroupOwned>False</IsGroupOwned>
      <Bitmap>AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
       AAAAP//fwAAAAAA//9/AAAAAAD//38AAAAAAP//fwAAAAAA//9/AAAAAAD//38AAAAAAP//f
       wAAAAAA//9/AAAAAAD//38AAAAAAP//fwAAAAAA//9/AAAAAAD//38AAAAAAP//fwAAAAAA/
       /9/AAAAAAD//38AAAAAAP//fwAAAAAA//9/AAAAAAA=</Bitmap>
      <Description />
      <Flags>505454667</Flags>
      <LandingType>0</LandingType>
      <Name>Arrabal Tango Club</Name>
      <Status>0</Status>
      <LocalID>4</LocalID>
      <MediaAutoScale>0</MediaAutoScale>
      <MediaID>00000000-0000-0000-0000-000000000000</MediaID>
      <MediaURL />
      <MusicURL>http://xx.xx.xx.xx:8000</MusicURL>
      <OwnerID>69af4428-cec0-48d8-a0d9-c05268facca5</OwnerID>
      <ParcelAccessList />
      <PassHours>0</PassHours>
      <PassPrice>0</PassPrice>
      <SalePrice>0</SalePrice>
      <SnapshotID>00000000-0000-0000-0000-000000000000</SnapshotID>
      <UserLocation>&lt;0, 0, 0&gt;</UserLocation>
      <UserLookAt>&lt;0, 0, 0&gt;</UserLookAt>
      <Dwell>0</Dwell>
      <OtherCleanTime>0</OtherCleanTime>
    </LandData>

That’s the xml file for Arrabal Tango Club in Condensation Land — I’ve only obscured the music URL and wrapped the part between the <Bitmap> and </Bitmap> tags, which originally used only one line, with no intervening blanks. This string encodes the shape and position of the parcel in the region. How do we know that? Well, the parcel map has to be stored somewhere, and it has to be quite large, so the only candidate is the “Bitmap” string :-) Most probably this long string is a text encoding of binary data.

Indeed this is the case: the string is using the Base64 encoding (you can find that quickly by browsing the Wikipedia). After decoding the string, we get a binary chunk of exactly 512 bytes. I checked that all the terrain files have strings of exactly the same length, so this number had to mean something. Continue reading

December 18, 2009 Posted by | OpenSim, Tech News, Tutorials, ZOE | , , , , , | 6 Comments

ZOE version 0.1 released

Zoe

Zoe (Zonja’s OAR Editor) is a simple offline OAR manipulation tool that I wrote recently. You can find a detailed description of its functionality in my recent post “Moving and rotating OARs”. In this post I’ll describe how to install and use Zoe. I’ve designed Zoe with simplicity in mind, so that currently it only implements some very basic operations — by combining these operations, however, a number of pretty cool things can already be done. I’d love to get some feedback about the functionality of Zoe — at the end of the post I’ll list some of the things that I want to add to it, but I’m open to suggestions.

ZOE version 0.1 released
Continue reading

December 16, 2009 Posted by | OpenSim, Tech News, ZOE | , , | 15 Comments

Moving and rotating OARs [UPDATED]

[Update 20091216: ZOE v 0.1 has been released. You’ll find a manual and the download link here.]

…and some filtering too

Opensim Archives (OARs) are nowadays the standard tool to share and interchange full Opensim regions. Justin Clark-Casey, the core Opensim developer in charge of OARs, has lately added merge functionality to the loading of OARs. This means that you can merge (i.e., add) the contents of an OAR file to an existing region, without deleting its contents. This opens a plethora of interesting possibilities, but it also is the source of a lot of new problems. What if I’m very interested in merging an OAR with one of my existing regions, but I’d prefer the new contents to be placed differently from their original position? For example, I can be given an airboxed level at 1000 meters, but I might want to put it at 2000 m (for example, because I already have contents at 1000 m). Or I might be interested in an OAR with (say) some nice palace and gardens, but it’s located in the NE corner of the sim and I’d like to install it in the SW part of the sim.

At present, the “load oar” command does not allow for this kind of transformations. Indeed, it’s debatable whether Opensim should include code for this kind of application at all, or whether such manipulations should better be handled offline.

0705 - ZOE- A test run

In this article I present a very simple (and highly experimental) OAR manipulation tool called ZOE (for Zonja’s Oar Editor :-)) that implements some very basic forms of filtering, relocation and rotation for OARs as a whole.

Continue reading

December 14, 2009 Posted by | OpenSim, Tech News, ZOE | , , , , , | 23 Comments

2889: Working with very large linksets in Opensim

[Update 20090721:] Added links to source code and a reference, corrected some typos.

[Update 20090723:] You can download an OAR with the sample bottle and the generator at rexxed.com: http://www.rexxed.com/2009/07/klein-bottle/

[Update 20090725:] OSGrid users: the sample bottle and the generator are now available at my store in OSGrid.

Introduction

Most objects in Opensim and Second Life are made of prims. Indeed, if we exceptuate avies, wearables (i.e., non-attachable clothes and body parts), Linden trees and particles, everything you see is made of prims. Prims are small, often uncomplicated geometrical shapes, like a cube, a sphere or a cylinder. By combining (“linking”) prims, builders create more complex objects like houses, vehicles, shoes, hair, jewellery, etc. When prims are linked, the resulting object remembers the order in which they were linked (which is important, for example when an object is scripted) and some other properties; the set comprised of the linked prims and these additional properties is called a link set or linkset.

Linksets in Second Life have several limitations: two prims can’t be linked if they are too far apart, and an object cannot contain more than 256 prims. This is the cause all kind of problems. For example, if you buy a big house, i.e., a house that has more than 256 prims or that it is very large (and therefore its prims are very far apart), the house can’t be assembled into a single object, because of the above rules. Then the house is normally packaged with a rezzer, a scripted assistant that helps you to position the house; once the house is positioned to your liking, you normally ask the rezzer to freeze the house in place, and then delete the rezzer.

0628 - A Klein bottle of 2889 prims (2)

Opensim (when used with an appropriate viewer, like the Hippo viewer) supports large linksets, i.e., objects comprising more than 256 prims. In this post I’ll describe some experiments I made with large and very large linksets. In one of these experiments, I created a single object consisting of 2889 prims. Such extremely large linksets are difficult to manage and create, but once you get them built they work wonderfully.

Continue reading

July 20, 2009 Posted by | OpenSim, Second Life, SL Exodus, Tech News | , , , , , , | 16 Comments

Grider: Prototypying the next generation Hypergrid

Hypergrid

I’ve already blogged (here and here) about hypergrid, a virtual world architecture that allows the federation of Opensim grids (or “worlds”), and in particular teleports between grids. Using hypergrid, you can teleport, for example, from OSGrid to my grid, the Condensation Land Grid, or between OSGrid and Sciencesim, or between Sciencesim and the Cyberlandia grid, etc. Wagner James Au reported in the New World Notes blog about a demo run by Tom Murphy at the Metaverse University in which hypergrid teleports were shown between grids located in different parts of the US. Hypergrid is developing very fast, and once it’s stabilized and distributed widely, it will implement the vision of a world-wide network of connected grids, where big corporate or educational grids will coexist with small, home-based grids. And thanks to the hypergrid teleport, all these grids will appear as a single world like Second Life is today — hence the name “hypergrid” :-)

Here’s a sample video showing hypergrid teleports in action:

In this article I’ll discuss some details of the current hypergrid architecture, some of the security risks involved, and how a new, more secure hypergrid architecture (sometimes called “safe hypergrid” or “hypergrid 2”) is being developed.

Continue reading

June 10, 2009 Posted by | OpenSim, SL Exodus, Tech News | , , | 12 Comments

Dynamic hypergrid links: the new Metaverse

Would you like to have your own SL-like micro-world in-house, to be able to make backup copies of your islands and your inventory at will, to be able to clone your avatar with all your inventory into your friend’s micro-world when needed, to be able to pack a whole island and share it with friends or with the world, to be able to use fantastic buildings and landscapes others have created, to be able to add an island instantly, to decide how much computer power you’re going to allocate to your regions, to be able to run 10+ islands with 20+ avatars in a single computer, …?

“Sure,” you’ll say, “all these things are very nice, but this has to come at a price, right? Everybody will be isolated in their own micro-world, and what’s the fun of that?”

Up to recently, that was it. Either you were in a “big” world (Second Life, Open Life, OSGrid, etc), or you had your own micro-world, and then you were isolated in it. This started to change when Christa “Diva” Lopes, from the OpenSim dev team, implemented the first phase of the Hypergrid concept, what we can now call static hypergrid. I blogged about it one month ago; here’s an older article from Vint Falken; of course it all started with Justin Clark-Casey‘s seminal post “Could there be a future without big grids?”.

With static hypergrid, grid owners decide whether their grids will be accesible from outside (i.e., if they’ll run their grid in hypergrid mode) and if so they inform other grids; other grid’ operators then use a simple command to link their grid to the first. This architecture gets complemented with the possibility to use prebuilt .xml files to link several (potentially thousands) of new regions at once with a single command.

That’s a great architecture, but it puts the power of jumping from one grid to another into the grid owner’s hands. And it’s good that it is so, because the foreign regions appear on the origin grid’s map, and if I were a grid operator I’d love to be able to control what appears on my map. However, it has several disadvantages for the end-user: it precludes the possibility to jump to an arbitrary grid; if grid A links to grid B and grid B links to grid C but grid A does not link to grid C, you have to jump to B to get to C, which is preposterous; in general, the major problem is that you’re limited in where you can teleport to by your grid owners. Of course SL works that way, but SL is not an open grid, at least for the moment. And if OpenSim has to become the 3D web, you have to be able to teleport, to jump, anywhere. The very same notion of your ISP determining which web pages you can visit and which you can’t visit is against the fundamental idea of the internet.

Ideally, what I’d love to be able to do would be something similar to the following: assume that I’m working in my soon-to-be-opened OpenSim Condensation grid, I get tired of building, and I decide to teleport to my friend’s Ludmilla grid (note: to another grid, not to an island inside my grid) to see whether she’s available to go exploring, so that I tp to her world, and, lo!, there she is; I greet her, and we both tp to OSGrid (the free OpenSim grid) and start exploring from there… Impossible? No! Here’s a video that demonstrates exactly that:

  Continue reading

February 8, 2009 Posted by | OpenSim, SL Exodus, Tech News | , , | 7 Comments