Tuesday 25 September 2012

Squinchpix’s experience converting to Pleiades-compliant names


The process of using Pleiades names consists of getting access to the specific name IDs.  Pleiades provides a site that will return these IDs here: http://pleiades.stoa.org/places/

On that page the user can type in the required name and retrieve the ID in the resulting URI.
SquinchPix actually maintained no location information for the pictures in its DB.  The place names are embedded in the captions, of course, and in the tag or keyword tables.  But the tag ‘Rome’ is not treated any differently from the tag for, e.g., ‘concrete’ or any other tag.  As a result there is no easy way to specifically pick out place name tags in an automated fashion.  What SquinchPix has done all along is maintain a pretty accurate lat/long pair for every picture.  It’s the lat/long pairs that drive location services on SquinchPix such as the Google map that gets generated dynamically for every image. 

In order to participate in the Pelagios project SquinchPix decided to make two changes to the DB.  In the table which contains information for each picture (‘PI’) a field was added for an unambiguous modern name for the location of the picture. 

Then came the work of actually rooting out the place names from the keyword table and associating the right place name with the right picture.  We wrote a script that looked for all the pictures that were keyworded ‘Rome’.  Those that were keyworded ‘Rome’ had the word ‘Rome’ entered in the new dedicated place name field by the script.  The script just dumped out the captions for those which were NOT keyworded ‘Rome’.    Then we inspected those captions looking for more place names.  Next came ‘Athens’, then ‘Mycenae’, ‘Naples’, ‘Tiryns’ and the rest.  For each new place name the script labeled that many more pictures and forced out fewer and fewer captions.  From 20,000 pictures without place names we used iteration to reduce that number to about 300 after two days of work.  By the end of that time each locatable picture had a specific place name associated with it.  The remainder were almost all pictures of artifacts with no secure find spot.  That remainder could probably be identified with some larger Pleiades-compliant name such as ‘Syria’, or ‘Mediterranean’ but that work is for another stage.

The second big change to the DB  was the creation of a separate table that used that same modern place name established in step 1 as an index to a set of doubles.  The doubles were simply the corresponding Pleiades-compliant name and the Pleiades ID.  This table was populated by hand, entry by entry.  On SquinchPix there are about 170 distinct and unambiguous place names so that there are that many records in this new table.  In addition to using the Pleiades look-up facility we made use of the .kml which we ran in Google Earth in parallel.  If we couldn’t find the place in Google Earth then we used the look-up facility.  Even though dealing with a much smaller number of records this hand-population took about four days.

Once that table was populated we had a secure way of going from the specific picture to its modern place name and then to the Pleiades-compliant name/ID pair. Now we simply wrote a script that would traverse all the pictures, get the Pleiades-compliant name and number and use it to write out the Turtle-compliant record. In this way (the extra table, that is) we could confine the fluctuating nature of the Pleiades project to a ‘localized’ corner of the DB. We anticipate that this table in our DB will change and will be maintained and updated on an ongoing basis. The reason for this is that Pleiades is dynamic and also our ideas about specific places and names may not mesh cleanly with theirs in all instances thus necessitating the occasional negotiation. To their credit they are very responsive to questions and suggestions about place names. I would urge anyone engaged in a conversion project to communicate with them whenever better ideas about place names or locations should surface.

Thursday 20 September 2012

Geographical information retrieval - finishing touches

In my last post back in July I wrote about the development of a set of APIs and an interface for geographically querying historical places and their annotations, which allows users to browse a long lost territory and retrieve information about historical artefacts. However, even an application as simple as a map visualisation wouldn't be possible, if services, data, and tools weren't made available to the community by a number of different parties. Naming all those upon whose work I have built is no easy thing. But the following have been especially helpful:
Even this short list gives a measure of the collaborative nature of research and development in this area. I owe a debt of gratitude to all of those who have provided the above resources and obviously to all of those who enabled them to do so. Below I go into a bit more detail about my finishing touches to the interface for retrieving geographical information about the ancient world. But, for those of you impatient to see the result, you can go straight to the heat map by clicking here.

Correction of box annotations

A few corrections have been made on the annotation API in order to return only annotations for which the actual geographical context was a point. It was in fact made known to me, thanks to Leif Isaksen, that in many maps the heat spots were strangely clustered at the intersection of nodes in a grid.    

Grid effect on heat spots
The effect is quite clear when visiting a province like XI, in Italy, where the spots are seen clearly positioned in an organised grid. Not only. It seems in fact that for regions like these, the majority of the annotations are grouped in this way. Clearly, the presence of numerous annotations like these undermine the purpose of having a heat map in the first place since the information about the original place associated to an annotation is lost and the contribution from the precise annotations is somewhat shadowed.

For this reasons all annotations whose geographical context is not a point has been ruled out as contributors for the final heath map, obtaining as a result a more informative map where hot spots are grouped around historical settlements like in the figure below.

Heat map without box annotations

Integration of Historical tile sets

An interesting addition to the interface is the adoption of a particular tile set for historical regions developed by Johan Åhlfeldt in his project Regnum Francorum Online (a description of his work can be read in this blog here). The tile set allows users to provide a background for historical maps which includes names of ancients settlements and depicts also well known roman roads (like the Appian way) alongside known mines and sanctuaries.

Seeing the actual historical landscape with the original names and connections among settlements can only increase the allure of exploring archaeological artifacts and in fact provides the best context in which to put what can be accessed via the different APIs from the Pelagios data galaxy.

This work has been supported by Pelagios and I'd like to thank Leif Isaksen, Elton Barker, Rainer Simon and Johan Åhlfeldt for sharing their ideas, support and resources.

Gianluca Correndo
Research fellow WAIS group Electronic and Computer Science University of Southampton
   

Wednesday 19 September 2012

A digital map of the Roman Empire

Background

The Barrington Atlas of the Greek and Roman World was published in 2000 as part of an international effort to create a comprehensive map and a directory of all ancient places mentioned in sources and a selection of important archaeological sites. Since then two digitization efforts based on the Barrington Atlas have come into being, Pleiades, which started off as a historical gazetteer,and the DARMC project, which is a layered historical atlas. In 2010-2011, as part of a common project, the geodata of DARMC was transferred to Pleiades, though, unfortunately, not all the places in the original Barrington directory could be matched between DARMC and Pleiades, resulting in many places without precise coordinates and feature data. Nonetheless, ever since, the Pleiades gazetteer has had the ability to display most ancient places on a map, individually and with their immediate surroundings, using Google Maps API and Google Maps as background layer. In March 2012 the Ancient World Mapping Center launched a first version of an online GIS application called Antiquity À-la-carte, covering the entire Greco-Roman World. This application is also based on the Barrington Atlas, on geodata from Pleiades/DARMC, and its own digitization efforts (roads, aqueducts, ancient coastlines).

Yet, while the DARMC and Antiquity à la carte initiatives provide geographical coverage and exiting possibilites to compose custom maps in layers, until now there has been no digital map that can be used as background layer for use in a fashion similar to modern mapping applications like Google Maps. Thanks to Pelagios, this is work that I have undertaken, with a view to aiding any archaeological or historical research interested in or using online mapping. We are releasing the map with a CC-BY license, allowing anyone not only to browse and consult it but also to use it for representing their own data or to build on it their own applications, provided that they include a proper scholarly attribution. What is more, the map can be used with OpenLayers, Google and Bing maps, so that anybody, who already has these systems in place, can easily swap out the map tiles for these historical ones.

To see the basic background map (using Google Maps API), click here (default setting is Rome, zoom level 7 of 11). For information about the making of the map, sources of geodata, and a legend to the symbols, click here. For those of you interested in finding out about how the map came into being, keep reading!

Aim
The aim of my work with Pelagios has been to create a static (non-layered) map of the ancient places in the Pleiades dataset with the capacity to serve as a background layer to online mapping applications of the Ancient World. Because it is based on ancient settlements and uses ancient placenames, our map presents a visualisation more tailored to archaeological and historical research, for which modern mapping interfaces, such as Google Maps, are hardly appropriate; it even includes non-settlement data such as the Roman roads network, some aqueducts and defence walls (limes, city walls). Thus, for example, the tiles can be used as a background layer to display the occurrence of find-spots, archaeological sites, etc., thereby creating new opportunities to put data of these kinds in their historical context.

The ancient places and their names have been rendered on a topographical map created from elevation data, originally from the Shuttle Radar Topography Mission (SRTM) project at NASA. The map itself is created as a tiled mapset in the Spherical Mercator projection (EPSG:3857), used by most webmapping services. It is compatible with Google and Bing street and satellite maps, OpenStreetMap, and can easilly be implemented with a javascript application programming interface (API), such as Google Maps and OpenLayers API. Work has taken two different forms: 1) preparation and improvement of the data; and 2) the rendering of the digital map. I elaborate on each item below.
Rome and Central Latium at zoom 9, click to display the full image.

Limitations
In a departure from the original Barrington Atlas and the Pleiades dataset, our digital map does not try to implement time periods when places are attested, nor does it speculate on the certainty (or otherwise) of locations: only precise locations from the Pleiades dataset can be rendered on the map. Nevertheless, since many places lacked precise coordinates and/or feature data, a good deal of effort has been made to improve the data. For sake of clarity, we have displayed only one of an ancient place's possible names, based on its primacy and importance in the Barrington Atlas. The digital map is presented at eight different zoom levels (3-10: zoom 10 corresponding to a scale at approx. 1:500,000), and one additional zoom level (zoom 11, which corresponds to a scale at 1:300,000) for maps of Central and Southern Italy, Northern Tunisia, Greece, Turkey, Syria, Lebanon, Israel, Palestine, Egypt and Jordan. Due to the lack of precise coordinates in the original dataset for regions, cultural (tribes, people) and natural features (mountains). thus far only places have a complete rendering on the map, while major rivers and lakes have been labelled with Latin names.
These considerations have been made to keep the map simple and easy to understand, which is a necessity for online publications, especially for interactive maps that users may want to click on, pan and zoom.

Licence
The transmission of the tiles to any web-mapping application is permitted under a Creative-Commons 3.0 (CC BY-SA) licence. Attribution to the Digital Atlas of the Roman Empire (DARE) project at http://dare.ht.lu.se is required and linking to this blogpost is encouraged. See below for implementation instructions.


Making of the map
There is a short description how the map was made at a separate page. Here you will also find the legend of the map and a listing of datasources for the geodata of the map. The map can be viewed in fullscreen mode at this page.


Preparation and improvement of the dataset

Georeferencing the Barrington Atlas

The Barrington Atlas used maps at three different scales. The central Mediterranean provinces of the Empire were rendered at 1:500,000, which corresponds to zoom level 10 on the digital map (which is 1:545,979 to be exact), peripheral provinces like Britannia, Germania, Belgica and Gallia were rendered at 1:1,000,000 (zoom level 9). There were also special maps of Rome, Carthage, Athens, Constantinople and their surroundings at 1:150,000 scale (zoom level 12). The accuracy of georeferencing the 1:1,000,000 scale Barrington maps, given that the places are located correctly on the printed map, is around 3 kilometers, which is illustrated in the online example 1, where georeferencing using different sources of information is described.
Example 1: Georeferencing the sanctuary Sanxay, dep. Vienne, France using different sources.

Over the past few years, the conditions for successful georeferencing of ancient places has increased. Nowadays there is better satellite imagery; national mapping agencies offer APIs to quality maps (e.g. Géoportail, IGN, France and OpenSpace, Ordnance Survey, Great Britain); georeferencing services have better accuracy, with the result that e.g. Google Maps Geocoding Service can return places that aren't even visible on their own maps (e.g. small hamlets); main archaeological monuments throughout Europe and the Middle East have their own articles on Wikipedia, often with highly accurate coordinates; national heritage agencies offer online database services, etc.

Additional data from Barrington Atlas

From the beginning it was clear to me that, since we were creating a map, we would require information that was not included in the original Pleiades dataset, and also that it was necessary to render ancient and modern placenames in a different way. One such example was the relative importance of a place. In the Barrington Atlas, places were represented in a hierarchy of importance through the use of font and its size, so that, for instance, those places considered of greatest significance - capitals, settlements with the legal status of colonia and municipia, legionary bases, important sanctuaries and mines - were rendered using capital letters. On our map, we let those places appear first, so that users can immediately locate themselves in the landscape and grasp at a glance those places which have greatest prominence in our sources.
The Barrington Atlas and the use of font and font-size to indicate relative importance among places. Minturnae (Minturno, Italy) is the most important place on this map, allthough all placenames with capital letters (Sinuessa, Suessa Aurunca and Interamna Lirenas) are assigned to the category of major settlements on the digital map.
After careful and repeated study of the Barrington Atlas, I found a total of 1488 places indicated as important on the map by its font and size (capital letters), mainly settlements, but also legionary fortresses, rural sanctuaries (in fact there are three of them, all Greek: Olympia, Nemea and Isthmia), to a residence, the villa of emperor Hadrianus, near Tivoli (outside Rome). In our dataset, these places have been assigned a major/important property allowing them to appear on the map before other places (i.e. at lower zoom levels) and depicted with bigger symbols and font. Even given the time restriction of the project, we decided that it was important to get the features and coordinates of these places as accurate as possible. Some belonged to those cases where Pleiades and DARMC had trouble aligning their datasets, especially those places sharing the same ancient name. For instance, only one of the two important places in Britain names Isca had precise coordinates in the Pleiades dataset (that named Isca = Caerleon, Exeter), and only one of three of the cities (that named Venta = Caerwent, Winchester, Caistor-by-Norwich).

Implementation of the map
I have prepared two simple examples of the implementation of the background map, one using the Openlayers API, example 2, and the other using Google Maps API, example 3. (Feel free to study the HTML code, which contains all the Javascript. The path to the directory on the Pelagios server where the tiles are stored is located at
http://pelagios.org/tilesets/imperium/{z}/{x}/{y}.png
Implementation in Google Maps uses the ImageMapType class, which is used to define a custom map type with the same behaviour as the default map types, whereas implementation in OpenLayers uses the Layer.XYZ class. The path must be specified in the JavaScript code, see the documentation for details.
There are several ways to get your data on the map, though it is beyond the scope of this article to go into the detail. The simplest way is just to place a marker at a specific location on the map: with its political and cultural infrastructure, our presents offers a favourable comparison to modern streetmaps in the display of information. Another popular way is to load a Keyhole Markup Language (KML) file into your webmapping application. This is demonstrated in example 4 (OpenLayers) and example 5 (Google Maps), where a KML-file containing the Roman milestones in Western Continental Europe is loaded.

I have set up a fully interactive implementation of our digital map at  dare.ht.lu.se. The web application uses the OpenLayers API. There is a two way interaction between the map and a backend SQL database, containing data about the ancient places. Information about the places can be retrieved in three different ways:
  1. directly, by clicking the symbols on the map;
  2. listing the places currently displayed on the map;
  3. by entering a search expression of an ancient or modern name or part of a name.
Results of your query appear in the right sidepane. It is also possible to display a detailed view of the site using Bing Satellite Maps or OpenStreetmap.

Conclusion
This is the first version of a digital map inspired by the majestic Barrington Atlas. Our map cannot really do it justice, first because of the complex nature of the original, second for the reason that much work remains to be done, with respect to both developing digital mapping techniques and to improving the quality of the feature data. In another way, however, this map goes beyond the scope of the Barrington Atlas, because - as a digital product - it can be expanded to much higher zoom levels and be connected interactively with users, other maps and georeferenced data of any kind (plans, drawings, photos, etc.)

For the future I envisage more places being included in the database: a digital map has no restriction in space, whereas the Barrington Atlas was constrained by the physical size of its pages (which even then were big, as any owner of the Barrington knows!). In this version I have stopped at zoom 11: however the freely usable OpenStreetMap continues to zoom 18 and already contains highly accurate data about the structure and shape of many archaeological sites. In addition, the OpenCycleMap, among others, has shown that elevation data from the SRTM dataset can be rendered at zoom level 14 with some interpolation. Finally, national agencies (e.g. IGN, France) are continuing to release land-use data that are being incorporated in the OSM dataset. Therefore I see a growing archaeological map of the Ancient World, not only a historical map.

Meanwhile the author is looking forward to hearing about your experience of this map, so that together we can improve our understanding of ancient world geography using twenty-first century technologies.
This work has been supported by the Pelagios project, and I would like to thank Leif Isaksen, Elton Barker and Sean Gillies (Pleiades), who provided ideas, support and feedback.
Johan Åhlfeldt
Digital Atlas of the Roman Empire (DARE)
Department of Archaeology and Ancient History, Lund University, Sweden

Monday 17 September 2012

SquinchPix joins Pelagios


When in Rome a few years ago, we came across the Santa Costanza, a beautiful 4th century AD church with the most amazing mosaics. But, when we tried to find pictures of them on the Internet afterwards, we drew a blank. Then and there we resolved to photograph the mosaics in their entirety and put the results on line for other people to enjoy. Thus SquinchPix.com was born.

SquinchPix.com is our archival site for photographs of historical cultural artefacts in Europe, including archaeological sites, buildings, artworks—basically anything that has historical interest and looks great! We aren’t affiliated with any institution or school: but, the more images we have captured, the more we have become convinced that the salient characteristics of digital photography—they’re easy (to take and upload), free and resilient—along with the potential for the Internet to bring these pictures to a mass audience, must be leveraged to assist researchers and students.

The real transformative potential of digital photography has yet to be fully realised. For example, whereas previously a single photograph of the Temple of Hephaestus in Athens was all that the book publisher’s budget would allow, it's now possible to photograph every capital, every sculpture, every cut in the marble, and put it all online for no increase in cost (for no cost at all, really). We also understand that the researcher needs clarity, so we exercise significant post-processing on the photos to make enhance their quality. The result is that often even very prosaic objects, such as column bases, can reveal a new and unexpected beauty. Of course, we can always do better: but, as the technology improves, so do our pictures.


Protome of a griffin.  Bronze cauldron attachment.  Olympia, Greece.  After 650 BC.  


Yet, despite this breakthrough in technology, there are many internet sites aimed at researchers where the proprietors still think like book publishers: what large-scale inclusion requires is a corresponding increase in the sophistication of the user interface. On SquinchPix every photograph is extensively tagged (keyworded). Currently our database holds a third of a million keywords for about 21,000 photos, which enables powerful searching across and aggregation of the records. In addition, every search term can be displayed as a tag- or keyword-map that shows what other terms on the site are associated with your search term and with what frequency: a click of the button brings up the corresponding image subsets. So, from finding one picture to relating a picture to many others—that has been SquinchPix’s journey.


Terracotta vessels arranged by century on SquinchPix.

It is for this reason above all that we are delighted to join Pelagios. Now our photographs can be related to many other kinds of data­—including archaeological data about them or text documents that refer to the place where they’re found—which will help provide fascinating layers of context for our images. We hope too that our photos may prove useful to be read alongside textual descriptions and archaeological records, thereby providing another dimension to the data already part of the Pelagios cloud. Our first drop of 1500 pictures taken in and around Athens, Greece will soon be available, so watch this space!

Just a short note on usage: researchers, lecturers, teachers and students may use our pictures at no cost. We only request to be credited with some such line as ‘Courtesy of SquinchPix.com’ (every photograph has the photographer’s name on it or beneath it). Those who wish to use the pictures for commercial purposes should inquire. In addition we always like to know how our pictures are being used, so please do get in touch! Our e-mail is bob@squinchpix.com.

You can read more about us here:
http://kbender.blogspot.com/2012/02/photo-archives-old-and-new.htm 

http://www.wanderingeducators.com/artisans/lives-artists/exploring-classical-world-photographer-robert-consoli.html

Robert H. Consoli, Susan Hynnes.
SquinchPix.com