Thursday 27 October 2011

InUse - ISWC 2011

Lessons learnt from chairing ISWC InUse track 2011

Recently I had the pleasure of organising ISWC 2011 In-Use track, which I co-chairs with Jamie Taylor from Google.

I've been taking some notes about the chairing process of this track, in the aim of passing them on to next year's chairs. Then I thought it might be useful to share these notes more openly on this blog so any future chair could make use of.

Abstract submissions or not?
One of the early decisions we made for this track, and inline with previous InUse tracks, was not to ask for abstract submissions, but rather to shoot for full paper submissions straightaway. Abstract submissions are useful for gauging number of submissions and to manage review assignments. However, number of submissions for ISWC doesn't fluctuate much, and remedies for lower or higher numbers of submissions are quite straightforward. On the other hand, assigning reviews based on abstract submissions is too problematic. Reviewers will be placing their bids based on abstracts, many of which will never turn into full paper submissions. About a third of abstract submissions in Research Track this year didn't turn into full papers! This normally messes up most reviewers' bids, which will wipe out any benefit from this abstract submissions step.

Submission address
Submissions to Research and to In Use tracks were made though completely separate EasyChair installations. However, a few people got confused with which submission address to use for which track. Future chairs of these tracks may want to consider having a multitrack EasyChair installation (ie single submission address) which will give authors options for submitted to the various tracks within the same EasyChair address.

Reviewing period
Selecting the right PC is obviously crucial. Because of the general deadlines of ISWC, most of the reviews had to be done in July, which is a common holiday month for many. So when you invite reviewers, make sure they know when the reviews will need to be done, so hopefully they can check their availability before accepting your invitation.

Logging review delegations
One thing some reviewers did was delegating their reviews to others without logging this in EasyChair. A much better way to delegate reviews is to send such requests through EasyChair, so that track chairs can see who's handling the reviews, which is especially important when the main reviewer is unreachable. Also make sure you remind PC members that they must read the delegated reviews and agree to their content and recommendations for acceptance or rejection.

Discussing reviews
As chairs we obviously needed to read every single review to make acceptance/rejection decisions. For many papers the reviewers did not agree on whether they should be accepted or not, and so a discussion between the reviewers was necessary. However, because of the various conference deadlines such discussions had to be done in August, when most PC members were away on holidays! Might help to send the reviewers an early reminder to be available for discussing, and editing or defending their recommendations. My main recommendation is to start asking reviewers to discuss their opinions as soon as the reviews are available, and not to delay this until all the reviews are in or until the reviewing deadline expires.

Number of reviews
As in previous years, we aimed for 3 reviews per submission which should be the minimum. There were a few cases were we had to bring in a fourth reviewer, or to add a meta review, to verify and/or sway the decision more firmly towards accept or reject. Some reviewers are bettermans more reliable than others, and it's always useful to ask previous chairs for their experience before selecting your PC candidates.

Metadata Generation
Metadata export in EasyChair leaves a lot to be desired! To get the metadata files from EasyChairs, you need to apply for a premium account. Note that the metadata file you will eventually get contains EVERYTHING! Including names of reviewers, review texts, reviewer decisions, comments, for rejected as well as for accepted papers. You must delete such info before passing the files on to other chairs. Best way to do this is with the Excel metadata files and not with the XML metadata ones. With Excel, just delete the sheets, columns, and rows that contain information that should be kept private. Takes 2 minutes!

Also note that these metadata files will contains information on submitted papers and not for final proceedings, so you need to check the data in these files and update them to match the proceedings version which could have different titles and author list or order of authorship. Failing to do so will result in inaccurate information on the data.semanticweb.org server.

Also make sure that paper listings on the conference website are also updated when the camera ready versions are uploaded to reflect any changes. Make sure you double check everything that goes on the site, program, list of accepted papers, session dates and times. Errors can easily crop up at every step.

Best paper nominations
Always tricky. Jamie and I read most of the papers submitted to our track, so we picked up a few papers along the way that we thought deserved to be nominated for a best paper award. That list of course was too long, so there was a second round of discussions between ourselves on each of the papers, and so we managed to shrink the list to 4 nominations. We did not rely on the reviewers scores when making our selection, since the range of scores vary a lot from one review to another, and some of them are not as reliable as others. Also, we didn't ask reviewers to recommend best papers, since each reviewer only read just a few of the submissions. For this process to be fair, chair (with help from vice chairs if any) must read and debate every potential paper.

We also decided to make public the rational behind each of our nominated papers. When the nominations were announced again before giving the award, we added keywords explaining why each of the nominations were made. These were also explained by Jamie when he presenting the slide and award. I hope that future chairs will continue to be transparent about the reasons behind their nominations.

Our InUse best paper nominees, and reasons for nominated the, were:

How to "Make a Bridge to the New Town" using OntoAccess
Matthias Hert, Giacomo Ghezzi, Michael Würsch and Harald Gall.
(important problem, novel solution)

Leveraging Community-built Knowledge for Type Coercion in Question Answering
Aditya Kalyanpur, J. William Murdock, James Fan and Christopher Welty.
(Tricky problem, novel solution, mature work, proven to work) (WINNER)

Mind Your Metadata: Exploiting Semantics for Configuration, Adaptation, and Provenance in Scientific Workflows
Yolanda Gil, Pedro Szekely, Sandra Villamizar, Thomas Harmon, Varun Ratnakar, Shubham Gupta, Maria Muslea, Fabio Silva and Craig Knoblock.
(Real world problem, focused solution, adaptable framework)

Using Semantic Web Technologies to Build a Community-driven Knowledge Curation Platform for the Skeletal Dysplasia Domain
Tudor Groza, Andreas Zankl, Yuan-Fang Li and Jane Hunter.
(Usability focused, requirements gathering, pragmatic solutions)







Sunday 6 June 2010

Live Social Semantics at ESWC2010



Live Social Semantics goes live once again. Following the successful runs at ESWC 2009 and HyperText 2009, the Live Social Semantics team was invited to relaunch the application at ESWC 2010, Crete. The application was very well received and generated a lot of attention and interest at the conference, where more than 130 people registered on our site to use the service. Credit goes to Wouter van den Broeck, Martin Szomszor, and Ciro Cattuto, for working so hard to get the application up and running on time for the conference.

Tuesday 20 April 2010

Job Vacancy

Researcher/Senior Researcher

Knowledge Media Institute (KMi)
£27,319 – £35,646, Ref: 6375
Based in Milton Keynes
Contract until 30 June 2012


KMi has an opening for a Research Assistant/Associate (depending on experience and qualifications) to undertake research on Social Semantic Web and Web Science in the context of the EU-funded project WeGov. WeGov aims at developing a toolset that interacts with common social networking systems (e.g. Facebook, Twitter, Delicious, YouTube) to better track and understand the opinions and behaviour of citizens during governance and policy-making processes. The tools will allow discussions on these social networking systems to be linked, seeded, and stimulated through injection of policy discussion points into relevant communities.

You should have a strong interest and expertise in Social Media and Software, Semantic Web, and Web Science and holds a degree in Computer Science or in other relevant disciplines. You must be able to combine scientific research with practical research and development in the project, and are willing to travel and work with the project’s academic and industrial partners.

For detailed information and how to apply go to here, call the KMi Recruitment Coordinator on +44 (0) 1908 654774 or email quoting the reference number. Closing date: 6th May 2010.

Tuesday 6 April 2010

PerCom 2010

Just got back from the 8th Annual IEEE International Conference on Pervasive Computing and Communications (PerCom 2010) in Mannheim, Germany. Some nice work was presented, but strangely there was almost nothing about semantics. There was quite a bit on social web and pervasive devices. Ciro Cattuto and I gave a talk at the First International Workshop on Communication, Collaboration and Social Networking in Pervasive Computing Environments (PerCol). The paper [1] focuses on the RFID side of our work on Live Social Semantics.

[1] Wouter Van den Broeck, Ciro Cattuto, Alain Barrat, Martin Szomsor, Gianluca Correndo and Harith Alani. The Live Social Semantics application: a platform for integrating face-to-face presence with on-line social networking. Proceedings of the PerCol Workshop, 8th IEEE Int. Conf. Pervasive Computing and Communications (PerCom), Mannheim, Germany, 2010.

Talk went very well and it was the highlight of the conference ;)

Thursday 25 February 2010

EU WeGov project has started






WeGov is a new EC STREP project under the ICT for Governance and Policy Modelling work programme.

The project aims to develop a toolset so full advantage can be taken of well-established social networking sites (Facebook, Twitter, Delicious, YouTube, WordPress, etc) when engaging citizens in two-way dialogs as part of governance and policy-making processes. The tools will make it possible to detect, track and mine opinions, behaviour, and discussions on policy oriented topics including their origins, bias and evolution. The tools will allow discussions to be linked, seeded, and stimulated through injection of policy discussion points into relevant communities.

Will link to the WeGov website when it goes live. A job ad for a researcher to work on WeGov will be coming out very soon.

Tuesday 11 August 2009

Moving to KMI

From the 1st of October 2009, I will be leaving ECS at the University of Southampton after 9 years of service, to take on the position of a senior lecturer at the Knowledge Media institute, where I'll be starting a Social Semantic Web research group.

Saturday 6 June 2009

LiveSocialSemantics experiment is over

In spite of the bad internet connection at ESWC, the experiment has been a great success. A great result for month and months of hard work. Looking forward to the next one!