plevsy

Two weeks ago i got an “urgent” request from my boss. They needed an application for the 27th National Congress of IT (Turkish > 27. Ulusal Bilişim Kurultayı) where they can get the Reports about who attended which session. I had only 4 days and it was 5 pm that means only 3 days left. I called a friend (@agaoglu) and asked if he could get a hand on this. We are working on a project called plevsy (Plus Event Systems) together. It is a Congress Management System.

He came with the idea that we do it with playframework. Java? just for a couple of forms? I am coding for over 10 years now. I used mostly PHP on my web based projects. In the last couple of years with Django too. I even write CGI on the beginning. I know how Java works, but never done a project with it. It’s to complicated to run it. It’s just a couple of forms, thats all. I looked before at the playframework. On their site it seems simple as it can. So? “yep, why not!”

After 2 days he build up the system and we meet at my place. We tested the data import from the barcode scanner. The main role was to register the participant, build a barcode for him and finally print his batch. So I got the barcode reader, but I don’t have a printer. It was a laser barcode reader, it couldn’t read from the screen :) As you can imagine we done nothing.

Next day at work, I plugged a printer and tested the badge print out function. Our badge has a custom size so I wrote the style sheet for the print option. I gave the page size, width, height everything. As a result I had a blank badge. It wasn’t in the margin. As a developer I tried to fix it via the CSS and after a half day of research, rewrite of the styles I was still on the same step. It doesn’t work. Yes in the W3C CSS documentation it should be work. But the real life is different. I went to the easiest solution. Setup the printers for page size and the browsers print function. The best solution for this was to get the badge as pdf, but this will be done another day.

Next day I set everything up, our server was running and plevsy with playframework with it. The client machines were ready to go. Get the printers tested and  I gave the users a 5 minute instruction how they use plevsy. The first attendees came to the desk and the registration progress began.

I saw that on client computers that some problems occur. My friend (@agaoglu) wrote the code not IE friendly. What we got was, he wrote the search box, where they can search the name they have registered over the web, in html 5. HTML 5 ? We are talking about IE 6. So they had to click whit the mouse on the search button after they wrote the name in the search box. They couldn’t use the “Enter” key, because it didn’t work. You can say like him “so what?”. If you have one hundred people waiting before your desk every second is a life saver. Don’t forget that. Usability means that you do it for the user in the easy way and not in the hard way. You need to do it as simple as you can. 

While we two developing on Linux machine we don’t have IE on it so we only test on Firefox, Chrome or Opera. We forgot often how IE works. I remembered this day IE ‘s bar with this “click to activate” thing. We had a couple of prompt boxes there and every time the user had to click on this “click to activate” thing and than come back to his keyboard to write it. So I disabled the prompt boxes. I make it hidden div’s and the speed gone up for the users.

In the new attendee form there were the section of the Institution. So we can get later a report for this section. It was simple a select box of the list for previous entered Institutions and if it doesn’t exist a new Institution area. I saw that list grow and grow. The users gotta find the Institution from the list and it was time taking to do it. Mostly they entered it as new. So we used jquery and there will be a plugin for it! Of course it’s jquery. It builds from a select box a searchable list. That’s it another saved time in the process. 

At the end, it deserves the name “play”. It was really an easy process to write it with play framework. The hit and go and the speed was unbelievable. I am waiting for a new project to use the play framework again. As a web developer and designer I can only say that finally you can use Java for web applications without a lot of pain. Thanks guys to make our lives easier.

ps: sorry, it take a long time to publish this

Main focus of the plevsy+ops is the extension that it has: operations. It simply means the actual handling of the event as when the participants rush to the convention center and when the session are held and when they need accommodations. Operations environment is really different from a standard application environment. To explain this, figure two well-known software: Gmail and MS Excel. We refer to them as web applications and desktop applications respectively, which practically means one works over the internet with your web browser(IE of firefox or something like that) and the other one gets installed into your computer and works on its own.

You can read/send mails using gmail as long as you have a browser[ubiquitous] and an internet connection, its not dependent on any specific computer so that you can send an email from your work and read the replies from your home, its dead simple. On the other hand, Excel requires to be pre-installed and when you start to write a report and save it at your work, you cannot complete it at your home unless you have Excel pre-installed at your home too and you physically moved the file with a memory stick of some sort. These two environments with their capabilities and their limitations are very well-known to engineers and users alike. But that front desk, that’s different.

Problem

Its different from desktop applications because you need to access the data from multiple locations which may be the computers at the desk. It is usually required to have more than one computer at the desk so that you can serve more than one customer at a time, but no typical desktop application is capable of synchronizing the data across its different installations.

Desk is more like a web application in the sense that it requires to access the same data from different locations but it may have a major drawback: internet connection. Conferences are usually held in shiny hotels with crappy internet connections. Even if they have a great internet connection, things tend to go south in stressful situations and it is not easy to explain a paid customer that you don’t know where she will be staying because internet is down. So a web application sitting in a secure, cool data center is out of the option.

Solution?

With these limitations in mind, the simplest solution is making a web application and moving it from its data center to the front desk. Or under it to be precise. It is easy to install all of the web application requirements on a basic computer and physically put it under the desk. Then it is possible to connect other computers with a local network, so you get synced data as well as isolation from internet connection problems. This is a solution we successfully carried out over the years.

But not without a big problem: management of the data. It was no easy job to get the operations data from and to a central location for archiving or analytics. We struggled with ideas from simple scripts to full-blown data merging applications without any solid solution. Nevertheless, these tasks was for experienced technicians only in a sense that organizers need to employ at least one. So as any technical tool that requires you to use specialized workforce, it was heavy, thus hard to use.

We were lucky to not run into any but there was a huge risk too. That server under the desk is exposed to all kinds of physical damage. A kick, a fall or some electrical fluctuation is always near which may result in outages or data loss. And if your server is out, all of that desk computers are useless. This structure is called ‘single point of failure’ and never desired.

Decentralization

In a sense that a web application living under the desk is a centre, we propose a no-center solution. Every computer on the desk is capable of running software so we move the application onto them and let the data flow around. Every computer on desk is able to work independently as well as to work collaboratively. Using a unique infrastructure, plevsy+ops makes it possible to work over any number of computers without having a central server. If any computer fails, you can still work on other ones. It’s more like a synchronized desktop application that is able to work over any network. So you’re not limited in any desk or any number of computers. If you need to archive or analyze you can get your data to anywhere connected to internet. Or you can work in sync with distinct sites given that they have a network connection or both connected to internet. Or you don’t. You can get your data (and application) to your laptop or tablet and work offline, then synchronize it with other stations.

Conclusion

plevsy+ops is still in early stages but the infrastructure is already solid. As soon as we have something usable, all that reliable data and synchronization goodness will be in your service. Hang in there.

I was about to put up a guideline for what we are trying to accomplish with plevsy and i stumbled upon a recent DHH posting. He simply advices that we need to make our goals public so that we can start thinking and plotting about getting it done. Considering the fact that plevsy is almost 3 years old without any physical form, i think this might be what’s missing.

plevsy is addressing a lot of problems in the event management business. It is especially for professional conference organisers which have diverse needs ranging from abstract awards to online registration to vouchers. Although plevsy aims for a whole event management suite the current focus is on operations [as PCOs call it] which means actual handling of the event. Some examples of the main needs in this area are:

• reports and statistics about participants like number of arrived participants or attendees of a select meeting who also happen to be from the same institution
• in-house listings of hotels for correlation purposes or number of occupied rooms over the course of the operation for accurate prediction
• transfer logs for select hours and sources/destinations including flight numbers and/or PNR codes
• managing all registration, accommodation and transfer fees as sales of predefined products enabling participant sponsors and sponsor based invoices

Some of the topics do not apply to all operations and they definitely do not cover what may be possible. As far as we know, feature-completeness comes at a cost [no, not that cost]. That much detail does not easily fit on any screen without making it extremely hard to use. That’s where most feature-complete tools fail because frankly, it is not easy to use anything that contains so much detail. plevsy+ops is trying to cover every aspect and on the other hand it keeps things simple by disclosing just enough information. How we simplified things and how you can improve your event operations will be the priority of this blog for some time as plevsy+ops is the main tool to achive that.

As a reminder [to us and everyone around]: plevsy is not and is never going to be a silver bullet for event management or an event organiser itself. Software are there to help people, and the bests are the ones that stays out of the way while doing it. plevsy will be the best helper for event organisers who are willing to get help. That’s our goal, made public.

The week before last was an interesting one. Apart from the usual chaos at our workplaces, we finally had our chance to get some hands-on experience with play! Simply put, it was one of the most pleasing two days of time developing, and the result was probably the best, for two days that is.

First of all i (@agaoglu) wasn’t anywhere near any web application for about 6 months now, so it can be said that i’m somewhat rusty. But i was onto some serious hadoop stuff, so my java blade is sharper than ever. @burakdalgic on the other hand, was simply harvesting his crops of PHP for almost a year and looking for some other ingredients to open his next brewery.

Anyway, @burakdalgic called me on Thursday afternoon to inform me that they would be needing a registrations tool for their front desk at 27th National Congress of IT (27. Ulusal Bilişim Kurultayı[in turkish]). Organizer of the congress had recently bought some barcode readers and by utilizing them, they wanted reports about participants of select sessions. Along with barcode readers, they were supplied with a software to record some details of participants which is developed with Excel/Access and somehow is unable to generate any reports, but that’s another story. When i asked about our deadline, he simply answered the event would start on Wednesday so there were 5 days ahead. I said OK thinking some sleepness nights on weekend is nothing new. Prior to that OK, I didn’t consider any frameworks or languages apart from that it wasn’t going to be any flavor of JavaEE nor PHP.

That evening at home after some chillout music and some rest cured my headache, two choices shined out of the dark: Django and play!. We had experience with Django and a small application like that was nothing compared to what we had been able to achieve with it. Beyond capabilities, it was the fastest-to-work-with environment i had ever been into. Generating HTML form out of the Model class was a huge performance boost but the templating was making up for it. Then i remembered: working in django, i always needed 3 virtual desktops. One for browser, one fore IDE (which is vim) and one for terminal that ‘$ django runserver’ was outputting. Whenever a error happens, django was generating an HTML report to tell me about it in the browser, but since that almost never made sense, i had to go to the terminal to see that i was actually comparing a string with an int in an entirely different place. I know this is not a django problem but a dynamically-typed language problem but it is not my problem now is it? Another problem was ‘$ django runserver’ was not the way to go in production, so we would be forced to deploy some web server, configure it and have another point of failure which is not what we want in a front desk of a conference.

So as the night goes darker one option was becoming brighter: play! Which actually had a bigger problem: we had zero experience with it. Apart from some tests with CRUD module and regular reading of their mailing list, we had no idea how to work it out. Well, we knew it was on java, which is a statically-typed language, which in turn has some great IDEs like eclipse which tell you it is not possible to compare a String with an int, as soon as you write it. That’s what i call a performance boost by the way. On the other hand i was remembering things about play having its own netty powered web server and the recommended way of running a play application was ‘$ play run’ indeed. That way we would able to track down any problems at that front desk more easily, since there would be only one place to look at. So, around 9pm that night i overlooked the lack of experience and take a leap of faith by pulling play/1.1 branch from launchpad.

With the help of ‘$ play help’ project had a structure, eclipsified and was running in no time. After some layout with jquery-ui (which quite sums up what i know about UI), we got our first screen that would be simply the C in a CRUD for our participant model.

After a few trips to documentation i got the form wired to my application code and was able to save entries to database with just one line, thanks to Active-Record-like interface for JPA stuff.

public static void save(Participant participant){
  participant.save();
  show(participant.id);
}

Manually constructing jdbc connections in JNDI, wiring it to JTA and informing hibernate to create the EntityManagerFactory out of that mess, so that i can create my EntityManager to do the same save(), just went out of the window. I don’t even want to remember the things i should have done in a web.xml and any other framework related xml just to wire that form to my application code. I know almost all of the xml-required-frameworks have some eclipse plugins to automate these processes but in my experience they rarely work. And when they work, it’s just for a time to understand they are not capable enough so back to manual. Besides, there are times when it is not possible to use any IDE. play!, with its out-of-the-ordinary package naming scheme, really shines here. It is extremely optimistic to edit something in com/spp42/plevsy/track/controller/Participants.java, compile it to get a war (which you need a previously written ant build at best) and expect it to work just after deploying on tomcat. With play!, it is simply ‘$ vim controllers/Participants.java’ and hitting reload. No need to ant-build or deploy or even quit vim.

Next, we needed the first core functionality which is barcode generation. Quick googling lead me to barbecue and within minutes i was able to output barcodes in PNG. That completed R in that CRUD.

public static void barcode(Long id)
    throws BarcodeException, OutputException{
  Participant participant = Participant.findById(id);
  Barcode barcode = BarcodeFactory.createCode128(
    participant.code);
  ByteArrayOutputStream baos = new ByteArrayOutputStream();
  BarcodeImageHandler.writePNG(barcode, baos);
  ByteArrayInputStream bais = new ByteArrayInputStream(
    baos.toByteArray());
  renderBinary(bais);
}

I remember a good friend of mine struggling to modify a PDF generation in django for days without a result because library wasn’t in a good shape. I might remember that wrong, but i think noone’s gonna argue that the library diversity in java beats the hell out of anything out there.

I committed what i have so far to svn around 11pm and went to bed. That was one less sleepless night for me. Friday morning, at work, i had some free time to finish U and D of the CRUD which took about five minutes combined. I somewhat felt experimental and look into some HTML5 input types which resulted in our participant listing to be more like google instant! (OK that was a bit off). But i didn’t take me more than fifteen minutes.

First what seems to the user

<div class="toolbar search ui-widget-header ui-corner-all">
  <input 
    type="search" 
    name="q" 
    id="q" 
    class="search ui-widget ui-corner-all" 
    value="${session.q}"
    incremental="true" />
  <button class="search">Search</button>
</div>
<div class="list">
  <ul class="partlist clean">
  </ul>
</div>

Then what is on user but does not seem

$('button.search')
.button({text:false,icons:{primary:'ui-icon-search'}})
.click(doSearch);
$('#q').bind('search', doSearch);

function doSearch(e) {
  $.get("@{Participants.search()}", 
      {q: $('#q').val()}, 
      function(data){
        $('ul.partlist').html(data);
  });
}

And in server

public static void search(String q){
  List result = Participant.find(
      "lower(name) like ? or " +
      "lower(lastname) like ? " +
      "order by lastname", 
      q+"%".toLowerCase(), q+"%".toLowerCase()).fetch();
  session.put("q", q);
  render(result);
}

which outputs

#{list result, as:'p'}
<li>
  <a class="ui-widget ui-widget-content" 
    href="@{Participants.show(p.id)}" >
  ${p.name} ${p.lastname} (${p.id})
  </a>
</li>
#{/list}

And as a result

That afternoon i had some more time to implement barcode checkins so i called @burakdalgic to learn that we would be getting a text file out of that barcode readers and users will be uploading them. As the documentation says, it could not be more simpler.

public static void install(Hall hall, File dump)
    throws FileNotFoundException, ParseException{
  Scanner scanner = new Scanner(dump);
  while(scanner.hasNextLine()){
    String line = scanner.nextLine();
    String[] entry = line.split("\\|");
    Date parsed = new SimpleDateFormat(
      "dd.MM.yyyy hh:mm:ss").parse(entry[1]);
    CheckIn checkIn = new CheckIn(entry[0], hall, parsed);
    checkIn.participant = Participant.findById(
      Long.valueOf(entry[0].substring(13)));
    checkIn.save();
  }
  main();
}

In java that is. It would be more concise with functional programming paradigms and anyone who can’t live without them is free to use scala.

I filled my remaining free time trying to figure out how to generate a pie chart with flot which took something like 2 hours. Say, 3 minutes of that 2 hour went to write required jpa-ql so remaining 1 hour and 57 minutes seemed too much time for just a pie chart after all that work in play. Alright, lack of experience with flot is responsible for much of it but i knew almost nothing about play either. However could write a whole another application in that time. Wasting time on less important thing, i went to bed again on time, cause there were only a handful of things left to do.

Saturday morning, after a late breakfast, another 2 hours got me a fully working application with solid layout, barcode checkins, reports and CRUD screens. Sleepless nights aside, we went to the barbecue party at our friends’ house and got some alcohol into our systems. Sunday afternoon, we met to iron out some details about barcode data format, and i didn’t even need to tell anything about play. Because the code was just simple java methods, HTML and javascript; @burakdalgic was able to pick just where i left and move on[to his work of UI].

Well, i mentioned the effort took two days but it seems it wasn’t even whole two days. Just some hours scattered over a weekend and we had what we needed. I wasn’t at the operations or front desk so i don’t know how it went the first hand but that’s a good thing from another angle: they didn’t have major problems [apart from the problem that we started running it in development mode to have some crashes which was fixed as soon as we switched to production] so i didn’t have to be there. For a two day application that is a great technical outcome. And from what i have heard from my collegue, users were pleased with its usage and performance. ‘The bottle neck on the desk was the printer’ he said.