Planet ODF

I’ve been thinking some more on the past, present and future of documents.   I don’t know exactly where this post will end up, but I think this will help me clarify some of my own thoughts.

First, I think technology has clouded our thinking and we’ve been equivocating with the term “document”, using it for two entirely different concepts.

One concept is of the document as the way we do work, but not an end-in-itself.  This is the document as a “collaboration surface”,  short-lived, ephemeral, fleeting, quickly created and equally quickly forgotten.

For example, when I create a few slides for a project status report, I know that the presentation document will never be seen again, once the meeting for which it was written has ended.  The document serves as a tool for the activity of presenting status, of informing.  Twenty years ago we would have used transparencies (“foils”) or sketched out some key points on a black board.  And 10 years from now, most likely,  we will use something else to accomplish this task.    It is just a coincidence that today the tools we use for this kind of work also act like WYSIWYG editors and can print and save as “documents”.  But that is not necessary, and historically was not often the case.

Similarly, take a spreadsheet.  I often use a spreadsheet for a quick ad-hoc “what-if”  calculation.  Once I have the answer I am done.  I don’t even need to save the file.  In fact I probably load or save a document only 1 in 5 times that I  launch the application.   Some times people use a spreadsheet as a quick and dirty database.  But 20 years ago they would have done these tasks using other tools, not document-oriented, and 10 years from now they may use other tools that are equally not document related.  The spreadsheet primarily supports the activity of modeling and calculating.

Text documents have myriad collaborative uses today, but other tools have emerged  as well . Collaboration is moved to other non-document interfaces, tools like wikis, instant messaging, forums, etc.  Things that would have required routing a typed inter-office memo 50 years ago are now done with blog posts.

That’s one kind of document, the “collaboration surface”, the way we share ideas, work on problems, generally do our work.

And then there is a document as the record of what we did.  This is implied by the verb “to document”.   This use of documents is still critical, since it is ingrained in various regulatory, legal and business processes.  Sometimes you need “a document.”  It won’t do to have your business contract on a wiki.  You can’t prove conformance to a regulation via a Twitter stream.  We may no longer print and file our “hard” documents, but there is a need to have a durable, persistable, portable, signable form of a document.  PDF serves well for some instances, but not in others.  What does PDF do with a spreadsheet, for example?  All the formulas are lost.

This distinction, between these two uses of documents,  seems analogous to the distinction between Systems of Engagement and Systems of Record, and can be considered in that light.    It just happens that each concept happened to use the same technology, the same tools, circa the year 2000,  but in general these two concepts are very different.

The obvious question is:  What will the future being?   How quickly does our tool set diverge?   Do we continue with tools that compromise, hold back collaborative features because they must also serve as tools to author document records?   Or do we unchain collaborative tools and allow them to focus on what they do best?

You possibly have heard of WebODF already, the Open Source JavaScript library for displaying and editing files in the OpenDocument format (ODF) inside HTML pages. For ideas what is possible with WebODF and currently going on, see e.g. Aditya’s great blog posts about the usage of WebODF in OwnCloud Documents and Highlights in the WebODF 0.5 release.

The WebODF library webodf.js comes with a rich API and lots of abstraction layers to allow adaption to different backends and enviroments. There is an increasing number of software using WebODF, some of that listed here.

Those which are interested in the capabilities of WebODF, without needing a custom and highly integrated solution, can additionally go for ready-made simple-to-use components based on WebODF. This blog post is the first of a series to introduce you to those. It starts with the component that gives you embedded display of OpenDocument format files, that is text documents (ODT), presentation slides (ODP) and spreadsheets (ODS), in webpages by just a single (sic!*) line of HTML code:
* no-one would add a line-break there ;)

ViewerJS

ViewerJS is an Open Source document viewer that enables embedded display of ODF or PDF files directly in webpages, without any external server dependencies, just done with HTML, CSS and Javascript. It uses WebODF to display files in the OpenDocument format and PDF.js for files in the PDF format.

Deploying and using ViewerJS with your webpages can be done in a few minutes. Follow this guide and see yourself!

Quickly Added

Start with looking at the current time and noting it.

As example file take an ODP of your choice, otherwise let’s use the slides from a talk at KDE’s Akademy in 2013, akademy2013-ODF-in-KDE-Calligra-WebODF.odp.

If you do not have a webserver handy, create a mini one locally on your system:


# Create a folder structure to serve statically
mkdir htroot

# Put the sample ODP file into htroot, renamed as "example.odp"
cp akademy2013-ODF-in-KDE-Calligra-WebODF.odp htroot/example.odp

# Add a simple html file:
touch htroot/example.html


Open example.html in an editor and have this as content:

<!DOCTYPE HTML>
<html>
  <head>
    <title>example.odp</title>
  </head>
  <body>
    <div>We got an ODP file.</div>
    <div>Would be nice to show it here.</div>
  </body>
</html>

Start a simple webserver program serving that directory, e.g. the one built into Python. For that open a separate console and do:


cd htroot
python -m SimpleHTTPServer


Now browse to http://127.0.0.1:8000/example.html and make sure you see that HTML file.

The ODP file example.odp is not displayed yet, right. Not so nice.

Let’s change that and deploy ViewerJS for it.

In the first console now do:


# Download http://viewerjs.org/releases/viewerjs-0.5.2.zip
# (check if there is a newer version perhaps, then change
# all "0.5.2" below to the new version string)
wget http://viewerjs.org/releases/viewerjs-0.5.2.zip

# Unzip the file
unzip viewerjs-0.5.2.zip

# Move the folder "ViewerJS" to the toplevel dir of
# the folder structure statically served by the webserver
# (could also be a non-toplevel dir)
mv viewerjs-0.5.2/ViewerJS htroot


Now replace the “Would be nice to show it here.” in the example.html with this code (remove the REMOVEME, workaround to strange WordPress behaviour):

<REMOVEMEiframe id="viewer" src="/ViewerJS/#../example.odp" width='400' height='300' allowfullscreen webkitallowfullscreen></iframe>


(in the sources one line, as promised. But add line-breaks as you like ;) )

Now reload http://127.0.0.1:8000/example.html in your browser. And if everything worked out, you see the ODP file now embedded in the webpage, ready to be read or e.g. presented fullscreen.

Look again at the current time. How minutes did you need? :)

ODF or PDF

For publishing done documents that should be only read and not further processed, PDF is the better choice IMHO, because the format specifies the exact positioning of everything.
ODF (same with similar formats like OOXML) leaves the actual fine-layout to the program displaying/printing the document, which can differ between computer systems and setups, usually due to the used font engine. This makes sense, as it allows to create ODF files from code that has no clue about layout calculations, e.g. some Perl script generating a report. But it can result in frustrations if some document with manually optimized layout gets differently layout-ed elsewhere.

Thanks to PDF.js ViewerJS can also nicely display PDFs, so use whatever format suits the needs, be it preview of some document to further process or display of the final result.

Take a PDF file and change the above example to show that instead of the ODP file. Then try also with an ODT or ODS file.

Getting better week by week

The developers of WebODF are constantly enhancing its coverage of the ODF spec. See how the slides template for this year’s GUADEC (of course done in ODP :) ) are almost looking the same in LibreOffice and ViewerJS (v0.5.2):

Currently the Wiki hosting the GUADEC slide templates still has to say:

Current configuration does not allow embedding of the file lightning_talks.odp because of its mimetype application/vnd.oasis.opendocument.presentation

ViewerJS and WebODF hopefully can be a reason to change that soon :)

When giving talks about WebODF of course ODPs and ViewerJS are used. Knowing the pitfalls the slides can be done avoiding those. Still many real-life samples not designed for current WebODF capabilities are increasingly well displayed, also e.g.

or

In general are ODF documents with only formatted text and images in SVG, PNG, JPEG or similar no problem for WebODF and thus ViewerJS. But as can be seen next, e.g. native ODF graphic elements are still a TODO (and the result not related to any censoring code ;) ). But, the display is already good enough for a “preview” :) :

BTW, if you are browsing a website that does not yet use ViewerJS to display ODF files embedded but only provides them as links, there is another WebODF-based option for Firefox users: the ODF viewer Firefox Add-on, that allows viewing ODF documents directly in Firefox on any device, without the need of a (big) office suite.

More on ViewerJS.org

Learn more about ViewerJS on the website ViewerJS.org, e.g. how to support non-embedded custom fonts. Discover the ViewerJS plugin for WordPress. Think about how you and your websites could make use of ViewerJS and how you could help to improve ViewerJS and WebODF, and then contact the ViewerJS and WebODF developers about that! They are looking forward to working together with you as well.

‎Europe: added one citation (edited with ProveIt)

Not even two weeks passed after the first update to WebODF 0.5 and there is the second.

It brings improvements in the rendering of the ODP slides and makes nagivation in ODT documents via home/end keys, or up/down cursor keys more reliable in all browsers.

On Tuesday the news that the UK Government had decided to use ODF as its official and default file format started to spread. The full announcement with technical details may be found here; the Document Foundation published its press release on Thursday morning there.

This decision is a landmark for several reasons. First, it is not every day that you see an entire government migrate to a standardized file format. You may hear about government branches using this or that solution, but nothing that is so “abstract” than a file format.  This time the UK Government has made the conscious decision to define a coherent policy in handling its digital documents, from the stage where they are created, edited and circulated all the way to the archival phase. It also comes year after the decision of the State of Massachusetts. As such the decision covers a variety of standards (HTML, PDF and ODF); yet its scope, as Glyn Moody rightly reminds us, also means that the devil will lie in the details of the execution.

Most of the migrations from one office suite to another tend to happen without any coherent document management policy. Many organizations moving from, say, Microsoft Office to LibreOffice do not necessarily adopt ODF as their default format and will carry on supporting whatever version of the MS Office file format internally. This usually leads to frustrations and compatibility problems. This time, the UK Government decision takes a different approach. By deciding about the formats first, the UK creates the conditions necessary to have real choices for its government and its citizens, thus setting a level playing field for everyone. Many people have understood this decision as being a move against Microsoft. It is not or at least it should not be. Microsoft Office implements ODF files and its latest editions, as I’m being told are actually quite good at it. What this move does, however, is to ensure no other solution will be at a competitive disadvantage because of a technical or legal (aka patents) lock-in. Of course, it remains to be seen what concrete actions the UK Government will take in order to ensure a smooth transition between proprietary formats and open standards; and it remains to be seen how well it will ensure a proper change management across all of its departments so that its agents feel comfortable with ODF documents and whatever new office suites that may be adopted as a result of the decision. Much could be lost at that stage, but much could be gained as well. And of course, just like with the Netherlands, the decision itself might end up being toned down or take a somewhat different meaning.

While reading among the tea leaves is not my favourite past time, it is relevant to assume that this decision may change a few things around the IT industry as well. By way of an example, I have always been amazed at Apple’s clean support of ODF inside Mac OS X but its constant absence across the iWork editions. Perhaps Apple will feel compelled to introduce ODF files in iWork now? Only time will tell. Cloud solutions will also have to improve or implement ODF and in some cases PDF support in a proper way.

The decision might also have consequences for other European countries and perhaps for the European institutions themselves, as the UK will now be an actual example of a country that has migrated to ODF, and not just one of the countries that made the choice of Free and Open Source Software. This is rare enough to catch the attention of several member states CIO offices.

This move to open standards by the UK Government is also telling of a deeper change in IT industry. We may reach the stage where finally, the average user starts to realize that the old Windows + Office paradigm starts to get exhausted. What can you do with Office documents aside opening them imperfectly in alternatives and opening them in a more effective way with Microsoft software? Actually, not much. Unless you get SharePoint. But the whole point is that in 2014, trying to extract revenue by creating lock-in on office files is no longer acceptable. That, I think, is what the UK Government decision really means. And if I’m right, it’s only the beginning.

Last but not least, this post would not be over without thanking many people whom I’ve worked with for several years in my position at my former company, Ars Aperta, in my former role at OpenOffice.org, at the OASIS Consortium and even today when contributing to the LibreOffice project. I’m thinking about people at OpenForum Europe, the FFII, the APRIL, the AFUL, the OASIS, the now defunct ODF Initiative and everyone else I am forgetting right now but who should be remembered. It’s nice sometimes, after such successes, to turn back and look at the road behind us. It can only give more confidence to walk on the one ahead.

The ODF Technical Committee at OASIS has elected Oliver-Rainner Wittmann as its new co-Chair, joining Patrick Durusau. Mr. Wittmann is an employee of IBM and as erstwhile employee of SUN Microsystems has been a longstanding member of the ODF TC. Wittman is currently part of the team working on Apache OpenOffice, the well-known open source office suite.

Mr. Wittman is the successor of Rob Weir, who held the position of co-chair since 2007 until now. In his capacity of co-chair he played an important role in the tumultuous period around the ISO/IEC standardisation of IS29500. He also was a prominent figure in the various ODF plugfests, organised by OpenDoc Society together with a number of governments, academic institutions and open source communities.

Due to changes in his role at his employer, Weir is longer directly involved with document formats and editors. He will temporarily stay on as liaison from the ODF TC to ISO/IEC WG6, to provide some continuity. Weir's analytic skills, strong pen, in-depth knowledge of office applications and file formats, as well as his sharp wit will be missed by many in the ODF community.

The UK Cabinet Office is the latest government to officially adopt the Open Document Format as its office format of choice. The document file format is expected to be used across all government bodies for sharing or collaborating on government documents.

The Open Document Format offers superior interoperability compared to legacy proprietary formats, and best serves the needs of users:

https://www.gov.uk/government/news/open-document-formats-selected-to-meet-user-needs The move supports the government’s policy to create a level playing field for suppliers of all sizes, with its digital by default agenda on track to make cumulative savings of £1.2 billion in this Parliament for citizens, businesses and taxpayers.

Less than 3 weeks after the release of WebODF 0.5.0 there is the first update.

It brings improvements in the rendering of the numbering of multi-level lists, solves issues during the startup of the Wodo.TextEditor on Safari and more.

It has been a while since I have discussed open standards here, even though I have alluded to them in passing. There are currently a number of initiatives and policies ongoing at the European level that are bringing this topic back on the table, especially with regard to public procurement practices. Why does it matter? Because it shows that beyond any kind of advantage, convenience, or the mere ability to have a real choice of IT solutions suppliers, open standards are considered by much of the private and public sectors as some sort of nuisance.

Depending on how you see it, the “battle” for open standards is either won, or it is  still ongoing at the normative level (think about the DRM injection in HTML5 that happened at the W3C). Open Standards, more than ever before, rule the IT industry and the Internet. Cloud technologies rely on open standards to a large extent; purchasing music tracks online lets you increasingly download open file formats that, while they may not be exactly standardized themselves, have open specifications and are unencumbered wiith digital restrictions management (yes, that’s how DRM should really be called).

On the other hand, desktop technologies are still a major issue. One could assume it is because of the stranghold of an entrenched  monopoly, and perhaps it is, to some extent. We are in 2014 however, and both open standards and FOSS desktop offerings (LibreOffice, Firefox, Linux  distributions for the desktop) are legion. These have a real uptake among what is  often referred to as the consumers’ market and that’s great news, but when it comes to what’s going on in the workplace, there seems to be little choice aside the MS Office + Outlook + SharePoint on Windows stack. Why  is that the case? Why is the European Commission still trying to tackle the problem in 2014?

The Desktop is traumatizing

And more exactly, change is traumatizing. Technology changes very quickly, but the more structured the workplace you have, the less adaptative it will be for IT  solutions. If you add the specific culture of the organization that can sometimes be more or less rigid and centered on one vertical industry, you will find long cycles of deployment for any kind of IT technologies and a reluctance to “switch” to a new brand or a new kind of software. This could not be more true on the desktop. I’ve been writing this for years here, but there are reasons for that: the desktop is used by pretty much everyone in the organization. While it  is  somewhat changing with the arrival of tablets and smartphones, desktops are here to stay. The problem is that desktops are very complex systems -offering a graphical interface and tools for pretty much every kind of uses and situations one can imagine- and as such come with more quirks than other devices and other software platforms. These quirks end up being noticed by the users, who most of the time are not computer-savy and will be reluctant to change. Worse,  their  skills will directly or indirectly be challenged by the change. This fear of change ends up being passed on to the CIO level, who has to make the purchase decision, and does not want to be hold liable for having chosen that “weird, so called innovative solution no one gets”.

Just like with any fear, we are not talking about rationality. In 2014, people who use Twitter on a daily basis will shout if their desktop has changed overnight. It is not a good practice to do that kind of brutal change anyway, but the very concept of microblogging was unknown to them 5 years ago. They embraced it with no trouble at all. Their desktop, however, is a holy land, the solitaire game and their office suite their hallowed relics.

Open Standards can sometimes be hard to understand

It is hard enough for people to understand what protocols such as TCP/IP do. These open standards however are invisible to most of them, even if they’re using them on a daily basis. Other open standards, such as OpenDocument Format, are probably not conceivable by some people, who think that an office document is “an extension of Microsoft Office”. I have even heard of teachers, here in France, who refused to even mention ODF because such a thing “could not possibly exist”. The conceptual distinction between a file and an application has not permeated much, even in the twenty first century.

Yet, open standards are the way to go. They may not always be the superior technology, but they offer a level playing field for the industry to build on and innovate with. The Internet has been built on this, so does cloud computing. Desktop solutions are no different. Using open standards brings you back in control of your suppliers and IT infrastructure; it ultimately helps reducing costs and keep your data safe, reusable and sustainable  for dozens of year to come. You can read more about it in the excellent article by Bjorn Lundell published here. Ultimately, the lock-in of the desktop solutions will stop being meaningful as the state of the art will change so much the solutions that are seen as essential today will stop being that important. But the documents, the images, the data, all your content will still be locked in undocumented file formats that need to be reverse-engineered in order to edit them. No one should build such a silo for the future and then throw away the key. That’s what has been happening for more than a decade on the desktop, unfortunately. Where does that lead  us? I think we can already see where: vendor lock-in is here to stay on a more or less large extent; but so are open standards. There will then be people who are stuck with their vendors and constantly handle the legacy; there will be the others, who actually enable information technologies to help them innovate. For them, the story has only started.

Presentations and videos from the Open Document Editors Devroom at FOSDEM 2014 are now available.

The OpenOffice presence at the event was strong and well-received, with a number of interesting talks and many visits at the booth. And the devroom was a nice way to get together with other projects focused on the ODF format.

Make sure you don't miss the following talks about improvements that are coming (or already came!) to OpenOffice:

The full list of talks from the Open Document Editors devroom is available as well. Most have a presentation attached.

One week-end of Calligra sprint is currently going on in the old and cozy center of Deventer in the Netherlands.
Yesterday everyone safely arrived, and have been in discussions since then… Right now we are all sitting in the livingroom of Boudewijn and Irina (who are being great hosts to the sprint) around the coffee table, everyone a laptop on their lap, in after-lunch digesting mode, with the full discussions to be continued now every minute.

This is the last but one update to the 2.8 series of the Calligra Suite, and Calligra Active released to fix recently found issues. The Calligra team recommends everybody to update.

Why is 2.8.4 skipped? Shortly before 2.8.4 release we discovered bug that sneaked in 2.8.2 version and decided to skip the 2.8.4 entirely and quickly release 2.8.5 instead with a proper fix. The bug is related to not showing file formats in Save dialogs.

Issues Fixed in This Release

General

• Show file formats in Save dialogs
• Added a number of missing translations

Kexi

• Only move to initial top-left position in table view on initial show (bug 334577)
• Display proper record’s page number when coming back to report’s data view (bug 335269)
• Properly remember previous search keywords in the Find Dialog (bug 334514)
• Fix crash for SELECT queries containing (invalid) “ORDER BY 0″ parameter
• Properly sort names of newly added tables and queries in Form Designer’s data source drop-down list (bug 334093)
• Fix crash possible when accessing queries with duplicated table names. Example query “SELECT t.foo FROM t, t”. Display error message so user can fix the statement. (bug 315852)
• Fix squashed up toolbar buttons for windows-based styles (bug 335149)
• Fix crash when removing a Database Connection (bug 315288)
• Temporarily hide “sort” buttons in forms because sorting is not implemented there anyway (related to bug 150372)
• Make Ctrl+S keyboard shortcut work for Save action in all designers (bug 334587)
• Fix deselecting text form’s text editor widget when moving to other widget (bug 336054)
• Let the spreadsheet import plugin to be found by Kexi
• Fix crash on changing Auto Tab Order form property, set to true by default (bug 336927)
• Properly retrieve map element after saving and re-opening report (bug 336985)
• Removed resource leaks

Calligra Stage

• Re-enable KPresenter (KPR)-to-Open Document Presentation (ODP) document filter

Try It Out

• The source code is available for download: calligra-2.8.5.tar.xz.
• Calligra binaries are available for download for many operating systems.

About Calligra

Calligra Suite is a graphic art and office suite. It is a part of the applications family from the KDE community. See more information at the website http://www.calligra.org.

A couple of days ago I received an email from Cristian, with two screenshots of KMyMoney running on KDE Frameworks 5.
There’s still lots to be done, but it’s a good start.


One thing where we’ll need help is getting a Qt5 version of KDChart. That’s what we use for charts in our reports. Calligra uses it too, so hopefully we’ll see something come up soon.

Cristian will be attending the Randa meeting in August, to work further on the port. Please, help make the sprint happen, donate to the fundraising and spread the word about it.

Today, after a long period of hard work and preparation, having deemed the existing WebODF codebase stable enough for everyday use and for integration into other projects, we have tagged the v0.5.0 release and published an announcement on the project website.

Some of the features that this article will talk about have already made their way into various other projects a long time ago, most notably ownCloud Documents and ViewerJS. Such features will have been mentioned before in other posts, but this one talks about what is new since the last release.

The products that have been released as ‘supported’ are:

• The WebODF library
• A TextEditor component
• Firefox extension

Just to recap, WebODF is a JavaScript library that lets you display and edit ODF files in the browser. There is no conversion of ODF to HTML. Since ODF is an XML-based format, you can directly render it in a browser, styled with CSS. This way, no information is lost in translation. Unlike other text editors, WebODF leaves your file structure completely intact.

The Editor Components

WebODF has had, for a long time, an Editor application. This was until now not a feature ‘supported’ to the general public, but was simply available in the master branch of the git repo. We worked over the months with ownCloud to understand how such an editor would be integrated within a larger product, and then based on our own experimentation for a couple of awesome-new to-be-announced products, designed an API for it.

As a result, the new “Wodo” Editor Components are a family of APIs that let you embed an editor into your own application. The demo editor is a reference implementation that uses the Wodo.TextEditor component.

There are two major components in WebODF right now:

1. Wodo.TextEditor provides for straightforward local-user text editing,by providing methods for opening and saving documents. The example implementation runs 100% client-side, in which you can open a local file directly in the editor without uploading it anywhere, edit it, and save it right back to the filesystem. No extra permissions required.
2. Wodo.CollabTextEditor lets you specify a session backend that communicates with a server and relays operations. If your application wants collaborative editing, you would use this Editor API. The use-cases and implementation details being significantly more complex than the Wodo.TextEditor component, this is not a ‘supported’ part of the v0.5.0 release, but will, I’m sure, be in the next release(s) very soon. We are still figuring out the best possible API it could provide, while not tying it to any specific flavor of backend. There is a collabeditor example in WebODF master, which can work with an ownCloud-like HTTP request polling backend.

These provide options to configure the editor to switch on/off certain features.

Of course, we wholeheartedly recommend that people play with both components, build great things, and give us lots of feedback and/or Pull Requests. :)

New features

Notable new features that WebODF now has include:

• SVG Selections. It is impossible to have multiple selections in the same window in most modern browsers. This is an important requirement for collaborative editing, i.e., the ability to see other people’s selections in their respective authorship colors. For this, we had to implement our own text selection mechanism, without totally relying on browser-provided APIs.
  Selections are now smartly computed using dimensions of elements in a given text range, and are drawn as SVG polygon overlays, affording numerous ways to style them using CSS, including in author colors. :)
• Touch support:
  • Pinch-to-zoom was a feature requested by ownCloud, and is now implemented in WebODF. This was fairly non-trivial to do, considering that no help from touch browsers’ native pinch/zoom/pan implementations could be taken because that would only operate on the whole window. With this release, the document canvas will transform with your pinch events.
  • Another important highlight is the implementation of touch selections, necessitated by the fact that native touch selections provided by the mobile versions of Safari, Firefox, and Chrome all behave differently and do not work well enough for tasks which require precision, like document editing. This is activated by long-pressing with a finger on a word, following which the word gets a selection with draggable handles at each end.

Drawing a selection on an iPad

• More collaborative features. We added OT (Operation Transformations) for more new editing operations, and filled in all the gaps in the current OT Matrix. This means that previously there were some cases when certain pairs of simultaneous edits by different clients would lead to unpredictable outcomes and/or invalid convergence. This is now fixed, and all enabled operations transform correctly against each other (verified by lots of new unit tests). Newly enabled editing features in collaborative mode now include paragraph alignment and indent/outdent.

• Input Method Editor (IME). Thanks to the persistent efforts of peitschie of QSR International, WebODF got IME support. Since WebODF text editing does not use any native text fields with the assistance of the browser, but listens for keystrokes and converts them into operations, it was necessary to implement support for it using JavaScript using Composition Events. This means that you can now do this:

Chinese – Pinyin (IBUS)

and type in your own language (IBUS is great at transliteration!)

Typing in Hindi

• Benchmarking. Again thanks to peitschie, WebODF now has benchmarks for various important/frequent edit types.

• Edit Controllers.  Unlike the previous release when the editor had to specifically generate various operations to perform edits, WebODF now provides certain classes called Controllers. A Controller provides methods to perform certain kinds of edit ‘actions’ that may be decomposed into a sequence smaller ‘atomic’ collaborative operations. For example, the TextController interface provides a removeCurrentSelection method. If the selection is across several paragraphs, this method will decompose this edit into a complex sequence of 3 kinds of operations: RemoveText, MergeParagraph, and SetParagraphStyle. Larger edits described by smaller operations is a great design, because then you only have to write OT for very simple operations, and complex edit actions all collaboratively resolve themselves to the same state on each client. The added benefit is that users of the library have a simpler API to deal with.

On that note…

We now have some very powerful operations available in WebODF. As a consequence, it should now be possible for new developers to rapidly implement new editing features, because the most significant OT infrastructure is already in place. Adding support for text/background coloring, subscript/superscript, etc should simply be a matter of writing the relevant toolbar widgets. :) I expect to see some rapid growth in user-facing features from this point onwards.

A Qt Editor

Thanks to the new Components and Controllers APIs, it is now possible to write native editor applications that embed WebODF as a canvas, and provide the editor UI as native Qt widgets. And work on this has started! The NLnet Foundation has funded work on writing just such an editor that works with Blink, an amazing open source SIP communication client that is cross-platform and provides video/audio conferencing and chat.

To fulfill that, Arjen Hiemstra at KO has started work on a native editor using Qt widgets, that embeds WebODF and works with Blink! Operations will be relayed using XMPP.

Teaser:

Other future tasks include:

• Migrating the editor from Dojo widgets to the Closure Library, to allow more flexibility with styling and integration into larger applications.
• Image manipulation operations.
• OT for annotations and hyperlinks.
• A split-screen collaborative editing demo for easy testing.
• Pagination support.
• Operations to manipulate tables.
• Liberating users from Google’s claws cloud. :)

If you like a challenge and would like to make a difference, have a go at WebODF. :)

WebODF text editor

KO GmbH and the developers of WebODF today released WebODF 0.5. After a long period of hard work, this new version features editing of OpenDocument Text files. It also contains lots of display improvements. WebODF still uses only HTML, CSS and JavaScript without any need for a special server software.

Almost two years ago (uh, already?) I went to Prague for a developer sprint of the team of Marble, the virtual globe and world atlas. My goal was to work on a proper maps shape plugin for Calligra.

We are ready to render, or? … Or?

Just, it showed that Marble at that time was not properly supporting the needs such a plugin has. Biggest showstopper was that Marble did not expose the information if and when all external data needed for rendering was available. Think e.g. of map pieces (so called tiles) still having to be downloaded from a server like OpenStreetMap‘s one. Not knowing the state is not good e.g. if a document with a map is to be rendered to a real printer.

Telling the File Manager some more about Marble and Geo Data Files

At that sprint then I worked instead on making sure that there are proper mime-types registered for the types of geo data files that Marble can read and display (like OSM data files, ESRI shapefiles and GPX files). So that such files are nicely displayed in the file manager with matching icon & type description and can get Marble assigned as handler. And:

[...]
Next step is to finish the thumbnail plugin, so one can also have nice previews of the content of these filetypes.
[...]

The thumbnailer was started in the train back from Prague, but Dresden was arrived before it was properly working. Then… the code went out of focus and mind, that step frozen in the middle of the air…

Getting ready to render!

These days, Calligra still has no maps shape plugin. Now and then I nag^Wbeg the Marble developers for the issue (how lame, indeed), as there are other interesting use cases for rendering of map stills, like:

• Creating a jigsaw puzzle for Palapeli, the jigsaw puzzle gaming program
• Creating an animation for a movie in Kdenlive, the video editor
• Yes, even creating the thumbnails for geo data files, if not using the simple installed map theme

Last week luckily earthwings found some personal interest to draft code for renderer state. Time to add a bit of momentum, by reviving the thumbnailer code, to serve as a use case. Still found in its old place, after some bit-rot cleanup and finally solving the problems of last time (different mindset, and things seem so obvious) it works now good enough for serious usage:

Thumbnails for Geo Data files

Some things need some more thinking (which map theme to use when, how to properly detect the sky object the data is for, how to deal with thumbnail generation in offline-state), but it’s a good first complete step :) It does not yet use the new render state feature, but should be soon.

So look forward to Marble 1.9, bringing previews for geo data files to your filemanager and in the file dialogs! (And do not forget to enable them in Dolphin > Settings > Configure Dolphin… > General > Previews, disabled for all types by default).

And perhaps, perhaps, one day we finally also will have a maps shape plugin for Calligra…

BTW, if you want to help development of Marble, other KDE Edu projects and more, consider to donate for enabling the Randa Meetings 2014. Every little amount will help, as it sums up. Best do it now! :)

During the move of KDE’s software projects from Subversion to Git most projects split their subprojects over multiple Git repositories. Calligra did not, but is keeping all code of all apps and extras in one single repository. That is all of the apps Author, Braindump, Flow, Karbon, Kexi, Krita, Plan, Sheets, Stage and Words as well as all of the extras like the file format converter, the Okular generators, file thumbnailers and other file manager integration.
One of the reasons is that many libraries and plugins are shared among the different programs, and the API of the libraries is still changing a lot between releases. By having API-changing commits to be atomic to all of the Calligra code, all developers have less problems to have consistent revisions of the libs and programs.

All can be too much

A downside is: people interested in only one of the Calligra programs still have to get all of the Calligra code and also are faced with possibly having to build all of Calligra. Such people could be developers working only on e.g. Kexi, users wanting to only build the bleeding edge of their favourite program, e.g. Krita, or packagers/integrators only interested in e.g. viewer components for office file formats.

To support the different building needs, by the time more and more all kind of if(SOMEFLAG) [...] endif(SOMEFLAG) were added to random CMakeLists.txt files. And with additionally the conditional building due to optional external dependencies, things got complex. And thus sometimes broken.

Products, Features, and Product Sets

To get things more structured again, the concepts of “product”, “feature” and “product set” have been introduced to describe the stuff that gets build and their internal dependencies:

A “product” is the smallest functional unit which can be created in the build and which is useful on its own when installed. Examples are e.g. libraries, plugins or executables. Products have external and internal required dependencies at build-time. Internal dependencies are noted in terms of other products or features (see below) and could be e.g. other libraries to link against or build tools needed to generate source files. A product gets defined by setting an identifier, a descriptive fullname and the needed internal build-time requirements. Any other product or feature listed as requirement must have been defined before.

Example:

calligra_define_product(BUILDTOOL_RNG2CPP "rng2cpp")
calligra_define_product(LIB_CALLIGRA "Calligra core libs"  REQUIRES BUILDTOOL_RNG2CPP)

A “feature” is not a standalone product, but adds abilities to one or multiple given products. One examples is e.g. scriptability. Features have external and internal required dependencies at build-time. Internal dependencies are noted in terms of other products or features and could be e.g. other libraries to link against or build tools needed to generate source files. A feature gets defined by setting an identifier, a descriptive fullname and the needed internal build-time requirements. Any other product or feature listed as requirement must have been defined before.

Example:

calligra_define_feature(FEATURE_SCRIPTING "Scripting feature")

A “product set” is a selection of products which should be build together. The products can be either essential or optional to the set. If essential (REQUIRED), the whole product set will not be build if a product is missing another internal or external dependency. If optional (OPTIONAL), the rest of the set will still be build in that case.
The products to include in a set can be listed directly or indirectly: they can be named themselves, or another product set can be included in a set, whose products will then be part of the first set as well.
Products and product sets can be listed as dependencies in multiple product sets. As with dependencies for products, they must have been defined before.

Example:

calligra_define_productset(STAGE "Full Stage (for Desktop)"
    REQUIRES
        APP_STAGE
    OPTIONAL
        # extras
        FILEMANAGER
        # plugins
        PLUGIN_DEFAULTTOOLS
        PLUGIN_ARTISTICTEXTSHAPE
        PLUGIN_DOCKERS
        PLUGIN_PATHSHAPES
        PLUGIN_VARIABLES
        PLUGIN_CHARTSHAPE
        PLUGIN_PICTURESHAPE
        PLUGIN_TEXTSHAPE
        PLUGIN_PLUGINSHAPE
        PLUGIN_FORMULASHAPE
        PLUGIN_VECTORSHAPE
        PLUGIN_VIDEOSHAPE
        # filters
        FILTERS_STAGE
)

There are a number of predefined product sets, but everyone can add their own custom product set by adding a file locally in the folder cmake/productsets, named with the name of the productset in lowercase and the extension “.cmake” and containing simply the definition as described above.
The ids of products and features (but not sets) are used to generate CMake variables SHOULD_BUILD_${ID}, which then are used to control what is build and how.

Deciding what to build

The product set(s) to build are passed via the CMake flag PRODUCTSET and are a whitespace separated list of product sets, products and features, though usually just a single product set, e.g. the predefined “ALL”, which is also the default.

Based on the dependency tree that is resulting from the definition of all products, features and product sets, then the internally required products and features are estimated for the requested set.
Following that it is checked whose of those have all needed external dependencies or must be disabled from the build. Finally then the internal dependencies are checked again, and the final set of products and features that will be really built is collected.

Seeing the dependencies

With the knowledge about the internal dependencies available, one is tempted to export this data in a format that can further processed, e.g. to visualize it. And thus now when running CMake, a file product_deps.dot in DOT notation is generated in the top-level build directory. This one can e.g. be transformed on the commandline into a PNG file, by

dot -Tpng product_deps.dot > product_deps.png

The following is generated for me currently when “ALL” products and features should be build (I am missing a few external dependencies for some filters):

Calligra product set “ALL”

If I would like to only build the KEXI and SHEETS product sets, by passing -DPRODUCTSET="KEXI SHEETS" to CMake, the graph changes to this, showing that only the products will be built which are required or optional in the dependency trees of both product sets:

Calligra product set “KEXI SHEETS”

More ideas

Besides creating pretty graphs to look at to get a (better) picture, other use cases might be possible:

• packagers could get some template file created for the packages they would create from all of Calligra
• Libs which are dependencies to other libs or app products could be automatically added to the target_link_libraries and their headers dirs to the include_directories
• CI build servers could only build those products and features which would be affected by the new commits

Then a use-case seems to be that people would like to select a pre-defined product set, but blacklist a few of the products/features. Support for that still has to been developed and done.

I wonder how much of all this might make sense enough to be moved into CMake itself. Currently though this whole system still needs to prove its usefulness by being adapted in more detail by all of Calligra, not only most parts. There is also a chance of having over-engineered things :) And instead Calligra should be simply split over multiple repositories as well. Not sure.

Be enlightened and inspired

This blog post basically should explain a little what all this product sets stuff is about to both all Calligra contributors who have yet not looked into details as well as externals with perhaps similar problems as the Calligra project.

If this approach could be also a solution for you, have a look at the macros in cmake/modules/CalligraProductSetMacros.cmake, they should be reusable outside of Calligra as well, only the new macro calligra_product_deps_report has Calligra specific code inside and could be made generic as well, if there is interest.

Release Notes: Support for multi-column type converters for MongoDB, Neo4j, and HBase was added. Column length control for UUID support was added for RDBMS. A bug was fixed in the L2 cache update procedure for a transaction rollback. HBase internal API usage was upgraded to support current versions of HBase. The generic table/column model used by other datastores was adopted for RDBMS support also. The Eclipse plugin was fixed. Assorted fixes and minor improvements were also made.

Release Tags: JDO, JPA, Persistence

Tags: Database, API, ORM, JDO, JPA, Persistence

Licenses: Apache 2.0

You might know that Okular has a plugin system, for adding support for more document formats. And you might know that Calligra since years also provides a plugin to Okular, which adds support to view slides from files in the OpenDocument Presentation (ODP) format. And not only for the ODP format: by simply using the Calligra import filters for PPT and PPTX you can also view the slides locked away in those formats.

The different apps of Calligra used to be built on the KParts system, so any files in formats supported by them would be also viewable in KPart-embedding programs like Konqueror or KDevelop. But due to the currently on-going creation of a new MVC-oriented foundation for the Calligra programs this has changed, the Calligra modules are no more KParts.

Now I happen to now and then read ODT files directly in KDevelop. And Okular has some native support for ODT. But it is not as powerful as what I am used to from Calligra, so I wanted that back. The best thing of course would be to write a Calligra-plugin directly for KDevelop (like done for the Okteta integration). But I wanted something quicker, and with less work. Writing KPart-wrappers around the Calligra modules would have been the next option. But then I remembered that Sven, who did the Calligra ODP generator plugin for Okular, had also once started an ODT generator, but left it in a branch. And Okular also has an UI optimized for document reading. So a commit cherry-pick and some bit-rot fixes later I had ODT files nicely displayed in KDevelop again, thanks to the chain Calligra Words engine -> Okular KPart -> KDevelop :)

See here Okular displaying the ODT 1.2 spec, of course in ODT format:

And like the ODP generator plugin adds support for PPT and PPTX by simply using the existing filters, the very same is possible with the ODT generator plugin and the import filters for documents locked away in DOC and DOCX formats. A link on a webpage to a file in DOCX format? Click it and view the file directly in the Okular KPart, powered by Calligra’s ODT generator plugin and DOCX import filter:

Currently the generators are just simple rendering ones. Of course we want the generators to be proper extended ones, including TextPage support, so you get all the comfort as you have when reading PDF files in Okular. Come and join the coding fun: navigate your editor to the generators’ code in extras/ and your browser to the excellent Okular generator Howto.

You instead would like to extend the support to other formats that Calligra has import filters for (Stage, Words)? Then take a look at the commit which added support for DOC and DOCX: just adding desktop files, per format one for the Application, one for the KPart and one for the generator itself :)

Waiting for your review request on the Review board (group “calligra”)!

Your preferred format is not yet there? Consider adding it, e.g. by joining the Document Liberation Project and adding also a import filter to Calligra.

So look forward to Calligra 2.9 later this year, bringing a better ODT viewer and some for DOC & DOCX to an Okular near your fingertips :) And perhaps more, at your will!

GSoC 2014 is running and I am working now on implementing outliner for Calligra Author.

Calligra Author is a special application to make creating novels and textbooks, those are usually big in size and require some specific tools during their creation to help their authors. As planned, Author should support 4 basic steps of developing a book:

1. Concept and Planning
2. Writing
3. Review
4. Publishing

Section boundary indication.

Section handling in action

Release Notes: Performance for some cases was improved by 60%. Support for USING was added to Cassandra schema generation. Support for some PostgreSQL types was improved. Support for more java.time classes was added. Many bugs and minor features were fixed.

Release Tags: JDO, JPA, Persistence

Tags: Database, API, ORM, JDO, JPA, Persistence

Licenses: Apache 2.0

And again, another KDE-AppStream post If you want to know more about AppStream metadata and why adding it to your project is a good idea, you might be interested in this blogpost (and several previous ones I wrote).

Originally, my plan was to directly push metadata to most KDE projects. The problem is that there is no way to reach all maintainers and have them opt-out for getting metadata pushed to their repositories. There is also no technical policy for a KDE project, since “KDE” is really only about the community right now, and there are no technical criteria a project under the KDE umbrella has to fulfill (at least to my knowledge, in theory, even GTK+ projects are perfectly fine within KDE).

Since I feel very uncomfortable in touching other people’s repositories without sending them a note first, I think the best way forward is an opt-in approach.

So, if you want your KDE project to ship metadata, follow these simple steps:

1. Check if there already is metadata for your project

That’s right – we already have some metadata available. Checkout the kde-appstream-metadata-templates repository at Github. You can take the XML file from there, if you want. Just make sure that there are no invalid tags in the description field (no <a/> nodes allowed, for example – the content is not HTML!), check if you have an SPDX-compliant <project_license/> tag, check if the provided public interfaces in the <provides/> tag match your project and also test if the URLs work.

Then you can copy the modified AppStream metadata to your project.

2. Write new metadata

How to write new metadata is described in detail at this TechBase Wiki page. Just follow the instructions.

In case you need help or want me to push the metadata to your project if you don’t have the time, you can also write me an email: matthias [{AT}] tenstral . net – or alternatively file a bug against the Github project linked above.

Don’t forget to have CMake install your shiny new metadata into /usr/share/appdata/.

All metadata you add to your project will automatically get translated by the KDE l10n scripts, no further action is required. So far, projects like Ark, K3b and Calligra are shipping metadata, and the number of AppStream-aware projects in KDE is growing constantly, which greatly improves their visibility in software centers, and will help distributions a lot in organizing installed software.

If you have further questions, please ask! .-)

KDE's first update of its 4.13 series of Applications and Development Platform is now available in Chakra's stable repositories.

This release was targeted at updating translations and bug fixing, with over 50 bugs being resolved. According to the official release announcement, these "include improvements to Personal Information Management suite Kontact, Umbrello UML Modeller, the Desktop search functionality, web browser Konqueror and the file manager Dolphin".

In addition, this move includes:

‎Standardization: added references and parts

This is the third (and last but one) update to the 2.8 series of the Calligra Suite, and Calligra Active released to fix recently found issues. The Calligra team recommends everybody to update.

Issues Fixed in This Release

General

• Add support for line breaks when reading the OpenDocument Format.
• Fix a bug in the style manager for filters.
• Make the text shape (specifically the dock panel) work on Windows 8.1.
• Excel document support: add support for 1904-based XLS files, typically created by Excel on Mac OS X.

Kexi

• Fix crash when closing form design window and color property isn’t saved. (bug 333551)
• Make context menu work for tables in Query Designer.
• Make it possible to set solid and inherited background color in form widgets. “Auto Fill Background” property has been added for this. It’s set to true when user selects background color. (bugs 333548, 333549)
• Fixed crash in Debug GUI when trying to modify table field. (bug 333933)
• Updated list of SQL keywords reserved for KexiSQL, SQLite, MySQL and PostgreSQL.
• Make it possible to use reserved words as names for table columns. (bug 332160)
• Fix crashes when closing tabs or windows (bug 334234)
• Removed a number of resource leaks.

Krita

• Translation fix in the Multihand tool: Axis -> Axes.
• More precise translations. (bug 333135)
• A fix for the outline of invert selection.
• Krita no longer closes immediately when a file is corrupted but gently warns the user.
• Fix bug: deleting Group-Layer and contents when there are no other layers prevented user from creating new layers (crash). (bug 333496)
• Improved detection of supported image formats.
• Removed a number of resource leaks.
• Add support for selection in GMIC filters. (bug 325771)
• Fix crash when GMIC filter is applied to the layer which was moved. (bug 327980)
• Add search box for filter names of the GMIC plug-in. A text box below the filters tree can now be used to find the GMIC filter by name.
• Select only paint layers when gathering all Krita layers from layer stack.
• Remember the last used preset across sessions.
• Fix invalid recalculation of width and height between units.
• Ensure that the channel flags are always reset when they are set to full, otherwise compositing will not be will work not efficient. (bug 333080)
• Fix painting grid on lower zoom levels. (bug 333234)
• Fix a triangular brush outline for a 1-pixel brush. (bug 334408)
• Fix pixel-alignment of the Rectangle and Ellipse tools, perform alignment exactly how the user expects. (bug 334508)
• Make layer actions such as “Delete the layer or mask” listed in the in Configure Shortcuts dialog. (bug 332367)
• Fix handling a tablet when “Pan/Scroll” mode is assigned to a button. Note: Wacom’s “Pan/Scroll” feature supports only vertical wheel scroll, so using usual Middle-button panning is recommended. (bug 334204)

GMIC filter search in Krita

Calligra Words

• Export to text: fix missing number of spaces in the text:s tag.
• Export to text: fix missing newlines after a heading.
• Export to text: add support for line breaks.

Try It Out

• The source code is available for download: calligra-2.8.3.tar.xz.
• Calligra binaries are available for download for many operating systems.

About Calligra

Calligra Suite is a graphic art and office suite. It is a part of the applications family from the KDE community. See more information at the website http://www.calligra.org.