I've blogged about some of the more prominent changes in this new Nepomuk release. I thought it would be a good idea to document all the changes, most of which I haven't publicly blogged about.

File Indexing

As the release announcement has been saying, the file indexer has undergone the maximum number of changes.

New Double Queue Architecture

We've split the working of the indexer into two parts - The first basic indexing and second full file indexing. The basic indexing quickly indexes the basic information about the file such as the filename and mimetype. This allows us to always at least answer simple queries. The other queue, which is only run when the user is idle, extracts the full information about the file.

New File Indexer

We've had some problems with Strigi earlier. With 4.10, we have finally decided to release our own solution. Our solution is arguably technologically inferior, but it's more maintainable and, for now, provides a better user experience.

Mimetype Filtering

One of the advantages of moving to this new file indexing architecture is that mimetypes are a very important part. All of the file indexing plugins use mimetypes to identify which types of files they can index. With this, we decided to allow the user to control the type of files that are indexed.

By default, source code is now no longer indexed. Common stuff like Documents, Images, Audio and Videos are.

KioSlave changes

Till the 4.9 release, the kioslave code hadn't changed much. With 4.9.1, we managed to optimize some of the code. The 4.10 release however takes this to an entirely different level.

Massive Optimizations

The 'nepomuksearch' tagging slave could initially show both non-file and file data. This means that it would also occasionally show contacts, albums and other details. Selecting any of those would result in another search for resources related to that contact. For this release, we decided to optimize for the most common use case of listing files.

The 'nepomuksearch' kioslave, and all other nepomuk kioslaves, now no longer show any result which does not have a URL. This coupled with a LOT of other optimizations, has now yielded a super fast kioslave which can display thousands of results in under a second.

There is also some interesting userbase documentation about custom queries on the nepomuksearch kioslave.

Tagging KioSlave

As previously stated, we are also introding a new tagging kioslave. This slave allows you to easily manage you Nepomuk tags, and browse files based on the different tags it contains.

File MetadataWidget

One of the largest part of the Dolphin Information Panel was the KFileMetadataWidget which was provided by kdelibs/kio. This widget was one of the last parts of Dolphin that still used Nepomuk1. Since kdelibs was frozen, we couldn't port it to Nepomuk2. Thus emerged the Nepomuk2::FileMetadataWidget in nepomuk-widgets.

The KFileMetadataWidget historically fetched all the data in another process. This was done because Strigi was a little unreliable. With KDE Workspaces 4.10, we are no longer using Strigi in Nepomuk. This means the widget now uses the nepomukindexer, to extract the data. It also no longer uses this multi-process architecture when loading the Nepomuk data. This result in a massive performance improvement cause we can rely on Nepomuk cache in Dolphin, instead of recreating it each time.

In terms of appearance, the widget has become a little more uniform, and by default only shows the properties that really matter.

Nepomuk Cleaner

The Nepomuk Cleaner originated from a series of scripts I was writing to clear up my own database. It eventually occurred to me that other people might suffer from the same problem. The scripts were eventually combined into a cohesive form, and released. The application is very simple right now, but that will change in future releases. I even contemplated not releasing it for 4.10, but it clearly provides some value, even if it doesn't look that great.

Nepomuk has long required a convenient way of managing tags. I've previously tried this with a simple Tag Managing Application, but that wasn't something that we wanted to ship. For the KDE Workspace 4.10 release, we are releasing a Nepomuk Tags kioslave.

Listing Tags

The kioslave provides a very convenient way of listing all the tags. You can even rename and delete tags, just like you would for any other folder.

Browsing Files

Nepomuk has always provided users a way to browse tags, but it was only one tag at a time. This seemed fairly limiting. Once could browse by more tags, but then you would have had to write the query yourself.

With this kioslave, you can finally browse the files based on the tags, and then filter the search even more by selecting more tags.

Nepomuk has a unique problem of maintaining an RDF store. Unlike traditional SQL based stores, RDF offers a very loose schema, which is a HUGE advantage. Unfortunately all of the current RDF stores do not support any form of schema enforcement. It's up to the client code to make sure that the data being pushed is valid.

This has resulted in a number of problems such as strings being stored where an integer should go.

With the KDE Workspace 4.7 release, we started employing our own form of schema enforcement in the Nepomuk Storage Service, but the old incorrect data still remains. Also, as Nepomuk has evolved as a project, we have found better ways to store data. Since the schemas are so loose, we could easily store both the old and the new data without any problems on the database level. This obviously results in more complex client code which has to handle both legacy and new data.

For this release, we decided to clean up the code to a certain extent and stop supporting some of the legacy data. We also decided to ship a very basic application called the "Nepomuk Cleaner".

This application is responsible to port any legacy data, clear up incorrect data, and merge duplicate data. We recommend that all users run it at least once. It will result in a performance upgrade of all areas of Nepomuk, including a significant impact in the indexing speed of emails.

With the 4.11 release, we're planning to improve the interface, add more cleaning jobs, and make running this application mandatory. That way we can safely remove all the legacy code paths.

Over the last 6 months, while working for Blue Systems, Nepomuk has undergone a number of changes. The most public and noticeable change has been the major refactoring of the file indexer. One large part of this has been the migration from Strigi. The other large part is the introduction of 2 phase indexing.

With 4.9 and earlier, the file indexing service used to just have one queue, whose speed could be controlled. This queue was filled on startup by comparing the mtime of the file with the one stored in the database. This would involve scanning through all the indexed folders. Once the scan was complete it would also listen to the file watcher to be notified when a file is modified or created.

This architecture had some shortcomings -

• Indexing each file is a time consuming process, and it involves extracting and pushing large amounts of data in Nepomuk.
• Since this process was slow and we did not want to annoy the users, artificial delays were introduced which were changed based on if the user is idle.
• The entire indexing process was suspended when on battery
• Faulty files which cannot be indexed do not have any information stored, and could not even be searched by filename

With this new release, we have split the indexing into 2 parts - Basic Indexing and File Indexing. The basic indexing just extracts the stat information and mimetype of the file. Whereas, the file indexing actually extracts data from the file.

This basic indexing is always enabled, and is very fast. It can process around 10-20 files per second. Also, it consumes very little cpu. Extracting this basic information first allows us to search on the basis of type, file and enabled the timeline kioslave to work properly.

The file indexing is the relatively heavy process that is only run when the user is idle, by default.

This two phase architecture allows us to still index all the files, while providing a relatively light burden to the user. It also allows us to provide finer control than a simple on/off switch. For example - Now when on battery, file indexing is disabled, but the simple indexing still continues.

This new approach will also allow us to provide more user feedback in future releases, such as an indexing progress bar.

Strigi has always been a large part of Nepomuk. In fact a lot of users still do not understand the difference between the two. It's quite common to see bug reports saying mentioning "Strigi/Nepomuk". Lots of blog posts do the same.

Strigi consists of a number of different parts. In Nepomuk we just used to use libstreams and libstreamanalyzer. These were pure C++ libraries. The great thing about Strigi is that it is based on streams, instead of files. So one can theoretically even extract metadata from the album image embedded inside an audio files. It's very powerful. Unfortunately, everything comes at a price, and this increased "awesomeness" comes with increased complexity. Additionally with it being a pure C++ ( no Qt or KDE ) library, contributing is harder.

For 4.10, We decided to take a very drastic change and move away from Strigi. There are a large number of reasons for doing so. Apart from the technical ones there was also an economic one - A large code base like Strigi is difficult to maintain and comes with a lot of added complexity.

Our own solution is based only on files (not streams) thereby making it a lot simpler. It directly uses the Nepomuk and KDE libraries, thereby making integration very simple. Integrating Strigi in Nepomuk required a lot of code.

This new file indexer currently resides in the nepomuk-core repository and does not have a public interface. I'm currently still debating if it should be public for 4.10. Write about if it gets a public interface, one can theoretically write plugins in other languages.

So far we have 5 indexers -

• Image File - Based on Exiv2
• Video Files - Based of ffmpeg (We might move to gstreamer)
• Audio Files - Taglib
• PDF Files - Poppler
• Plain Text files

Writing file indexers for Nepomuk is now very simple. In fact these 5 indexers combined are just 500 lines. Here is the important part of the plain text extractor -

QTextStream ts( &file );
QString contents = ts.readAll();

int characters = contents.length();
int lines = contents.count( QChar('\n') );
int words = contents.count( QRegExp("\\b\\w+\\b") );

SimpleResource fileRes( resUri );
fileRes.addType( NFO::PlainTextDocument() );
fileRes.addProperty( NIE::plainTextContent(), contents );
fileRes.addProperty( NFO::wordCount(), words );
fileRes.addProperty( NFO::lineCount(), lines );
fileRes.addProperty( NFO::characterCount(), characters );

The current file indexers cover most of the commonly used files, but they still need to be polished. So, if you're interested in contributing to Nepomuk, here is your chance.

I've managed to catalog some of the different files that I know we support. Our current indexers support many more formats, they just need to be properly tested.

If you're interested in helping, you can start by running nepomuk-core, and manually indexing the different file formats and updating this page. If you're a developer, feel free to checkout nepomuk-core, and start writing extractors. I've written a simple guide.

Btw, all of this Nepomuk awesomeness is powered by Blue Systems!

Last week 4.9.1 was tagged, and it should release any day now. A large part of my time was spent bug fixing and stabilizing some of the most user visible features. This blog post highlights some of the more user visible changes.

File Indexer

The File Indexer has also gone through a number of fixes -

Do not check for new files on startup

One of the most annoying things I found about file indexing was the initial check for new files on startup. Even though this check runs silently in the background, it was very annoying. Specially for people who run Nepomuk all the time.

Now, we only check if the strigi version has changed, and only then do we check for new files / check for any files that could not be indexed the last time. If someone wants to forcibly check for new files, they can do so with a dbus call -

$ qdbus org.kde.nepomuk.services.nepomukfileindexer /nepomukfileindexer updateAllFolders false

Or wait till 4.10, when I add an option to do so in the KCM.

Secondary Indexer

This patch technically made it into 4.9.0, but I never got the chance to blog about it. I've introduced a SimpleIndexer which serves as backup when the strigi indexers provide incorrect data. This way instead of having no data about the file we at least have the basic information such as filename, url and mimtype. I have plans for a more concrete solution for 4.10, but that's for another blog post.

Nasty Deadlock

Another annoying bug that was not caught with 4.9.0 was a deadlock that rendered the file indexer service useless.

This only happened during the first run of Nepomuk, and led to some unfortunate publicity. In in the end it was a very simple fix once the proper backtrace had been provided.

Strigi Analyzers

Another big change which probably has a large impact on indexing times is the blacklisting of certain file indexing (strigi) plugins. Currently by default, we only blacklist the SHA1 hash generator. With it blacklisted, we do not (hopefully) need to read the full contents of the file. This results in a noticeable performance improvement for large files.

Queries

File Queries

One of my favorite features of Dolphin is their Places Panel with all of those defaults - Documents, Images, Audio and Video files. Unfortunately, those defaults were normal Nepomuk queries and not file queries. With 4.9.1, they have been changed to file queries which lets them access a number of optimizations on our end, and allows them to benefit from my minor optimizations in file queries.

Query Updates

Most applications which currently use Nepomuk utilize the QueryService in order to run queries. The Query Service can then provide updates on those queries if there is some change in the results - It used to do so by running ALL open queries when any data in Nepomuk changes. This was not a good approach, as it obviously does not scale.

With 4.9.1, we are now using some simple heuristics and the Resource Watcher to only re-run queries when affected data changes. So indexing a file will not result in the updating of all custom Nepomuk magic folders.

With KDE 4.9, we introduced a new repository called nepomuk-core. This contained a combination of kdelibs/nepomuk and kde-runtime/nepomuk. It was created because of the API freeze present in kdelibs. Considering that most of the client libraries are thin wrappers over the runtime components, it made sense to combine them in one repository..

In order to be compatibile with kdelibs, the new library is installed with the Nepomuk2 namespace.

Now with KDE 4.10 we are going to have another new repository nepomuk-widgets. This repository contains the remaining GUI parts of kdelibs/nepomuk that were not moved to nepomuk-core.

I have been working a lot to improve Nepomuk's technical documentation on the techbase. And it now finally reached the point where I think it covers most of the major aspects of Nepomuk. If you're using Nepomuk in your applications, you should read it.

It's a little hard for me to objectively state if I've explained stuff properly considering that I have been so involved in the development. If you feel some section is lacking or you're having trouble understanding it, please contact me. I'll be happy to update it.

And with this, I can (somewhat) tick off one item from the Nepomuk BOF todo list.

Nepomuk has a very decentralized architecture where the different components exist as different processes. They are all variants of the same executable - nepomukservicestub. This servicestub loads appropriate service plugin. The main reason for doing this was stability. If one of the components crashes, then it doesn't take all the other components with it.

Unfortunately this architecture doesn't hold very well when the different components need to communicate with one another. In that case they need to use complex methods such as dbus or local sockets. Another problem is the increased memory consumption cause each process has its own internal cache (Nepomuk stuff) and other KDE specific stuff.

If you ignore file handling in Nepomuk, we have two main services -

• Storage Service
• Query Service.

The Storage Service is responsible for managing the ontologies, initializing virtuoso, and other data management functions. The QueryService exists for caching queries and running them in a separate thread.

Now the Query Service obviously need to access the virtuoso database, and for that it needs to go through the storage service. This communication happens through a local socket. The same socket which all other applications use to access Nepomuk.

Last week, I finally merged the query service into the storage service.

I was aiming for a small memory decrease, and a slight performance upgrade on the queries. Boy, was I wrong! The additional local socket seems to have been a huge bottleneck.

Here are some benchmarks listing about 12,500 resources.

There are still many more performance upgrades that can be done, but this seemed like a good place to start :)

Most people assume that if they switch off file indexing in Nepomuk, then all the nepomuk file services will get disabled. This is however not the case. Nepomuk consists of two services which are used to deal with files -

• Nepomuk File Watcher
• Nepomuk File Indexer

The Nepomuk File Indexer is responsible for calling the strigi plugins to index the files, whereas the FileWatch service is a general service that monitors file move, creation and deletion events. Even when the File Indexer does not exist, files may have metadata attached to them - Tags, Rating and Comments. We need the File Watcher to update our database whenever the url of a file changes.

The File Watcher internally uses a kernel API for file monitoring - inotify. This API, while quite easy to use, does not allow us to recursively watch directories, and more importantly, does not provide file move events unless we are watching both the source and the destination directory.

We need file move events in order to track a file's url. This results in us having to create inotify watches for every single directory in your $HOME folder. This causes a large disk load on startup and is the cause of one of the critical bugs in Nepomuk. And we have no solution, until the kernel provides us with a better API.

Anyway, Nepomuk is being used in KDE PIM and Telepathy (development version), and none of those use cases have anything to do with files. It doesn't make sense to subjugate others to pay the price of the file watcher, when they are not doing anything related to files. So, with that in mind, please add the following lines to your nepomukserverrc, if you do not care about files at all -

[Service-nepomukfilewatch]
autostart=false