Comments on: Candidate for the new Zeitgeist DB http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/ I like potato chips Sat, 31 Jul 2010 07:52:05 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Leo Sauermann http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2613 Leo Sauermann Wed, 10 Jun 2009 15:08:37 +0000 http://seilo.geekyogre.com/?p=686#comment-2613 hey, fyi from the "freedesktop" perspective beyond the current task at hand: the situation here is similar to what happened in kde 4.0 in 2007/2008. There, the NEPOMUK backend was added as a metadata/search store with similar goals as tracker has. Also the technology is very similar (nepomuk's store is also rdf+sparql+nepomuk, as is tracker). The result is that on KDE the performance of the store is "ok" and is currently improved to "better", but many applications are now moving to the new metadata store. so thinking about this early can help two years later, when a whole desktop/mobile platform can be integrated. also, if the db is too slow, its possible to combine it with a relational db, such as openlinksw's open source virtuoso store does (hardcore combination of sparql and sql in one query language and store implementation) hey, fyi from the “freedesktop” perspective beyond the current task at hand:

the situation here is similar to what happened in kde 4.0 in 2007/2008. There, the NEPOMUK backend was added as a metadata/search store with similar goals as tracker has. Also the technology is very similar (nepomuk’s store is also rdf+sparql+nepomuk, as is tracker).
The result is that on KDE the performance of the store is “ok” and is currently improved to “better”, but many applications are now moving to the new metadata store. so thinking about this early can help two years later, when a whole desktop/mobile platform can be integrated.

also, if the db is too slow, its possible to combine it with a relational db, such as openlinksw’s open source virtuoso store does (hardcore combination of sparql and sql in one query language and store implementation)

]]> By: Philip Van Hoof http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2609 Philip Van Hoof Sun, 07 Jun 2009 08:33:58 +0000 http://seilo.geekyogre.com/?p=686#comment-2609 Seif, Natan: that's great to hear. Thanks. Seif, Natan: that’s great to hear. Thanks.

]]> By: Natan Yellin http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2605 Natan Yellin Sat, 06 Jun 2009 19:25:45 +0000 http://seilo.geekyogre.com/?p=686#comment-2605 Hi Philip, As Seif said, we're very excited by the work that's been done with tracker lately, but we don't want to lock ourselves into one backend. (Especially not before Tracker 0.7 is released.) In another two weeks, as soon as my exams are over, I'm going to get back to working on the Tracker backend. We'll definitely consider using it by default on computers where Tracker is already installed. Hi Philip,
As Seif said, we’re very excited by the work that’s been done with tracker lately, but we don’t want to lock ourselves into one backend. (Especially not before Tracker 0.7 is released.)

In another two weeks, as soon as my exams are over, I’m going to get back to working on the Tracker backend. We’ll definitely consider using it by default on computers where Tracker is already installed.

]]> By: Seif Lotfy http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2600 Seif Lotfy Sat, 06 Jun 2009 13:15:02 +0000 http://seilo.geekyogre.com/?p=686#comment-2600 Again at this point we have some depanents. So until October we have to stick to what we know. Dependng on tracker at the moment even the storage is to risky. Especially that we have a gsoc project running and some deadlines. The current DB will help a merge later. But please understand that full dependencies are not in our interest at he moment. We do intend to work with you guys just else we would not have Natan working on the optional tracker backend. Again at this point we have some depanents. So until October we have to stick to what we know. Dependng on tracker at the moment even the storage is to risky. Especially that we have a gsoc project running and some deadlines. The current DB will help a merge later. But please understand that full dependencies are not in our interest at he moment. We do intend to work with you guys just else we would not have Natan working on the optional tracker backend.

]]> By: Philip Van Hoof http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2598 Philip Van Hoof Sat, 06 Jun 2009 11:20:09 +0000 http://seilo.geekyogre.com/?p=686#comment-2598 Note that in branch tracker-store we are working on separating the storage and query engine from the indexing, crawling and monitoring. This means that the tracker-store service will be separately packagable and wont even have any inotify things running. It's merely a SPARQL+Nepomuk, and SPARQL UPDATE+Nepomuk serving service that'll be activatable by DBus and that can even be taken down during unactivity. Which means that you can see tracker-store as a database. Except that instead of SQL and a normalized schema, it talks SPARQL, RDF and Nepomuk. It means that it's designed to cope with truly large amounts of data within a mobile setting (we decomposed, or denormalized, our storage for better performance - we opted for fast query, but a bit slower storage -). Meanwhile its being designed specifically for mobile purposes. This of course doesn't mean that Tracker ain't useful for a desktop. It just means that our (Nokia) team's focus is a mobile one. For GNOME I'm not sure why you'd still want your own store. I'd just opt for Tracker, personally. That way you can focus on your creativity and ideas for Zeitgeist instead of on the boring tasks of storing things yourself. Zeitgeist should for example start using SPARQL internally too. You probably don't want to implement SPARQL yourself. As for raw throughput speed I have been experimenting with a light unix-socket IPC mechanism which allows you to INSERT data at a throughput rate of 100,000 items per 1.3 seconds (scales linear). Ensured storage (with the COMMIT) is of course a bit slower than this throughput, but we'll enqueue your store requests in a queue (and we call you back when final storage and COMMIT have succeeded). Take a look at the branch tracker-store-ipc for that experimental stuff. Note that in branch tracker-store we are working on separating the storage and query engine from the indexing, crawling and monitoring. This means that the tracker-store service will be separately packagable and wont even have any inotify things running. It’s merely a SPARQL+Nepomuk, and SPARQL UPDATE+Nepomuk serving service that’ll be activatable by DBus and that can even be taken down during unactivity.

Which means that you can see tracker-store as a database. Except that instead of SQL and a normalized schema, it talks SPARQL, RDF and Nepomuk. It means that it’s designed to cope with truly large amounts of data within a mobile setting (we decomposed, or denormalized, our storage for better performance – we opted for fast query, but a bit slower storage -).

Meanwhile its being designed specifically for mobile purposes. This of course doesn’t mean that Tracker ain’t useful for a desktop. It just means that our (Nokia) team’s focus is a mobile one.

For GNOME I’m not sure why you’d still want your own store. I’d just opt for Tracker, personally. That way you can focus on your creativity and ideas for Zeitgeist instead of on the boring tasks of storing things yourself. Zeitgeist should for example start using SPARQL internally too. You probably don’t want to implement SPARQL yourself.

As for raw throughput speed I have been experimenting with a light unix-socket IPC mechanism which allows you to INSERT data at a throughput rate of 100,000 items per 1.3 seconds (scales linear). Ensured storage (with the COMMIT) is of course a bit slower than this throughput, but we’ll enqueue your store requests in a queue (and we call you back when final storage and COMMIT have succeeded). Take a look at the branch tracker-store-ipc for that experimental stuff.

]]> By: Seif Lotfy http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2595 Seif Lotfy Sat, 06 Jun 2009 09:01:45 +0000 http://seilo.geekyogre.com/?p=686#comment-2595 Some people don't use tracker! How should they use zeitgeist! This DB module allows us to use tracker as an OPTIONAL backend! Some people don’t use tracker! How should they use zeitgeist! This DB module allows us to use tracker as an OPTIONAL backend!

]]> By: Daeng Bo http://seilo.geekyogre.com/2009/06/the-db-candidate-for-zeitgeist/comment-page-1/#comment-2594 Daeng Bo Sat, 06 Jun 2009 06:27:11 +0000 http://seilo.geekyogre.com/?p=686#comment-2594 What's the reason for creating your own database instead of using, for example, Tracker's? I think tight integration with Tracker would be a big plus. What’s the reason for creating your own database instead of using, for example, Tracker’s? I think tight integration with Tracker would be a big plus.

]]>