Comments on: First attempts on improving the Zeitgeist DB before GUADEC

By: Seif Lotfy

Seif Lotfy — Thu, 04 Jun 2009 08:44:14 +0000

@Milan Bouchet-Valat
We already intend to use the .desktop files for applications as it already contains the executable as well as other info.
renamed/moved files will be monitored by a new process i will introduce to zeitgeist that will update the info manually thus i wanted to give the data ID’s instead of using their path.
I think i will use one table for all apps to work out the issues.

We should go through the DB again. Let us talk online

By: Milan Bouchet-Valat

Milan Bouchet-Valat — Thu, 04 Jun 2009 08:38:00 +0000

For what I really understand, you design sounds good to me. A few details though:
- to identify applications, we just agreed yesterday on #gnome-shell that we should use the full path to a .desktop file – rather than the executable. (Jason Smith is going to add put code into gnome-settings-daemon to add a window manager hint on every window that would store that information. But more on that on IRC if you want.)

- to identify documents, I guess you could use a hash in addition to the path. I’m not sure whether indexers do that already, but that allows you not to lose track of a renamed/moved file, which would be silly. Though I’m not a specialist, I let you weight the pros and cons.

- any reason why timestamps are float in your model? To me, there are always integers…

- while giving every app its own table if needed can help integration, I think you should avoid as much as possible to do so. You could just see with UNR what they need, and create the DB for that, so that Zeitgeist itself knows how to deal with this data. And you could also list the cases where external apps would need their own table, and try to give them what they want in your original design.

And I’m particularly happy since my desktop contexts can now be considered as tags, which makes the whole thing cleaner!

By: James Henstridge

James Henstridge — Thu, 04 Jun 2009 08:11:04 +0000

A few comments on the schema itself:

1. It is common to just name the primary key of a table just “id”, and foreign keys refering to that table as “$table_id” or just “$table”.

2. Use a consistent naming scheme for tables. I’d suggest the singular form of what rows represent. For linkage tables, use a combination of the two table names. So that’d involve renaming “tag_ids” to “tag”, “tags” to “data_tag” and perhaps “timetable” to “event” and “data” to “uri”.

3. In some cases where you’ve specified “{KEY}” for multiple fields it is clear that you are using a compound primary key (e.g. the “tags” table), but in others you don’t (e.g. for the “data” table you probably don’t want two rows with the same “uri” value but different “uriid”). Perhaps specify “{UNIQUE}” or similar where you just mean that a field value is unique but not part of the primary key?

4. As you seem to be using sqlite for storage, you probably want a simple integer primary key for every table. Sqlite always includes such a key, and will try to name it to match a field in your schema. So add a primary key to “tags” (you don’t want it to treat “tags.tagid” alone as the primary key) and “timetable”.

By: James Henstridge

James Henstridge — Thu, 04 Jun 2009 01:50:09 +0000

I’m not sure I understand why you are “giving every registered application its own table”. If they all have the same schema as described by your diagram, why not just have a single table with foreign keys to both the “app” and “data” tables?

Also, you might want to consider using an ORM to make access to the database easier. Storm might be a good choice (disclaimer: I am a Storm developer).

By: Seif Lotfy

Seif Lotfy — Wed, 03 Jun 2009 21:46:51 +0000

@Christian Hergert agree we will need to have a migration schema! Using JSON to transfer from ver 2 to ver 3 is a good deal

By: Christian Hergert

Christian Hergert — Wed, 03 Jun 2009 21:42:34 +0000

I guess to clarify, this is where using a serialization format such as JSON or pickle and shoving in a b-tree really helps. Because you can just add a version tag to the object. And then have object migrations that translate from a ver2 to a ver3 object and vice-versa.

By: Christian Hergert

Christian Hergert — Wed, 03 Jun 2009 21:41:00 +0000

Obviously you have a plugin contract, but what happens when you find that you need to change your internal storage schema? How are you going to handle upgrading the database running on users desktops to your new schema without deleting all of there existing information?

By: Seif Lotfy

Seif Lotfy — Wed, 03 Jun 2009 21:18:35 +0000

@Christian Hergert
The idea is that the applications will be sending its data
so when the application changes it has to change its plugin to send the data in the format we accept

By: Christian Hergert

Christian Hergert — Wed, 03 Jun 2009 21:10:15 +0000

Do you have a plan to be able to migrate your schema as the application changes? I’ve yet to find an application that didn’t have to adjust it’s sql schema as they add new features and refactor over time. It’s why many still consider using basic b-trees and multiple indexes.

Eager to use it, Cheers