As Trever blogged yesterday, the Zeigeist team has been busy with tweaking the DB and the engine. During that process tools and benchmarks have been developed to make the tweaking and testing more interesting. Trever will be blogging about that tomorrow so make sure to check his blog.
Our end goal is trying to scale the engine to be able to handle a few billion events just as fast as it can handle a few hundred thousand. While we are not there yet we managed to have some pretty nice stable results for the first iteration. A lot of results show more than 100% speed enhancement. In other words a lot of queries from our standard benchmarks now consume more than 50% less time to execute. Here are some graphs of our benchmarks.
Green indicates the 0.9 release
Yellow indicates the new trunk
Most notable performance enhancement is querying Zeitgeist with a specified timeframe (from data x to date y).
Same queries with an open timeframe also improved
We also have a copy of the Synapse queries benchmarked
The queries here are typical queries used to extract info from Zeitgeist. So right now the team is really happy with the initial results. For Synapse on my local DB (over a year old), all my synapse queries perform under 0.08 seconds. We still can get more out of this. The trick here was improving our indexes and our sql query generator.
Next month we will be going through another iteration.
I think it is safe to assume that as an open source project, striving to grow our community, we should make first interactions of new contributors with us a pleasant one.
So when commenting on a new contribution or attempt, we should try to either say something positive or don’t say anything.
If the contribution is not useful then sugar coat you criticism. Because while the contribution might be bad, the person who did the contribution showed initiative to help out. We need to harvest this momentum and help integrate the person into the community.
If the contribution is useful then praise the developer. Making him feel useful. I learned from Lydia Pintscher to not only praise good contributors, but grant them autonomy to play around. This would allow them to master their craft, which will lead to them feeling like they have a purpose to stay.
For as long as Zeitgeist has existed it has been hosted on Launchpad. We made use of almost every Launchpad feature and grew to love Bazaar. The tight integration between Bazaar and Launchpad should be a taken as a grand example to the community, and the the team hacking on Launchpad and Bazaar has been always very responsive, supportive and above all incredibly approachable. The Zeitgeist team hosted all of its projects on Launchpad which made our development process very agile. I really need to thank the Launchpad team for hosting the project from its inception.
During the course of the last 3 and half years, Launchpad served us well and allowed us to handle:
ca. 748 unique bugs reported
ca. 231 merges
ca. 53 code contributions
We hope that many other great projects will find a valuable home on Launchpad. Without Launchpad, Zeitgeist would have not reached the level it has now.
But time changes and projects grow. The Zeitgeist team has a stable developer base of minimum 7 people putting several hours a week into Zeitgeist development and deployment. With these efforts comes the need of a vendor neutral platform hosted and serviced in a way that serves the interest of the project best.
Due to the deployment efforts being made to get Zeitgeist integrated with applications, our developers got more and more exposed to git. A stream of developers expressed their desire to work with us, however they were very uncomfortable or unfamiliar with bazaar and Launchpad, due to the popularity of git. That is something we do not understand in the core team, but still something we respect and will act on.
We couldn’t find a good direct replacement for Launchpad honestly, but since Bugzilla is something like the defacto standard for most people we decided to host Zeitgeist on Freedesktop.org (I must admit in-patch commenting in Bugzilla is sweeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeet).
We needed to wait until we finished the 0.9 release of Zeitgeist as well as narrow down the bugs to be less then 10 to make the transition as smooth as possible.
Now that Zeitgeist 0.9 is released, all our development will take place on Freedesktop.org and will be mirrored to launchpad, this means no code will be pushed directly to Launchpad, but rather Launchpad will pull from the git repositories. There the Zeitgeist Ubuntu guys (including me) will be syncing bugs and keeping everything intact. So bugs reported in Launchpad will be ported to FDO. But not the other way around.
We did not turn our backs on Launchpad, its just time for us to remove any barriers to getting the widest adoption of and contributions to Zeitgeist
On the road toward Zeigeist 1.0, FDO will be a great home for the project and we look forward to many great new contributions.
Since the Zeitgeist team is kinda recovering from last month, and no one really feels like blogging, I am here to sum up what has been up the last 2 months. (Also I think a blog post from my side is long overdue)
Privacy Manager
A lot of users are scared of “logging”, and since Zeitgeist is considered an event logger, we ended up having some users trying to disable Zeitgeist all in all breaking Unity searching and so on.
Fact is, if you want searching to be sorted properly, the search provider needs to know what files you recently or frequently used. Also some people want to have some of their stuff logged and others things not (blacklisting folders).
Activity Log Manager which we already had developed in python last year has been ported to Vala. This nifty tool allows you to blacklist application, folders, mime-types from being logged as well as the option to delete parts of your log. The port to Vala happened so we can patch it into the Settings Manager to give it an more integrated feel with the System in Ubuntu and Dawati.
It has received very good reviews and there are some very interesting new development efforts/ideas in the make (right now one in a state of flux which I hope I can get out of soon).
Here are some screenshots of the current stand of ALM
(dialog for deleting history)
(dialog for blacklisting folders and file types)
(dialog for blacklisting applications)
Zeitgeist 0.9
Last year we started porting Zeitgeist from python to Vala. Mostly for the sake of easier deployment as well as improving startup time. As a side effect also memory consumption decreased and during the development more optimizations have been made.
It was a very intense operation sponsored by Collabora and Canonical as well as the awesome unpaid Zeitgeist developers.
So a big thanks goes to Collabora, Canonical, Michal Hruby, Siegfried Gevatter, Manish Sinha, Mikkel Kamstrup, Trever Fischer, Stefano Candori, Moritz Neeb, as well as everyone else who tested Zeitgeist, provided bug reports or contributed in any other way.
We released some alphas and a beta. But now the real deal is out. We still have some nice optimization in the queue that we will release with the next Zeitgeist version. So don’t hesitate to try it out.
I have been absent from blogging a bit
So here are the highlights of the past month:
GNOME Clocks is shaping up nicely, a nice example of prototype becoming product. It is a very nice experience for me in developer <-> designer communication. Something that I have been working on with Allan Day, Jakub Steiner, Garret LeSage and Hylke Bons. (I will wrap up the code and push it up somewhere tomorrow). I will be giving a talk about this experience along with Allan Day (including prototyping) at FOSDEM.
The prototyping spirit within the GNOME design team is growing. After playing with Jakub Steiner’s HTML/JS prototype I got inspired to play with HTML/JS too.
The Zeitgeist team set up a nice TODO list for the next release and beyond (blog post coming soon). Some awesome commit by Michal Hruby managed to reduce the memory consumption by a big chunk by “optimizing allocations of variants”. Our bug list for the next release has only 4 more bugs open so take a stab if you dare…
Thanks to Collabora, I got a N9. After ruining my SIM card trying to build a Micro-SIM I ended up running to O2 getting myself one from there that got activated after 30 minutes. Now I am playing around with QML again.
I started playing with Plasma Active and have very mixed feelings about it. Tech wise its pretty good, but I don’t find it intuitive enough like GNOME 3 or Unity on a tablet. I seem to be missing the concept of activities really.
I started getting into XBMC and now Ubuntu TV is announced… will be trying it out soon.
I decided to prototype an extensions based on Jimmac’s mockups. I still need to fix some minor issues and add animations. It is not perfect yet and is a bit different from the original design:
Applications Icon is on the top of the bar instead of the bottom (seemed more comfortable for me)
Searchbar is centered (I think it was Hylke Bons who told me to do that a while back)
but I am still working on it and hopefully can upload it to extensions.gnome.org soon, so we can get feedback! I will leave you with some screenshots and a video for now…
This month Collabora has been sponsering zeitgeist for one year, here is the story about how that came to be and what happened during that year.
How I got in contact with Collabora?
I met Rob McQueen, Phillippe Khalaf, Sjoerd and some from the Telepathy team during Gran Canaria Desktop summit. I needed help with an idea (Teamgeist) and a couple of months later I was introduced to Youness Alaoui (until this day I consider a very good and close friend) as well as Sumana Harihareswara (from whom I learned a lot about organizing).
How I got sponsored?
Collabora saw Zeitgeist as a promising technology and observed the development for a while. They wanted to leverage it in GNOME more. After I tried to do some work on GNOME Shell last year, I was approached by them and they offered to sponsor my work, integrating Zeitgeist into GNOME as well as work on Zeitgeist. They also put two of their developers, Abner Silva and JP Whiting, to work on the Zeitgeist Qt bindings for a while.
What I did during the year?
I mainly get sponsored for working on Zeitgeist and related stuff for GNOME and KDE (development, bug fixing and specs definition). Some of the things I worked on were:
Zeitgeist extensions for GNOME Shell: The purpose of the extensions is to integrate some Zeitgeist experience into GNOME Shell. The experience comes in form of three extensions (you can grab the source from here)
Jumplists: Displays the recently/most used files used with the apps displayed in GNOME Shell. The extension has been uploaded to extensions.gnome.org
Journal: Zeitgeist based activity-history journal as a gnome-shell overview tab. It allows you to access you files via GNOME Shell and sorts them for you in chronological order. The extension has been uploaded to extensions.gnome.org
Search: Makes the Shell search via Zeitgeist and and categorize the results better. I will upload it to the extensions website when the highlighting issue is fixed.
El Loco: A Tool to browse you activities at a current Location. Go on the map and click on the town or street and you will be able to see what/who you interacted with there. It is pretty unique in what it offers but needs more UI design, since until now it served primarily a demo application, Ted Gould and I ported it to Gtk3 (https://code.launchpad.net/~seif/+junk/ellocco) as well as to Qt (https://code.launchpad.net/~seif/+junk/qt-lav).
Activity Log Manager: Due to popular demand we needed to hack on a privacy tool. Along side Manish Sinha, JP Lacerda, Stefano Candori and Siegfried Gevatter, we developed ALM. It is a graphical user interface which lets you easily control what gets logged by Zeitgeist. It supports setting up blacklists according to several criteria (such as application or file types), temporarily stopping all logging as well as deleting recent events. ALM will be seeing deeper integration into Ubuntu and hopefully GNOME too soon. You can grab the code from here.
Gedit Dashboard: Michal Hruby’s idea during the hackfest has become reality, after Hylke Bons and I spent almost a month working on it from scratch. The idea was to have a “Google Chrome”-like dashboard for gedit as well as a quick search. The patient and kind Gedit folks helped me clean up the code and now its part of gedit-plugins. This is my favorite plugin for gedit now. You can get it from http://git.gnome.org/browse/gedit-plugins.
libqzeitgeist (now maintained by the amazing Trever Fischer): libqzeitgeist was developed by Abner Silva and JP Whiting. It allows easier development for KDE apps. It is used in Trever’s plugin for KDE Activity Manager to push events directly into Zeitgeist. Trever is now maintaining the library and added QML support. You can find the work here.
Zeitgeist Engine: Of course since all the previous work is powered by Zeitgeist we had to work on getting Zeitgeist better and better. They sponsored my development and maintenance with the team for the 0.7 and 0.8 series and backed up our GNOME sponsored hack-fest. The biggest accomplishment was done when we ported Zeitgeist to Vala, resulting in less memory consumption and much better start-up times.
Epiphany Zeitgeist History (under development): I am working on patching Epiphany to use Zeitgeist as it main storage for History and “Frecency Search”. The Epiphany guys already have a history rewrite branch in which I am sneaking in some code to replace their SQLite back-end. In the end its less code then what they have
Gtk-RecentManager (worked on designing the API with Federico, Siegfried and Michal Hruby): One of the tasks that emerged during the Desktop Summit was replacing the Gtk.RecentManager logic (which has some flaws). We started designing and discussing the issues at hand on LGO. Implementation is now being done by Siegfried and Federico.
What is next?
Collabora currently also sponsors Siegfried Gevatter for his work on Zeitgeist since last March. I will be trying to do some more GNOME related work. Currently I am trying to work on prototypes for the GNOME Design team to help them evaluate their designs. I will also be experimenting with some internal Zeitgeist algorithms stuff soon. I don’t really know what is next. But I hope we can go further into integrating more with GNOME applications and becoming a blessed GNOME module.
Fun fact?
I used GAJ to write down my Zeitgeist development hours for Collabora. Pretty helpful tool. I can tell how long a file was open and how many times I modified it.
This post is by no means meant to be criticism, but rather an opinion on raising questions we couldn’t get answers to.
So on day 2 I got Federico Mena and Garrett LeSage to try out my prototype. While we noticed some nice features there were also some very unclear ones too. Today we concentrated on the “Tabs” and “Top Panel”, which is what is covered by the prototype.
The Good:
Minimalism. It really is easy on the eyes in a way. Reducing the chaos of tabs is relaxing.
Having a shrunk version of of the pages you got open in the “Pages” helps you navigate easier, than navigating via page titles. Its links switching apps in GNOME Shell pretty cool.,
Blends in with the other designs of the core GNOME apps.
It is not clear which one are really the currently open pages and which one are the “Recent”, since the “Recent” button is highlighted. We are not even sure if the upper one represents the currently open pages. If not then where are the tabs? If yes then based on the mockup there is this horizontal line splitting the view in two:
Our interpretation of these mock-ups which looked the following (keep in mind its a functional prototype which means some things are not fully implemented):
(the implementation doesn’t show screenshots of recent stuff since its hard to query those, I replaced them with empty buttons).
So the way to differ according to the (real looking) mock-ups in the wiki is to give the upper section of the view a toolbar style. Seems fair and IMHO opinion it looks nice. Problem however is how distinguishable is it from the the lower “Recent” section? How big does this view get? If I have 20 tabs open that is a lot of scrolling to be done to find one of my open pages, and even more to be done to get to the “Recent” stuff (which is not opened anymore)…
That being said the lower recent section (the real recent stuff) has to be defined better. Is it:
a) going to represent “History”: Bad idea because my history is huge and this won’t even scale.
b) just a predefined of maximum amount of recent pages. How do I find my History then?
Navigating through Pages:
While the mock ups introduce a new and very nice way for recognizing the websites you have open. It kinda takes you out of the focus. One thing I like about tabs is that I have what I am working on in Focus. And by just moving my eyes I can read the label of the tabs and decide if I want to navigate to a new tab or not. With the current mock up I would have to zoom out of my current page and back in. Not sure how good or bad this is.
As far as we could tell the designs are inspired by several other projects by Apple, litl, Pre, Chrome, etc… Some use tabs some don’t… Honestly all of those that use tabs are more intuitive in our opinion. Maybe because I am used to them, this might change.
Creating new Pages:
How do I create pages? Based on the mock ups I will need to use the search bar as seen below.
How we understood it is the search bar will search through the history and open pages too to find what I want… This is just brilliant (and I am not being sarcastic). If I can’t find what we are looking for in the History, there is a “Search” via Google (or anything else of your choice). This is very common for non-techies to now use the url bar and instead do the following:
Always use the search entry next to the URL bar in Firefox
Type “Gmail” or “Facebook” or “Youtube” or “cuevana” there, in the search entry
Pick the first link from Google, which is “of course” the right one, and goes on with life
The only corner case I see here is if I know the URL. How does this work? Do I really need to go through Google. Is there no easy way to create a new page (please don’t tell me menus).
As for the sorting according to the wiki will be done wither via recent or frequent and will the results will be either presented as a grid or linearly… Its still on the Tod o list.
Closing new Pages:
There was no obvious way but I guess having a small x on the corner of a page button to close it would make sense. So no biggie.
Viewing a page:
Its pretty standard and the reduced noise in the panel makes it comfortable however it still feels like wasted space. While I agree with Mathias Hasselmann on the principal of the flaws that he addressed in the following illustration,
he must have not gone through the whole wiki… When you click on the Facebook label it will expand into a URL bar. Not sure what to think of this idea. The button on the VERY right is confusing. It is supposed to be a share button but since there is so much empty space I expected it to be a close button. The panel does not give the impression of a real panel. However this might change since a URL bar might replace it indefinitely due to some comments by the WebKit-Gtk team on “phishing”.
Last comments:
I still can’t make up my mind on the design whether it is good or bad. It has potential but needs to be tested through, before it is decided that its going to be the “Web browser for GNOME”. In a lot of way the ideas make sense but they don’t seem practical all the time. Visually appealing does not always make the functionality usable. I know some other ideas emerged during the Desktop Summit by Hylke Bons and Garrett LeSage on an alternative view for Epiphany.
What I like about this is I understand it from the first look. I know how to create a new page how to navigate through stuff etc… However I do miss the thumbnails from the Web design a bit. And again where is my URL bar? Yet I LOVE THE SIDEBAR. You can check out more of the duos work here.
I think (my stupid opinion) it is to early to decide that “Web” should be the new Web browser for GNOME. I can not see where “Web” will offer me something better than Chrome or Firefox other than integration with the GNOME platform. Web won’t make me ditch Epiphany. But if Epiphany is gone I am not sure I want switch to Web yet. It is missing the Umpf!
That being said I am done with the first phase of prototyping and you can get my code from git (you kinda need Zeitgeist for the history part since I don’t feel like hacking a DB for a prototype). You can grab the prototype from https://github.com/seiflotfy/gnome-prototypes… Again I need to underline that the implementation is a prototype which means limited (very limited) functionality to just play with the designs. The code will not be maintained and will be thrown away.
Often I can not find myself able to criticize or make up my mind about mockups. I need something interactive to work with. While a lot of time things seem logical, the concepts don’t reflect on reality and some ideas need fine tuning.
So in the spirit of allowing others to look into the usability of the design and point out the good and bad I decided to prototype Web. It will be done tomorrow, since I still have some unresolved issues. Technically the prototype (browsing through the site works and switching pages) works, I just need to work on some minor issues and getting screenshots of the each page as well as store history to populate the dialup thingie. From a technical aspect everything can be done very quickly. I also found a place for Tracker (Boomarks) and Zeitgeist (History/Queue) in the prototype.
So what I need is a way to make a screenshot of a webkit webview? Any hints are welcome!
I am prototyping in Python. The end product will either be in JS or Vala or C. This discussion needs to be done with devs.
Here is a teaser…
UPDATE: I was informed that it will be the new design for Epiphany YAY
The great and fun gedit crew have accepted my proposal to include the Dashboard into their plugins repository.
Now you will find it under:
git clone git://git.gnome.org/gedit-plugins
We removed some dependencies and relied on GNOME common tools. We will continue fixing minor issues with the plugin as well as fixing some gedit bugs (the gedit crew is pretty motivating)
Feel free to ping me (with technical issues) or Hylke (with usability issues) if something is bugging you or you have any suggestions.
I will leave you with a small video and hope you like it and some screenies:
YAY
