Google Summer of Code 2010: Automatic Synchronization between Sahana Instances

This project is part of Sahana Eden Google Summer of Code

Executive Summary

  • Abstract: Automatic sync between Sahana instances has received much attention from Sahana community. It aims to sync resources between multiple Sahana instances. I will work on resolving its current issues and adding new features:
    1. Modifying sync process to work with S3XRC resources instead of DB tables to maintain full references
    2. Offline Sync facility (USB)
    3. Add the facility to schedule multiple custom sync jobs (custom - weekly/daily/one-time with selected or all resources)
    4. Make sync peer discovery work over WANs & LANs by combining ZeroConf with Beacon Service (an online directory of Sahana instances)
  • Student: Amer Tahir
  • Mentor(s): Fran Boon and Michael Howden

Code

How To Use

Downloading and Installation

Do “bzr checkout lp:~amertahir/sahana-eden/sync” (without quotes) to get the latest source-code from repository. Then rename the sync directory to eden and place it under your web2py applications directory. That's all, now fire up your browser and open http://127.0.0.1:8000/eden/ to get started :)

NOTE: the demo instances (specified above) could also be used to test out sync (use username: admin@example.com and password: admin to login)

Getting Started

After setting up the instance, you have to first Register (or Login with administrative account) and specify a few settings for sync. For this, open Administration page and then click Sync > Sync Settings from the menu. You will see the settings page for sync. Here you have to specify you instance URL (e.g. http://sync.eden.sahanafoundation.org/eden), description and so on.

Then proceed to the Synchronisation main page. From there you can sync manually (click “Sync Now”), add sync partners, subscribe to sync pools, resolve sync conflicts and so on. The page is self-explanatory.

How to Sync?

Synchronisation process works automatically in the background (as Cron jobs) and you can use the Sync Scheduler to manage sync jobs. However, if you wish to bypass that and manually perform sync, you can click “Sync Now” button to start syncing with sync partners that you have specified.

Sync Partners

Sync Partners are instances that you wish to sync with. They could be deployed on SahanaEden, SahanaAgasti, Ushahidi or possibly any platform that SahanaEden can talk to (currently Eden, Agasti and Ushahidi are supported).

Before you can start syncing, you need to define sync partners. This can be done by going to the Sync Partners page. You can either search for partners or add them manually (right now only manual additions are supported). Every sync partner has a UUID (0 for non-Eden instances), instance URL (e.g. http://sync.eden.sahanafoundation.org/eden), instance type (Eden/Agasti/Ushahidi/etc.) and Sync Policy. Sync policies other than “No Sync” only come into play when the data that is imported also exists locally. It could be the case that local data is modified after the last sync with this peer (Local is newer), similar for remote data (Remote is newer) or both have been modified (either of them is newer). So, to decide how to resolve such 'conflicts' automatically, you can select one of the sync policies. If you want to manually resolve such conflicts, select “Choose Manually”.

Sync Schedules

To automate the sync process, you can define sync jobs. Sync Jobs are individual syncing processes with sync partners that you like. So, if I want to sync “Person Registry” with some sync partner, I would just create a new sync job and select “Person Registry” and the sync partner I want to sync with. Along with that, I would specify the scheduled time for the job to run, either daily, weekly, hourly or just once.

You can define multiple sync jobs for a particular sync partner, specifying resources to sync and sync policies individually for each of the sync jobs. By default, only resources modified after the last sync operation (new data) are synced. This is similar to differential backups. However, if you want to sync all data, you can specify “Complete Sync” in the job settings.


This section is a work in progress and would be updated as development is carried out on this project

Progress

Weekly Meetings
  • Thursdays at 1400 UTC Calendar at IRC (this is tentative schedule, *would* change)
Progress Reports
Project Plan and Timeline

Following are the chosen SMART Goals that were recommended during the application process, and reviewed over the bonding period:

Midterm Evaluation Milestones: 24 May - 12 July
GoalMeasureDue DateStatus
Modify the existing sync process to work with S3XRC import/export instead of working directly with DB tables (and records)Prototype demo of the S3XRC powered sync system06/09/2010Completed
Incorporate automatic conflict resolution policies in the sync systemDemo of a working (possibly buggy) implementation06/20/2010Completed
Finalizing conflict resolution policies after bug fixing and preparation of test scenarios to verify itDemo of the sync system with conflict resolution including Test Plan and Test Report that would be approved after review by the mentor(s)06/29/2010Incomplete
Finalize the sync system with bug fixing and changes suggested by the mentor(s) along with testing of the entire systemDemo of the sync system with testing procedures defined (Test Plan); approval by the mentor(s)07/11/2010Incomplete


Final Evaluation Milestones: 13 July - 9 August
GoalMeasureDue DateStatus
Implement the Offline Sync (USB flash sync) functionality in the sync systemPrototype demo of the offline sync (possibly buggy)07/18/2010Incomplete
Incorporating Beacon service into the sync implementation to work along with ZeroConf daemonDemo of a working (possibly buggy) implementation of sync that utilizes both Beacon service and ZeroConf along with approval by the mentor(s)07/20/2010Incomplete
Implementation of multiple sync pools with automatic discovery of sync partners using Beacon service and ZeroConfDemo of the working system along with approval from the mentor(s)08/01/2010Incomplete
Bug fixing of the offline sync and other components of the sync systemProgress report submitted to the mentor(s), then reviewed and approved by them08/05/2010Incomplete
Final integration of the entire system, preparing it for the demoDemo of the entire sync system, approval by the mentor(s)08/09/2010Incomplete

Navigation
QR Code
QR Code foundation:gsoc_tahir (generated for current page)