Building BitSyncHub

Fredrik Håård

fredrik@metallapan.se

Whoami

Freelance developer

  • Specialized in nothing
  • (Getting Stuff Done)

Easily bored ==> automation

PyCon Sweden

  • keep May 9-10, 2016 free

BitSyncHub

Service to sync Bitbucket -> Github

travis.png

  • Created for my own use
  • Now used by 'a lot' of projects

(metrics would have been good here…)

Tech choices

Python 3

  • "hand typed" WSGI good enough

DIY WSGI with routing!

def reghandler(regex, fn):
    def select(environ):
        return re.compile(regex).match(
            environ['PATH_INFO'])
    return (select, fn)

HANDLERS = [
    reghandler('.*/bitsynchub/', bsh),
    # other apps were planned to go here...
    ]
def application(environ, start_response):
    handler = None
    for candidate in HANDLERS:
        if candidate[0](environ):
            handler = candidate[1]
            break
    if handler is None:
        status = "404 Not Found"
        out = [b"Not Found"]
    else:
        status = '200 OK'
        out = handler(environ)
    headers = [('Content-type', handler.content_type)]
    start_response(status, headers)
    return out

WSGI ain't hard

def bsh(environ):
    post = cgi.FieldStorage(
        fp=environ['wsgi.input'],
        environ=environ,
        keep_blank_values=True
        )
    email = post['email'].value if 'email' in post else None

Off the shelf

  • Celery
  • MongoDB (for Celery)
  • Gunicorn (WSGI server)
  • hgapi with hggit (Mercurial integration)
  • gitapi (Git integration)

hgapi

  • Pure-python Mercurial API
  • Original USP: not GPL
>>> import hgapi
>>> repo = hgapi.Repo("test_hgapi")
>>> repo.hg_add("file.txt")
>>> repo.hg_commit("Adding file.txt", user="me")
>>> str(repo['tip'].desc)
'Adding file.txt'
cmd = ["hg", "--cwd", path, "--encoding", "UTF-8"] + list(args)
proc = Popen(cmd,
             stdout=PIPE, stderr=PIPE, env=env)

out, err = [x.decode("utf-8") for x in proc.communicate()]

Integration

bitbucket.png

  • BitBucket supplied data in json format
  • Extra stuff per repository in URI parameters
http://www.metallapan.se/services/bitsynchub/?github=haard/hgapi
if scm == 'git':
    repo = gitapi.git_clone(source, slug)
    repo.git_push(target)
repo = hgapi.hg_clone(source, slug)
for hgb, gb in branches:
    repo.hg_command('bookmark', '-r', hgb, gb)
repo.hg_command("--config", "paths.default=" + target,
                "--config", "extensions.hggit=", "push")
notifications.send_message(email, traceback.format_exc(),
                           "Exception when synching %s" % (source,))

Github account

github.png

  • A single Github account with permission to push

Issues

Troubleshooting

  • No user interface
  • No registration
  • No (deterministic) way to find the user

Re-triggering on error

  • Dummy commits are ugly

New repositories

  • Git is funny, but not 'ha-ha' funny

Lessons learned

People have expectations

  • Source code repository
  • Uptime

Small automations are really useful

$ wc worker.py bsh.py
32  103 1140 worker.py
67  199 2068 bsh.py
99  302 3208 total
  • This presentation is ~150 loc

Future

  • Metrics (fun)
  • Polling (convenient)
  • Private repositories (why?)
  • Open source (why?)

Fin