CHANGELOG

Changelog
=========

0.6.0 (2015-09-04)
------------------

- Refactored the ManageSF CLI. [George Peristerakis]

  The goal is to test and normalize the CLI to resemble the REST API.
  Also adds debug for replication errors.

0.5.0 (2015-08-13)
------------------

- Only Administrators should be allowed to create new projects.
  [Christian Schwede]

  Setting enabled by default (=only admins allowed).
  Can be disabled in the configuration if needed.

- Set WebOb requirement to v1.5.0a1. [Matthieu Huin]

- Add sshconfig REST API. [Christian Schwede]

  Add a basic interface to add or remove SSH configurations. Commands have to be
  sent to the endpoint /sshconfig/<aliasname>/. A JSON dictionary in the following
  format has to be sent with a PUT request to create or update a hostentry:

  {
      "hostname": "Hostname",
      "identityfile_content": "TheActualKeyOfTheHost",
      "userknownhostsfile": "UserKnownHostsFile",
      "preferredauthentications": "PreferredAuthentications",
      "stricthostkeychecking": "StrictHostKeyChecking",
      "username": "Username"
  }

  The identityfile_content is written to a keyfile, and the keyfile name is added
  using the setting "identityfile" to the host configuration.

  The following is an example how to use curl to add a new ssh configuration named
  "sample", using the example JSON from above saved in a file named "sample.json":

      curl -v -b ./cookiefile -c ./cookiefile -X POST
          --data "username=user1&password=userpass&back=/"
          http://tests.dom/auth/login

      curl -v -b ./cookiefile -c ./cookiefile -X PUT \
          http://tests.dom/manage/sshconfig/sample \
          -H 'Content-Type:application/json' \
          --data-binary "@sample.json"

  The managesf CLI has been updated too and includes two commands to add or delete
  a SSH config entry. Example usage:

      ssh-keygen -f private_ssh.key

      sfmanager --url http://managesf.tests.dom \
          --auth http://managesf.tests.dom \
          --auth user1:userpass \
          gerrit_ssh_config add --alias aliasname --key private_ssh.key --hostname sample.github.com

      sfmanager --url http://managesf.tests.dom \
          --auth http://managesf.tests.dom \
          --auth user1:userpass \
          gerrit_ssh_config delete --alias sample

- Pin requirements to the latest working versions. [Matthieu Huin]

- Fix test incompatibility with current WebOb library (1.5.0a1)
  [Matthieu Huin]

  We keep the reference to WebOb in requirements.txt for future version
  pinning.

- Change RCIP git repos. [Matthieu Huin]

- Pin WebOb to fix Exception handling error. [Matthieu Huin]

- Fix bug in sfmanager. [Christian Schwede]

  There is an unitialized variable in sfmanager. An exception is raised using the
  following command:

   sfmanager project create --name sample

0.4.0 (2015-07-10)
------------------

- Use pysflib to fetch auth cookie. [Matthieu Huin]

- Introspection REST API endpoint. [Matthieu Huin]

  The server advertises the library version and the available services
  depending on whether they need authentication or not. Example:

  GET /about/

  {'service':
   {'name': 'managesf',
    'version': 'x.y.z',
    'services': ['zuul', 'jenkins', 'lodgeit', 'etherpad', 'managesf'],
    'auth_services': ['gerrit', 'redmine'], }
  }

- Update replication documentation. [Christian Schwede]

0.3.2 (2015-06-22)
------------------

- Add managesf documentation. [Christian Schwede]

  - add sphinx requirement
  - add missing pygerrit and python-redmine requirements
  - add sphinx documentation directory
  - split documentation into sfmanager and REST API parts
  - add installation documentation
  - build and install manpage
  - update README.md
  - enhance sfmanager documentation

- Fix managesf replication list/get-all subcommands. [Christian Schwede]

  The commands list and get-all are basically broken in the REST API.

  In the REST API it is not possible to get a list of configs with a URL like this:

  http://managesf.tests.dom/manage/replication

  There is always a sectionname and setting required. The following URL thus
  works:

  http://managesf.tests.dom/manage/replication/sectionname/setting

  However, the CLI always calls a sys.exit(0) if the command "list" is used; and
  even if this is fixed there is sometimes a "None" sent as section name and/or
  setting. This is obviously not the right way to do it, so let's fix this and
  make the REST API itself handle URLs without a section and setting name.

  With this patch the following URLs are all valid and return expected data:

  List of complete config:
  http://managesf.tests.dom/manage/replication/

  List only settings of section "sectionname":
  http://managesf.tests.dom/manage/replication/sectionname
  http://managesf.tests.dom/manage/replication/sectionname/

  Only return setting in sectionname
  http://managesf.tests.dom/manage/replication/sectionname/setting

  Also fixed:

  - Filter the projects a user can see correctly, before it was possible to guess
    section and setting names and retrieve the values even if the user was not a
    member of that project

  - Fixed a mismatch between the replication config; the "real" replication config
    always uses a prefix "remote"

  - Return 404 if a project is not found

- Add missing requirements. [Christian Schwede]

0.3.1 (2015-06-11)
------------------

- Bump to 0.3.1. [Matthieu Huin]

0.3.0 (2015-06-10)
------------------

- Prepare lib for pypi. [Matthieu Huin]

- Add CLI command for managing Gerrit API password. [Fabien Boucher]

- Add htpasswd controller. [Christian Schwede]

  This controller allows to create a htpasswd entry that is used to protect the
  Gerrit API access.

  Usage is quite simple:

  - PUT: creates a new random password entry and returns the password to the client.
         Response code is 201
  - GET: returns either 204 (user exists) or 404 (user does not exist)
  - DELETE: returns 204 if user is deleted, 404 if not found

- Fetch auth cookie from browser cookie jar if available. [Matthieu
  Huin]

  When using the CLI, if the user doesn't provide authentication infos,
  we try to look for an existing authentication cookie in known cookie
  jars. If one is available, we use it to authenticate.

  Currently supported browsers/OS combinations:

  * Firefox/Linux
  * Chromium/Linux

- Improve logging and CLI output. [Matthieu Huin]

- Set behavior on user subcommands consistent (pwd) [Fabien Boucher]

  - Handle the password from the command line if present
    if not use iteractive handling.
  - Improve help related to user commands

0.2.0 (2015-06-01)
------------------

- Fix missing content-type header. [Matthieu Huin]

- Add some info when user creation/update fails. [Matthieu Huin]

- Add support for user management in CLI. [Matthieu Huin]

  This patch introduces a new syntax for project related actions, especially
  project users management, to prevent ambiguities. The old syntax is
  preserved but marked as deprecated.
  Basic CLI testing is added as well.

- Return public info upon successful localdb user auth. [Matthieu Huin]

- Add the possibility to disable SSL verification (good for tests)
  [Matthieu Huin]

- Add localuser db for authentication. [Fabien Boucher]

0.1.1 (2015-05-20)
------------------

- Minor change to test gating conf. [Matthieu Huin]

- Fix path ! [Fabien Boucher]

- Fix group format that request a tab a separator. [Fabien Boucher]

0.1 (2015-05-15)
----------------

- Set gerrit project ACL to allow public readable gitweb. [Fabien
  Boucher]

  Furthermore add:
  - gitignore
  - fix flake8 errors

- Adds unit testing. [Matthieu Huin]

- Adds a console script entrypoint for sfmanager. [Matthieu Huin]

- ManageSF commit. [user1]

- Move pysflib to an external dependency. [Matthieu Huin]

  pysflib is now hosted as its own project, so the code gets downloaded
  from the right repo when building roles.
  Add the possibility to override the repo, will be useful for cross-project
  testing.

- Get members of a given group in gerrit with pysflib. [Matthieu Huin]

- Add upstream-ssh-key option. [Christian Schwede]

  Now it is possible to use a SSH key for authentication if you want to create a
  new project with an upstream repository that requires authentication (for
  example a private on Github).

  Example:

  1. ssh-keygen -f mykey

  2. Register mykey.pub as deploy key in project settings on github

  3. sf-manage
      --url http://tests.dom/
      --auth user1:userpass
      create --name sample
      --description "Sample project"
      --upstream git@github.com:somewhere/private-project.git
      --upstream-ssh-key mykey

- Refactored the CLI of managesf. [George Peristerakis]

- Add project membership UI in the dashboard. [George Peristerakis]

  - Unifies how user are added/removed from groups at
    project creation and after.
  - User addition/removal is done by user email instead of
    username.
  - Add a membership UI management in the Dashboard.

- Trigger a ssh-keyscan for the target host for gerrit replication.
  [Fabien Boucher]

  - Remove the creation of .ssh/config by the puppet gerrit manifest
  - Add a ssh-keyscan in the manaesf replication configurator
  - Add a copy od id_rsa in .ssh of gerrit

- Fix default layout.yaml for zuul. [Fabien Boucher]

  - This patch allow a patch to enter the gate pipeline
    when a commment is added on the label code-review.
  - Also deactivate the submit button for project core
    group. Only the PTL is allowed to bypass the gating
    system.

- Managesf: use URLs instead of hostnames. [Christian Schwede]

  Using fixed "http://" prefixes and default port combinations makes it quite
  difficult to use a "https://" prefixes without breaking anything.

  Let's use URLs instead of hostnames, ports and internal hard-coded prefixes.

- Fix import of requests. [Christian Schwede]

  Using a different name for a package is misleading, let's avoid this.

- Confirm project delete in dashboard. [Christian Schwede]

  Also disables the deletion of the config project in managesf.

- Fix requirement install (session argument) + others. [Fabien Boucher]

  - Need to avoid the usage of this really old virtualenv
    version 1.10.1. (Installed by sfstack)
    sudo pip install -U tox virtualenv
  - Use sorted in unittest at some place because
    after the update of tox and virtualenv order returned
    in tests changes. So consolidate tests.
  - Fix missing file for new edeploy version
  - Fix jenkins-jobs usage for new PermissionDenied on .cache directory

  We need to update our slaves, sfstack code and SF doc.

- Improve dashboard cache. [Christian Schwede]

  Cache is now specific for every auth token, and is cleared on POST/DELETE
  requests.

- Add a basic dashboard. [Christian Schwede]

  This adds a basic dashboard using AngularJS and the managesf REST API.
  Currently listing, creating and deleting projects is supported and the open
  reviews and bugs are shown.

  An additional method was added to managesf to GET project infos including open
  issues and reviews. The data for this call is internally cached for 15 seconds
  because this triggers at least four internal REST API calls.

- Fix broken managesf replication controller test. [Christian Schwede]

  Pecan now returns a 404 for GET requests with incorrect path arguments.
  This only affects GET requests, because the method _handle_bad_rest_arguments in
  pecan/rest.py is only used in _handle_get in pecan/rest.py.

  Affecting change: https://github.com/stackforge/pecan/commit/548ac35e

  Included in release 0.8.1 from Oct 14 2014:
  https://github.com/stackforge/pecan/commit/c7f241fd

- Managesf-cli: use host as auth-server if not specified. [Tristan
  Cacqueray]

- Managesf-cli: add authentication checks and ask password from stdin.
  [Tristan Cacqueray]

- Add the force option for project deletion on Gerrit. [Fabien Boucher]

  Add by default force=True to managesf for project
  deletion. A project not empty (with open patchsets)
  is not removed on Gerrit it the force option is not given
  to the API of Gerrit.

- Fix managesf unitests really long to run. [Fabien Boucher]

  Mock a function that add really long delay
  during tests.

- Enabling checker in backup restore tests. [anil.venkata]

  Resource checker script for backup and restore was under condition,
  which was never succesful, as the variables in the condition were
  obselete. So removed that condition to enable checker all the time.

  Also added a check in backp_start and restore, which will check for
  gerrit and jenkins services before exit.

- Disable TestManageSFAppReplicationController.test_get blocking gate
  bug. [Tristan Cacqueray]

- Managesf uses pysflib.sfgerrit.GerritUtils. [Fabien Boucher]

  - Change managesf gerrit controller to user pysflib
  - Update managesf gerrit controller unittests
  - Add pysflib GerritUtils unittests

- Provide pysflib with RedmineUtils. [Fabien Boucher]

  - Introduce usage of python-redmine lib instead
    of re-writting the wheel.
  - Install pysflib on the install-server role
  - Install pysflib on the softwarefactory role
  - Modify test_01,08,10,11 to use the new pysflib for Redmine
    API calls.
  - Remove RedmineUtil class from utils.py file of
    functional tests.
  - Use pysflib for managesf Redmine API calls
  - Adapt managesf redmine controler tests
  - Gerrit hooks now uses pysflib
  - Cauth userdetail now uses pysflib
  - Fix cauth requirements file not used

- Adding gate pipeline to SF. [anil.venkata]

  Used gate pipeline from sf.dom
  Renamed "Approved" lable as "workflow" and added more values
  added submit permissions to jenkins user
  added tests for gate pipeline

- Add unittest for managesf utils.py. [anil.venkata]

- Move fake config for test_redmine managesf. [Fabien Boucher]

  - Remove the fake config from test_redmine.py
    and put it in the common fake config.

- Add unittest for managesf backup.py. [Fabien Boucher]

- Add global manageSF tests using test_app. [Fabien Boucher]

  - Add tests of the REST API of managesf
  - Change root.py to report real errors to
    the caller
  - Fix HTTP return codes
  - Fix managesf cli calling to add/remove user

- Add unittests for managesf's redmine module. [anil.venkata]

  Added unit tests for redmine module in managesf.

- Managesf unitest gerrit controller. [Fabien Boucher]

  - Add unitests for gerrit controller of managesf.

- Base patch for managesf tests. [Fabien Boucher]

  - Add basic to run tests with tox on managesf
  - Add test on managesf in run_tests.sh

- Add an option in managesf to add/remove user from project.
  [anil.venkata]

  managesf should allow PTLs and COREs, to add or remove
  the user from Redmine and Gerrit.

  managesf 'add_user' command will allow PTL/CORE user, to add
  a user to ptl, core and dev groups, for example
  ./sf-manage.py --auth user1:password add_user --name proj1 --user user2 --groups ptl-group,core-group

  managesf 'delete_user' command will allow PTL/CORE user, to delete
  a user from all groups of the project, for example
  ./sf-manage.py --auth user1:password delete_user --name proj1 --user user2
  if group is provided, it will delete only from that group
  ./sf-manage.py --auth user1:password delete_user --name proj1 --user user2 --group core-group

- Fix run_tests.sh for SF. [Fabien Boucher]

  - Fix all current flake8 errors
  - Add tox.ini to cauth. This allow to
    run tests in virtualenv

- Add functional test for Github oauth login. [anil.venkata]

  Mock github rest APIs using httmock. When a request is made to github,
  this library calls our mocked api, which processes the request and
  send mock replies for the requests.
  This mock server keeps user6 and user7 as github users, along with their
  mail,surname,ssh_keys, oauth code and oauth token.
  If the login is succesful, user will be created in gerrit & redmine like LDAP.

  This mocked login will be only enabled for tests.dom domain and only when user
  credentials are passed as a query string to github login url,
  as this mocking is only required for testing.

  Add a test which will login using github oauth(with github user user6),
  test checks if login works and name, email and ssh keys are fetched and available in Gerrit & Redmine,
  test also creates a project in gerrit and redmine using managesf, and checks
  for the project in both redmine and gerrit.

  Enabled Managesf cli tool to access cookie as a CLI option, and
  skip get_cookie if cookie is alredy provided(and directly use it)

  In tests/functional/__init__.py, no need to explicitly create
  user, using create_gerrit_user, as assign_cookies already does that.

- Provisioner and checker tools. [anil.venkata]

  This patch provides a provisioner and checker script. The
  intend is to provide a way to validate backup restore
  mechanism in case of disaster. It will be useful also
  for validate update by allowing the update test
  process to populate a SF with user data and verify that
  those data still exists after an update.

  How to run those scripts:
  cd SoftwareFactory/tests/backup_restore/
  ./run_provisioner.sh

  this script will ssh to puppetmaster and runs
  tools/provisioner_checker/provisioner.py

  ./run_checker.py
  this script will ssh to puppetmaster and runs
  tools/provisioner_checker/checker.py

  Both the scripts return exist status, which can be used
  to check fail or success.

  There is no cleaning step. Thats means random user data
  will remains on the SF. Those must be only use during
  validation process.

  Functional tests for backup/restore can be started
  by ./run_functional-tests.sh backup_restore_tests

- Backup/restore software factory through managesf. [anil.venkata]

  Use managesf cli for start/get/restore backup
  To manually start backup of SF(cron jobs already setup to take backup)
  sf-managesf.py backup_start
  To get the backup tar ball(i.e  to sf_backup.tar.gz)
  sf-managesf.py  backup_get
  To restore SF from the tar ball
  sf-managesf.py  restore --filename sf_backup.tar.gz

  How it works?
  Each service will have a script for backup and restore.
  When managesf is called for backup/restore, it  ssh to all the services
  and run these scripts. In each script of the service, the files to be
  backed up will be there along with backup and restore commands.
  Each service will use mysql node as a backup server to
  backup to/restore from for its files/data.

  what files/data will be backed up?
  puppetmaster - /etc/puppet/environments/sf /root/puppet-bootstrapper
  gerrit - git repos
  jenkins - jobs
  ldap - ldap directory(/var/lib/ldap)
  mysql - all databases(i.e gerrit, redmine and all)

- Fix publics URLs and small other stuff. [Fabien Boucher]

  - Remove auto gen of common.yaml (useless) and complete it
    to report public URLs of services
  - Zuul reports the public zuul status URL
    in gerrit change messages.
  - Jenkins report the public jenkins URL in
    gerrit change messages. We are able to access
    the console.
  - Mails sent by Jenkins should have the right pub URL now.
  - Mails sent by Gerrit should have the right pub URL now.
  - Mails sent by Redmine should have the right pub URL now.
  - Fix URL fragment (navigating) to gerrit.
  - Add template for providing the right URL for the new
    mail registration.
  - Fix the .gitreview for new project to point to
    top_domain:29418
  - Fix link in gerrit commit message to redirect to pub URL
    of redmine.
  - Add a socat port forward from commonservices:29418 to
    gerrit:29418
  - Add zuul to the top menu and remove the drop down menu.

  A problem still remain, sometime topmenu is nesteed in top
  menu. Need some help on that.

  No way to change the gerrit URL report by zuul in jenkins
  build logs (available on the Jenkins UI) baseurl in zuul.conf
  does almost nothing.

- Gerrit project replication via managesf. [anil.venkata]

  With this patch, managesf allows users to modify/get replication config
  and trigger replication to remote replication host.

  Use the below command to modify/get replication config
  sf-manage.py replication_config <command>
  Supported commands are :
  list, get-all, add, unset, rename-section, remove-section

  To trigger the replication
  sf-manage.py trigger_replication --project <project_name>

  change-Id: Ia174f9db4428864f578c920c572cc26524d1f581

- Re-set ProxyPreserveHost to On. [fabien boucher]

  - Setting ProxyPreserveHost to on tell apache to
    use the provided Host header to build the ProxyPath
    uris.
  - Remove useless port specification in the gateway ProxyPass
    rules.
  - Avoid cauth and managesf app to be called by Virtualhost
    Servername. Use WSGIScriptAlias instead.
  - Completly avoid the use of managesf hostname in functional
    test as well as auth hostname. Only use commonservices.

- Avoiding the need for public IP for managesf. [Babu Shanmugam]

- Bug fix - admin should be able to delete any project regardless of
  whoever owns it. Managesf's redmine interface was not permitting admin
  user to delete a project created by other users. This patch fixes that
  bug. [Babu Shanmugam]

- This fixes the problem where redmine UI is not loaded completely.
  [Babu Shanmugam]

- Add topmenu for SF. [Christian Schwede]

  The top menu is built using bootstrap and an iframe.
  To get the iframe working a few modifications are required:

  - build customized gerrit.war that doesn't escape iframe (see README how to
    build gerrit; the required modification is located in
    gerrit-gwtexpui/src/main/java/com/google/gwtexpui/user/client/UserAgent.java
    and simply uncomments/removes code within

      public static void assertNotInIFrame() {}

    The built gerrit.war is now located in a public Swift container in
    enocloud.

  - use paths for all services (for example /jenkins/ instead of /); this
    required some changes, for example urls.py of lodgeit
  - add an apache proxy on the commonservices node that serves
    the iframe and forwards requests to the correct nodes
  - removed custom css and logo from gerrit
  - and some more minor changes.

  Etherpad and Paste service don't show the URL (due to the iframe); for
  Etherpad URLs just use the second buttom from the right and for Paste use the
  displayed links.

  It is now also possible to add SSL to this Apache and restrict access to all
  webservices because all requests are proxied.

- Single sign-on and sign-off for the softwarefactory services. [Babu
  Shanmugam]

  The services which rely on the LDAP authentication such as redmine
  and gerrit are authenticated via a cookie which is created by the
  authentication service when a user logs in to any of the service.

  When a service is launched it will redirect the user to the central
  authentication service which validates the user's credentials against
  an LDAP server or through github servers via OAUTH and sets a cookie
  that would be visible across all the services that operate on the
  same 'top domain'

  Similarly, when user logs out of any of the service, the request will
  be redirected to the central server which signs out from all the services
  through a series of redirection thereby unsetting all the cookies.

  Right now, the central authentication service is designed to run inside
  the managesf node but with its own hostname alias.

  The functional tests are updated to absorb cookies based authentication.

  Through this commit, there is no way we allow an anonymous user to access the
  softwarefactory services.

- Add log level to managesf server. [anil.venkata]

- Removes SF_PREFIX and introduces SF_SUFFIX which adds support to
  extend the domain names. [Babu Shanmugam]

- Assign core group user the Redmine Developer role. [fabien boucher]

  This patch fix the miss of core users.
  Now we take care of the core users too.

- Bootstrap config repo. [Babu Shanmugam]

- Added 'Approved' label and a submit_filter mandating 2 +2s for 'Code-
  Review' label. [Babu Shanmugam]

- Fixed bug because of which a project with open issues is not deleted.
  [Babu Shanmugam]

- 5630 pre-register users from LDAP to Redmine. [anil.venkata]

  Change-Id: I77534ba600509e4359130536309100c4c71e6be1

  Conflicts:
  	puppet/modules/redmine/manifests/init.pp

  This patch
  1)installs Redmine 2.4.2 version from backports
  2)clones Redmine ldap sync plugin
  3)provides and setups default settings for this plugin, to sync users from ldap
    (i.e changes in redmine.yaml and post-conf-in-mysql.sql.erb)
  4)installs the plugin using puppet "plugin_install" exec type
  5)Default installation will have one issue, plugins can't find assets, to fix this
    created a soft link to pluggin asset folder in puppet
  6)Run the rake test which syncs users from ldap, using puppet Exec(ldap_sync_users)
  7)This rake test syncs users whenever puppet agent is run in redmine node,
    so users from ldap are synced periodically

- Redmine private projects implementation. [Babu Shanmugam]

- Enable redmine for functional tests. [anil.venkata]

- Deploy managesf role/node. [Babu Shanmugam]

  Added gerrit debug as well

- Remove the old managesf and rename the new one. [fabien boucher]

- Add a import-pr.py for import PR from Github. [fabien boucher]

  - Update the Readme about the migration of project from Github
  - Add subdirs in tools dir

- Add ability to create a project with an history. [fabien boucher]

  - upstream can be specify in the configuration file
  - if upstream exists git objects are fetch and push on gerrit
  - only master branch is proceed
  - remove dulwich as dependency and use git command instead

- Add flake8 tests. [Christian Schwede]

  Also fix errors to pass tests.

- Kick Jenkins JJB starter. [Fabien Boucher]

- Fix group creation with wrong name. [Fabien Boucher]

  Add creation of config project on Redmine
  Some PEP8 fixes

- Add the option to create the config repo (jjb) [Fabien Boucher]

- Add setup.py for manage sf tool. [Fabien Boucher]