Skip to content

Classes and scheduled tasks to automate the import of regularly updated online spatial data into GeoNode.

License

Notifications You must be signed in to change notification settings

OpenGeoscience/dataqs

Folders and files

NameName
Last commit message
Last commit date

Latest commit

305ddba · Jun 8, 2017
Apr 27, 2017
Apr 27, 2017
Apr 12, 2016
Feb 10, 2017
Mar 11, 2016
Jan 26, 2016
Feb 18, 2016
Sep 25, 2015
Feb 10, 2017
Feb 10, 2017

Repository files navigation

Data Queues

dataqs (Data Queues) is a simple Django app to download, process, and import spatial data into GeoServer/GeoNode.

Quick start

  1. Add "dataqs" to your INSTALLED_APPS setting like this:

    INSTALLED_APPS = (
        ...
        'dataqs',
        'dataqs.forecastio',
        'dataqs.gfms',
        'dataqs.airnow',
        'dataqs.wqp',
        'dataqs.aqicn',
        #etc
    )
    
  2. In your settings.py or local_settings.py file, add a CELERYBEAT_SCHEDULE setting to specify when Celery should run data_queues tasks:

    from celery.schedules import crontab
    CELERYBEAT_SCHEDULE = {
        'gfms': {
            'task': 'dataqs.gfms.tasks.gfms_task',
            'schedule': crontab(minute='3'),
            'args': ()
        },
        'forecast_io': {
            'task': 'dataqs.forecastio.tasks.forecast_io_task',
            'schedule': crontab(minute='1'),
            'args': ()
        },
            'task': 'dataqs.aqicn.tasks.aqicn_task',
            'schedule': crontab(hour='*/6', minute='0'),
            'args': ([],)
        },
    }
    
  3. Also add the following settings:

    #Location of GeoServer data directory
    GS_DATA_DIR = '/usr/share/geoserver/data'
    
    #Directory where temporary dataqs geoprocessing files should be downloaded
    GS_TMP_DIR = GS_DATA_DIR + '/tmp'
    
    #AirNow API username:password
    #(sign up for a free account at http://airnowapi.org/account/request/)
    AIRNOW_ACCOUNT = 'your_airnow_username:your_airnow_password'
    
    #NASA GPM FTP ACCOUNT
    #(sign up at http://registration.pps.eosdis.nasa.gov/registration/)
    GPM_ACCOUNT = 'your_gpm_email_account'
    
    #HIFLD: Dictionary of layers to process in the form of:
    HIFLD_LAYERS = [
        {
        'name': 'US State Boundaries',
        'url': 'https://hifld-dhs-gii.opendata.arcgis.com/datasets/718791120f6549708cb642dac6ff0dbf_0.geojson',
        'table': 'state_boundaries',
        'sld': 'polygon'
    },
    {
        'name': 'Cities and Towns NTAD',
        'url': 'https://hifld-dhs-gii.opendata.arcgis.com/datasets/6a1e404a10754e59bac4bfa50db3f487_0.geojson',
        'table': 'cities_towns',
        'sld': 'point'
    },
    {
        'name': 'Roads and Railroad Tunnels',
        'url': 'https://hifld-dhs-gii.opendata.arcgis.com/datasets/2f6abb736360437ba363e0a1210b4d36_0.geojson',
        'table': 'roads_tunnels',
        'sld': 'line'
    }
    ]
    
    #Time to wait before updating Geoserver mosaic (keep at 0 unless Geoserver
    #is on a different server. In that case, there will need to be an automated
    #rsync between GS_TMP_DIR where celery is running and
    #GS_DATA_DIR where GeoServer is running.
    RSYNC_WAIT_TIME = 0
    
  4. In order to run the spei processor, the following must be installed:

    sudo apt-get install netcdf-bin
    sudo apt-get install cdo
    

About

Classes and scheduled tasks to automate the import of regularly updated online spatial data into GeoNode.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published