Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPBv v2.0 #10

Merged
merged 81 commits into from
Feb 7, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
8101680
use v2.1 of spb data generator
vpapako Oct 17, 2017
6074337
switch to local docker images
vpapako Oct 17, 2017
078a389
switch to spb for spbv-v2.0
vpapako Oct 17, 2017
8be97db
update Makefile according to local images
vpapako Oct 17, 2017
99a3374
add function to equally distribute dbpedia versions to the total ones
vpapako Nov 6, 2017
c414a92
add functionality for sending dbpedia external data
vpapako Nov 7, 2017
b2e399d
add 5 extracted sub-versions of dbpedia data.
vpapako Nov 7, 2017
c488f8f
add constants for new parameters
vpapako Nov 7, 2017
410798c
add required ontologies and dbpedia data
vpapako Nov 7, 2017
379abf2
make BenchmarkController to see the new parameters
vpapako Nov 7, 2017
5893180
increase core dependency to 1.0.7
vpapako Nov 7, 2017
b3098f7
change insertion/deletion ratio datatypes
vpapako Nov 17, 2017
86c77b7
fix ontologies
vpapako Nov 17, 2017
d3930ff
remove unused script
vpapako Nov 17, 2017
cb279d8
implement additions
vpapako Nov 17, 2017
85e3e9d
change initial parameters
vpapako Nov 17, 2017
969bedc
reset ServerThreads and MaxClientConnections to default
vpapako Nov 17, 2017
f31b28f
recover load_to_virtuoso.sh
vpapako Nov 20, 2017
7870bbb
sort and only keep XMLSchema#integer
vpapako Nov 22, 2017
2470050
keep ontologies in their initial ttl format
vpapako Nov 22, 2017
79fec9d
add final dbpedia datasets
vpapako Nov 23, 2017
86faa5a
directly load dbpedia versions, change ontologies to ttl
vpapako Nov 23, 2017
142aa11
restructure dbpedia files
vpapako Nov 23, 2017
0785b57
gold std: load dbpedia final instead of changesets
vpapako Nov 23, 2017
a724bb2
add support for deletions
vpapako Dec 1, 2017
065926c
remove useless debug messages
vpapako Dec 4, 2017
2bcb7f7
minor change to log message
vpapako Dec 4, 2017
92f7b39
update image version tags
vpapako Dec 5, 2017
cc542d0
semi-parallelize deleted triples export
vpapako Dec 5, 2017
0edd556
optimize deleted triples export, by parallelizing the process
vpapako Dec 6, 2017
dfb737f
change to virtuoso/tenforce
vpapako Dec 6, 2017
26afd2b
keep 1000 dbpedia entities
vpapako Dec 6, 2017
dae2e07
minor changes
vpapako Dec 6, 2017
0d09660
reset parameters
vpapako Dec 8, 2017
4a4424e
fix deletion bug. implement parallel deletion through java.
vpapako Dec 8, 2017
aae3e40
fix export script name
vpapako Dec 8, 2017
4c7bc45
handle cases when deleted triples exceed the total random
vpapako Dec 8, 2017
69e2200
compute delete-set much more efficient based on statistics
vpapako Dec 11, 2017
f1e537f
minor changes
vpapako Dec 12, 2017
cad5577
fix typo
vpapako Dec 12, 2017
62f252c
remove debug messages
vpapako Dec 13, 2017
f051ac3
Merge pull request #8 from hobbit-project/prl-delete_spbv2.0
vpapako Dec 13, 2017
2ee1ff9
fix sent task from task generator to system
vpapako Dec 22, 2017
e212970
incorporate ontology files to the initial version data files
vpapako Dec 22, 2017
5531a22
remove support for choosing deleted triples from clusterings or
vpapako Dec 22, 2017
d67e145
switch to the new hobbit.core version 1.0.8
vpapako Dec 22, 2017
fb8db65
make structured queries based on DBSB
vpapako Jan 16, 2018
f7313f1
add more query types
vpapako Jan 18, 2018
fb9486a
add support for new stractured queries, fix bug on dbpedia versions
vpapako Jan 18, 2018
b9c0508
fix query syntax errors, add missing prefixes
vpapako Jan 18, 2018
bd84bba
convert ontologies in order to be in the same format as data (n-triples)
vpapako Jan 18, 2018
018b7b3
add data files preparation script for FM versioning strategy
Jan 23, 2018
ad04d5e
remove old constants - add a new one for sent data form
Jan 24, 2018
76e9b21
add build and push images scripts
Jan 24, 2018
5ac0f45
add README
Jan 24, 2018
5c90f2e
replace all '-' chars with '_' on env variables
Jan 24, 2018
72d9bad
add support for sending data as independent copies or change sets
Jan 25, 2018
ffff349
add possible annotation entities file
Jan 25, 2018
bf0188e
simplify generateData
Jan 26, 2018
1acd14c
implement loading of different versions for SPBv v2.0
Jan 26, 2018
c110f89
add log message
Jan 26, 2018
7aa2fa9
change task log messages
Jan 26, 2018
bb1799d
remove debugging sleep
Jan 26, 2018
118a104
change isql to isql-v
Jan 29, 2018
c0d4f67
remove prepare loading script
Jan 29, 2018
03f4bda
add possible template values
Jan 29, 2018
c5e800a
make change speed to be based on both deletions and additions
Jan 29, 2018
25c350a
add functionality for supporting resultsets compare when
Jan 29, 2018
b18289f
fix gYear format to a valid one
vpapako Feb 1, 2018
742e136
make cross-version queries to return distinct values
vpapako Feb 1, 2018
7193f8c
minor changes
vpapako Feb 1, 2018
5a508c0
restore virtuoso.ini for getting complete results
vpapako Feb 1, 2018
50b102f
add query subtype information to task
vpapako Feb 1, 2018
a522a45
change response evaluation
vpapako Feb 1, 2018
59dcf10
update SPB dependency
vpapako Feb 2, 2018
df135d4
add virtuoso JDBC dependency
vpapako Feb 2, 2018
56bbbdc
add local maven dependencies
vpapako Feb 2, 2018
83f8b66
add constants for dbpedia stats
vpapako Feb 7, 2018
0a5302c
change evaluation storage to 1.0.7-SNAPSHOT for supporting large results
vpapako Feb 7, 2018
6a0b48b
change virutoso.ini parameters for returning correct results
vpapako Feb 7, 2018
de3e1bc
Merge branch 'master' into spbv_2.0.backup
vpapako Feb 7, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 12 additions & 14 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,20 +1,18 @@

default: build dockerize

build:
build:
mvn clean package -U -Dmaven.test.skip=true

dockerize:
docker build -f docker/versioningbenchmarkcontroller.docker -t git.project-hobbit.eu:4567/papv/versioningbenchmarkcontroller:1.0 .
docker build -f docker/versioningdatagenerator.docker -t git.project-hobbit.eu:4567/papv/versioningdatagenerator:1.0 .
docker build -f docker/versioningtaskgenerator.docker -t git.project-hobbit.eu:4567/papv/versioningtaskgenerator:1.0 .
docker build -f docker/versioningevaluationmodule.docker -t git.project-hobbit.eu:4567/papv/versioningevaluationmodule:1.0 .
docker build -f docker/versioningvirtuososystemadapter.docker -t git.project-hobbit.eu:4567/papv/versioningsystem:1.0 .
# docker build -f docker/versioningr43plessystemadapter.docker -t git.project-hobbit.eu:4567/papv/versioningr43plessystem .
docker build -f docker/versioningbenchmarkcontroller.docker -t git.project-hobbit.eu:4567/papv/versioningbenchmarkcontroller:2.0 .
docker build -f docker/versioningdatagenerator.docker -t git.project-hobbit.eu:4567/papv/versioningdatagenerator:2.0 .
docker build -f docker/versioningtaskgenerator.docker -t git.project-hobbit.eu:4567/papv/versioningtaskgenerator:2.0 .
docker build -f docker/versioningevaluationmodule.docker -t git.project-hobbit.eu:4567/papv/versioningevaluationmodule:2.0 .
docker build -f docker/versioningvirtuososystemadapter.docker -t git.project-hobbit.eu:4567/papv/versioningsystem:2.0 .

docker push git.project-hobbit.eu:4567/papv/versioningbenchmarkcontroller:2.0
docker push git.project-hobbit.eu:4567/papv/versioningdatagenerator:2.0
docker push git.project-hobbit.eu:4567/papv/versioningtaskgenerator:2.0
docker push git.project-hobbit.eu:4567/papv/versioningevaluationmodule:2.0
docker push git.project-hobbit.eu:4567/papv/versioningsystem:2.0

docker push git.project-hobbit.eu:4567/papv/versioningbenchmarkcontroller:1.0
docker push git.project-hobbit.eu:4567/papv/versioningdatagenerator:1.0
docker push git.project-hobbit.eu:4567/papv/versioningtaskgenerator:1.0
docker push git.project-hobbit.eu:4567/papv/versioningevaluationmodule:1.0
docker push git.project-hobbit.eu:4567/papv/versioningsystem:1.0
# docker push git.project-hobbit.eu:4567/papv/versioningr43plessystem
14 changes: 14 additions & 0 deletions dependency-reduced-pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,20 @@
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>eu.ldbc</groupId>
<artifactId>semanticpublishing</artifactId>
<version>2.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/semanticpublishing-2.0.jar</systemPath>
</dependency>
<dependency>
<groupId>org.virtuoso</groupId>
<artifactId>virtjdbc</artifactId>
<version>4.2</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/virtjdbc-4.2.jar</systemPath>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
Expand Down
33 changes: 2 additions & 31 deletions docker/versioningdatagenerator.docker
Original file line number Diff line number Diff line change
@@ -1,35 +1,6 @@
FROM ubuntu:trusty
FROM tenforce/virtuoso

# get Virtuoso
ENV VIRT_HOME /opt/virtuoso-opensource

ENV VIRT_BUILD_OPTS --with-readline

ENV VIRT_DB /var/lib/virtuoso/db

RUN apt-get -y update && \
apt-get -y clean && \
apt-get -y install dpkg-dev build-essential && \
apt-get -y install autoconf automake libtool flex bison git gperf gawk m4 make libxml2-dev libssl-dev libreadline-dev wget && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /tmp/*

WORKDIR /opt
RUN git clone git://github.com/openlink/virtuoso-opensource.git virtuoso-opensource.src

WORKDIR /opt/virtuoso-opensource.src
RUN git checkout stable/7
RUN ./autogen.sh && ./configure --prefix=$VIRT_HOME $VIRT_BUILD_OPTS && make && make install

ADD virtuoso.ini.template /var/lib/virtuoso/db/virtuoso.ini
ADD virtuoso.ini.template /versioning/virtuoso.ini

RUN export PATH=$PATH:/opt/virtuoso-opensource/bin

EXPOSE 1112
EXPOSE 8891

# get java8
# get java 8
RUN apt-get update && \
apt-get -y upgrade && \
apt-get -y install software-properties-common && \
Expand Down
14 changes: 10 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,23 @@
<dependency>
<groupId>eu.ldbc</groupId>
<artifactId>semanticpublishing</artifactId>
<version>0.1</version>
<version>2.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/semanticpublishing-2.0.jar</systemPath>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.8.8</version>
</dependency>


<dependency>
<groupId>org.virtuoso</groupId>
<artifactId>virtjdbc</artifactId>
<version>4.2</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/virtjdbc-4.2.jar</systemPath>
</dependency>
<dependency>
<groupId>org.aksw.jena-sparql-api</groupId>
<artifactId>jena-sparql-api-core</artifactId>
Expand All @@ -95,7 +102,6 @@
</exclusion>
</exclusions>
</dependency>

</dependencies>

<build>
Expand Down
Loading