Quadstore use cases

[jacoscaz]

What is the community building with quadstore? This is the place to show and tell!

[retog]

I posted this in the wrong place: #109 (comment)

The original idea (8 month ago): https://viewer.scuttlebot.io/%25MGKpQ3qNLDv3QM2ddia5v74Qr2C8FUhkwSqqhvFLlW4%3D.sha256

The goal is to have a decentralized collaborative knowledge graph on Scuttebutt. What will be passed around over the network are the update statements and each client constructs the graph(s) according to the statement they get via their social connections.

So far I've been working on the UI to combine direct-editing using rdfjs-svelte with SPARQL operations using YASGUI.

This is the construction site on Scuttlebutt:

And here's a tweet pointing to the UI on the web:

https://twitter.com/txreto/status/1323033951477399553

[jacoscaz]

Thank you @retog for sharing! @gsvarovsky, this might be of interest?

[gsvarovsky]

Thanks for the tip Jacopo, looks interesting @retog!

[gsvarovsky]

I'm building the m-ld Javascript engine using node-quadstore, and it's been great to work with you @jacoscaz!

m-ld is a decentralised live information sharing component, with an API based on JSON-LD.

From the documentation:

m-ld is used by including it in a software application (an 'app'). The app gains the power to share live data with other copies of itself, without any need to store or coordinate the data centrally in a database, or otherwise cope with the hard problem of keeping distributed data up to date. A bit like Google docs, but for any structured data specific to the app: designs, bookings, observations, plans or anything else; and also, without the need to entrust the data to a third party.

On the website there's also a demo and a playground.

We're working towards a Beta and interested to hear any ideas about use-cases or projects that anyone has in collaboration or collective intelligence!

[jacoscaz]

Thank you for all the fixes, ideas and chances to brainstorm on things! Looking forward to more.

[michaelcpuckett]

Hi! I've been working on a proof-of-concept that allows querying Quadstore with GraphQL syntax. I'm not sure what I'll end up doing with it, but wanted to share it in case someone else is in my situation.

https://github.com/michaelcpuckett/node-quadstore-webpack-bundle

Here's the context that led me to this solution:

I'm a front-end engineer, so I'm developing mostly for the browser. (I really like that Quadstore works in the browser!)

I work on a web app that requires lots of data to be sent over the wire, up front, right when a user logs in. Then the client side is responsible for displaying the data in various views, including sorting/filtering/querying the data.

Currently we use Vuex (like Redux) to store all the data, split up into modules, in pretty much the same way it's stored in the backend database. (The data could fit into a graph, but the backend has it set up into regular relational tables.)

The problem is that each view/component ends up doing a lot of work getting bits and pieces of data from various modules, then matching on IDs to connect them into a format that makes sense, then sorting, etc...

I have been searching for a way for the views/components to make a single query to a client-side data store to get back all the required data in the specified format.

Quadstore is a great starting point for this use case, because it allows arbitrary SPARQL to be fed into it.

Over time, I've learned a lot about RDF and SPARQL, but I still don't feel very comfortable writing SPARQL syntax, and I don't expect anyone on my team to learn the details. I figured we could use GraphQL and an additional layer to hide some of the RDF-specific "weirdness" (like entities having to be a URL).

All I needed was a way to convert GraphQL syntax to SPARQL. I found a few libraries to help with that.

The end result isn't perfect, and the queries are pretty slow, but I think it's a cool demo!

[jacoscaz]

Very interesting @michaelcpuckett ! Perhaps you might want to have a look at Comunica’s GraphQL actor. Comunica is the modular framework for building SPARQL query engines that quadstore uses. We provide a custom configuration of Comunica that is optimised for size and dependency count which does not include the GraphQL actor but ever since version 8.0.0 we support usage of other Comunica configurations. Forking ours to include the GraphQL actor should be relatively easy.

[antoine-morvan]

Our solution uses SPARQL as a backend in a Javascript environment. I cannot say much more about the general picture of our work, but in this setting we use the Quadstore along with the the SPARQL Engine from Comunica in a custom test framework. Indeed, in order to have relevant tests running without starting a whole endpoint (fuseki / virtuoso / ...), our test framework starts/stops Quadstores/SPARQL Engine with minimal data sets on the fly.

As of today we use an old version, with limited SPARQL support, thus we are pretty limited in the scope of our tests and still rely on more classical mocks & stubs. But having a test endpoint started on the fly with minimal dataset when possible is much more convenient :)

[jacoscaz]

Interesting use case! Thank you for sharing @antoine-morvan. Happy to see we’re helping others test their own stuff.

[KonradHoeffner]

We investigate using Quadstore in the frontend to insulate our RDF Graph visualization research prototype SNIK Graph against downtimes in our SPARQL endpoint and allow easier migration to other domains by not requiring a SPARQL endpoint, which is easier to setup, e.g. for students that write a bachelor thesis and don't want to learn how to install and setup a server and a SPARQL endpoint. Given that our main data is only ~6 MB in Turtle files, we think this should work well enough, and maybe even be more performant. Also, this would allow bundling the data with the visualization so that a former state of both could be replicated more easily, for example with persistant URIs in research papers.

[jacoscaz]

Very interesting use-case. A couple of thoughts:

1. We're currently working on improving query performance by pushing filters from Comunica (the SPARQL engine we're using) down to quadstore so that less needs to happen in-memory. Let me know if you encounter particularly slow queries.
2. Depending on how you're approaching things, you might be interested in @gsvarovsky 's m-ld for real-time backend/frontend synchronization.

[rasendubi]

I'm playing with building a personal data lake: exporting stuff from my online accounts, converting it to JSON-LD (for storage/archival), and importing it into quadstore for analysis/exploration. The project is pretty much ill-defined so I'm figuring it out as I go.

What are my expectations so far:

• There won't be too much data. The data lake only covers my activity, so unlikely to grow into tens- and hundreds-Gb range. I will likely be fine importing all data every time I boot the app, and data will likely fit in memory. (I would like to keep it in-memory as long as feasible.) Though I may be dreadfully wrong here and data will explode quickly once I import something high-frequency (e.g., I imagine my browsing or email history could easily take multiple Gbs.)
• Highly heterogeneous data and wide vocabulary. Not 100% sure but I will likely end up importing a lot of stuff from third-party services, so will have to go with wide and weakly-defined vocabulary. Data should be mostly small but may occasionally contain larger blobs (e.g., I imported all my GitHub comments, and while most are small, a couple of them triggered the 10k limit).
• Data enrichment. Will likely need to enrich data to bridge various sources and link my various accounts to a single "me."
• Mostly browsing use-case. I don't expect to run complex analysis on the data. The main use case is browsing my activity, so it would mostly traverse properties from one entity to another. Occasionally ranged/sorted by dateTime.
• Throw-away apps. I may build multiple apps for slicing and dicing various aspects of my data. Some may use quadstore, some may use timeseries db, or something else for full-text search

Anyway, I don't have a scope defined so am very likely to wonder off and breaking all my assumptions