-
Notifications
You must be signed in to change notification settings - Fork 0
/
gephi-and-the-streaming-module.html
42 lines (35 loc) · 18.6 KB
/
gephi-and-the-streaming-module.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Gephi and the Streaming module - Toto do stuff</title><meta name="description" content="As you may have remark if you follow me, I'm a big fan of Gephi with the streaming module. In this article, I want to…"><meta name="generator" content="Publii Open-Source CMS for Static Site"><link rel="canonical" href="https://totetmatt.github.io/gephi-and-the-streaming-module.html"><link rel="alternate" type="application/atom+xml" href="https://totetmatt.github.io/feed.xml"><link rel="alternate" type="application/json" href="https://totetmatt.github.io/feed.json"><meta property="og:title" content="Gephi and the Streaming module"><meta property="og:image" content="https://totetmatt.github.io/media/posts/11/StreamArchi.jpg"><meta property="og:site_name" content="Toto do stuff"><meta property="og:description" content="As you may have remark if you follow me, I'm a big fan of Gephi with the streaming module. In this article, I want to…"><meta property="og:url" content="https://totetmatt.github.io/gephi-and-the-streaming-module.html"><meta property="og:type" content="article"><style>:root{--body-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--heading-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--logo-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--menu-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol"}</style><link rel="stylesheet" href="https://totetmatt.github.io/assets/css/style.css?v=825c89ac06c7215b642eda05e8a14751"><script type="application/ld+json">{"@context":"http://schema.org","@type":"Article","mainEntityOfPage":{"@type":"WebPage","@id":"https://totetmatt.github.io/gephi-and-the-streaming-module.html"},"headline":"Gephi and the Streaming module","datePublished":"2013-09-16T19:18","dateModified":"2020-06-20T00:35","image":{"@type":"ImageObject","url":"https://totetmatt.github.io/media/posts/11/StreamArchi.jpg","height":846,"width":1096},"description":"As you may have remark if you follow me, I'm a big fan of Gephi with the streaming module. In this article, I want to…","author":{"@type":"Person","name":"Totetmatt","url":"https://totetmatt.github.io/authors/totetmatt/"},"publisher":{"@type":"Organization","name":"Totetmatt"}}</script></head><body><div class="site-container"><header class="top" id="js-header"><a class="logo" href="https://totetmatt.github.io/">Toto do stuff</a></header><main><article class="post"><div class="hero"><figure class="hero__image hero__image--overlay"><img src="https://totetmatt.github.io/media/posts/11/StreamArchi.jpg" srcset="https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-xs.jpg 300w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-sm.jpg 480w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-md.jpg 768w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-lg.jpg 1024w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-xl.jpg 1360w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-2xl.jpg 1600w" sizes="(max-width: 1600px) 100vw, 1600px" loading="eager" height="846" width="1096" alt=""></figure><header class="hero__content"><div class="wrapper"><div class="post__meta"><time datetime="2013-09-16T19:18">13/09/16</time></div><h1>Gephi and the Streaming module</h1><div class="post__meta post__meta--author"><a href="https://totetmatt.github.io/authors/totetmatt/" class="feed__author invert">Totetmatt</a></div></div></header></div><div class="wrapper post__entry"><p>As you may have remark if you follow me, I'm a big fan of Gephi with the streaming module.</p><p>In this article, I want to share how it can be use "officially" (I mean, on the way it was designed I think) and some tricks that can be used ;-) .</p><h1>Presentation Gephi Streaming</h1><p>To be simple, the Gephi Streaming module allow to stream graph over HTTP(S) between 2 gephi instances. Basically, you send some JSON that say add a node, remove an edge, change a node etc... to gephi. There is a <a href="https://marketplace.gephi.org/plugin/graph-streaming/">marketplace</a>, a <a href="http://wiki.gephi.org/index.php/Graph_Streaming">wiki </a>and some <a href="http://wiki.gephi.org/index.php/Specification_-_GSoC_Graph_Streaming_API">articles</a> that explain the concept and a beginning of a protocol.</p><p>If you're using the API, you will probably ask "Why ? the Java API can do everything and more". And I agree with you, but the API is quite confusing on the way that it's not "simple" to boot it up for a simple project (ok I didn't spend a lot of time on it but I tried and I wasn't convinced).</p><p>I also bet that you (Data scientists, Data journalist, Data pornographists ,etc ...) use python or ruby or scala or R or whatever other programming language. Gephi Streaming is based on HTTP(S) / REST , it means that it's open to almost every language, as long at you have a minimum support of HTTP transaction.</p><p>It sounds also possible to share and work on live graphs (live streaming of twitter status e.g) with people all over the world via internet.</p><h1>How it works, how it reacts</h1><p>First look at the module after you download it, you will see that it's quite simple and there is only 2 "mode":</p><ul><li><strong>Client</strong> : This part makes connection and synchronisation with other gephi instance possible (see <strong>Gephi to Gephi </strong>below)</li><li><strong>Master</strong> : Most important part, this is where you can create a master server that will received graph action and update other clients that register to it.</li></ul><p>AS I already say, the concept of the graph streaming is to send action for nodes or edges (add, change, delete). What's important here is that for each action, you are using an ID to refer to a gephi element (node or edge).</p><h1>Gephi to Gephi</h1><p>Gephi streaming allows to connect 2 gephi instances (even workspace since 0.8.2).</p><p>It's pretty simple to put in place, let say you have 2 gephi instance running (one in 192.168.0.1 (short name <strong>One</strong>) and another in 192.168.0.2 (Shot name <strong>Two</strong>), to be "network funny").</p><p>Idea is, a <strong>Master</strong> provide graph for a <strong>Client</strong>. So let's completly not democraticaly decide that <strong>One</strong> is master and others are client.</p><p>In <strong>One </strong>just go to the Gephi Streaming module, click right to <strong>Master server</strong> and <strong>Start</strong>. You're now running a master at http://192.168.0.1:8080 (you can configure the default and even put some SSL. I assume that you are on <strong>workspace0</strong>.</p><figure class="alignnone size-full wp-image-216"><img loading="lazy" style="color: #333333; font-style: normal; line-height: 24px; margin-top: 0.4em;" alt="ScreenHunter_01 Jul. 30 08.39" src="https://totetmatt.github.io/media/posts/11/ScreenHunter_01-Jul.-30-08.39.jpg" sizes="(max-width: 48em) 100vw, 768px" srcset="https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_01-Jul.-30-08.39-xs.jpg 300w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_01-Jul.-30-08.39-sm.jpg 480w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_01-Jul.-30-08.39-md.jpg 768w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_01-Jul.-30-08.39-lg.jpg 1024w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_01-Jul.-30-08.39-xl.jpg 1360w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_01-Jul.-30-08.39-2xl.jpg 1600w" width="278" height="212"></figure><p>In <strong>Two</strong> now, click right on <strong>Client</strong> and <strong>Connect to Stream</strong>. A window opens, don't worry, and enter <strong>http://192.168.0.1:8080/workspace0</strong> and click ok. Congratulation, the 2 gephi instance are connected and if everything is working, all structural change in the Master (add, change, delete) should be reflected in the Client.</p><figure class="alignnone size-medium wp-image-215"><img loading="lazy" style="color: #333333; font-style: normal;" alt="ScreenHunter_02 Jul. 30 08.41" src="https://totetmatt.github.io/media/posts/11/ScreenHunter_02-Jul.-30-08.41-300x182.jpg" sizes="(max-width: 48em) 100vw, 768px" srcset="https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_02-Jul.-30-08.41-300x182-xs.jpg 300w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_02-Jul.-30-08.41-300x182-sm.jpg 480w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_02-Jul.-30-08.41-300x182-md.jpg 768w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_02-Jul.-30-08.41-300x182-lg.jpg 1024w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_02-Jul.-30-08.41-300x182-xl.jpg 1360w, https://totetmatt.github.io/media/posts/11/responsive/ScreenHunter_02-Jul.-30-08.41-300x182-2xl.jpg 1600w" width="300" height="182"></figure><h1>External to G<span style="font-size: 26px;">ephi</span></h1><p>One of the possibility is to send action from <strong>external tools</strong> (crawler, web-spider, script that analyse a csv etc...) to a gephi instance.To do that, just open Gephi and launch a <strong>Master Server</strong>. On tool side, just send some HTTP requests like describe <a href="http://wiki.gephi.org/index.php/Specification_-_GSoC_Graph_Streaming_API">here</a>.</p><p>What's interesing here is that you can add value to your tool by having a real-time graph representation of your data. So, instead to have a gexf temporary file, you could just use this fonctionnality to import whatever graph to gephi.</p><p>In terms of code you can check my <a href="https://github.com/totetmatt/GephiStreamer">GephiStreamer </a>python class on github :</p><pre>#GephiStreamerManager
def send(self,action,iGraphEntity):
if type(iGraphEntity) == Node or type(iGraphEntity) == Edge:
postAction = {action:iGraphEntity.object}
else:
postAction = {action:iGraphEntity}
params = json.dumps(postAction) # Create a json that respect the message form
aSendURL = self.name() #http://somehost:someport/someworkspace?operation=updateGraph
r= requests.post(aSendURL, data=params) # Send a Post Requests. If Streamerlaunched, It should display element.</pre><p>Another example in Java:</p><pre>public void sendAction(IGephiStreamAction a) throws Exception
{
OutputStreamWriter writer = null;
URL url = new URL(this.getTargetWorkspaceInstance()+"?operation=updateGraph");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
writer = new OutputStreamWriter(conn.getOutputStream());
writer.write(a.getJson()); //e.g {"an":{"A":{"label":"Streaming Node A","size":2}}}
writer.flush();
conn.getInputStream();
}</pre><p>I think you got the point now ;).</p><p>What is nice is the "handeling of errors" which is quite permissive :</p><ul><li>If you define twice a node, it won't do anything</li><li>If you define a edge with a node that doesn't exist, it won't do anything</li><li>If you update a node that doesn't exists, it won't do anything</li></ul><p>You have to take care about all the special cases, but I find it quite practical to not be psycho-rigid ;-)</p><p><em>I use this technics to convert almost everything to a gephi graph (Data Handcrafted in txt file, log server information converted live with shell scripts etc...) .</em></p><h1>Architecture</h1><p>By knowing these two behaviors, we can think about a streaming architecture where software are talking to gephi that are talking to other gephi.</p><p>Simple exemple, we saw that in Gephi to Gephi that a <strong>Client</strong> can register to a <strong>Master</strong>. But you can register as <strong>many Masters</strong> as you want. So the <strong>Client</strong> can compile<strong> 2 differents graph</strong> at the same time.</p><p>This leads to a open idea of collaborative network analysis and generation. 1 year ago, I've done some experiment at my university. With one topic given (websphere of "abortion" (I would prefert lol cats, but teachers are grumpy in France :D )) , half where crawling pro-website and other half anti-website.</p><p>Architecture used here is simple, each working-pair uses <a title="Arrow V – Command line crawler" href="http://matthieu-totet.fr/Koumin/tools/arrow-v-command-line-crawler/">ArrowV </a>(crayzay advertising isn't it ?), a command line web crawler, on their computer that have Gephi running as a <strong>Master</strong>. We also have a computer used by nobody but that also run Gephi as a <strong>Master</strong> (which is the <strong>super master</strong>). Each pair configures the crawler to stream their crawling result to their<strong> local gephi</strong> and to the <strong>super master gephi</strong>. So that, they can have a view to what they crawled in their <strong>localhost</strong> and the <strong>super master gephi</strong> has the <strong>whole view</strong> of everyone compiled in <strong>one graph</strong>.</p><p>The result was nice to see that 15 people in 30 min are able to map a quick web map of one topic. Ok wasn't perfect and still some works to do, but it was a good proof of concept.</p><p>It raises a lot of (new) questions and constrains, but it opens gate to powerfull tools that smartly mix technology & human.</p><figure class="wp-image-219"><a href="http://matthieu-totet.fr/Koumin/wp-content/uploads/2013/07/StreamArchi.jpg"><img loading="lazy" title="Streaming Architecture" alt="StreamArchi" src="https://totetmatt.github.io/media/posts/11/StreamArchi-1024x790.jpg" sizes="(max-width: 48em) 100vw, 768px" srcset="https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-1024x790-xs.jpg 300w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-1024x790-sm.jpg 480w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-1024x790-md.jpg 768w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-1024x790-lg.jpg 1024w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-1024x790-xl.jpg 1360w, https://totetmatt.github.io/media/posts/11/responsive/StreamArchi-1024x790-2xl.jpg 1600w" width="1024" height="790"></a></figure>This is what is (in theory) achievable if we mix the different way to use the Streaming Plugin.<p></p><p>That's all for the moment. I'll try to keep this article up-to-date if new stuff appears.</p></div><footer class="wrapper post__footer"><p class="post__last-updated">This article was updated on 20/06/20</p><ul class="post__tag"><li><a href="https://totetmatt.github.io/gephi/">Gephi</a></li><li><a href="https://totetmatt.github.io/graph/">graph</a></li><li><a href="https://totetmatt.github.io/real-time/">Real-time</a></li><li><a href="https://totetmatt.github.io/stream/">Stream</a></li></ul><div class="post__share"></div><div class="post__bio bio"><div class="bio__info"><h3 class="bio__name"><a href="https://totetmatt.github.io/authors/totetmatt/" class="invert" rel="author">Totetmatt</a></h3></div></div></footer></article><nav class="post__nav"><div class="post__nav-inner"><div class="post__nav-prev"><svg width="1.041em" height="0.416em" aria-hidden="true"><use xlink:href="https://totetmatt.github.io/assets/svg/svg-map.svg#arrow-prev"/></svg> <a href="https://totetmatt.github.io/augentwitt-pictures-grabber-from-twitter.html" class="invert post__nav-link" rel="prev"><span>Previous</span> Augentwitt - Pictures Grabber from Twitter</a></div><div class="post__nav-next"><a href="https://totetmatt.github.io/lets-play-gephi-understand-degree-weighted-degree-and-betweeness-centrality.html" class="invert post__nav-link" rel="next"><span>Next</span> Let's Play Gephi : Understand Degree, Weighted Degree & Betweeness centrality </a><svg width="1.041em" height="0.416em" aria-hidden="true"><use xlink:href="https://totetmatt.github.io/assets/svg/svg-map.svg#arrow-next"/></svg></div></div></nav><div class="post__related related"><div class="wrapper"><h2 class="h5 related__title">You should also read:</h2><article class="related__item"><div class="feed__meta"><time datetime="2016-04-25T20:33" class="feed__date">16/04/25</time></div><h3 class="h1"><a href="https://totetmatt.github.io/twitter-streaming-importer-naoyun-as-a-gephi-plugin.html" class="invert">Twitter Streaming Importer : Naoyun as a Gephi Plugin</a></h3></article><article class="related__item"><div class="feed__meta"><time datetime="2015-11-05T08:55" class="feed__date">15/11/05</time></div><h3 class="h1"><a href="https://totetmatt.github.io/the-final-hack-of-the-gephi-streaming-plugin.html" class="invert">The Final hack of the Gephi Streaming Plugin</a></h3></article><article class="related__item"><div class="feed__meta"><time datetime="2015-09-06T22:47" class="feed__date">15/09/06</time></div><h3 class="h1"><a href="https://totetmatt.github.io/lets-play-gephi-dbpedia-rdf-sparql-and-your-favorite-actors.html" class="invert">Let's Play Gephi : Dbpedia, RDF, Sparql and your favorite Actors</a></h3></article></div></div></main><footer class="footer"><div class="footer__copyright"><p>Powered by <a href="https://getpublii.com" target="_blank" rel="nofollow noopener">Publii Static CMS</a></p></div><button class="footer__bttop js-footer__bttop" aria-label="Back to top"><svg><title>Back to top</title><use xlink:href="https://totetmatt.github.io/assets/svg/svg-map.svg#toparrow"/></svg></button></footer></div><script>window.publiiThemeMenuConfig = {
mobileMenuMode: 'sidebar',
animationSpeed: 300,
submenuWidth: 'auto',
doubleClickTime: 500,
mobileMenuExpandableSubmenus: true,
relatedContainerForOverlayMenuSelector: '.top',
};</script><script defer="defer" src="https://totetmatt.github.io/assets/js/scripts.min.js?v=f4c4d35432d0e17d212f2fae4e0f8247"></script><script>var images = document.querySelectorAll('img[loading]');
for (var i = 0; i < images.length; i++) {
if (images[i].complete) {
images[i].classList.add('is-loaded');
} else {
images[i].addEventListener('load', function () {
this.classList.add('is-loaded');
}, false);
}
}</script></body></html>