A random graph containing 500,000 nodes and 500,000 edges. You can pan and zoom around the map seamlessly.
shingle.js is designed to publish large interactive graphs online.
It consists of two parts: an off-line tool that chops your large network graph into tiles stored in json files, and a Javascript library which loads these tiles when in view and inserts and removes them from the DOM of a single svg element.
Steps for designing large graphs for online visualization
For visualization or large network graphs through shingle.js, a few preliminary steps are advisable.
The data
It only makes sense to use shingle.js if you have a huge graph, in the millions of nodes and edges between the nodes. In addition, the nodes and edges may be annotated with weights, labels and categorizations.
Layouts
In order to draw the graph onto a two-dimensional plane, a "layout" has to be created: each node needs to be assigned a position in a two-dimensional plane. Shingle currently does not offer facilities for creating layouts, so external software needs to be used for that.
There are various algorithms that can do it, but OpenOrd seems to be the only program that can handle really huge datasets, so OpenOrd seems to be the only option currently.
OpenOrd allows for tuning parameters, and it is useful to play with these parameters, to get the visualization just right. It may take several iterations to get the parameters right.
Node and edge categorizations
The nodes and edges may come with their own categorizations. Otherwise, some heuristic method may be used to determine categorizations. The current algorithms for community detection do not seem to be capable of handling truly large data sets.
Visual design
The graph needs to be designed visually also to help communicate the information presented. The default design might not suit the purposes. Node and edge sizes and colors and associated icons and labels need to be determined, as well as what information needs to be shown when clicking on a node or an edge.
Importing the data into the shingle toolchain
A front-end program needs to be written to convert the graph data into a format the Shingle processing tools can deal with.
Deployment
In the end, after that, the Shingle processing tools can run, reading in the graph and generating the tiles needed for the Javascript code to display the graph.
Usage instructions
The package contains an utility that chops up a graph into tiles, and a Javascript framework that allows display of the tiled graph, using svg.
Build instructions
- Download source code here.
- Edit the 'makeinclude' file: modify at least OUTPUT_PATH so that it points to a directory where the resulting output files can be stored.
- Type 'make'
That's it! The OUTPUT_PATH directory should now contain a directory 'random-graph_500000'. Open the file '_sample.html' inside that directory, and you should see the above map.
How it works
There are two executables in the bin/ directory inside the OUTPUT_PATH directory:
- json2bin.a - converts a json file containing a graph into a binary form, preparing it for the next step
- quadbuilder.a - splits up the graph into tiles
These commands are invoked as follows: to for example convert a graph in json_input.json and have the results stored in the directory graph_output_path/, invoke the commands:
$(OUTPUT_PATH)bin/json2bin.a json_input.json nodes.bin edges.bin
$(OUTPUT_PATH)bin/quadbuilder.a nodes.bin edges.bin graph_output_path/
In addition, the files shingle.js and shingle.css are needed.
A minimal html file that contains the graph:
<html>
<head>
<link rel="stylesheet" href="shingle.css">
<script src="shingle.js" type="text/javascript"></script>
</head>
<body>
<div id="shinglecontainer" data-width="100%" data-height="6in" data-graph-path="graph_output_path/" ></div>
</body>
</html>
With the basic parameters to a shinglecontainer div:
- data-width="..." - specify width of the graph viewer
- data-height="..." - specify the height of the graph viewer
- data-graph-path="..." - specify the path to the data. Useful for providing more graphs, or for destroying the browser cache.
- data-node-field="..." - the name to use in the link to specify what node to highlight initially.
- data-debug-quads="true" - show the tile boundaries, for debugging purposes.
Shingle can display info about active or hovered nodes in an info panel in the graph. The info panel is a separate supplied component that can be synchronized using callbacks.
The JS code to show the graph on a page, including info panel:
/*
* create an info component in the graph panel, here we use the supplied shingleInfoPanel.js
*/
var theInfoData = new shingleInfoPanel();
/*
* create the graph
*/
var myGraph = shingle.new({
// use the created element
el: document.getElementById("shinglecontainer"),
// supply info panel content
infoContentEl: theInfoData.el,
onClear: function() {
//
// clear the info panel
theInfoData.clear();
}, onFocus: function(quadid, nodeid, data) {
//
// when clicked on node in graph set this node in the info panel
theInfoData.setMainNode(quadid, nodeid, data);
}, onFocusRelatedNode: function(quadid, nodeid, data) {
//
// when clicked on node in graph this is called for each related node
// set the related nodes in the info panel
theInfoData.appendRelatedNode(quadid, nodeid, data);
}, onHoverIn: function(quadid, nodeid) {
//
// when hovered (mousover) on node in graph highlight in the info panel
theInfoData.highLightNode(quadid,nodeid);
}, onHoverOut: function() {
//
// when blur (mousout) on node in graph unhighlight in the info panel
theInfoData.unHighLightNode();
}
});
/*
* set the info panel hover behaviour
*/
theInfoData.onHover(function(quadId, nodeId) {
//
// highlight the hovered node in the graph
myGraph.hoverIn(quadId, nodeId);
});
/*
* set the info panel click behaviour
*/
theInfoData.onClick(function(quadid, nodeid) {
//
// make the cliched node active in the graph
myGraph.changehighlightTo(quadid, nodeid);
});
To run without the info panel simply delete the 'theInfoData' logic and omit the 'infoContentEl' parameter. To create your own info panel or custom behaviour attached to graph UI actions you can create your own components, and wire it up to shingle using the callbacks specified in shingle.new.
Parameters can be supplied as data attributes of the shinglecontainer, or passed to the JS shingle.new call without the 'data' part and in camelCase. So for instance we could replace the 'data-graph-path' attribute with a graphPath parameter in the call to shingle.new:
...
/*
* create the graph
*/
var myGraph = shingle.new({
// use the created element
el: document.getElementById("shinglecontainer"),
// supply info panel content
infoContentEl: theInfoData.el,
graphPath: 'graph_output_path/',
...
More advanced parameters:
- initialZoom: "..." - specify initial zoom level (1-100)
- scrollZoom: true | false - scrolling in the graph zooms (mousewheel), default true
- scrollZoomInitDelay: "..." - milliseconds to wait after scrolling to trigger zoom, to prevent long page scrolls to get stuck in a zoom action when passing the graph, default 400
- fontFamily: "sans" - font to use for text in the graph
- fontColor: [r, g, b, a] - color for text in the graph, specified as array of rgb or rgba values
- nodeColors: [[r, g, b, a], .. ] - color scheme, array of colors to use in the graph for the communities, the colors are specified as arrays of rgb or rgba values, and cycles if the number of communities exceeds the number of colors
- renderWhenNodeNotFound - when false no map is displayed, default true
- selectNodes - can the user select nodes by clicking or touching them? default true
- panning - dragging the map is allowed? default true
- zoomSlider - can the user zoom in / out using a zoomslider? default false
- staticMap - disable zooming, panning and selecting nodes, default false
- showRelatedNodeNames - false, true (display related node names), 'hover' (display on hover of related node)
- fontSize - option to set a fixed fontsize in px for display, default 18
- textGrow - will the text grow when the zoomlevel increases above initialzoom?
default true
- textShrink - will the text shrink when the zoomlevel increases above initialzoom? default false
- textGrow and textShrink increase or decrease the font size displayed linear to the zoomlevel
- relatedFontSize - font size for related nodes names when displayed
- translateOffsetX - added SVG CSS X translate offset
- translateOffsetY - added SVG CSS Y translate offset
More advanced callback function parameters:
- onNodeNotFound - callback function called when node specified by nodeid parameter was not found
- onQuadsDrawn - called each time quads are drawn
- onFirstQuadDrawn - called the first time a quad is drawn
Functions to use with the shingle object (returned by shingle.new):
- zoomIn - zoom in using step supplied in option zoomStep, default 5%
- zoomOut - zoom out using step supplied in option zoomStep, default 5%
- zoomReset - reset zoom level to option intitialZoom
All CSS classnames start with shingle-, if this still would collide with your css you can overrule the classnames by supplying custom classnames in parameters: containerClass, containerWithFocusClass, scaleElClass, translationElClass, nodeClass, nodeTextClass and mapClass.
See
_sample.html
for an example of how to then display the graph, or an example where an initial node is highlighted:
_sample.html?nodenumber=499600,
or an example showing the edges of the tiles:
_sample.html?debugquads=true
The primary input file for the tool chain is a json file that contains an array of nodes and an array of edges between the nodes, in the following format:
{
"nodes":
[
{
"nodeid": "0",
"name": "Node number 0",
"x": -4.69657,
"y": -111.763,
"size": 1,
"community": 644
},
...
{
"nodeid": "99999",
"name": "Node number 99999",
"x": -345.672,
"y": -154.746,
"size": 1,
"community": 334
}
],
"relations":
[
{
"nodeidA": "0",
"nodeidB": "489087",
"strength": 1.000000
},
...
{
"nodeidA": "499957",
"nodeidB": "499964",
"strength": 1.000000
}
]
}
This distribution comes with one example graph, 'sampledata/random-graph_500000.json', which is the one you see in the banner above.
License
Contributors
- Ingrid Halters
- Rene Buhrs