shingle.js

Publish large interactive graphs online

A random graph containing 500,000 nodes and 500,000 edges. You can pan and zoom around the map seamlessly.

shingle.js is designed to publish large interactive graphs online.

It consists of two parts: an off-line tool that chops your large network graph into tiles stored in json files, and a Javascript library which loads these tiles when in view and inserts and removes them from the DOM of a single svg element.

Steps for designing large graphs for online visualization

For visualization or large network graphs through shingle.js, a few preliminary steps are advisable.

The data

It only makes sense to use shingle.js if you have a huge graph, in the millions of nodes and edges between the nodes. In addition, the nodes and edges may be annotated with weights, labels and categorizations.

Layouts

In order to draw the graph onto a two-dimensional plane, a "layout" has to be created: each node needs to be assigned a position in a two-dimensional plane. Shingle currently does not offer facilities for creating layouts, so external software needs to be used for that.

There are various algorithms that can do it, but OpenOrd seems to be the only program that can handle really huge datasets, so OpenOrd seems to be the only option currently.

OpenOrd allows for tuning parameters, and it is useful to play with these parameters, to get the visualization just right. It may take several iterations to get the parameters right.

Node and edge categorizations

The nodes and edges may come with their own categorizations. Otherwise, some heuristic method may be used to determine categorizations. The current algorithms for community detection do not seem to be capable of handling truly large data sets.

Visual design

The graph needs to be designed visually also to help communicate the information presented. The default design might not suit the purposes. Node and edge sizes and colors and associated icons and labels need to be determined, as well as what information needs to be shown when clicking on a node or an edge.

Importing the data into the shingle toolchain

A front-end program needs to be written to convert the graph data into a format the Shingle processing tools can deal with.

Deployment

In the end, after that, the Shingle processing tools can run, reading in the graph and generating the tiles needed for the Javascript code to display the graph.

Usage instructions

The package contains an utility that chops up a graph into tiles, and a Javascript framework that allows display of the tiled graph, using svg.

Build instructions

That's it! The OUTPUT_PATH directory should now contain a directory 'random-graph_500000'. Open the file '_sample.html' inside that directory, and you should see the above map.

How it works

There are two executables in the bin/ directory inside the OUTPUT_PATH directory: These commands are invoked as follows: to for example convert a graph in json_input.json and have the results stored in the directory graph_output_path/, invoke the commands:
$(OUTPUT_PATH)bin/json2bin.a json_input.json nodes.bin edges.bin
$(OUTPUT_PATH)bin/quadbuilder.a nodes.bin edges.bin graph_output_path/
In addition, the files shingle.js and shingle.css are needed.

A minimal html file that contains the graph:

<html>
  <head>
    <link rel="stylesheet" href="shingle.css">
    <script src="shingle.js" type="text/javascript"></script>
  </head>
  <body>
    <div id="shinglecontainer" data-width="100%" data-height="6in" data-graph-path="graph_output_path/" ></div>
  </body>
</html>

With the basic parameters to a shinglecontainer div:

Shingle can display info about active or hovered nodes in an info panel in the graph. The info panel is a separate supplied component that can be synchronized using callbacks.

The JS code to show the graph on a page, including info panel:

/*
 * create an info component in the graph panel, here we use the supplied shingleInfoPanel.js
 */
var theInfoData = new shingleInfoPanel();

/*
 * create the graph
 */
var myGraph = shingle.new({
  // use the created element
  el: document.getElementById("shinglecontainer"),
  // supply info panel content
  infoContentEl: theInfoData.el,
  onClear: function() {
    //
    // clear the info panel
    theInfoData.clear();
  }, onFocus: function(quadid, nodeid, data) {
    //
    // when clicked on node in graph set this node in the info panel
    theInfoData.setMainNode(quadid, nodeid, data);
  }, onFocusRelatedNode: function(quadid, nodeid, data) {
    //
    // when clicked on node in graph this is called for each related node
    // set the related nodes in the info panel
    theInfoData.appendRelatedNode(quadid, nodeid, data);
  }, onHoverIn: function(quadid, nodeid) {
    //
    // when hovered (mousover) on node in graph highlight in the info panel
    theInfoData.highLightNode(quadid,nodeid);
  }, onHoverOut: function() {
    //
    // when blur (mousout) on node in graph unhighlight in the info panel
    theInfoData.unHighLightNode();
  }
});                                        

/*
 * set the info panel hover behaviour
 */
theInfoData.onHover(function(quadId, nodeId) {
  //
  // highlight the hovered node in the graph
  myGraph.hoverIn(quadId, nodeId);
});

/*
 * set the info panel click behaviour
 */
theInfoData.onClick(function(quadid, nodeid) {
  //
  // make the cliched node active in the graph
  myGraph.changehighlightTo(quadid, nodeid);
});

To run without the info panel simply delete the 'theInfoData' logic and omit the 'infoContentEl' parameter. To create your own info panel or custom behaviour attached to graph UI actions you can create your own components, and wire it up to shingle using the callbacks specified in shingle.new.

Parameters can be supplied as data attributes of the shinglecontainer, or passed to the JS shingle.new call without the 'data' part and in camelCase. So for instance we could replace the 'data-graph-path' attribute with a graphPath parameter in the call to shingle.new:

...

/*
 * create the graph
 */
var myGraph = shingle.new({
  // use the created element
  el: document.getElementById("shinglecontainer"),
  // supply info panel content
  infoContentEl: theInfoData.el,
  graphPath: 'graph_output_path/',
  ...

More advanced parameters:

More advanced callback function parameters:

Functions to use with the shingle object (returned by shingle.new):

All CSS classnames start with shingle-, if this still would collide with your css you can overrule the classnames by supplying custom classnames in parameters: containerClass, containerWithFocusClass, scaleElClass, translationElClass, nodeClass, nodeTextClass and mapClass.

See

_sample.html

for an example of how to then display the graph, or an example where an initial node is highlighted:

_sample.html?nodenumber=499600,

or an example showing the edges of the tiles:

_sample.html?debugquads=true

The primary input file for the tool chain is a json file that contains an array of nodes and an array of edges between the nodes, in the following format:

{
  "nodes": 
  [
    {
      "nodeid": "0",
      "name": "Node number 0",
      "x": -4.69657,
      "y": -111.763,
      "size": 1,
      "community": 644
    },
    
    ...
    
    {
      "nodeid": "99999",
      "name": "Node number 99999",
      "x": -345.672,
      "y": -154.746,
      "size": 1,
      "community": 334
    }
  ],
  "relations": 
  [
    {
      "nodeidA": "0",
      "nodeidB": "489087",
      "strength": 1.000000
    },

    ...

    {
      "nodeidA": "499957",
      "nodeidB": "499964",
      "strength": 1.000000
    }
  ]
}

This distribution comes with one example graph, 'sampledata/random-graph_500000.json', which is the one you see in the banner above.

License

Contributors