|
|
# Monterey Phoenix Graph Trace Import in Neo4j
|
|
|
|
|
|
This guide explains the import of Monterey Phoenix (MP) graph traces into Neo4j using either the web-based MP Firebird or the stand-alone MP Gryphon version. Neo4j (https://neo4j.com/) is a native graph database with both an open source “Community Edition” and commercial “Enterprise Edition”. The same methodology presented can be used to import the MP graph traces into other graph databases, but the syntax provided is specific to Neo4j’s scripting language called Cypher. This guide assumes the reader has basic working knowledge of both MP and Neo4j. If you have any questions or feedback, please reach out to the author at Michael.senft@nps.edu.
|
|
|
|
|
|
Import of the event trace graphs generated by MP into Neo4j is not needed to use the powerful sanity checking tools, reusable architecture patterns, reusable assertions and queries provided by the Monterey Phoenix framework [Auguston, 2020]. However, the combination with Neo4j opens new opportunities to combine behavior models and data. Because the MP schema are already visualized as a graph, a direct one to one mapping of the MP schema within Neo4j is possible.
|
|
|
This guide provides the syntax needed to import the MP traces from both the web-based MP Firebird and stand-alone MP Gryphon. Because the two platforms export different versions of the same information, separate syntax is need.
|
|
|
|
|
|
# MP Firebird Import in Neo4j
|
|
|
|
|
|
Once you have compiled your MP model, you can export the graph traces using the Export feature. Clicking on the EXPORT button at the top of the MP Firebird screen brings up the menu below. Click on the “Code and Event Trace” as illustrated in Figure 1.
|
|
|
|
|
|
![Figure1](uploads/99ae34eaae593c4532e5656d0ec1220c/Figure1.jpg)
|
|
|
Figure 1 – MP Firebird Export Option
|
|
|
|
|
|
This will download a *.wng file to your default Download directory. Opening the file will reveal JSON formatted data as illustrated in Figure 2. This file contains information about the nodes and links (edges) for each of the MP graph traces.
|
|
|
|
|
|
![Figure2](uploads/5b565e803089d1a933ff5412ad002f72/Figure2.jpg)
|
|
|
Figure 2 – MP Firebird *.wng File Export Example
|
|
|
|
|
|
This file next needs to be copied to the “import” directory of your Neo4j database instance. On a Windows system this is typically “C:\Users\<<Username>>\.Neo4jDesktop\neo4jDatabases\database-<<UUID>>\installation-<<Version>>\import”. Each database is assigned a Unique Identifier (UUID) when it is created so it is important you put the file in the correct database instance.
|
|
|
|
|
|
This import also uses the “APOC” plugin for Neo4j, which must be installed. More information can be found at https://neo4j.com/docs/labs/apoc/current/ Finally, you must enable file import in your Neo4j configuration. For a Windows system, you only need to add “apoc.import.file.enabled=true” to the end of your “Settings” file illustrated in Figure 3. On a Linux system, you need to create a file named “apoc.conf” file with “apoc.import.file.enabled=true” on the first line.
|
|
|
|
|
|
![Figure3](uploads/a3aa701ce117a224eb7210102d0b4f48/Figure3.jpg)
|
|
|
|
|
|
Figure 3 – Neo4j Setting Files
|
|
|
With these conditions met, the MP traces can be imported into Neo4j. The four Cypher commands below (in red) need to be executed. The only change needed is replacing <<filename.wng>> with the full name and extension of the file you copied to the Neo4j import folder. A file named car_race.wng would be imported as CALL apoc.load.json("file:///car_race.wng")
|
|
|
|
|
|
##### Create Nodes
|
|
|
*CALL apoc.load.json("file:///<<filename.wng>>") YIELD value UNWIND value.entities AS valueitem UNWIND keys(valueitem.nodes) AS element CALL apoc.merge.node([valueitem.nodes[element].type],{guid:valueitem.nodes[element].guid},{name:valueitem.nodes[element].name,collapsed:valueitem.nodes[element].collapsed,hidden:valueitem.nodes[element].hidden,order:0,probability:0}) YIELD node RETURN node *
|
|
|
|
|
|
### Match nodes and relationships
|
|
|
*CALL apoc.load.json("file:///<<filename.wng>>") YIELD value UNWIND value.entities AS valueitem UNWIND keys(valueitem.links) AS element MATCH (a),(b) WHERE a.guid = valueitem.links[element].source AND b.guid = valueitem.links[element].target CALL apoc.merge.relationship(a,valueitem.links[element].type,{guid:valueitem.links[element].guid,order:0,probability:0,text:CASE WHEN valueitem.links[element].text IS NULL THEN 'null' ELSE valueitem.links[element].text END},{},b) YIELD rel RETURN COUNT(*)*
|
|
|
|
|
|
#### Set order and probability for nodes
|
|
|
*CALL apoc.load.json("file:///<<filename.wng>>") YIELD value UNWIND value.entities AS valueitem UNWIND keys(valueitem.traces) AS element UNWIND valueitem.traces[element].nodes as part MATCH (a) WHERE a.guid = part SET a.order = valueitem.traces[element].order, a.probability = valueitem.traces[element].probability RETURN a*
|
|
|
|
|
|
#### Set order and probability for links
|
|
|
*CALL apoc.load.json("file:///<<filename.wng>>") YIELD value UNWIND value.entities AS valueitem UNWIND keys(valueitem.traces) AS element UNWIND valueitem.traces[element].links as part MATCH ()-[a:FOLLOWS|IN|NAMED]-() WHERE a.guid = part SET a.order = valueitem.traces[element].order, a.probability = valueitem.traces[element].probability RETURN a*
|
|
|
|
|
|
Successful execution of the first two commands will result in the nodes created being returned as a visualization and a count of the relationships created for a given MP trace file. As illustrated in Figure 4.
|
|
|
|
|
|
![Figure4](uploads/3d15c4ddbbf40a2f654962838bf8ca65/Figure4.jpg)
|
|
|
Figure 4 – Neo4j Result from Cypher Command Execution
|
|
|
|
|
|
Neo4j Cypher Note: "value", "valueitem" and "element" are just arbitrary variable names. The name after the period refers to a key within the .wng JSON format. UNWIND is a command to unwind a list in a sequence of rows. “UNWIND keys(valueitem.traces) AS element” takes all of the “traces” in the .wng file (second row of Figure 2) and returns them individually for parsing into nodes and relationships.
|
|
|
|
|
|
Neo4j Cypher Note: "CALL apoc.load.json("file:///<<filename.wng>>") YIELD value UNWIND value.entities" is Cypher syntax for loading a .wng file, returning the file contents as "value" then unwinding the list of "entities" within the file contents and returning each list element as a "valueitem".
|
|
|
|
|
|
Neo4j Cypher Note: "CALL apoc.merge.node([valueitem.nodes[element].type],{guid . . .” creates nodes of the same type as Monterey Phoenix (ATOM, COMPOSITE, ROOT, SAY) with the properties of “name”, “collapsed”, “hidden”, “order” and “probability”. For order and probability, a default value of 0 is set because the actual values will be added later since these values come from the list of "traces" within the file contents instead of the list of "entities" being unwound earlier. MP graph traces that do not include “order” information will result in all nodes having an “order” of 0.
|
|
|
With the command
|
|
|
|
|
|
*MATCH p = (n)-[]-(),(l) WHERE n.order = 1, l.order=1 RETURN p,l*
|
|
|
|
|
|
This will return the first MP graph trace from the Car_Race.mp mode. The same MP graph trace is shown from MP Firebird on the right. The MATCH p=(n)-[]-() command returns all of the connected nodes and relationships for the 1st MP graph traces. Adding MATCH (l) returns the unconnected nodes as well.
|
|
|
|
|
|
![Figure5-1](uploads/1cce93daaa9badd7ff872d64214b59f8/Figure5-1.jpg)
|
|
|
|
|
|
![Figure5-2](uploads/cba57bb43f4c89f55556f86e0482668b/Figure5-2.jpg)
|
|
|
Figure 5 – Neo4j and MP Firebird Comparison
|
|
|
|
|
|
# MP Gryphon Import in Neo4j (stand-alone)
|
|
|
|
|
|
The process to export MP graph traces from MP Gryphon is similar to MP Firebird. Click on “File” as depicted in Figure 6 then on “Export”, which will bring up the screen in Figure 7. Next save the file in the desired directory.
|
|
|
|
|
|
![Figure6](uploads/bfbab2955ba9ee02effe842a836f486e/Figure6.jpg)
|
|
|
Figure 6 – MP Gryphon File Options
|
|
|
|
|
|
![Figure7](uploads/bcf162d6c1e41dd10aac7498202efb9b/Figure7.jpg)
|
|
|
Figure 7 – MP Gryphon File Export
|
|
|
|
|
|
The file format of the MP Gryphon trace graph export is similar to the MP Firebird trace graph export, but lacks the Unique Identifiers assigned to each node and relationship.
|
|
|
|
|
|
![Figure8](uploads/bb4fd19a288142546f47bd50e565e73a/Figure8.jpg)
|
|
|
Figure 8 - *.gry File Format
|
|
|
|
|
|
The *.gry file can also be directly imported into Neo4j by copying the file to the “import” directory of your Neo4j database instance. On a Windows system this is typically “C:\Users\<<Username>>\.Neo4jDesktop\neo4jDatabases\database-<<UUID>>\installation-<<Version>>\import”. Each database is assigned a Unique Identifier (UUID) when it is created so it is important you put the file in the correct database instance.
|
|
|
|
|
|
There Cypher syntax to import the MP Gryphon graph traces is below. Because of the different formatting, only two statements are needed to import all of the information. There is one key limitation as only a single set of MP Gryphon graph traces can be imported. Also, all of the nodes are created with the type “MP_Node” with a “type” property of their respective Node type from MP. The ability to import multiple sets of MP Gryphon graph traces is provided by a Python script discussed in the next section.
|
|
|
|
|
|
Create Nodes
|
|
|
|
|
|
*CALL apoc.load.json('file:///<<filename.gry>>') YIELD value UNWIND value.graphs AS item UNWIND item.nodes AS test CALL apoc.merge.node(['MP_Node'],{traceIndex:item.index, label:test.label, id:toInteger(test.id), type:CASE WHEN test.type = 'R' THEN 'ROOT' WHEN test.type = 'A' THEN 'ATOM' WHEN test.type = 'C' THEN 'COMPOSITE' ELSE 'SAY' END, probability:item.probability, traceMP_Code: substring(value.mp_code, 3,40)}) YIELD node RETURN node*
|
|
|
|
|
|
Create Relationships
|
|
|
|
|
|
*CALL apoc.load.json('file:/// <<filename.gry>>') YIELD value UNWIND value.graphs AS item UNWIND item.edges AS test MATCH (a:MP_Node {traceIndex:item.index, id:toInteger(test.source)}), (b:MP_Node {traceIndex:item.index, id:toInteger(test.target)}) CALL apoc.merge.relationship(a,test.relation,{probability:item.probability, labels:CASE WHEN item.labels IS NULL THEN 'null' ELSE item.labels END, traceMP_Code: substring(value.mp_code, 3,40)},{},b) YIELD rel RETURN rel*
|
|
|
|
|
|
![Figure9](uploads/21537fb84eb2d1998d4fdab22c2f4279/Figure9.jpg)
|
|
|
Figure 9 – Result of import of MP Gryphon graph traces into Neo4j
|
|
|
|
|
|
# MP Gryphon Import in Neo4j (Python 3.x)
|
|
|
To address the limitation of being able to import multiple MP Gryphon graph traces into Neo4j, I wrote a Python script that leverages the Neo4j Python Driver (https://neo4j.com/docs/api/python-driver/current/) to create Unique Identifiers (UUIDs) for each of the traces as well as functionality to refactor nodes into their respective MP schema types. The Python script can be used as a stand-alone Python program (.py) or as a Jupyter Notebook (.ipynb). The code is well commented and below is an example of the Cypher scripts that are created and executed within Neo4j for a MP Gryphon graph trace export file named “Car.gry”
|
|
|
|
|
|
#Creates an index for the MP Gryphon trace graph nodes imported to greatly increase the speed of refactoring the nodes to their type as found in MP
|
|
|
|
|
|
*CREATE INDEX mpindex FOR (a:MP_Node) ON (a.id, a.traceIndex, a.traceUUID)*
|
|
|
|
|
|
#Creates the nodes from the MP Gryphon trace as "MP_Node" types with properties for "traceIndex, “label”, “type”, “probability”, “traceUUID” and “mp_code”, a 37 character excerpt of mp_code entry
|
|
|
|
|
|
*CALL apoc.load.json('file:///Car.gry') YIELD value UNWIND value.graphs AS item UNWIND item.nodes AS test CALL apoc.merge.node(['MP_Node'],{traceIndex:item.index, label:test.label, id:toInteger(test.id), type:CASE WHEN test.type = 'R' THEN 'ROOT' WHEN test.type = 'A' THEN 'ATOM' WHEN test.type = 'C' THEN 'COMPOSITE' ELSE 'SAY' END, probability:item.probability, traceUUID:'ff4f5e67-e3d0-4310-b21d-c35d25d47260', traceMP_Code: substring(value.mp_code, 3,40)}) YIELD node RETURN node*
|
|
|
|
|
|
#Creates the relationships (edges) between the MP Gryphon trace nodes with properties for "probability”, “label”, “traceUUID” and “traceMP_Code”
|
|
|
|
|
|
*CALL apoc.load.json('file:///Car.gry') YIELD value UNWIND value.graphs AS item UNWIND item.edges AS test MATCH (a:MP_Node {traceIndex:item.index, id:toInteger(test.source), traceUUID:'ff4f5e67-e3d0-4310-b21d-c35d25d47260'}), (b:MP_Node {traceIndex:item.index, id:toInteger(test.target), traceUUID:'ff4f5e67-e3d0-4310-b21d-c35d25d47260'}) CALL apoc.merge.relationship(a,test.relation,{probability:item.probability, label:CASE WHEN item.label IS NULL THEN 'null' ELSE item.label END, traceUUID:'ff4f5e67-e3d0-4310-b21d-c35d25d47260', traceMP_Code: substring(value.mp_code, 3,40)},{},b) YIELD rel RETURN rel*
|
|
|
|
|
|
#Refactors the "MP_Node" type to their repective ROOT, ATOM, COMPOSITE or SAY types
|
|
|
|
|
|
*match (n:MP_Node) WHERE n.type = "ROOT" with collect(n) as nodes CALL apoc.refactor.rename.label('MP_Node','ROOT',nodes) yield errorMessages as eMessages return eMessages
|
|
|
match (n:MP_Node) WHERE n.type = "ATOM" with collect(n) as nodes CALL apoc.refactor.rename.label('MP_Node','ATOM',nodes) yield errorMessages as eMessages return eMessages
|
|
|
match (n:MP_Node) WHERE n.type = "COMPOSITE" with collect(n) as nodes CALL apoc.refactor.rename.label('MP_Node','COMPOSITE',nodes) yield errorMessages as eMessages return eMessages
|
|
|
match (n:MP_Node) WHERE n.type = "SAY" with collect(n) as nodes CALL apoc.refactor.rename.label('MP_Node','SAY',nodes) yield errorMessages as eMessages return eMessages*
|
|
|
|
|
|
#Deletes the index created initially for the "MP_Node" type because all of these nodes have been refactored into their appropriate type
|
|
|
*DROP INDEX mpindex*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\ No newline at end of file |