Applying version control to ArcGIS ModelBuilder models?

Question

Version control is an indispensable tool for software development, allowing one to reliably and cleanly step back in time to the last time X did it's job exactly right, or to see what changed between then and now -- typically used when trying to figure out why X is no longer working exactly right.

However all the tools I know about for this work only on plain text files. Toolboxes (standard ones, not the python toolboxes introduced in 10.1), and thus their models, are binary. Does anyone have a workable method to bring versioning to them?

Note: versioning is different from backup. There are any number of simple methods to create snapshots of files for a particular date/time -- Windows backup, previous versions, xcopy /s d:\foobar\ x:\foobar_%date%, zip stuff_%date%.zip stuff\*, and so on.

Applying a tool such as git, fossil, mercurial, subversion, or ... to a binary file is one step better than using xcopy or zip in so much as one can add a commit message, "Model foobar %date% now overwrites previous results only if Baz doesn't exist", but is still anemic compared to what that same toolset can do applied to text files: e.g. show me exactly what was changed between last year and today.

Answer 1

Every mainstream version control software, be it central central version control like SVN or distributed solutions like Git, Mercurial, Bazaar etc. allow storage of binary files. They are all quite effective both performance-wise and also in terms of occupied space.

Inspecting differences between revisions/versions of a file is of course different story. There is not much you can do when you want to compare ArcGIS models, which indeed are binary. I do not know of any diff tools for models, I doubt there will be any as most people do not need this specific functionality.

Answer 2

Currently I have the workflow of ArcCatalog: opening toolbox > selecting model > editing > file > export > to python, switch to SCM tool > refresh changes > commit changes (enter log comment).

It's cumbersome so I don't do it so much, and thereby lose many of the benefits of versioning.

Answer 3

ModelBuilder is old, clunky, and is not getting any significant updates with ArcGIS Pro, if this tweet is any indication. I have never been a big fan of it (though begrudgingly still use it when I have to), so you might consider this answer as a sidestepping of the question and a recommendation to look at alternatives.

FME is arguably the most obvious ModelBuilder alternative, as it has a similar flow diagram UI. One relevant advantage is that its documents are in plain text format, so they can diffed (although there is often a lot of meaningless, auto-generated cruft that you have to learn to ignore). It is commercial software, however, so its cost may be out of reach for some.

Others which I'm less familiar with include Orfeo Toolbox, Whitebox Geospatial Analysis Tools, and QGIS's graphical modeler (based on SEXTANTE). These are all open source modeling environments with GUIs.

A big push that I have observed at GIS and open data conferences in recent years is towards the idea of "reproducible research", that is, data and processes that can be easily shared with and reproduced by others. That often means using open data formats, open source software, and shared repositories. Python and R are very popular for this.

I thought Dharhas Pothina's presentation on Python and GIS earlier this year made a good argument for this. I agree quite strongly that over-reliance on a GUI is detrimental to reproducibility. With code, so long as you are familiar with the language, you can scan through it pretty quickly and figure out what's going on, whereas with a GUI you have to click and scroll around through lots of different windows, often nested deep within one another, to get at values and settings.

Certainly, there is a tradeoff here, but in my opinion, anyone doing any serious work (scientific research, policy-making, etc.) should be using tools which facilitate reproducibility.

Sorry that this doesn't answer your question directly (though I don't believe there is an easy answer).

Answer 4

The introduction of Python toolboxes at ArcGIS 10.1 for Desktop invalidates your four year old statement that all:

Toolboxes, and thus their models, are binary.

Standard toolboxes are binary but Python toolboxes (*.pyt) are text files.

Consequently, I think Python toolboxes should be considered if version control of source code trumps the requirement for a model building GUI.

You are aware of this from your answer on Why learn/use Python Toolboxes over Python Script Tools? but I thought I should include this as an answer here so that the option to use Python toolboxes (to gain easy access to version control) rather than Standard toolboxes is not overlooked by future readers of this Q&A.

Answer 5

In many ways, I am actually glad ESRI did not overhaul the entire Geoprocessing Framework and Modelbuilder with the transition to ArcGIS Pro. There are many (research) organizations who invested heavily in building giant custom models that would undoubtedly need a complete overhaul if ESRI had broken compatibility.

As with the transition to Python / Geoprocessing from the background of Arc/Info AML macro's, this would undoubtedly mean a giant impact and many people at loss. Even some 5 - 8 years after the first release of ArcGIS, there were still researchers and government organizations occasionally referring to AML models on forums like this one, that they hadn't yet been able to convert to Python, due to time, money or other constraints. This just illustrates the potential impact of such a transition, which undoubtedly would be huge.

I agree ModelBuilder can be "clunky" at times if you don't know it well, but since I truly started learning Python, and started understanding the Tool Validation programming (http://resources.arcgis.com/en/help/main/10.2/index.html#//00150000000v000000), much of the "pain" has been releaved. I now way better understand some of the "cryptic" error messages toolvalidation can throw up, and leave you at loss as to which part of the model is broken, and I can now more effectively fix them or prevent them by writing proper tool validation code. This is especially valueable when "integrating" non-Python toolboxes / models with Python scripts.

One wish though I still have, and that would make life with ModelBuilder much easier, is if the automatic toolvalidation actually highlighted the tools, and automatically opened the models, that cause the warnings or errors related to variables. Alternatively, at a minimum, the error and warning listings that pop-up once you click "OK" on an invalid model, should include the exact tool name, and the name of the model, where the tool resides that is invalid. If you have many nested models, finding the tool that caused a particular validation error, can be tedious at times with just a listing of errors that doesn't include the tool or (sub-)model name but just the invalid variable name. I am actually at loss as to why ESRI didn't include the tool and model name in the listing, it seems like an easy fix for this problem.

Also, a kind of "Search" functionality, to find tools by name as defined in the models, would be handy.