I was recently asked to migrate a large database with data from one demo server to another. That is a simple task with the MLCP copy command. It even allows you to add collections, permissions, migrate to a different database root, etc. Unfortunately, I was asked to not only migrate the database, but the associated app servers as well, and impose a permission structure that allowed access to the data from a REST api instance. I decided to capture all relevant details in a Roxy project, and use that to automate deployment to the target environment.
Initializing a Roxy project
What you need first is an empty Roxy project structure. I’ll be assuming a project name myapp, running against MarkLogic 7. Get hold of the ml script if you don’t have it yet (you can download it from https://github.com/marklogic/roxy/tree/dev), and run the following command:
ml new myapp --server-version=7 --branch=dev --app-type=rest
This will create a Roxy project in a subfolder called myapp. It takes the dev branch of Roxy to utilize the latest cutting edge features, which we will need for the capture functionality that we are going to use here. It also creates a REST-type project, which gives you the emptiest Roxy project structure you can currently get with the ml new command.
Setting up environments
The next step is to add details about the relevant environments. In my case it concerned a development, and a production environment, so I used the pre-existing dev and prod environment labels. You can add your own ones as well. Just edit the environments property in deploy/build.properties. Create a environment-specific properties file. For dev you create a deploy/dev.properties. I typically put the following lines in such a file:
user=gjosten password= app-port=8058 xcc-port=8059 content-forests-per-host=3 dev-server=mydev.server.com
Roxy will ask for the password if you keep it empty. Note: make sure that the name of the server property matches the environment. So it is dev-server for dev.properties, but prod-server for prod.properties.
Capturing ml-config
Once this is done, you are ready to take the first step in capturing MarkLogic settings, and code. Just run the following command:
./ml dev capture --full-ml-config
Replace ‘dev’ with the appropriate environment. The above command will create a new file named deploy/ml-config-dev.xml. It will contain a list of all app servers, databases, amps, users, roles, etc from the specified environment. You don’t want to bootstrap that, and luckily Roxy normally ignores the new file. Go into this file, and isolate all parts that are relevant for your application. Copy these over to deploy/ml-config.xml.
You probably want to replace the default parts generated by Roxy, but put them next to each other first. Roxy can use placeholders to insert values from the properties files. If you matched your project name with (partial) names of databases and app servers, you could decide to copy some placeholders over. One useful case could be to use app-port and xcc-port placeholders to aim for different ports per environment. Add more properties to the properties files, if you have additional app servers.
You could in theory replace the entire ml-config.xml with the captured ml-config, but usually that is not advisable. In case you do start off with the captured one, make sure to remove the XML processing-instruction at the top.
Testing ml-config
A large benefit from Roxy here is that you can easily do some dry runs against a local VM, or your own laptop. Run the following command to create app-servers, databases, and anything else you selected on your local environment:
./ml local bootstrap
Tweak the ml-config until bootstrap runs flawlessly. Then open the Admin interface, and verify everything looks complete and running correctly. Once here you are ready to bootstrap the target environment. That is just a matter of running bootstrap against a different environment.
Capturing modules and REST extensions
Just capturing and deploying the ml-config will likely not result in a fully functioning application. It very likely depends on additional code, like modules or REST extensions. Roxy provides two additional capture functions to get hold of those. If you have a more traditional application, not using the more recent REST api, you can run this:
./ml dev capture --modules-db=mymodules
Replace ‘mymodules‘ with the appropriate modules database name. All files in that database will be written to the src/ folder.
If your app is in fact a REST api instance, like applications generated with the App Builder, use this command instead:
./ml dev capture --app-builder=myappserver
Replace ‘myappserver’ with the name of the app-server that is the REST api instance. This will capture modules into the src/ folder, but also isolate REST transforms, REST extensions, and REST options into the rest-api/ folder.
Roxy by default assumes there is just one project-specific modules database. There are ways to deploy multiple sets of sources to different modules databases. But you might consider capturing those in separate projects. That is probably easier.
Testing deploying modules
You are close to having reproduced an entire MarkLogic application with just a few commands! Test the capture of modules and REST extensions by deploying them locally:
./ml local deploy modules
This will deploy both src, and all REST artifacts. After this you should be able to go to the newly created app servers, and have running applications! Repeat above against the target environment to get them up and running there as well.
Copying documents
Last step in the process if of course copying the contents of the document databases, and maybe also schemas, and triggers. MLCP is a very useful tool for that. You can either use separate MLCP export and import. You can use ml {env} mlcp for that! Or use MLCP copy to transfer directly between source and target. Unfortunately, you can’t use ml {env} mlcp for that (yet)..
Good luck!
