--- alias: operation-guide-deployment description: "This documentation outlines the deployment process, detailing package structure, installation stages, and step types used to manage software installations across" --- # Deployment Overview A Package contains artifacts and metadata about them. Package metadata describes the overall sequence of actions required to install it in a given system across one or multiple machines known as targets. A package describes its installation process with sequence of steps having an identity and a type. The identity allows identifying them and the type is associated with a class that will take care of executing that particular action. Complex systems can be divided into multiple Packages held together by a root package. This root package has no specific requirements but it's a good practice to create a root package without artifacts that contains only the metadata to allow discovering the list of packages that compose the system. The overall list of packages to install is determined by collecting all the package dependencies and all explicitly deployed packages. A package is considered explicitly deployed when it is referenced by a `DeployPackage` step type. ## Installation Process Overview The installation process is handled by a processing pipeline that goes through several stages. The following diagram illustrates all of them in sequence. Each step in every package will go through each of the stages. The classes associated with the step type might add code to handle that stage or if they have nothing to do can ignore it. ![Installation Pipeline][installation_pipeline] When beginning the deployment process, the deployment framework (DF) attempts to determine if any of the packages to be installed has been deployed to the target. If at least one is found, the installation is considered under upgrade mode, otherwise it is considered under clean mode. In either clean or upgrade mode, packages are deployed by migrating from the earliest version to the one referred either by dependencies or by explicit deployment. When selecting what versions to deploy, the DF checks the migration strategy for each of them. The migration strategy can either be `LatestOnly` or `MigrationPath`. For `LatestOnly`, it will scan the package sources for the highest version number of that package and will deploy just that one. For `MigrationPath`, it will scan the package sources for the packages with versions between the installed version and the latest version. If there is no installed version, it will start at the lowest version number found. Steps can set target versions for packages to control the topmost version that will be considered. They can use wildcards to indicate the latest package compatible with version x.1 (should be x.1.*). Steps have an extensible syntax because they allow arbitrary inner elements. This feature allows custom step handlers to include any kind of metadata they require. ## Pipeline stages Each pipeline stage is executed for all packages to be deployed. The process only advances to the next stage after passing through all steps. All steps will first run initialize actions, then all steps will run subsequent acquire actions until cleanup is reached. The next points summarize what will be happening at each stage of the pipeline. ### Initialize 1. Parse the root package manifest, flatten the dependency tree and dedupe it 1. Read the `parameters.json` file 1. Read the packages and detect the variables 1. Detect the migration strategy for each package id (`LatestOnly`, `MigrationPath`) 1. Assemble the execution plan (the packages to install and the order to install them) 1. Assemble the pipeline (all the steps in the packages to install in a tree like structure based on the ordered list of packages to install) 1. Build a list of the deployment targets (machines, containers) 1. Connect to the deployment agent on the target machine ### Acquire 1. Detect what is installed and set the Installation Mode 1. Check if preparation is required and output the preparation notice (what will be created or changed) ### Validate 1. Do we have all the required variables? 1. Do we have all the prerequisites installed? If no, can we install them? 1. Can we establish the required connections? 1. Do we have the required authorizations? 1. On error can we continue, or should we abort? ### Prepare 1. Install the prerequisites 1. Create or modify the environments that will accept the packages (databases, folders, windows services and so one) ### Execute 1. Deploy all the packages in sorted order by running the assembled pipeline. ### Complete 1. For databases, complete tasks such as finalizing **AlwaysOn** configurations 1. Start Windows Services 1. Register uninstall scripts to remove what was installed if required 1. Update the deployment journals so we can keep track of what was installed ### Cleanup 1. Delete the working directories and temporary files that were created by the installation. ## Package Type and Package Step Type Package Type and Package Step Type link a manifest element with the class that will take care of interpreting it. The package type handlers are also responsible for collecting the variables that they require to operate and to customize the environment to receive the artifacts. Inside the package there might be variations on how to handle each content part and that is when step types come into play. They allow the package handler to deploy the content that is inside the package. ## Package Manifest Every package must have one and only one manifest file. The manifest file is always named manifest.xml. The following example illustrates the format: ```xml Cmf.Online.Mes Database Creates the MES database objects. 5.0.0 ``` ## DeploymentPackage The root element of the package manifest. 1. version - an attribute that identifies the version of the package manifest format. ## PackageId An element holding the identifier for the package. ## PackageType An element that groups related packages together. It is currently used to tie the package to one of the product tiers. In terms of the deployment framework it allows running logic before the first package of that given type and after the last one. ## Description An element that allows entering text to describe the package content or purpose. ## Dependencies A list of optional or mandatory dependency packages. ## Version A version number according to the [Semantic Version](https://semver.org/) specification. ## Steps A list of the steps required to install the package. ### Step One task to execute when deploying the package. 1. type - an attribute that indicates the kind of handler that will be called. 2. contentPath - an attribute that indicates the path to the artifact to be delivered to the handler. The step can take any extra number of XML attributes besides these. The deployment framework provides an API to allow the step handler to read them. The step can also take any number of inner XML elements. Some standard step types use this feature to add extra deployment options to the manifest. ## UI The UI metadata allows declaring how the variables required by the package will be prompted to the user. ```xml ... ``` ### Wizard Step The wizard step will be rendered as a step in the setup ui: ![Wizard Steps][breadcrumb] ```xml ... ``` 1. id - creates an unique identifier for the step that will be used during validation. each step should have an unique id 1. order - the order of the step in the wizard 1. title - the text that will be placed in the wizard breadcrumb 1. type - used to identify screens that required special treatment. In version 1.0 customization of screens is not supported and generic should be used for non-product wizard steps. 1. requiresValidation - used to indicate if the validation API should be called when the user clicks next ### Variables and Variable A variable will be an input for the user to supply. It can then be used in step attributes. Generally, the `$(VariableName)` expression will get replaced by the variable value in the variables dictionary. Some variables are handled in a special way to simplify step declaration. One such example is the product database connection string. In these cases, the developer will refer to something like `$(Product.Database.Online)` and the framework will assemble the connection strings using the information in the variable dictionary. ```xml ``` Attributes: 1. name - the string that will identify the variable in the variable dictionary. On step attributes the `$(VariableName)` expression will get replaced by the value in the dictionary. 1. order - used to sort the inputs in the corresponding group or wizardStep 1. readOnly - if true the variable will not be editable by the user 1. isRequired - if true it will not be possible to advance to the next step until a value is supplied 1. valueType - describes the type of data that will be captured by the input. 1. label - the value for the label that will be rendered next to the input of the variable Variable types: 1. Text 1. FilePath 1. FolderPath 1. NetworkPort 1. IpAddress 1. Integer 1. Password 1. Boolean ### Groups and Group Creates a container for variables that has independent validation and represents some sort of logic relationship between the variables contained within it: ![Groups][group] ```xml ... ``` 1. id - creates a unique identifier for the group that will be used during validation. each group should have a unique id 1. order - used to sort the corresponding group in the wizardStep or in a parent group 1. title - the label for the visual container of the group 1. type - used to give special treatment to some specific groups in the product. In version 1.0 customization of visual appearance is not supported and you should use generic. 1. requiresValidation - used to indicate if the validation API should be called when the user clicks test ## Databases Each database is assigned a logical name that represents it. The logical name is not the actual name of the database but a way to refer to it without depending on the name chosen for the database. ## Step Types The deployment framework supports the following base steps: ### DeployPackage This step explicitly deploys a given package. ```xml ``` ### DeployFiles Allows copying files from the package to a destination. The current version only supports two destinations that are indicated by the `PackageType`. If it is `Presentation` then the `PresentationFileLocation` element in the Global Manifest. If it is `Business` the `BusinessFileLocation` element is used. 1. contentPath - indicates what will be copied. It is possible to use Minimatch patterns to filter content. ### RunPowershell Executes the PowerShell script given in contentPath: ```xml ``` ### RunSql Executes the SQL script against the database declared in the targetDatabase attribute. Support values are Online, Ods and Dwh. ```xml ``` ### RunClickHouseSql Executes an SQL script against the ClickHouse database specified in the targetDatabase attribute. Supported values are: `$SYSTEM_NAME`, `$SYSTEM_NAME`CDM, `$SYSTEM_NAME`ODS and `$SYSTEM_NAME`DWH. At the end of execution, the specified patchId is automatically inserted into the `T_DatabasePatches` table, unless it already exists. This happens regardless of whether the condition (if provided) evaluates to true or false, unless the script itself performs the insertion. ```xml ``` This step has the following attributes: 1. **contentPath** - the path to the SQL file(s) to execute. 2. **targetDatabase** - the target ClickHouse database on which to run the script. 3. **conditionPath** - (optional) the path to a condition SQL script. The main script will only execute if this condition evaluates to `true`. 4. **patchId** - an unique identifier for this script. The script runs only if this patchId is not already recorded in the `T_DatabasePatches` table of this database. 5. **patchDescription** - (optional) a description to include when automatically inserting the patch record. 6. **replaceTokens** - enables token variable replacement in the script before execution. ### RunXmla Executes the XMLA script against the Analysis Services database. ```xml ``` ### TaggedFile Replaces tokens in the file indicated by the contentPath: ```xml ``` ### RestoreDatabaseFromBackup Takes a database backup and restores it. ```xml ``` ### ConfigureEntityTypeOperations Adds the required operations configuration for a list of entity types. ```xml ``` ### ConfigureServices Adds the required service configuration for a list of assemblies holding services. ```xml ``` ### EnqueueSql Sends SQL to be executed later by placing it on the command queue. ```xml ``` The system will use the connection to Product.Database.Online to execute a procedure named Control.P_EnqueueCommand. The procedure must accept the following parameters: 1. BatchExecutionId - A unique identifier to group commands that have to be executed in a batch. 2. QueueName - A varchar to accept the name of the queue where to place the script. The framework will pass the logical name of the database that is passed as target database. (See the database section to read about logical names) 3. CommandText - The text of the command. ### ProductLicense Installs the product license. The product license should be set in the variable Product.Licensing.LicenseRawData. ```xml ``` ### ProductConfiguration Deploy Config entries. ```xml ``` ### DeployRepositoryFiles Deploys the files specified into the IoT file repository. 1. contentPath - indicates what will be copied. It is possible to use Minimatch patterns to filter content. ```xml ``` ### GenerateRepositoryIndex Generates the IoT file repository index file. This step is required if new files were deployed with a **DeployRepositoryFiles** step. ```xml ``` ### IoTAutomationTaskLibrariesSync Syncs the automation task libraries of the generated iot packages. Prevents the user having to manual sync the packages. ```xml ``` ### TransformFile Transforms a target file by inserting, replacing, merging or removing data based on a source transformation file where the latter one can also contain tokens to be replaced by the StepType TaggedFile. ```xml ``` This step has the following attributes: 1. *file* - The name of **both** the target file and the source transformation file. It is mandatory. 2. *tagFile* - Optionally set so that the TaggedFile step is applied to the source transformation file before applying transformations. 3. *relativePath* - Optionally set to indicate that the target file is not in the package's TargetDirectory but in a relative path to it. Note that the source transformation file is **always on the package's main folder** and not in any subfolder. This step is useful when we want to change some information in a file that is already shipped in a product release but we don't want to replace the whole file with ours as this can become hard to maintain. So, instead of replacing the file with another one, we just declare a transformation file with what we want to do, e.g. add a configuration, remove another, change a value, etc, and these changes will then be reflected in the target file. These transformations are made by using Microsoft's jdt (for json) and xdt (for xml), each one having their own syntax. Please refer to their documentation for more information: * [Jdt](https://github.com/Microsoft/json-document-transforms/wiki) * [Xdt](https://docs.microsoft.com/en-us/previous-versions/aspnet/dd465326%28v=vs.110%29) Currently, only json and xml files are supported to perform this actions. **Examples:** * Transforming the UI Html config.json to include and remove packages: #### Package step declaration ```xml ``` ##### Target file ```json { ... "packages": { "available": [ "cmf.core.shell", "cmf.core.business.controls", "cmf.core.controls", "cmf.core.checklist" ], ... }, ... } ``` ##### Source transformation file ```json { "@jdt.remove": [ { "@jdt.path": "$.packages.available[?(@ == 'cmf.core.checklist')]" } ], "@jdt.merge": [ { "@jdt.path": "$.packages.available", "@jdt.value": [ "cmf.newavailablepackage" ] } ] } ``` ##### Final Target file ```json { ... "packages": { "available": [ "cmf.core.shell", "cmf.core.business.controls", "cmf.core.controls", "cmf.newavailablepackage" ], ... }, ... } ``` * Transforming the UI Web.config to include and remove rules: ##### Package step declaration ```xml ``` ##### Target file ```xml ... ... ... ... ``` ##### Source transformation file ```xml ``` ##### Final Target file ```xml ... ... ... ... ... ``` ### MasterData Creates an integration entry that will import master data, by creating and loading a MasterDataPackage instance. The attributes represent the same configurations that the [[master-data-packages-tutorial]] supports. CreateInCollection is currently not configurable. ```xml ``` The master data to be imported can either be a plain master data file alongside its additional files or a zip archive that already contains everything. Consider the following setup package folder structure: ```bash My.Custom.Package.1.0.0 ├── DEEs │ ├── FirstCustomDee.cs │ └── SecondCustomDee.cs ├── XMLObjects │ └── MyUIPage.xml ├── MasterData │ ├── MasterDataFile.xlsx │ └── MasterDataArchive.zip └── manifest.xml ``` In order for the MasterDataPackage to be created correctly, the step in the manifest.xml must be set with the following data: ```xml ``` The setup will capture everything defined in the step xml node and create the zip archive to be set as the MasterDataPackage's Package property. The master data file is going to be placed in the root of the zip archive, but the additional files will be have the same folder/file structure as above. The archive will have the structure as shown below: ```bash MasterDataFile.zip ├── DEEs │ ├── FirstCustomDee.cs │ ├── SecondCustomDee.cs ├── XMLObjects │ ├── MyUIPage.xml └── MasterDataFile.xlsx ``` However, as said before, it is also possible to use a zip archive that is already built. In this case, it is the user's responsibility to ensure that it is well formed as it will be set **as is** to the MasterDataPackage's Package property. The configuration in the manifest file becomes more simple as the the only attribute that needs to be set is the filePath. All the other ones aren't considered. ```xml ``` ### ExportedObjects Creates one or more **Integration Entries** that will import xml objects. ```xml ``` ### ProcessRules Creates one or more **Integration Entries** that will execute DEEs. ```xml ``` ### CreateIntegrationEntries Creates one or more **Integration Entries** with the contents of the file/s specified for the **Integration Message**'s content. ```xml ``` The attributes represent the same configurations as defined in section **Integration Handler Resolution** of [Systems Integrations](../userguide/administration/system-integrations/index.md). Example: ```xml ``` ### UpdateConfiguration Updates an existing config value. ```xml ``` ### Generic A generic step handler that performs no specific action by itself. This step type is typically used to run scripts in specific [Pipeline Stages](#pipeline-stages) ```xml ``` This step has the following attributes: 1. **scriptHandler** - (optional) defines the handler responsible for executing the script. If not specified, it is inferred from the file extension (.sql → SQL Server, .ps1 → PowerShell). Supported values: ClickHouse, SQLServer, PowerShell. 2. **on{PipelineStage}** - (optional) specifies the script to be executed during the referenced [Pipeline Stages](#pipeline-stages). 3. **targetDatabase** - (optional) the target database on which to execute the script. 4. **patchId** - an unique identifier for this script. The script runs only if this patchId is not already recorded in the `T_DatabasePatches` table of the target database. 5. **replaceTokens** - enables token variable replacement in the script before execution. [installation_pipeline]: images/installation_pipeline.png [breadcrumb]: images/breadcrumb.png [group]: images/group.png