Views:

Overview

The North52 BPA Data Packager functionality allows you to easily move data stored in records from one Dynamics 365 CE/CDS instance to another. 

 


In this article we describe the components of the Data Packager and how it works. 

Data Packages

Data Packages consist of the Data Package record and xCache (of Data Package type) records. The combination of these records allow you to move data from one instance to another, or duplicate the data on the same instance with different record GUIDs. 

Data Packager licensing

To use the North52 Data Packager functionality you need to install the North52 Business Process Activities solution (v573 or higher) on both the source and target instances and have a valid license. Contact sales@north52.com for further information.

Data Package records

Data Package records are very similar to North52 BPA Decision Table Formulas. They have access to all the Functions available in a standard North52 BPA Formula which makes it very easy to combine logic (e.g. for transforming data, matching records) and data in a single package. This makes it a very powerful solution for transporting and manipulating data.

A Data Package record can contain multiple sheets; you would usually create one for each entity, however there are situations where you may need to update records after the initial create so in this case you would have one sheet for the record creation and one for the update. 

When you execute a Data Package, the sheets execute from left to right and within the sheets from top to bottom. This means that you are able reference a record (e.g. to populate a lookup field) that has been created above or in a sheet to the left when creating or updating another record - this makes it easy to set up a Data Package to create records which are 'children' of another (e.g. Accounts/Contacts or Web Pages) or even dependent on a child record being created before the parent can be updated (e.g. Primary Contact on Account or First Web Form Step on the Web Form). When adding data to a Data Package these relationship updates are automatically handled.

Register tab

  • Formula Settings
    • Name: specify the name of the Data Package
  • Deployment Settings
    • Trace Level: Select the Trace Level to Information to output the Data Package execution into a detailed trace log. Off is the default setting.
    • Solution Deployment: Select the unmanaged solution to associate to the Data Package. All the xCache records created by the Data Package will be associated to this solution too.
  • Advance Settings
    • Category: Specify a category here to be able to group related Data Packages together
    • Sub-Category: Specify a sub-category here to be able to group related Data Packages together
    • Execution Order: when executing Data Packages as part of an automated process (e.g. DevOps) the execution order can be used to determine the order in which they should be executed

Functions tab

The Functions tab provides access to all the functions available in North52 BPA. You can use them in Data Package sheets as you would in a standard North52 formula.
 

Source tab

The Source tab provides access to xCache records, Global Calculations, Global Actions and Decision Table Calculations. You can use them in Data Package sheets as you would in a standard North52 formula. For example if you wanted to use the GUID of a record created in a preceding sheet, you could expand the Decision Table Calculations, expand the Sheet Name and then click on the row - this useful if you are creating a record without specifying the GUID and need to reference the newly created record in a lookup field of another record.
 

Explore tab

The Explore tab provides access to data held in the current instance (Explore System Data section) and the data model schema  (Explore Data Model section). The Explore System Data section is key to how your build out your Data Package by using data from the current instance.

Searching for records:

  • Select an Entity from the drop down list
    • After selection the first 50 records found are displayed
      • 50 is the default number returned, you can change this by setting the DP - Configuration - Entity Retrieve Count on the Global Calculations sheet - see more details below
    • Type a search term in the Search records... input box and press Enter. The first 50 records found matching the criteria will be displayed
      • Pressing enter with the cursor in the input box again will reverse the result list
    • Records which return the same name will be appended with (n) where n is the n-1th record found with the same name (see image above and the Adventure Works account)
    • Records which don't have a value in the Primary Field will show the GUID of the record

Manually adding records to the Data Package

  • Select one or more records (indicated by the 'file' icon - see image above)
    • Note: All selected records must be of the same entity type before you can add them to a Data Package
    • Expand related records by clicking the triangle icon for the individual records
  • Click the lightning bolt icon next to the search input 
  • The selected records' detail will be added to Data Package sheets
    • If the sheet does not already exist for the entity it will be created
    • Only fields with data will be added to the sheet
    • Some of the record data will be created in xCache records (see DP - Configuration - TextFieldsToXCache section below for the rules)

Automatically adding records to the Data Package

To speed up the process of adding records to the Data Package it is possible to define Fetch-Xml queries which determine the record set to be added to the Data Package. These queries can be parameterized so that you can easily return results based on conditions like 'records modified in the last X days'. The Fetch-Xml queries are defined on the Global Calculations sheet (see next section below). To add records to the Data Package:

  • Select DP - Fetch-Xml Queries from the drop down list
    • Check the Fetch-Xml Queries you wish to add
  • Click the lightning bolt icon next to the search input 
  • The records' detail returned by the Fetch-Xml queries will be added to Data Package sheets
    • If the sheet does not already exist for the Name of the Query it will be created
    • The sheets will be created in the same order as the list of Fetch-Xml queries
    • Some of the record data will be created in xCache records (see DP - Configuration - TextFieldsToXCache section below for the rules)

Fetch-Xml tab

The Fetch-Xml tab is not currently used by the Data Packager and is there for future functionality.
 

Global Calculations sheet

Just like a standard Decision Table in a North52 BPA formula the Global Calculations sheet allows you to specify named calculations (variables) for use in your Decision Tables. However, there are some default calculations provided when you create a Data Package and some reserved names.

DP - Configuration - ExcludeFields

List of field names to exclude from any data generation. When data is being added to a Data Package sheet from a record these fields are excluded as you don't normally need them e.g. createdon

DP - Configuration - TextFieldsToXCache

List of entity.field names which automatically create an xCache reference. To ensure that the Data Package formula remains valid and does not exceed the current limit of 1 million characters, fields which contain large amounts of data should be specified here. For example when moving Web Page records the 'adx_webpage.adx_copy' field should be specified. 

Some data will by default be stored in xCache records:

  • Fields longer than 100 characters
  • Fields containing line breaks (\r, \n, \r\n)
  • Fields containing the characters { } [ ] | or the string /*

DP - Configuration - Default Mode

When the Data Package sheets are being created the Sheet Option will default to the mode specified here. Normally, this would be to create a record only if it does not already exist based on the GUID supplied - 'createifnotexistsbyid'.  The modes you can specify here are: createifnotexistsbyid, updatecreatebyid

DP - Configuration - Entity Retrieve Count

Number of rows to retrieve when navigating the System Data tree view and searching for records. For example, '50' will return up to fifty records.

DP - Configuration - Enable BD Updates

When adding records to a Data Package the Enable Bi-directional updates setting will automatically create the necessary sheets to update the referenced data. For example, Account-Contact via Primary Contact field on Account. Allows parent and child records to get created first with no dependencies and a separate update builds the dependencies. Set to 'true' to activate this functionality, and 'false' to deactivate.

DP - Configuration - Enable SR Updates

When adding records to a Data Package the Enable Self-referencing updates setting will automatically create the necessary sheets to update the referenced data. For example, Parent Account and Account, Web Page and Parent Web Page. Allows parent and child records to get created first with no dependencies and a separate update builds the dependencies.  Set to 'true' to activate this functionality, and 'false' to deactivate.

DP - Configuration - Enable DR Updates

When adding records to a Data Package the Enable Dynamic Reference updates setting will automatically replace references to fixed GUIDs with dynamic references to sheet row numbers. For example, Contact-Account via Company Name on Contact. Instead of the Company Name field being set to a GUID if possible take a Dynamic Reference to the associated Account row ('account|795487dd-4c1a-e911-a972-002248010929' versus 'account|' + {Account-Row-0004}). Set to 'true' to activate this functionality, and 'false' to deactivate.

DP - Configuration - Last X Days

Used in Fetch-Xml queries to find only recently modified data

DP - FetchXml - Accounts

This is a sample Fetch-Xml query containing a reference to the DP - Configuration - Last X Days variable above. See the section on Defining Fetch-Xml for use in Data Packages below.
 

Global Actions sheet

Define functions for use when executing Data Packages. For example, disabling SDK processing steps and Processes before the Data Package executes records creates or updates and then enabling them again afterwards. 

Defining Fetch-Xml for use in Data Packages

Define Fetch-Xml queries on the Global Calculations sheet to speed up the process of creating Data Packages. There is a sample query provided when you create a new Data Package - this can be modified or deleted. To add a new Fetch-Xml queries for use in the Data Packager:

  • In the next available Global Calculation row (or insert a new row) specify the Calculation name with a prefix of DP - FetchXml - e.g. DP - FetchXml - Web Pages
  • In the corresponding Calculation Value column enter your Fetch-Xml. You can reference any preceding DP - Configuration - [Name here] calculations in your Fetch-Xml (add them using the Source tab >  Global Calculations > [Calculation Name]), for example: 
    '<fetch> 
<entity name="account"> 
<all-attributes /> 
<filter type="and"> 
<condition attribute="statecode" operator="eq" value="0" /> 
<condition attribute="modifiedon" operator="last-x-days" value="GetVar('DP - Configuration - Last X Days')" /> 
</filter> 
</entity> 
</fetch>'
  • Note: You are not able to use any dynamic functions here e.g. UtcDate(), only references to static values contained in DP - Configuration - [Name here] calculations

Important information about Data Package Fetch-Xml

  • The Fetch-Xml queries are displayed on the Explore System Data list in the same order as they are defined in the Global Calculations. This also determines the order the records are added to the Data Package sheets.
  • The name of the automatically generated Data Package sheet is based on the Calculation Name: DP - FetchXml - Web Pages creates a sheet with the name Web Pages
    • This allows you to create multiple sheets for records of the same entity - useful when post creation updates are required
  • A Fetch-Xml query with a Calculation Name containing the word update will always result in a sheet being created with the Sheet Options set to updatecreatebyid 
    • This is used when you need to create updates to records like the Primary Contact on an Account or the First Web Form Step on the Web Form

Data Packages and Solutions

When you create a Data Package you need to associate it with an unmanaged Solution. The Data Package and associated xCache records are stored in this solution as XML web resources and allows you to easily transport data from instance to another. Because it is the same way you transport other customizations you can combine your Data Package with customizations like entities, forms, processes, etc. and move all related customizations and data in one solution. 

You can export your solution as either unmanaged or managed and import it on the target system. After importing you need to click the Publish All Customizations button to 'unpack' the Data Package records and associated xCache records. 

Once the Data Package has been 'unpacked': 

  • Navigate to Settings > North52 BPA > N52 Data Package
  • Open the Data Package record
  • Click on the lightning bolt button in the formula editor navigation bar to open the execution pane
  • Click the Execute button to run the Data Package
    • Any messages or errors will be shown in this pane

Guid Switcher

The Data Packager has functionality to replace all record GUIDs with new GUIDs. This allows you to ensure that unique records are created and there will be no clashes with existing records when you execute the Data Package. Each unique GUID is replaced with the same new GUID to ensure that references to other records are maintained. To use the GUID Switcher:

  • Ensure that the Data Package has been Saved after adding new records
  • If you only want to replace specific GUIDs, highlight the cells containing the GUIDs you wish to replace. If in the selected cell range there exists a GUID(s) it will change this/these GUID(s) throughout all sheets. Otherwise the GUID Switcher will change all GUIDS across all sheets
  • From the Commands menu in the editor navigation bar select Guid Switcher
  • After the process is completed (the Commands menu closes) save your Data Package before executing or exporting

Clone

To create a copy of a Data Package:

  • From the Commands menu in the editor navigation bar select Clone
  • Navigate to Settings > North52 > N52 Data Package
  • The cloned Data Package will be listed with the name of the cloned Data Package prefixed with :: Clone ::  and suffixed with a random number e.g. :: Clone :: Data Package - Portal Template 1634759149

Note: Cloned Data Packages do not clone the associated xCache records. It is primarily a mechanism for creating new Data Packages from template Data Packages

Known limitations

The known limitations for the Data Packager are:

  • It is designed for transporting configuration data and not large scale data migrations
    • Each Data Package is limited to 1 million characters (for reference all the records required to transport the default Community Portal is about 900,000 characters) 
  • Notes with file attachments over 750kb are not supported

Related Articles (4)

Data Packager - Disable/Enable Plug-ins and Processes Data Packager - How To - Create a Data Package (Dynamics Portals) Data Packager - How To - Import and Execute (Dynamics Portals) Data Packager - Introduction