Views:

Overview

The North52 BPA Data Packager functionality allows you to easily move data stored in records from one Dynamics 365 CE/CDS instance to another. 

 

https://www.youtube.com/watch?v=xgnQMuAWoYs


In this article we describe the components of the Data Packager and how it works. 

Data Packages

Data Packages consist of the Data Package record and xCache (of Data Package type) records. The combination of these records allow you to move data from one instance to another, or duplicate the data on the same instance with different record GUIDs. 

Data Packager licensing

To use the North52 Data Packager functionality you need to install the North52 Business Process Activities solution (v573 or higher) on both the source and target instances and have a valid license. Contact sales@north52.com for further information.

Data Package records

Data Package records are very similar to North52 BPA Decision Table Formulas. They have access to all the Functions available in a standard North52 BPA Formula which makes it very easy to combine logic (e.g. for transforming data, matching records) and data in a single package. This makes it a very powerful solution for transporting and manipulating data.

A Data Package record can contain multiple sheets; you would usually create one for each entity, however there are situations where you may need to update records after the initial create so in this case you would have one sheet for the record creation and one for the update. 

When you execute a Data Package, the sheets execute from left to right and within the sheets from top to bottom. This means that you are able reference a record (e.g. to populate a lookup field) that has been created above or in a sheet to the left when creating or updating another record - this makes it easy to set up a Data Package to create records which are 'children' of another (e.g. Accounts/Contacts or Web Pages) or even dependent on a child record being created before the parent can be updated (e.g. Primary Contact on Account or First Web Form Step on the Web Form). When adding data to a Data Package these relationship updates are automatically handled.

Register tab

  • Formula Settings
    • Name: specify the name of the Data Package
  • Deployment Settings
    • Trace Level: Select the Trace Level to Information to output the Data Package execution into a detailed trace log. Off is the default setting.
    • Solution Deployment: Select the unmanaged solution to associate to the Data Package. All the xCache records created by the Data Package will be associated to this solution too.
  • Advance Settings
    • Category: Specify a category here to be able to group related Data Packages together
    • Sub-Category: Specify a sub-category here to be able to group related Data Packages together
    • Execution Order: when executing Data Packages as part of an automated process (e.g. DevOps) the execution order can be used to determine the order in which they should be executed

Functions tab

The Functions tab provides access to all the functions available in North52 BPA. You can use them in Data Package sheets as you would in a standard North52 formula.
 

Source tab

The Source tab provides access to xCache records, Global Calculations, Global Actions and Decision Table Calculations. You can use them in Data Package sheets as you would in a standard North52 formula. For example if you wanted to use the GUID of a record created in a preceding sheet, you could expand the Decision Table Calculations, expand the Sheet Name and then click on the row - this useful if you are creating a record without specifying the GUID and need to reference the newly created record in a lookup field of another record.
 

Explore tab

The Explore tab provides access to data held in the current instance (Explore System Data section) and the data model schema  (Explore Data Model section). The Explore System Data section is key to how your build out your Data Package by using data from the current instance.

Searching for records:

  • Select an Entity from the drop down list
    • After selection the first 50 records found are displayed
      • 50 is the default number returned, you can change this by setting the DP - Configuration - Entity Retrieve Count on the Global Calculations sheet - see more details below
    • Type a search term in the Search records... input box and press Enter. The first 50 records found matching the criteria will be displayed
      • Pressing enter with the cursor in the input box again will reverse the result list
    • Records which return the same name will be appended with (n) where n is the n-1th record found with the same name (see image above and the Adventure Works account)
    • Records which don't have a value in the Primary Field will show the GUID of the record

Manually adding records to the Data Package

  • Select one or more records (indicated by the 'file' icon - see image above)
    • Note: All selected records must be of the same entity type before you can add them to a Data Package
    • Expand related records by clicking the triangle icon for the individual records
  • Click the lightning bolt icon next to the search input 
  • The selected records' detail will be added to Data Package sheets
    • If the sheet does not already exist for the entity it will be created
    • Only fields with data will be added to the sheet
    • Some of the record data will be created in xCache records (see DP - Configuration - TextFieldsToXCache section below for the rules)

Automatically adding records to the Data Package

To speed up the process of adding records to the Data Package it is possible to define Fetch-Xml queries which determine the record set to be added to the Data Package. These queries can be parameterized so that you can easily return results based on conditions like 'records modified in the last X days'. The Fetch-Xml queries are defined on the Global Calculations sheet (see next section below). To add records to the Data Package:

  • Select DP - Fetch-Xml Queries from the drop down list
    • Check the Fetch-Xml Queries you wish to add
  • Click the lightning bolt icon next to the search input 
  • The records' detail returned by the Fetch-Xml queries will be added to Data Package sheets
    • If the sheet does not already exist for the Name of the Query it will be created
    • The sheets will be created in the same order as the list of Fetch-Xml queries
    • Some of the record data will be created in xCache records (see DP - Configuration - TextFieldsToXCache section below for the rules)

Fetch-Xml tab

The Fetch-Xml tab is not currently used by the Data Packager and is there for future functionality.
 

Global Calculations sheet

Just like a standard Decision Table in a North52 BPA formula the Global Calculations sheet allows you to specify named calculations (variables) for use in your Decision Tables. However, there are some default calculations provided when you create a Data Package and some reserved names.

DP - Configuration - ExcludeFields

List of field names to exclude from any data generation. When data is being added to a Data Package sheet from a record these fields are excluded as you don't normally need them e.g. createdon

DP - Configuration - TextFieldsToXCache

List of entity.field names which automatically create an xCache reference. To ensure that the Data Package formula remains valid and does not exceed the current limit of 1 million characters, fields which contain large amounts of data should be specified here. For example when moving Web Page records the 'adx_webpage.adx_copy' field should be specified. 

Some data will by default be stored in xCache records:

  • Fields longer than 100 characters
  • Fields containing line breaks (\r, \n, \r\n)
  • Fields containing the characters { } [ ]

Related Articles (4)

Data Packager - Disable/Enable Plug-ins and Processes Data Packager - How To - Create a Data Package (Dynamics Portals) Data Packager - How To - Import and Execute (Dynamics Portals) Data Packager - Introduction