BIZTALK Mapping

Biztalk provides mapping which can be developed with VS2010. BT can transform an XML into another XML using a mapping in XML-format.

Problems

1.  If the target XML is a SQL-Server INSERT request with over 100 properties the request could fail in some cases. The request should be debatched.
2. The debatch is difficult to validate because a delete cannot be executed for the current debatched bulkinserts based on a dataset which has no key. If there are keys the delete of rows would lead to a complex orchestration or flow for debatching.
3. Advanced transaction is needed over hundreds of debatched bulkinserts handling failures if 1 debatch is rejected. How should an occasionally locked (delete) transaction be detected and afterwards freed in BT if a failure has occured?
4. Validation of the message values is not part of any requirements. But this should be for datasets:
   • a) You cannot back trace or validate each value that was inserted without implementing complex separate flows for each dataset.
   • b) The real world does not provide always a superkey a.k.a  non-nullable set of attributes defining the uniqueness of a row/set of business values. There could exist tables within a set of rows ( functional dependencies within superkeys).

Solution

Custom Mapping with a .NET-APP

Step 1 :domain object mapping

We consider 2 kinds of mapping in OO Design between the problem domain universe and C#: ORM (Object Relational Mapping with Entity framework, EF) and LINQ to XML (Object Hierarchical Mapping). The latter is cumbersome if the XML is subjected to change or there  are too many XMLs related to different schemas. The large number of properties in each XML/Object adds also to the complexity. Therefore, the OHM is implemented with Altova object Mapper. Altova can perform a mapping to business objects by generating C# code from a schema.

Step 2 : business object mapping

BT uses XSLT to map a business entity to another business entity. The .NET-App  is made with Plain Old CLR objects (POCO) interfacing with the BT-server. Because the BT-message is represented in a typical BT schema the mapped object cannot be simply  mapped with ‘=’ and other operators from System.Reflection namespace, i.e. accountlines[0] = accountlinesCollection[0] and the naive property fields operators will fail without custom querying each node. The .NET-App implements a custom mapping which detects the properties that can be mapped from 1 hierarchical business entity to another relational business entity.

SYSTEM RULES

Association

Every XML business property class must have 1-1 relation with a property of an entity framework class. This means an XElement or element are associated to only zero or 1 property per EF-class and every property of an EF-class is related to exactly 1 XElement or element per XML.

Interface

Every dataset object has its own implemention of a key. You cannot determine in advance if the attribute Id or ID is unique or even exists in the problem domain. To satisfy and enforce uniqueness an interface (IObject)  must be implemented with only readonly properties:

• GUID (global UID)
• Id (local UID that may be dependent on another property )

Example:

public string GUID
{
get { return <datasetUID>.First.Value.ToString(); }
}
string IObject.Id
{
get { return GUID.ToString(); }
}

The GUID is for completeness and is a NON-dependent key whereas the Id is a dependent key. This Id-property is reserved to tackle the problem 4b if needed otherwise it is a redundant key which should be implemented explicitly private for the EF class. Every object is identified by its GUID to interact with the outer world (global) objects.  The system identifies every business object by its IObject interface. All system operations involving mapping, validation, evaluation and comparing must adhere to this constraint: IObject.

New Array of GUIDs

If a dataset has no key and no GUID or ID attribute then through its IObject interface GUIDs will be generated by the ToArray() function and Mapping function : MapValues<TTarget>() or its overloads. If a domain entity has no Id the Id is equal to GUID. The interface implementation must look like this:

///For the XML record :

public string GUID { set; get; }

public string Id
{

get { return GUID; }

}

///For  the EF object:

public string GUID
{
get { return Id.ToString(); }
}
string IObject.Id
{
get { return Id.ToString(); }
}

It is required to implement and to use an extension method in a helper-layer and not in the generated Schemas-layer. Otherwise a new build will be prevented by design. Altova does not use arrays for the recordsetcollection. So a redesign may be needed to extend an existing generic interface as long as  it has the IObjects constraint.

Non-functional requirements

Evolution qualities are considered for the .NET-App : testability, maintainability, extensibility and scalability: the software can handle in a stable way multiple large bulk inserts, validates inserts, is composed of loosely coupled , high cohesive objects layered in different assemblies. At least 1 assembly is needed for UNIT-TESTING to guarantee testability. It is required to pass ALL TESTS before release & checking in code and to create a new test for every new dataset in <TestCallbackDatasetInsertValues>. Not doing so endangers the implementation of IObject and Mapping.

If no evolution is considered then the .NET-App is hard to maintain, test, extend and scale .

The supported value mapping uses NULLABLE-TYPES:

• System.Byte
• System.Boolean
• System.Int32
• System.Int64
• System.Decimal
• System.DateTime

Supported REFERENCE-TYPE:

• System.String

BIZTALK Receive/Send Configuration Pattern

The dataset walks through 3 stages. There are 3 pairs of receive/send ports with a starting and final state. It is recommended to use different folders to prevent an eternal loop in receiving and sending messages.

1. Start State:
   1. receive port SFTP_GET (for SFTP) or CSVReceiver (for Filesystem)
      1. every location uses the RcvStreamAnalyzer pipeline to filter and check the stream for valid UTF-8 flatfile message.
   2. send port SendAnalyzed filters on SFTP_GET:

1. Filtered State:
   1. receive port CSVDatasetReceiver
      1. every location uses the correct pipeline to create the corresponding XML bulk insert message:

1. send port SendXMLized filters on CSVDatasetReceiver
2. XMLized state:
   1. receive port XMLreceiver
      1. every location uses the RcvStreamAnalyzer pipeline to insert, validate and export the dataset.
   2. send port SendValidation filters on XMLreceiver
3. Final state:
   1. The dataset is now inserted into 2 databases and is validated for each value.
   2. 3 validation messages can be found.