SAP Integration Suite integrates with AWS Kinesis Data Stream for Data Analytics

by Dai (Bato) Quach

In this blog, we will experiment with extracting data from SAP Integration Suite API and using AWS Kinesis Data Stream for data ingestion and then have an insight into the collected data with Apache Flink. Also, reacting to anomaly detection of the data.

SAP Integration Suite API provides different sets of APIs such as Integration Content which has enough functions to build a CI/CD pipeline for interface development, Partner Directory provides a flexible role base logical, Security artifacts, and Message Processing Log... We use a few OData endpoints in the Message Processing Log and Integration Content in this blog, consuming API sets is the same except the required role for each API set is different – the role is to authorize the OAuth client to consume APIs. You should consult SAP doc for more detail on the required roles of each API set.

We have 2 parts. SAP Integration Suite (CPI) is the focus of this part where we implement data collecting and reacting - undeploy the artifact which causes permanent errors that we get notification from AWS. In the second part, we will implement an AWS data pipeline to ingress and analyze data and then send out a notification (AWS SNS) if there is something that exceeds the predefined threshold. With collected data, we can also send it to a data lake to study (ML) and can make a better decision in SAP Message Monitor or SAP Security artifact rotation.

The collecting workflow

In a period, we call Message Processing Log OData API to get all messages and for each error message we will call another endpoint to enrich the message with more detail of the error, and then streaming messages to AWS Kinesis Data Stream by using AWS adapter.

Let’s say we have a 2-minute timer, and we have 5000 messages which include 1000 error messages (just an example), in the ideal case we must make 1 + 1000 API calls in a 2-minute window. What if it could not make it? We may increase the timer to have a bigger window to collect data, but it won’t be lucky all the time.

Like any long-run workflow, it’s better to decouple it into asynchronous parts. We can use any event/message service to leverage the decoupling, there are many options in the market, and it depends on each company. We are using AWS and prefer to use Event Bus of Amazon EventBridge as it natively integrates with many target services in AWS. As I know so far there are 3 AWS adapters available on SAP Integration Suite but only the Advantco AWS adapter supports EventBridge now.

In the download package, we also provide an implementation of Event Mesh for your reference, but you will see why we choose a cloud native event service over Event Mesh. With Event Mesh, it simply does the decoupling that why at Integration flow #2 we consume both Success and Error messages from the broker to route them to AWS Kinesis. Amazon EventBridge Event Bus is also an event router where we can route the Success messages to Kinesis inside AWS (we will have details in part 2 of the blog), so we only need to consume the Error messages in Integration flow #2 to enrich with detailed error and route it to the final destination AWS Kinesis.

Another reason and the most important why we chose the Advantco AWS adapter for the implementation is because it’s the only adapter that supports the IAM role – the security best practice in AWS.

We have 2 integration flows:

  • Integration flow 1: Collect all messages and split them one by one then send it out to a SQS topic.
  • Integration flow 2: Asynchronously receive messages from the SQS topic, enrich the message with the error detail, and then send to a Kinesis Data Stream. Multiple threads are counted here, which helps to handle the high throughput.

Some screenshots of the integration flows are below. You can download the whole package of integration artifacts [here](https://github.com/quachtd/sap_is_aws_data_integration/blob/main/AWS Data Integration 1.0.1.zip) for more details of the implementation. It’s just a proof of concept, you have to handle error gracefully and add more logic if needed.

Integration flow 1 – MessageProcessing_CI2AWS_OB

Integration flow 2 – MessageProcessing_CI2AWS_GetMoreDetail_OB
As I mentioned above with EventBridge Event Bus we route Success messages directly to Kinesis in AWS, for the Error messages we route them to AWS SQS that is why we use AWS SQS as the sender adapter here.
We are supposed to make the integration flow simpler with OData adapter, but the OData receiver adapter doesn’t support navigation of the API which is why we use HTTP adapter and require an additional Integration Process for the enrichment step.

The reacting workflow

Any exceed of the predefined threshold will trigger an SNS message in AWS, and then the subscribed SQS topic will receive the message (configured in AWS). In SAP Integration Suite, we consume the message and consider it as a notification for remediation.

In this implementation, if any integration flow keeps repeating errors for 4 hours continuously then a notification is triggered (in AWS which we will cover in part2) and the integration flow will be undeployed and a notification message will be sent out to inform the support team.

We have 1 integration flow to accomplish it:

  • Integration flow 3: Receive a notification message from an SQS topic, then undeploy the integration flow where the integration id is from the message payload, and finally send out an SNS message to inform the support team. Email/Mobile notification from the SNS topic is configured in AWS, not a part of this integration flow.

The screenshot of the integration flow is below.

Integration flow 3 – IntegrationContent_AWS2CI_UndeployRuntimeArtifact_IB

That’s it for the first part. We will continue with the second part of working on AWS.
All you need to try out this implementation:

Hope you find something new from this blog and enjoy it.