Using AWS Services to Perform Data Analysis With SAP SuccessFactors

A successful experience has been enjoyed by an organisation in performing unique data analysis with SAP SuccessFactors, using AWS services.

SuccessFactors is a suite of applications from SAP, a renowned player in enterprise and business applications. SuccessFactors mainly covers Human Resource core services, Payroll services, Recruiting Management, Performance Management, Workforce Planning, Learning and Development Management and other related Human Capital Management functions. 

While SAP offers data analytics, the organisation does have unique requirements in what they wanted to achieve in data analytics which led them to look into other possibilities. 

Eventually, following setup helped to deliver the desired outcomes. 

Extracting Data from SAP SuccessFactors Using AWS Glue

AWS Glue is a fully managed, serverless data ETL (Extract, Transform, Load)  service with efficient operations. Its high performance enablement is suitable for extracting data from Successfactors and there are available connectors for AWS Glue.

A designated IAM Role is needed for the AWS Glue job. The role enables access to all resources used by the Glue job, including Amazon S3 for any sources, targets, scripts, temporary directories, and AWS Glue Data Catalog objects. The role also grants access to the Glue Connector for SAP SuccessFactors.

As a quick summary, following policies are associated with the IAM Role:

  1. AWSGlueServiceRole (accessing Glue Studio and Glue Jobs)
  2. AmazonEC2ContainerRegistryReadOnly (accessing the Glue Connector for SAP SuccessFactors)
  3. AmazonS3FullAccess (reading and writing to Amazon S3)
  4. SecretsManagerReadWrite (For accessing AWS Secrets Manager)


Authenticating with SAP SuccessFactors is done through OAuth Authentication. Following are specified:


  • Url: the URL of the server hosting Success Factors. 
  • User: the username of the account.
  • CompanyId: the unique identifier of the company.
  • OAuthClientId: the API Key that was generated in API Center.
  • OAuthClientSecret: the X.509 private key used to sign SAML assertion. This is obtained in the certificate that was downloaded in Registering your OAuth Client Application.
  • InitiateOAuth: set this to GETANDREFRESH.


Leveraging AWS Secrets Manager for access and parameter security

  • In the AWS Secrets Manager console choose store a new secret.
  • Choose Other type of secret. This option means you must supply the structure and details of your secret.
  • Add all required properties to connect to SAP SuccessFactors as well as any additional private credential key-value pairs required by the Glue Connector
  • The AWS Glue ETL job and the secrets need to be in the same region


After activating the Glue Connector and completing the configuration, a Glue job can be built


The Script tab can be used to review the script being created by Glue Studio. Following is a sample script (Sample only):


import sys

from awsglue.transforms import *

from awsglue.utils import getResolvedOptions

from pyspark.context import SparkContext

from awsglue.context import GlueContext

from awsglue.job import Job


## @params: [JOB_NAME]

args = getResolvedOptions(sys.argv, ['JOB_NAME'])


sc = SparkContext()

glueContext = GlueContext(sc)

spark = glueContext.spark_session

job = Job(glueContext)

job.init(args['JOB_NAME'], args)

## @type: DataSource

## @args: [connection_type = "marketplace.jdbc", connection_options = {"dbTable":"ExtAddressInfo","connectionName":"cdata-sapsuccessfactors"}, transformation_ctx = "DataSource0"]

## @return: DataSource0

## @inputs: []

DataSource0 = glueContext.create_dynamic_frame.from_options(connection_type = "marketplace.jdbc", connection_options = {"dbTable":"ExtAddressInfo","connectionName":"cdata-sapsuccessfactors"}, transformation_ctx = "DataSource0")

## @type: DataSink

## @args: [connection_type = "s3", format = "json", connection_options = {"path": "s3://PATH/TO/BUCKET/", "partitionKeys": []}, transformation_ctx = "DataSink0"]

## @return: DataSink0

## @inputs: [frame = DataSource0]

DataSink0 = glueContext.write_dynamic_frame.from_options(frame = DataSource0, connection_type = "s3", format = "json", connection_options = {"path": "s3://PATH/TO/BUCKET/", "partitionKeys": []}, transformation_ctx = "DataSink0")

job.commit()


The ETL-ed data is stored in a S3 bucket (as a data lake) for data analytics which further leads to:

  • Reporting 
  • Backup and Disaster Recovery 
  • Observability Enhancement


This setup also provides future potentials on Machine Learning and Artificial Intelligence, as the large volumes of data is readily there in S3 for utilising the comprehensive AI services in which AWS is a leader of. 


                                                                                                        Simon Wang


Comments

Post a Comment

Popular posts from this blog

Improve Output Quality When Streaming Log Data to Amazon QuickSight for Visualisation

Image
Amazon QuickSight is a commonly used business intelligence tool set that helps wide range of users to comprehend data by visualisation.  Log data from many sources, both from AWS services and non AWS services, can be streamed to Amazon QuickSight for data visualization, analytics, and for leveraging generative AI. This is a fairly typical use case of QuickSight.  But the quality of the Amazon QuickSight outputs for log data is influenced, or decided, before QuickSight even receives the data. This is because, often, log data has so much dimensions in it, or there are just simply so many fields, which significance is not equal. Insignificant data fields will significantly reduce the quality of the QuickSight visualisation and its readability. Some times, you also want to strip sensitive data before sending the data set for visualisation.  There are creative ways to improve the log data quality when streaming it to Amazon QuickSight using filtering and/or redacting.  Be...
Read more

A Problem Solving Experience on Amazon S3 Multi-Region Access Points

Image
Amazon S3 buckets can be used for storing large amount of data. AWS’ comprehensive tools on data analytics can then be leveraged for processing the data. Often, the post-analytics (processed) data is stored in another S3 bucket. Users can then access the processed data by fetching it from that S3 bucket.  This piece discusses a problem that was recently experienced on accessing the S3 buckets behind a Multi-Region Access Points setup and the problem solving experience.  The Setup A large amount of data is deposit into Amazon S3 service for analytics processing. The processed data is then stored in a different S3 bucket. A user base of around 10,000 accesses the processed data actively using a proprietary application. This application is latency sensitive. Most users are located in three geographical regions: Australia and New Zealand, North America and South America.  Due to the latency sensitive requirement, Amazon S3 Multi-Region Access Point setup has been used.  ...
Read more

AWS Storage Gateway File Gateway with S3 and FSx For Lustre with S3

Image
Cloud data storage is widely used not just by cloud native compute environments but also on-premises computer systems and applications. It has been so as there are tremendous benefits that cloud data storage brings, most of which are difficult or impossible for non-cloud data solutions. These include but not limited to: • Virtually unlimited capacity • The ease of provisioning • Being on-demand • Data durability (through the distribution and replication powered by the cloud) • Cost efficiency • Reduced or eliminated maintenance overhead AWS Storage Gateway is a common storage service for on-premises and AWS cloud hybrid environments – whether it is being in the transition phase of a migration from on-premises to the cloud or that both on-premises and cloud environments will continue to work together and coexist for the times to come. AWS Storage Gateway provides local caching for on-premises application’s latency-improved access to the cloud data storage.   Various...
Read more