Salesforce DAGs
Overview
The DAGs produced in this section were some of the first DAGs made during the internship. The DAGs here are made for Contact objects and Account objects, which are native to Salesforce. The DAGs originally were made during May-June of 2026 and intended for use in the Sandbox environment which only used test data. As of July 31st, 2026, the DAGs are being refit to function in a production environment.
TODO: LINK HERE
All of the code can be accessed in AWS and as well in the GitHub repository.
The general structure of each DAG is the following, each DAG follows an "ETL" or Extract, Transfer, Load design. Since PostgreSQL was chosen as one of the endpoints during the internship, data is fed between PostgreSQL or Salesforce and then fed into the intermediary service of Airflow. In Airflow, the data is "transformed" in whatever manner is necessary, and then is fed to the other service. If data was taken from PostgreSQL, then it would typically flow back out to Salesforce, and if data was taken from Salesforce it would generally go back to PostgreSQL.
Many of the DAGs contain their own respective documentation, however additional documentation is added here such that they can be understood at a higher level and more context can be given with respect to their uses in Salesforce.
Visual Studio Code AWS Extension
The Visual Studio Code (VSC) extension for AWS makes modifying and creating existing DAGs extremely simple. The utility allows users to create, delete, upload, and modify files in AWS' file hierarchy as if it were a normal file explorer in VSC. For example consider the current working directory for DAGs:
More information about the plugin can be found here. Installing the plugin in VSC and working with it is relatively simple. In VSC, the extension can easily be found by searching "AWS" into the extension marketplace. The top result "AWS Toolkit" should be installed.
Once installed, there will appear an "AWS" icon on the bottom left sidebar of VSC. Click on it.
In order to sign in once the sign in process is configured, begin by heading into myapps in URI's SSO and select AWS. Next, on the bottom left hand corner of VSC there will appear a red bar. Click on that bar.
After clicking on the bar, a menu will bring up the option to access "IAM Identity Center", click on it. This will bring up a dialogue box with the option to open a link to AWS which will verify your credentials and permit you access to AWS' services through VSC again.
Once the site has been correctly accessed, the following screen will appear, which will allow the user to close it and return to VSC.
Then, click IAM Identity Center again. If you have the corresponding permissions for AWS, then you should be able to hit AWS Full Permission, which at least should give you full access to AWS. Note that you will not be able to run Airflow inside this environment, merely modify files, delete them, and upload files in AWS from your VSC environment.
I have opted for the following file hierarchy for the Airflow environment (this will not change how it is viewed and accessed from inside Airflow) since it separates DAGs into their respective functions quite well:
S3 bucket (foundational-etl-poc-bucket)
|___ requirements.txt
|___ dags/
|___ create_tables/
|___ salesforce_dags/
|___ account_dags/
|___ contact_dags/
|___ affinaquest_dags/
|___ education_dags/
Bulk API
In Salesforce, there are four primary bulk operations, insertion, deletion, hard delete, updating, and upserting (update+inserting). Insertion adds a new entry to salesforce given some relevant fields about the object.
An object hard deleted in Salesforce is immediately deleted, whereas an object which is deleted in Salesforce lingers in the recycle bin for 15 days.
Airflow has bulk methods which make insertion and deletion of massive amounts of records efficient. I opted to use Simple Salesforce, which also has bulk operators that can be used in Airflow as well. For Further information on Bulk APIs in Simple Salesforce can be found here:
https://github.com/simple-salesforce/simple-salesforce
Rather than adding 50 objects individually, many are added at the same time. Each object in Salesforce has bulk methods. The bulk methods are all essentially identical in how they operate.
Test Connection DAG
This DAG simply tests that there is an existing Airflow connection and is entitled "salesforce_test_dag". This DAG is useful since it simply reads from Salesforce, which proves that Salesforce is connected without making any unexpected modifications to it or PostgreSQL. In the event that a connection must be re-established between Airflow and Salesforce, this DAG can be used to test that the connection is valid safely.
The DAG does not follow the ETL structure as it is just used to verify that a connection exists.
from datetime import datetime
from airflow.decorators import dag, task
from airflow.providers.salesforce.hooks.salesforce import SalesforceHook
@dag(
dag_id="salesforce_test_dag",
schedule=None,
start_date=datetime(2026, 1, 1),
catchup=False,
)
def salesforce_test_dag():
@task
def test_connection():
hook = SalesforceHook()
# Test with a simple query
result = hook.make_query("SELECT Id, Name FROM Account LIMIT 5")
print(f"Successfully connected to Salesforce!")
print(f"Retrieved {len(result['records'])} accounts")
for record in result["records"]:
print(f" - {record['Name']}")
return result["records"]
test_connection()
salesforce_test_dag()=6.3.0Create Table DAGs
These aren't much DAGs as much as they are files inside Airflow which are run following
Contact DAGs
Contact objects in Salesforce represent an individual associated with a business account, and as the name would suggest, contains that person's name, phone number, email address, and other useful fields by which that person can be contacted.