Another reason for a “Bad Request” null – error in Azure Data Factory
When creating a pipeline in Azure Data Factory to load data from Salesforce into a Fabric Lakehouse, I made contact with this glorious error message in ADF.
I could pin it to my “Copy data” activity by setting breakpoints in my pipeline. But I had no idea what the reason for this message was. So, research the internet, reading StackOverflow and helpful articles like this: ADF: Error trying to debug pipeline: BadRequest where people have similar errors, most related to leading or trailing spaces or characters in parameter or variable names. I could rule that out since I checked that twice in my pipeline.
So, I recreated the whole pipeline step by step and saved the resulting JSON in text files to see if anything changes in the background that I am unaware of in the GUI (you never know).
The moment everything broke was when I removed an unused variable from my pipeline.
Since I moved the object name from a variable to a parameter and replaced all occurrences in expressions and formulas, I removed the variable “objectname” from the pipeline. This is when the error popped up again—recreating the variable fixed it. So, the next logical step is “check if you forgot it somewhere in the pipeline”. And I didn’t… I checked the JSON of the pipeline, and it was nowhere.
But then I found it, in a default value of a parameter of a dataset, used in the pipeline. Even though the default value was overwritten one the pipeline and not referenced at all.
Lesson learned: If you remove a variable from a pipeline, ensure it is also removed from linked datasets parameter default values, even if you are not using the default value.
