CI/CD with Azure Synapse Notebooks - Error Resolved

 

Some features of Azure Synapse are mysterious. Recently, I was working on deploying Azure Synapse artifacts from development to production using the "Synapse Workspace Deployment" extension in Azure DevOps and received an odd error: 

2021-11-10T21:20:14.8670075Z For artifact: AzureSQLQueryTool: Checkstatus: 202; status message: Accepted

2021-11-10T21:20:44.9656242Z For artifact: AzureSQLQueryTool: Checkstatus: 200; status message: OK

2021-11-10T21:20:44.9661205Z For artifact: AzureSQLQueryTool: Artifact Deployment status: Failed

2021-11-10T21:20:44.9673543Z Error during execution: Error: Failed to fetch the deployment status {"code":"400","message":"Failed Component = DataFactoryResourceProvider, ErrorCode = 400, Error = BadRequest "}
2021-11-10T21:20:44.9723399Z ##[error]Encountered with exception:Error: Failed to fetch the deployment status {"code":"400","message":"Failed Component = DataFactoryResourceProvider, ErrorCode = 400, Error = BadRequest "}
2021-11-10T21:20:44.9945300Z ##[section]Finishing: Synpase deployment task for workspace: myWorkspace_prod

The new items I had added to Synapse were several spark notebooks for ingesting data. I had tested them individually and they all appeared to be working, yet Azure DevOps' CI/CD gave me error when it attempted to deploy the release to production. I had followed the instructions provided by Microsoft to set up the CI/CD pipeline, yet it was failing.

I attempted to add override parameters for the notebooks - each notebook was linked to the spark pool in dev which was named "sp_dev". The Production spark pool was called "sp_prod", so with parameters for the pool's name it should work, right? 

No. Same error. 

After numerous other unsuccessful attempts at deployment, I deleted the production spark pool and recreated it with the same name as the dev spark pool. The notebooks deployed without a hitch. 

If you see the above error messages in your CI/CD logs and have spark notebooks in your Synapse deployment, the fix is always give the same names to the spark pools in every environment. 

Data Community Fun: A visit with Paul Turley

 

A few weeks ago, Paul Turley, his wife Shirley, their two spunky dogs and anti-social cat pulled up at the "Letourneau Resort" for a 3-day visit. We spent some time working together on the patio while his dogs raced around my property. The cat hid in the Data Bus and only went out when forced to do so. We shared several meals and spent many hours working at our computers (he visited Thursday, Friday, & Saturday. We both had to work Thursday & Friday). 

"Fancy" patio set-up. Just the computer, but we do have internet out there and the weather was perfect for sitting outside enjoying the fresh air while also getting a lot of work done. 

View from patio when you look up from computer. :) 

On the last day, Saturday, my boss at 3 Cloud, Kathi Vick, and her husband joined us. Paul has a blog called "Data On The Road", so he interviewed both of us before getting back on the road heading eastward. 

Guy loaned Paul his "Director's clapper" - it was the award from the Sakuracon 2005 AMV competition and was signed by a bunch of the convention guests - for the filming. The interviews took place on my patio couch. The clapper was definitely a fun way to start the videos.

I got to put the "Arizona" sticker on their "Places We've Been" map attached to the side of the Data Bus.

In early 2020, Paul and I had planned on being co-presenters at the Phoenix SQL Saturday, so I made cloud shirts for both of us so we'd match. Unfortunately, COVID came along and the Phoenix SQL Saturday never happened. 

For the interview with me, Paul and I donned our matching shirts to demonstrate how cool it would have been if we had presented with matching cloud shirts. They look pretty spiffy, eh? And the cloud earrings are of course a necessary accessory for any cloud data professional!

The above is a screenshot of the video posted to Paul's blog. I recommend you go see the video on his page, and also check out the other interviews he did with a number of other wonderful people in the data community. He also has a schedule posted telling you where he'll be and when. If he's coming to your area, reach out to him and say, "hi". 
 

I can't wait to see the Data Bus again when it next passes through the Phoenix area. 

I'm Speaking at the PASS Data Community Summit Nov 8-12, 2021!

 

If you haven't heard, the PASS Data Community Summit is coming up Nov 8-12, 2021. It's a free conference hosted by Redgate Software. The Summit is the largest gathering of Microsoft Data Platform professionals. Speakers will cover data management, analytics, architecture, and professional development. 

The full session list is available here.

My session is "Hitting The Bricks: An Introduction to Databricks". It is intended for people who have never used Databricks but would like to learn about it to see if it's something that they should add to their systems. Since it's a pre-recorded session, you can watch it at any time during the PASS Summit. Since I am scheduled for a live 30-minute Q&A session related to the session on November 11th at 5 PM Eastern Time, I suggest you watch the session prior to that time so you can come to the Q&A session with questions. The Powerpoint and demo notebook will be available on the PASS Data Community Summit page around that time as well. 

Register here! and see you at the Summit.

Renewed my Azure Administrator Associate Certification!

 

I'm happy to say that I passed the renewal assessment for my Microsoft Certified: Azure Administrator Associate. 

In December 2020, Microsoft changed the renewal process to help learners stay current - those with an active certification which was within 6 months of expiring could take a free online assessment to renew their certification. The assessment can be retaken multiple times free, but you have to wait at least 24 hours before retaking it. On the renewal page for the certification, Microsoft provides links to Microsoft Learn training directly related to the assessment content, so you will know exactly what topics to study. 

It's definitely a less stressful way to take the exam: you can take it using own machine in the comfort of your home - no driving to a test center and using an antiquated "testing computer" covered in other people's schmutz. And you can't beat the price: free. 

Be sure to renew your certifications when they are close to expiration.