PGP Universal Server is an email (and other stuff) encryption application, primarily for enterprise use.  One component of PGP Universal Server is the Web Messenger.  Now, you might have seen the functionality of the web messenger before.  We (the corporate empire that has your personal information) want to email you about something pretty private and want to send you an encrypted email.  However, you quite likely don’t have any sort of encryption routine for your personal email, so instead we ask you to visit a server in our DMZ to read your email over a secure connection.  In this manner we’re doing a bit of CYA and protecting your private information.

Well in order for you to visit our DMZ server and read your email securely, we’ll need to drop an SSL certificate out there on the web server.  Sounds simple, right?  Not always.

In our particular case we were using a 2048 bit certificate issued by Entrust in order to create the security for the HTTPs connection.  Adding the certificate was not particularly hard (disregard the firewall hell I ended up in), but we continued to receive errors about the certificate not being trusted.  Digging around a bit, it was clear that we needed to get the root and intermediate certificate for Entrust installed onto the web server in order to provide the browser with the entire certificate chain.  While this isn’t a particularly difficult thing to accomplish on a normal server, Windows or Linux, it proved quite tough when we didn’t have access to the Linux operating system for the PGP server.

We tried exporting the certificate from PGP and into IIS, then back out of IIS with the PFX option to “include all certificates in the chain,” and finally into PGP.  That didn’t change anything.  We called technical support and stumped them for several hours until they gave us the glimmer of a solution that eventually got us into place.  That solution?

1. Log into PGP Universal Server with administrative rights (I was superuser, I don’t know what least permissions for this might be)
2. Go to the Keys > Trusted Keys tab
3. At the bottom of the list, choose “Add a trusted key”
4. Import / Paste your root certificate and select the top two check boxes: “Trust for TLS” and “Trust for SSL”
5. Repeat the process for the intermediate certificate
6. Go to the Services > Web Messenger tab
7. Disable the Web Messenger service, and once it refreshes, re-enable Web Messenger
8. Test your SSL connection via a new browser window to see if your error has gone away

An interesting observation was that while Entrust was already listed on the Trusted Keys screen, that certificate was for the 1024 bit chain, whereas the ones that I needed to import were for the 2048 bit certificates.

I would wager that this scenario would be the same for most any other intermediate certificate you need to add to PGP Universal Server.  It’s unfortunate that there isn’t anything clearly documented about where to go for this functionality, as it seems like plenty of people will (or do) stray from their default list of trusts.

I may take some time and share a few other exciting items from our recent trials with PGP…

I was working on an in place upgrade from SQL 2005 to SQL 2008.  The default instance’s upgrade ran just fine – a bit lengthy, but fine.  I restarted the server and everything was peachy.

I moved on to upgrading the named instance and wasn’t nearly as paranoid about the process.  It seemed to run a little faster than the default instance did – I put that to the fact the perhaps some shared files were already done in the earlier effort.  Eventually, though, the installed finished and showed failure.  Not cool, but I figured good ol’ Google would take care of me.

Wrong.

When you Google for my error code from the install logs:

Wait on the Database Engine recovery handle failed.

You get a number of results, but none of them addressed my specific error, which was:

Script level upgrade for database ‘master’ failed because upgrade step ‘provisionsystemaccounts.sql’ encountered error 15151, state 1, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.

This business of the “provisionsystemaccounts.sql” script error had only one result in Google and it didn’t help me.  I couldn’t find the script hidden on the computer.  I opened FileMon and couldn’t see it being referenced.

All of the results I did find talked about having changed the sa account to a different name and that you’ll need to rename it back to make things work.  They walked me through starting the SQL services without letting the upgrade scripts execute, and that was a significant help in the end.

Let me walk you through the solution and then I’ll explain what I think happened.

1. First, open your most recent SQL Error Log file
2. Look for “Starting provisionsystemaccounts.sql …”
3. Following that string will be a “Cannot find the user” value – this is your key

Using the following steps, start SQL and create the user / group:

1. Start SQL Services avoiding the upgrade scripts:
   net start mssqlserver /f /T3608
   Note: I would wager you actually want to start a named instance, so replace ‘mssqlserver’ with ‘mssql$<instancename>
2. Connect to SQL using the DAC.  The following command will get you to a sqlcmd prompt via a trusted connection:
   sqlcmd -E -A
3. Issue a ‘create login’ command to create the user / group you’re missing, followed by the all important GO
4. Type ‘exit’ in the sqlcmd prompt to return to the command line
5. Stop the SQL Server services you started earlier with:
   net stop mssqlserver (insert appropriate service name)
6. Run the Repair from the SQL 2008 installation media

The repair should run right along and fix everything for you.  If you encounter another error I suppose you can check to see if there is another “missing” login or you’ve run into something I didn’t see.

Now – why did this happen?  Well, in my situation it seems to be related to the fact that I upgraded the default instance first which also upgraded the Full Text Engine.  That, in turn, removed the Full Text Engine user group from the local groups.  In particular, I was missing “<ServerName>\SQLServer2005MSFTEUser$<ServerName>$MSSQLSERVER”.  Note the fact that it mentions the default instance, not the named instance!

The SQLServer2005MSFTEUser login refers to a local Windows group that is used for controlling access to the Full Text Engine.  I suspect that when I upgraded the default instance, the installer removed the group from Windows, but not necessarily from the named instance of SQL.  Once I recreated the group, granted my service account access to mirror the other similar groups, and added that login back into SQL via the DAC, everything went fine from there.

This took me 4 hours and I nearly did a full rebuild of the system databases, which would have set me back by a day or two.  I hope that I’ve saved you a few minutes or a few gray hairs with this one…

So, getting to the point, I recall the concept of SQL Saturday coming to Nashville slid into my knowledge sometime months prior to the actual date. The idea floated about, given that I still sit near the DBA team, and they are friendly enough continue to treat me as one of their own, despite my conversion to management. It wasn’t until the week of the event that I actually allowed it to materialize, and I registered.

From that moment, a re-awakening began. I downloaded the all-powerful TweetDeck and all those people I followed long ago gave the little bird inside it the breath of life, allowing it to chirp… constantly. I immediately scrambled to turn off the notifications to give myself a little bit of control.

As things cooled off a bit, I got back into a little rhythm sorting out who was who again. It wasn’t too tough – Brent Ozar (blog | twitter) and Thomas LaRock (blog | twitter) were there, of course. Those two were the pushers of the tweet-drug that got me into this whole thing 2 long years ago at the ’08 Summit. I spotted Michelle Ufford (blog | twitter) who I’d collaborated with on a cool little stored procedure that tracks the latency on a replication subscription using tracer tokens. I was able to pull out those stats just recently to help prove a point! There were the MVPs that I’ve not met, but listened to intently like Buck Woody (blog | twitter) and Denny Cherry (blog | twitter). I got to see that like Mr Rockstar, Jeremiah Peschka (blog | twitter) was mysteriously changing jobs… to then discover Brent had changed jobs, too… and Jeremiah had taken Brent’s. Ugh… my head hurts.

I also noticed I was following one of the best tweeters – Adam Savage (@DonTryThis) from the Mythbusters!

Sorry – I veered off from SQL Saturday, the point of this. When Saturday finally arrived, I was a bit put off by “losing” the day with my family, but I took it as an opportunity to get out on my motorcycle as well as to get re-acquainted with SQL Server. I arrived, got signed-in and worked out a game-plan with my co-worker Bill. It was clear we’d be side-by-side most, if not all of the day, as we had similar interests throughout the program. We went through back-to-back PowerShell sessions, learned about the internals of SQL server, DMVs, a first look at Data Collector, and then a nice framework for performance tuning.

In the midst of all that, it was a re-introduction to the people that put their knowledge out in front of everyone so we can all be better DBAs, no matter the scale of our environment, our seniority in an organization, or if we’re Ops or Dev. The camaraderie was fantastic, and it was a great to find a new set of faces to drop into Twitter so that annoying little TweetDeck bird can chirp a little more often – Glenn Berry (blog | twitter), Kendra Little (blog | twitter), and Aaron Nelson (blog | twitter).

Amongst those 500 words above, I’m not sure if I said much, but take this from them: SQL Saturday isn’t a ‘lost’ Saturday. If anything, it’s a great kick in the pants to get back to your database servers and make them bigger, stronger, faster. It’s an awesome way to see a larger population of database geeks and find out that a bunch of them stay in continual contact via the intertubes. And even better, because they are in such good contact, they’ll be in good contact if you need help with something.

So here’s hoping I can get the funds approved for the 2010 Summit, because there was this really interesting hashtag I saw on twitter and I really want to join in.

I initially encountered this “problem” at 3am in the middle of a high visibility database re-host where I had to relocate the data files from one server to another using a detach / attach process. After detaching the database through Management Studio, I remoted into the server to copy the file, only to receive the error “Access is denied. You don’t have permissions or the file is in use.” I thought to myself, of course I have permissions – I’m a local administrator on this server. Next, I thought why would the file be in use – I just detached it!

After about 30 minutes of chasing our tails, we discovered that with SQL Server 2005, when you detach a database, the file system object will have its NTFS security changed so that only the Windows account you used to detach the database will have any access to the file. This becomes an issue for my team because we log into the SQL engine through SSMS using one account, but when using remote desktop, we use a different individual account.

The fix, so long as you are a local administrator, is to right click on the file(s) and alter the security to re-grant the appropriate (likely administrators) Windows group(s) full access to the file. At that point, you’ll be able to perform the necessary actions like deleting, copying or moving the file(s).

I can’t say I have any idea what happens when you do this using SQL authentication, nor can I speak to whether or not this occurs in SQL Server 2008.

Anyone else experience this one? Does anyone know about SQL authentication?

First, the MSDN PBM blog that isn’t updated too often, but the (current) top post for “complex” policies seems pretty helpful.

Next, Lara Rubbelke, who had a presentation at PASS that I missed. She’s been working with Policies, PowerShell, and built an entire framework to help you get it done.

Third, Buck Woody, who seems to be a lot more of PowerShell kind of guy, but knowing PowerShell, I think, is going to be key to automating policy evaluation across your enterprise. (Of course Lara may have taken care of that for us…). Buck offers an awesome PowerShell framework that he presented at PASS on his blog as well.

As for my first hand experience, it came before PASS and it was a great experience. I used a policy stored on my sandbox SQL 2008 server and evaluated 8 instances that were a mix of SQL 2000 and SQL 2005. I had that policy check specific server configuration items (such as Max DOP) and correct the instances that didn’t meet the stated policy. After creating the policy, it was just a click of the mouse to evaluate. Like I said earlier, though, to automate, you’re going to need to know a scripting language like PowerShell.

As a final link, I’ll send you to the PASS Presentation Decks, where you can search through for the presentations and find a number of slide decks regarding PBM. Specifically:

• DBA-320-M
• DBA-412
• AD-301-M

Enjoy – and be sure to let me know if you figure it all out because learning PBM is high on my personal list… it’s just that I haven’t had any time to devote to that particular list!

So in the final days of 2008, we had plenty of scrambling within the company to meet deadlines before the big production freeze.  This lead to plenty of late night, last minute code reviews and releases.  Beyond that, we had several long-running critical incidents crop up – one associated with the MS CRM app we’re using, so not exactly a great place to see slowdowns.

Things kept quiet, though, on the final day of the year and I was able to finish my on-call cycle with a quiet night on the couch.

As 2009 starts, pretty official today, what with it being the first Monday, I’ve got plenty of things to stare down.

Biggest on the plate is a SQL 2008 cluster – 5 nodes using Veritas Clustering technology.  The business wants to see data from all over the enterprise in several different places as quickly as they can get it.  As such, we’re going to stand up a very sizable set of replication distributors so we can stop worrying about it.  Right now a lot of the work is done through self-distributors, but that’s only because one of the remote distributors is a Pentium III server.  Let’s just say that isn’t exactly speed demon these days.

While I’m waiting for all the infrastructure to get in place, I’ll be testing VCS with SQL 2008 in the lab.  I don’t expect much to happen there – the VCS SQL Agent doesn’t seem to be the most complex piece of work, but maybe I’m overlooking something.

I’m just polishing off a little C# applet to dig through the enterprise and collect physical configuration data about our servers.  Things like physical memory, vendor, etc.  This way we can identify the servers that aren’t meeting our standard and work toward bringing them in line.  We realized the other day that one of our clusters is operating with a lot less RAM than we thought… oops!

We’ve got a big datacenter move coming along, so I’ll play a role in all that planning.  We’ve got a server exhibiting CPU pressure issues, but I’ve deferred a lot of that to my teammates, since I think I’ve got plenty of things hanging around and I can’t dedicate myself appropriately.  Of course we still need to sort out this database mirroring problem – that hasn’t gone anywhere for weeks and we’ve just been running with log shipping as the alternative.

The boss is in town this week, too, so we’ll all get a little time to get the feel of things from him, which is nice.

I’m not really one for resolutions, but I do know that I’m going to re-dedicate myself to the blog, keeping it updated better.

I got tagged while I was standing in the middle of the Magic Kingdom in Florida with my family. I briefly considered answering the call, but realized I’d better focus on the Mouse first. I’ve been back for a week and I’ve still not gotten my head back 100%, so the reply just hasn’t come up until now.

Database Administration – isn’t that a little twisted?

I think the first challenge simply has to be my choice to take this career path, and the process of getting to where I am today.

From the age of 13, I knew for absolute certain that I would be a police officer. I was consumed by the culture and the idea. I bought a radio scanner and listened to their communication. I played every one of the Police Quest games on my computer. I even got a four year Criminal Justice degree because my parents told me I couldn’t skip college to take my “dream job.”

So many seeds of my current path had been sewn during that time without my quite realizing it. First, I was on or around a computer all the time. My grandfather had my parents drive to his office and pick up the Apple IIe each weekend for us to have around the house. I was five years old and from this point forward I was surrounded by computer technology.

My father ran an IT based small business from home when he was laid off from his engineering job. My grandfather and uncle amassed a collection of both Apple and IBM machines. I went to BASIC programming camp at least one summer. Yet the entire time I was growing up, I actively rebuffed people’s ideas that I would take a job in IT. I was going to be a cop and that was that.

Fast forwarding many years, I find myself in the middle of the 6 month long police academy in my hometown – my life’s goal achieved – and I realize that it isn’t anything like what I had dreamed of. While I had thoroughly enjoyed the people I worked with as a civilian there (doing, what else, Access database work), the daily grind of an officer was looking less appealing. The people were different here, and I looked inside myself to find that this was not my dream job.

I pulled out of the academy, an immensely difficult thing for me to admit, and I went back to my job at the mall – selling computer software and games – that I had done for more than 8 years prior to now. I dove into the certification process, grabbing up every book I could find for an alphabet soup of certifications. I self studied my way to an A+, I-Net+, Server+, MCSE, MCSA, and finally, the last one, a MCDBA. This was over a six month period of time where I balanced moving in with my younger sister, doing the mall gig and building a solution in MS Access for a family friend’s business.

I knew going in that despite a litany of certifications, I wasn’t going to get in anyplace with some experience. Luck, though, was on my side. I got a call one day at work about a 3 week contract with a local manufacturer – their DBA was going on vacation and they needed someone to cover for a week. To this day, I have no idea how I got the call, but I took it. I worked with their guy for a week, learning the environment as much as possible, and then it was mine for a week. I had backup from non-SQL DBAs, but it was basically mine. The third week, he came back and we reviewed what had gone on. They called me two weeks later and asked me to do another three weeks so they could speed up a project – I’d made an impression.

I can’t say that I was suddenly on board with this place, but I finally had my foot in doors – I finally had some experience. From there I landed a gig an old friends from my mall job… which leads me to the next challenge.

Repl-what?

Several months into this gig with a small business where we did bench work for a few other computers, they decided to take on developing software for the retail stores of our major customer. It was going to be backed out of SQL Server, and I was the DBA. Long story short – we decided that we’d drop a SQL Server into each of the 80 stores and replicate, over the public network, the data back to the home office.

Yeah – the public network. Cost savings. And the “cost savings” continued into the fact that we didn’t get static IP addresses on the end points in the retail stores. I fought for weeks to sort out merge replication, passing out ID seeds, and working with FTP replication in SQL 2000. We tested in the “lab” and when the day came, we all drove out to our first stores to set everything up.

There were problems – the app wasn’t quite right, our store setup routine was brand new and clunky, and then the replication just wasn’t working. At this point, remember, I’m the guy. We’re 10 hours from home, sitting in a retail store for 14 hours, a few days in a row, while the app code is tweaked and I hammer my head against the desk over replication.

Eventually we pulled the plug on the replication. It just wasn’t going to happen in the time frame we had, and we had to start moving forward. The first three stores were allotted more time than the others, and that time was running out. We sorted out a routine of seeding all the tables’ ID columns with the store ID and decided to figure out how to get the data centralized later.

All I know is that I was in the hot seat, pushing the very edge of my abilities and I had to make a nasty decision to say, I’m sorry – I can’t do what we wanted to do, but we can do [X] instead. I was lucky and my boss was understanding. She was technically oriented, understood the situation, and we moved on.

Once we got out of those first stores, my buddy Paul and I ended up working (or driving) for nearly 40 hours straight. I will never forget that whole process, and the lessons learned.

Into the shark tank

Eventually the retail business worked on my nerves, and I found a new job out of town. This was a relative start up and they had big contracts in the pipe. I was part of a two man team preparing to handle a good bit of health care data for some big customers. My partner was an experienced DBA and I felt very comfortable. I learned a tremendous amount from him about the day-to-day maintenance of servers.

Being a start-up with big ideas, things eventually unwound and I was in the market. Luck shone upon me once again and I had stumbled into a contact through playing ice hockey. This guy, a substitute player on my team, was a DBA manager in town and he was looking for somebody. I’d turned him down once, with the promise of the start-up still gleaming in my eye, but when I came back a couple of months later, he was still in the market.

I joined up, despite hearing stories of long hours, and I quickly realized I was going to get run over by six runaway trains if I didn’t get moving fast. I dove in, tearing through the ticket queue, learning about the hundreds of SQL instances in the environment, and generally flying low, but steady while I learned.

Eventually the team shrank, we changed directors and even managers, but by then, I realized I’d positioned myself pretty well. I suddenly found myself full of institutional knowledge – people on the team would ask about X job on Y server and I could tell them who requested, why they wanted it, what it did and when it ran. I knew the outage windows for dozens of servers by heart.

I worked really hard to be outgoing, friendly and approachable – something I’d wanted to do from the very start of working in IT. I never wanted to be the guy to push the user away from the mouse and do the work. This paid off – I got noticed, and I got added to critical support teams with huge visibility.

At this point, I’m a Senior DBA at this job and I’m working hard to be the mentor that the former teammates were to me. We pull some long weeks, but I’m working with technology and processes that I just don’t see being used around me. I’ll trade a few extra hours at work for the experience I’m gaining.

While we were discussing his interview process, I told the former manger, who is now an architect with the company, that I realized I never could have passed his interview. If I had walked in and faced him, I wouldn’t have seen past the interview room. His reply? “I have a greater faith in potential than I do experience. I knew from talking with you that you had a great potential and I wanted that on my team.”

OK… I was a little late to the party, and I was a lot long winded. Thanks to all that, I’m not sure that there is anybody left to tag… so – if you’ve read this far and haven’t been tagged, then I tag YOU.

Amongst plenty of other things, Kristine and I spent the week trying to re-create the errors we saw in our production environment.  (Giving credit, where credit is due – this was mostly Kristine’s work)  To our dismay, we haven’t been able to reproduce the errors!

According to Microsoft Support and to the previously mentioned forum post, the culprit is the sp_getapplock / sp_releaseapplock being used within the database.  However, we’ve identified the two stored procedures in the production database that make this call, and we’ve executed them multiple times.  The lock procedure have to be executed within a transaction, so we left a few transaction open, and ran our tests to see if this might help – not at all.

We’ve tried with an empty database, we’ve tried with a restore of the original database.  We’ve tried with manual tests via SSMS and we’ve tried to create the error with people using the application – absolutely no combination of tricks has given us the results we want / expect.  At this point we’re stuck wondering if CU10 will or will not fix our problem.

While I’m away, maybe Kristine will find the magic test that gives us a chance to confirm the fix as well as test the application and then we’ll be ready to move on the patch before the holidays are upon us.  Maybe.

Anyway, this session had broken, but the log was still building, so it wasn’t the end of the world – let’s just start it back up.  No – we couldn’t be so lucky.  Instead, we found on the target side a log full of minidumps and errors.  Now, for the Googlers (which included me and my teammate!), the errors:

Error: 17066, Severity: 16, State: 1
SQL Server Assertion: File: <loglock.cpp>, line=818 Failed Assertion = 'result == LCK_OK'. This error may be timing-related. If the error persists after rerunning the statement, use DBCC CHECKDB to check the database for structural integrity, or restart the server to ensure in-memory data structures are not corrupted.

Error: 3624, Severity: 20, State: 1
A system assertion check has failed. Check the SQL Server error log for details. Typically, an assertion failure is caused by a software bug or data corruption. To check for database corruption, consider running DBCC CHECKDB. If you agreed to send dumps to Microsoft during setup, a mini dump will be sent to Microsoft. An update might be available from Microsoft in the latest Service Pack or in a QFE from Technical Support.

Error: 1454, Severity: 16, State: 1
While acting as a mirroring partner for database '<DBName>', server instance '<ServerInstanceName> encountered error 3624, status 1, severity 20. Database mirroring will be suspended.  Try to resolve the error and resume mirroring.

Of course, with a database mirroring target, you can’t fire of a DBCC CHECKDB command – it’s a database in recovery!  Further, as I mentioned earlier, simply restarting database mirroring didn’t change a thing.

Amusingly, this was about 3 days after we’d returned from Seattle, so my first thought was to send out a plea for help from the twitter gang, but after no response, we turned to calling Microsoft.  Actually, most of the work from this point goes to my co-worker Kristine.  She spoke to support, passed the mini dumps and got confirmation of what we’d found in the one reasonable result of our Google search – the use of “sp_releaseapplock” in the source database.

Our source database is a third-party, shrinkwrap peice of sofware for Workforce Management named Aspect.  In a recent upgrade, one or more of the stored procedures for this applicaiton included the use of this particular command.  In turn, that command wrecked our mirroring process.

Just like the Google result above, Microsoft has suggested that CU8 will resolve the issue, but we were already looking for CU10 for this same server, for a different problem, so we’re headed down the route of testing CU10.  Once we’ve confirmed we can make the mirror break happen in test, we’ll see if CU10 is the fix, and I’ll post an update.

At the same time, it’s made me reflect on the things happening in my own team. Here I sit, technically on Thanksgiving Day, on my laptop, winding down from cleaning house, and I get a ping in IM from our DBA in Japan. It’s late in the afternoon for him and I know he was called in early. Beyond that, I know this guy is a really sharp DBA and if he’s asking for something, it certainly isn’t trival. I hesitate slightly, but I respond.

He’s hit a snag with replication – the distributor is complaining about the system generated replication procedures being missing on the target. Amusingly, I’d spent the plane rides from Nashville to Seattle typing up some internal training materials on replication (which I’m realizing are way too in depth for a start) and I knew exactly what he needed to do.

Unfortunately, I explained it in way too little detail, and he didn’t grab it right away. He toiled for a few minutes and I checked back in. I knew from his response that my answer hadn’t made sense. Suddenly I launched into a paragraph explaination of replication distribution and how to re-generate the procs that were missing. He plugged away for a minute and came out victorious. He was extremely thankful, given the time of night, and I told him it was pleasure enough knowing that I’d been able to teach him a bit more about replication. This is the spirit of PASS – sharing with your fellow DBA as peers.

So – if you didn’t already know, here’s (my take on) the short version:

Replication transfers data uses parametered stored procedures and these stored procedures are system generated by the snapshot agent. The snapshot agent leverages the system stored procedure sp_scriptpublicationcustomprocs to generate those procedures. It then applies that script only to the target environment and at a later time the distribution agent, when attempting to transfer data, calls these procedures with the appropriate parameters.

Since the distribution agent is complaining that these specific table’s stored procedures are missing, all you need to do is to run that system stored procedure which will generate a script for you. That script will have all of the replication stored procedures for every published table in the publication you indicate. From that large system generated script, you must locate the 3 procedures associated with the specific table that is creating errors and then run those procs on the target environment.

Those procs, for reference, are the sp_MSins_<tablename>, sp_MSupd_<tablename>, sp_MSdel_<tablename>, for INSERT, UPDATE, DELETE respectfully. Also, the procedure sp_scriptpublicationcustomprocs needs to be executed in the publishing database and it takes a single parameter of the publication name.

Our little hang up came when we changed the primary key constraint on a published table, but the snapshot agent didn’t run to replace the procs.

If you see anything I’ve stated wrong, feel free to correct me!