Prompt Critical Solutions (PCS) has been using 3CX for Voice Over IP (VoIP) solutions ever since the company was founded in 2016 (even before if we are being fully transparent).  As is the case with many solutions, one starts off simply using a product. Then, once you are sufficiently happy with it, you start promoting it more. And, when presented an opportunity... team up with them as a partner. That evolution and partnership between PCS and 3CX started early on. In fact, PCS has been a 3CX partner since 2018. As with most partnerships, there is a tiering status and 3CX is no exception. Over the years, the partnership levels for 3CX have evolved. With this latest evolution, PCS is now recognized as a Bronze level partner.

There's a plethora of reasons I could go into as to why PCS has chosen to use 3CX as our VoIP solution of choice. However, if you just want to take our word for it and start a trial of your own, feel free to use our Referral Link to download a fully equipped 3CX System today! Otherwise... keep reading on (content curtesy of the 3CX marketing team) to learn more about 3CX and some of the amazing things it can provide you with...

3CX - Your Communications System

3CX is a robust, fully equipped communications system, which allows you to communicate with your colleagues and customers - anywhere, anytime.

Here is a breakdown of why 3CX is the solution for your business:

• Telephony: Web client and mobile apps for remote working
• Video: Included in your system, no add-ons needed
• Live Chat: Talk with your customers in real time via your website or WhatsApp!
• SMS: Allow your customers to interact with you instantly
• Integrations: CRMs and Microsoft 365
• Low Cost: Save on your phone bills, add ons and hardware
• Flexible: Keep your numbers, choose your phones and your provider
• Secure: 3CX includes advanced security features
• Install anywhere: Hosted or on-premise, MiniPC, Hyper-V, VMware or KVM!
• Backed by the vendor, supported by us
• 600,000+ customers globally

3CX Apps & Features

3CX enables you to take your extension with you wherever you go. With apps for Android, iOS, web browsers and Windows, you are guaranteed to be kept in the loop, always and everywhere. What’s more, customers can instantly reach you directly from your website with the 3CX Live Chat and WhatsApp integration.

3CX Apps for Android & iOS Device

• Make and take calls from anywhere
• Video Conferencing
• Instant Messaging and Chat
• Answer WhatsApp messages
• Conference Calls
• Free VoIP calls with your smartphone
• Encrypted and Secure

User-friendly Web Client / Desktop Apps

• Work from anywhere - multifunctional web app with inclusive UC features
• Call from your browser, control your IP phone or smartphone
• Answer customer queries from live chat, WhatsApp and SMS from one interface
• Video conferencing at the click of a button
• Native Desktop App for calls
• Launch calls directly from your CRM
• Integrate seamlessly with Microsoft 365
• View the status of your colleagues

3CX Live Chat: Next Level Customer Communications

• Chat with your customers on your website
• Create Chat queues
• Answer from your webclient or 3CX App
• Transfer chats to other team members
• Elevate a chat to a call or a video with a click
• Chat reporting available
• Chat monitoring feature for Managers!

WhatsApp Integration

• Manage WhatsApp messages directly from one platform
• Messages are logged, to ensure no response violates company policies
• Staff do not need to give out their personal WhatsApp numbers
• Chat conversations are logged centrally
• WhatsApp messages can be sent to a queue of agents to share the load

Send & Receive Business SMS / MMS

• Send SMS / MMS to your customers
• Route SMS to multiple agents so no message goes unanswered
• Keep personal mobile numbers private

Integrated Video Conferencing

• Free for up to 250 participants
• Initiate with a single click
• Video conference dial-in
• Create as a conference or a webinar
• Remote control assistance for quick & easy troubleshooting
• Pre-upload PDFs before meetings
• Includes a whiteboard
• Share your screen
• Polling tool
• Record your videos
• Includes chat

The 3CX Call & Contact Center

With 3CX you get advanced call center features. Increase your customer service, sales, boost productivity and offer support with 3CX’s contact center technology.

• Switchboard and wallboard
• Call Center Reporting – user activity, team queue, SLA and more!
• Queue Strategies and real-time statistics – round robin, hunt by threes and more
• Call & Chat Monitoring available for Managers
• Integration with Microsoft 365
• CRM integration with popular CRMs – Salesforce, Freshdesk and more
• 3CX CRM API to develop your own CRM integration
• Call Recordings
• Hot Desking
• Create your own call flows with 3CX CFD

Slash Costs

With 3CX, your ongoing telco costs will be reduced significantly.

• Decrease your Phone Bill by up to 80%
• Licensing based on number of simultaneous calls and not extensions
• Make free interoffice calls and branch calls
• Free calls to remote workers with web client and iOS / Android apps
• Use prominent SIP trunks to reduce outbound call charges
• Eliminate 800 number phone bills with WebRTC integration
• Use International DIDs & IP Telephony so customers can call in at competitive rates

Security & Backup

When it comes to VoIP security, 3CX sets the standard. 3CX’s inbuilt security features have been exclusively developed to protect your PBX system from attacks.

• Increased secure web server configuration
• Automatic detection & blacklisting of SIP attack tools
• Global IP Blocklist automatically updated for participating PBXs
• Traffic to 3CX apps is encrypted via the 3CX Tunnel
• Voice traffic is encrypted via SRTP
• Automatic generation and management of SSL certs
• Qualys labs A+ rating and Immunilabs A Rating
• Automatic failover for instant backup
• Encryption between browser and the website ensured via HTTPS
• Limit access to 3CX management console based on IP

Call Flow Designer

By utilizing 3CX’s Call Flow Designer you can handle incoming calls effectively, efficiently and much faster. Here are just a few examples of how 3CX can improve your customer service:

• Call routing based on customer authentication
• Automatic outbound dialler
• Callback scheduler
• Credit Card Authentication
• Phone Orders
• Surveys
• Automated Text to Speech & Speech to Text in 120 languages

CRM Integration

3CX CRM integration can improve customer satisfaction, increase staff productivity and save you time. It supports all major CRMs including Salesforce, Microsoft Dynamics, Hubspot and more!

This is how it can help you:

• Call journaling
• Creates a new customer record for new numbers
• Saves agents time searching
• Chat journaling when using 3CX Live Chat
• Call logging in your CRM records - even when using 3CX Apps!

Hotel PBX

3CX’s hotel module enables you to provide high-end service, minus the cost. The Hotel Module is optimized for internal and guest communication with features such as:

• Check-in/out of Guests
• Sets extension to match guest name
• Billing of external calls
• Scheduling of wake-up calls
• Blocking of external calls in vacant rooms

This won't be a long-winded blog post about the evolution of cable standards over the years, nuances of wire gauge and twist rates, and all the various connectors that can be used. There are so many other sites that have covered this, and entire novels could be (and have been) written on the subjects. This is just a simple table to act as a quick reference for the various cable categories, their usable/rated speeds (not all of which are official), the distance that speed should be obtainable at (not all of which are agreed upon), and some notes. Enjoy!

Update on Jan 26, 2023 - I expanded the Cat 6 10Gbps entry to reflect shielded vs unshielded and clarify the exact part of the standard where it appears.

As one goes about shoring up their IT security, one aspect of the process is limiting (or authorizing) the methods in which a "secure connection" is established. A secured cryptographic connection basically has 2 components, the "protocol" portion, and the "cipher" portion. Together these create what is called a "cipher suite." And, as should be no surprise, as technology advances, these cipher suites fall out of favor and are flagged as being "weak" or "insecure" as vulnerabilities are discovered, and/or computational power renders the security provided as trivial to break. Unfortunately, as I found out this past week, even big players on the block *cough* Microsoft *cough* occasionally don't get the memo about what they should/shouldn't use.

But, before we dive into this past months adventures... first a very rudimentary primer on cipher suites. If you already know about cipher suites... skip this paragraph... otherwise read on...

While this is a great oversimplification, the protocol portion of the process is what sets up the initial hand shake between two systems and negotiates what kind of packet exchange will take place. Think of this a bit like saying you are going to ship packages back and forth via USPS, UPS, FedEx, etc.. The protocol doesn't deal with the data... just how the data is shipped back and forth (again... massive oversimplification). The cipher on the other hand is how the messages will be encoded/decoded... it handles the scrambling/obfuscation of the data so that anyone listening in can't intercept and make use of the data. The cipher could be thought of as a language that both systems agree upon... English, French, Swahili, etc.. But, just because you happen to know Swahili doesn't mean you can watch the traffic and make sense of it... the language (cipher) just sets the ground rules of the language... there's a set of encryption keys that both parties will use to do the actual scrambling of the data. The language (cipher) chosen just tells both sides how to use the keys to encode/decode the data while they are speaking Swahili. Stick with me a minute on this one... In English I say "cat" and you know what that is. Well, if we agreed upon Swahili as our language, I would say "paka" instead. But, I want to obfuscate what I really mean, so my super secret decoder book tells me that when I want to say "cat" in Swahili, what I should really say is "tumbili" (the Swahili word for monkey). That way when someone listens in on the conversation, even though they know Swahili, they won't understand what the conversation is actually about because they don't have a copy of the secret decoder book.

OK... that was a lot of oversimplification... but that helped to bring it down to where most people can understand cipher suites at a base level.

So.. "Fantastic!" you say "We have some cipher suites that are safe and some that aren't. How do we know which ones we can trust to keep us safe?" I'm glad you asked! A great site to check the status of a cipher suite is Ciphersuite Info which will show you the security status for over 300 different common cipher suites. If you visit that site, you can see see the current security status..

I don't want to get into what all the name mean and do a history of each. But, breaking down a few basics from the above screen shot you'll see stuff like CBC which is "Cipher-Block-Chaining" and SHA which is "Secure Hash Algorithms" and the like. If you click on any one of them in the list you'll be presented with a more detailed breakdown of each to include any reasons to hint as why it has been given a particular rating.

As for the general meaning of the ratings... Pulling straight from their FAQ section, here's what each categorization means...

• Insecure - These ciphers are very old and shouldn't be used under any circumstances. Their protection can be broken with minimal effort nowadays.
• Weak - These ciphers are old and should be disabled if you are setting up a new server for example. Make sure to only enable them if you have a special use case where support for older operating systems, browsers or applications is required.
• Secure - Secure ciphers are considered state-of-the-art and if you want to secure your web server you should certainly choose from this set. Only very old operating systems, browsers or applications are unable to handle them.
• Recommended - All 'recommended' ciphers are 'secure' ciphers by definition. Recommended means that these ciphers also support PFS (Perfect Forward Secrecy) and should be your first choice if you want the highest level of security. However, you might run into some compatibility issues with older clients that do not support PFS ciphers.

The first 3 in the list are fairly self explanatory. It's that last one where they talk about "Perfect Forward Security" (PFS) that throws people off a bit. Not going too deep into the weeds, PFS is basically a mechanism by which if a hacker takes a packet and manages to decrypt it and thereby discover the key being used, that the key cannot be used to decrypt new packets being transmitted... or to determine future keys to allow the decryption of new packets. So, for now, PFS stands at the panicle of encryption techniques.

"Fantastic!" you say. "We know know what's good/bad to use... but, how do we know what we are using?" I'm so glad you asked! 😁 While cipher suites can be used for so many different applications, the one most people use every single day without giving it a second thought is the ones being used to secure your web browsers connection to a given website. Any time you open up a browser and go to "https://www.example.com"... that "S" in "HTTPS" says it should be a secured connection. In many web browsers, if you see a padlock or shield next to the web URL after loading the site, that means the connection has been secured. There's an article by Microsoft that goes over what some of those symbols are and what they mean when visiting a website in Edge...

https://support.microsoft.com/en-us/microsoft-edge/securely-browse-the-web-in-microsoft-edge-c7beb47a-de9e-4aec-839d-28224a13a5d2

Now... that padlock or shield just says it negotiated a secured connection... it doesn't actually mean that the connection is "safe" to use. Why? Well... because the cipher suite that you used to negotiate with may be considered weak or insecure! In fact, at the time of this writing, the Microsoft Support page linked above uses a weak set of cipher suites! 😂 The way that you can see the cipher suite in use is to hit F12 in Edge or Chrome, then go to the Security tab and look at the connection properties. Viewing the Microsoft site in Chrome shows me that I'm connected using a weak suite...

"But, how can this be?!?" you say. "This is Microsoft! Certainly there must be a mistake. Surely you can connect to their site with a more secure cipher suite, right?" I love your optimism. Lets find out! As luck would have it, there's a website that you can use to check on the security of other websites. So, we can go to https://www.ssllabs.com/ and enter in the "support.microsoft.com" URL and it will test what levels of security their website supports. At the time of this writing... they get an "A" for security score. That sound good right? Well... not quite so fast... scroll down in the report to the section where they list cipher suites available for connection and you'll see the following...

You'll notice that they only support TLS 1.2... which is perfectly fine. But, all of the TLS 1.2 cipher suites that they support are flagged as being weak!!! 😲 There are no good/secure connection offerings available. For Shame!

"My organization said we are disabling all weak cipher suites. So, what will happen when I try to visit the Microsoft Support website?" It's funny you should ask! I've already done that myself and it's why I'm aware of the fact that Microsoft's site has all weak cipher suites in use. When I open up Edge and go to the support website I'm presented with the following error message...

So, on one hand, my settings worked exactly as intended. Disabling the known weak cipher suites prevented me from connecting to this site using a weak cryptographic connection. On the down side.... I can't connect to the site because there are no suitable/acceptable cipher suites available to choose from! 😛

Unfortunately, there's not much I can do other than complain. I can't force Microsoft to update their website to use stronger cipher suites. The only solution I have at my disposal is to lower the overall security posture of my browser. Or, in my case, launch an alternative browser that isn't honoring the Group Policy settings that disable the use of weak cipher suites *cough* Chrome *cough*. So... yeah... Edge will honor Group Policy settings that are created to disable specific cipher suites. Chrome on the other hand does not have complementary Group Policy setting. The only way to force Chrome to disable cipher suites is at the command line when you launch/run the browser. You have to pass it in as a command line argument/parameter which from an IT administrator perspective is 1) really annoying to deploy to users, and 2) near impossible to enforce that it always be used. One basically has to find every shortcut created by the system to launch Chrome and replace the run line to look something like this:

C:\Program\ Files\ (x86)\Google\Application\chrome.exe  --args --cipher-suite-blacklist=0x0088,0x0087,0x0039,0x0038,0x0044,0x0045,0x0066,0x0032,0x0033,0x0016,0x0013

Note: Each one of those hex values (eg. 0x0088) correlates to a cipher suite

And, if Chrome is set to be the default browser, just clicking on a link in an email will cause the program to launch without honoring the black list. As of this writing, there's no way that I'm aware to force the black list to be honored other than by command line arguments. Hopefully that changes some time in the future.

Long story short... nobody is perfect... not even Microsoft. (but, you didn't need me to tell you that, did you! 😉)

Note: Post migrated from old Blogger website.

Let us start with a bit of backstory before we dive into the "how to" portion shall we?

In the early days of using Veeam, the original backup and replication server I started with was a bare metal/physical server. The primary repository had a capacity of 20TB of usable space and my clients were barely using any of it at the time. Fast forward a few years and I was approaching 16TB of that storage consumed and it was looking like there was no end in sight for growth, so I needed to find a replacement solution. While there were tons of options available, what I chose to go with (for better or worse) was a server which had over 5 times the storage capacity as the original. But, this time I intended to use it as a hypervisor to pull double duty and make storage management of backups a bit easier to deal with... or so I thought at the time...and I was more or less right in the end.

Rather than stand up a new Veeam Backup & Replication server and start all over... and in doing so leave the existing server in service while backups aged off over the next two years... I chose to instead do a Physical-to-Virtual (P2V) conversion of the old/original server. I've done this dozens of times before using the Veeam Agent for Windows... Just never on anything quite so large before. But... Like so many other conversions... It worked like a charm. I took a backup of the server, stood up a fresh VM on the new hypervisor and did a restore. Boom! Other than the amount of time it took because of the sheer mountain of data being moved... everything worked fantastic. 😃

📌Side Note: I could have also simply moved the backup files over to the new server, but in doing so it would have lost all the ReFS block cloning benefits. While I did have enough available space to accommodate re-hydrating all the files, the thought of consuming roughly 60% of the new servers capacity right out of the gate was not appealing. Anyway... I digress...

With the P2V conversion out of the way, over the next few months each client was slowly transitioned over to using their own dedicated repositories... each residing on their own virtual disk. One advantage of this was that I found it was much easier to seed a repository for a new client during the onboarding process (trying to push 3TB of client data out of a 10Mbps DSL Internet connection is just not practical). This transition also significantly reduced the amount of data consumed on the original 20TB volume as short term retention policies quickly freed up a lot of space. The longer GFS retention settings however meant that there was still a substantial amount of data remaining on that volume.

Having freed up over 6TB of space (leaving me with ~10TB) I was eager to reclaim some of that space by trying to shrink and compact the virtual drive. With a volume formatted with NTFS, the process would have been quite simple... Defrag the drive to push the contents to one end of the volume, shrink the volume, then shrink or compact the virtual disk using any number of available methods. You don't really need to go through all of that just to shrink the virtual disk... but it does help from a house keeping perspective and it also prevents the drive from inadvertently growing back up to an unruly size. Drives formatted with ReFS are however limited to only being expanded... you cannot shrink an ReFS volume (at least not that I've ever discovered). I set out to do the other usual steps without first shrinking the volume and quickly ran into trouble. None of my usual methods of shrinking were working (I tried about 5 different methods with very limited success)... I was seemingly stuck with a +16TB virtual disk even though I was only consuming just under 10TB. But.. all hope was not lost... SDelete to the rescue!

The main problem I was having is that there was still non-zeroed data in the clusters on the virtual volume. With a non-zero value present, the compacting tools were refusing to free up any significant amount of space. (I did manage to recover about 1TB of space with my normal methods... but not the expected +6TB of space I was hoping for). As luck would have it though, the Microsoft Sysinternals team has a product called SDelete that can write zeros to all the free space on a volume to alleviate that issue. After using SDelete, I was able to compact the virtual disk down to about 10TB. All was good with the world.

Fast forward and now there is hardly any data remaining on the drive (relatively speaking). Due to aging of backups, manually moving some remaining backups to another volume, and just manual deletions in general, only ~1TB remains... and this remaining data is projected to stick around for the foreseeable future. So... Time for another shrinking session to free up another 9TB of space! I've learned a few extra tricks along the way and here's how I went about doing this latest one...

⏰Backing up a bit though, I actually started this process off with ~3TB of data... I'll explain a bit more on that in a minute.

I started off by moving the the 10TB virtual disk to a new storage pool. This was done as a live migration to avoid any downtime. The migration took the better part of a day. Once moved, I ran SDelete to zero out all the free space. In this case running "sdelete64.exe -z d:" was the magic command of the day. I then spent the next half day watching the 20TB volume completely fill to 100%.

Drive Full

But... don't panic... the virtual drive does not expand at this point. Because it's all zeros being written, the underlying virtual disk didn't budge in size. It stayed at 10TB the whole time. The zero data that SDelete is generating is written to a hidden file at the root of the volume. As soon as the volume hits 100% capacity, that file is deleted and the actual used space is once again properly reported (in my case just under 3TB). The second phase of SDelete is to purge the MFT files. This is where things get a bit weird. There's been a known bug in SDelete v2.x whereby the percent complete does not always report properly. And, in my case... by a LARGE amount...

Nothing quite like having a reported 1 BILLION percent complete! 😂 I'm not sure if this is something with the how SDelete interacts with this particular version of Windows Server (Server 2016), the version of ReFS version (ReFS version 3.1), an issue with it being such a large volume, something with it being on a virtual disk... the fact that SDelete was originally created to work with NTFS volumes (more on that later)... I just don't know to be honest. I just know it's been reported as an issue by many different people for a few years now and seems to remain a problem as of SDelete version 2.04.

Anyway, during this purge process you'll notice that the drive volume grows once again. This is due to the hidden file once again at the root of the volume (SDELMFT000000) which as of the above screenshot had already consumed over 1.5TB of space. It also took what was feeling like forever to do the MFT purge. Whereas it only took a shade over 7 hours to write all the zero data for the drive in the first phase... the MFT file purge writes much slower to the disk and as such was on track for taking 160 hours (over 6 days) to finish this second phase.😲 (I'm sure it doesn't help that this virtual disk sits on mirrored pool of spinning rusty metal if you haven't already jumped to that conclusion).

Several people have reported it was OK to let this second phase finish. But, I was growing impatient and killed the process with CTRL+C which cleanly closed the program and removed the hidden file it was creating. The reason I killed SDelete is that ReFS does not have a traditional MFT like NTFS. According to the software's documentation, what this phase is attempting to do is write files small enough that the data is stored within the MFT itself rather than written out to disk somewhere with a reference in the MFT. So, this phase... to the best of my knowledge... should be omitted from drives formatted with ReFS (again, SDelete was written for NTFS, not ReFS volumes). On an ReFS volume, if this phase is left to run to completion, it seems to simply re-zero the volume again, only this time much slower because it's doing so with very tiny file writes. With SDelete finished doing it's magic the virtual drive is still at the same 10TB it started at... but don't fret... it's magic time!

With zeroing the free space complete, I did another live migration of the virtual drive back to its original storage location. The migration back was significantly faster... correction... "shorter"...  because the only thing being moved is the actual data consumed at this point. All the free space, having been zeroed out, is ignored. Thus, in doing the move, the drive was auto-magically compacted and my final virtual disk size was reported at just over 3TB... Ok... So... Not really. It only shrunk it down to 7TB. 😞 And here's the reason why...

When data is written to a volume it is written to a "cluster" in the volume. That cluster has a fixed size associated with it. For NTFS and ReFS formatted volumes the default cluster size is 4kB. This size is set when you format the drive and cannot (to my knowledge) be changed thereafter. Here's an example of selecting the two cluster size options (shown as "Allocation unit size") for ReFS via the Windows Explorer drive format GUI, as well as an example of using the Disk Manager wizard showing the 8 (yes eight!) cluster size options available when formatting a volume with NTFS.

ReFS Cluster Options

NTFS Cluster Options

However, when I formatted this volume, per recommendations from Veeam, I formatted it with a cluster size of 64kB. Why does any of this matter? Well... if data is written to the cluster, but the cluster isn't full, the cluster still has to be tracked by the underlying VHDX file regardless of how full it is. This means that if even so much as 1kB of data is written into the cluster, the VHDX has to account for and track 63kB of empty space. Here's a perfect example of a text document with only six lines of text and is stored on a volume formatted with 64kB cluster size...

File Properties

This empty space in the cluster is called "Slack Space" and has been used by crafty hackers to store bits of malicious code.

This particular drive I knew was heavily fragmented from many years of use and it wasn't entirely surprising to know that I had over double the volume consumed in part because of having to track partially filled clusters (it wasn't the only contributor of course). I've used the Windows Defrag tool on the drive, but if I'm being honest... using the Windows Defrag tool is basically just a way of checking the box so you can tell your manager "yes, I defragmented the drive"... but it really doesn't do that fantastic of a job IMHO. So... not being completely happy with my initial space savings, I set out to "properly" defragment the volume to combine the clusters. The sad story of which was in my previous post "Lessons Learned In Pain: Disk Defragmentation"... So, I'll let you read that rather than rehashing it all again here. But, the short version of my initial defragmentation attempt was that it was an utter failure. The defrag program I used didn't know how to deal with Block Cloning and it re-hydrated all my backups bloating them up to roughly 6TB, and because of moving data around on the drive during the defrag process... it expanded out the VHDX file to just over 12TB! 😵 (talk about one step forward, two steps back!)

Being more than a little disgruntled about the drive having lost its block cloning benefits, I moved over a bunch of backups to their own volume to let them age off in another location. What I was left with was 1.1TB data that will stick around for the foreseeable future until the serve is decommissioned. So, I took another swing at defragmenting the drive. Now, the slack space as noted before is one reason the drive didn't shrink as far as it could/should have. Another reason is that the data was fragmented and scattered all over the drive meaning that there weren't large segments of zeroed data. Take for instance these two pictures here...

Fragmented Data Storage

Contiguous Data Storage

Even though the clusters are full in the first picture, there are segments in between that are empty. Now, it is to my understanding (I'm happy to be told I'm wrong) that those empty clusters have an entry for them too, and that entry is fairly small. But, the entry made is for a range of clusters, not just a cluster by cluster entry. So, if you have a large contiguous range of clusters, there would be a single entry for all of them in a row (e.g. clusters "30-35", rather than, "30, 31, 32, 33, 34, 35"), but, if you had the same number of clusters split into ten regions, then there would be ten entries to account for those ranges of clusters. As I understand it (again, I'm not a storage system engineer), these entries aren't anywhere nearly as significant as the issue surrounding partially filled clusters and slack space, but it apparently exists as part of the equation all the same. I've also read in a few places that if the empty clusters are few in number between two clusters filled with data, that rather than being flagged as empty, they'll be accounted for just as if they did hold data. Whatever the true story is, my personal experience has shown that it impacts being able to compress the drive.

So, with that in mind, I told the defrag utility to pack all the clusters sequentially, sorting the data by file names, which shoves them all to the front end of the volume, both minimizing slack space and maximizing contiguous clusters of real and empty data. I then waited patiently for another 20+ hours for the volume to be defragmented (I hate waiting). Here's the visual representation of cluster placement before and after the defrag...

Defrag Report

And, for what it's worth, one of the files that was listed in the pre-defrag analysis report was for an old company laptop, the backup file of which was only 190GB in size, but had a reported 17,002 fragments! 😝 Like I said... it was an old volume... and the native Windows Defrag didn't do much to ever help that volume stay clean. 🤷

📌Side Note/Tangent: Where I have found smashing all the data down to the front end of the volume has been most important for me has been when shrinking an NTFS volume (I'm talking about the actual volume, be it stored on a physical or virtual drive). The reason being is that you can only shrink the volume down to the last cluster that has data stored in it. If that cluser is near the end of the volume, you won't be able to shrink it very far. A small handful of partition manager applications will proactively move data for you prior to shrinking... but most wont. So, I always suggest using a good defrag tool like O&O Defrag to clean up the space before attempting to shrink a volume. As for "when/why do you shrink a volume?"... it is most often times done when I clone an HDD drive to an SSD drive and the HDD is significantly larger than the SSD. One could very easily argue that defrag is most important when optimizing the performance of an HDD (which is why it was developed in the first place), but I tend to convert more systems to SSD rather than leave them running on HDDs if I can help it. On with the story...

So, with the drive finally defragmented I once again ran through the same process as before... I ran the same SDelete command as before. This time I canceled it as soon as it hit the second phase where it tries to zero the MFT. And, like before, I migrated the drive which did the auto-magic compression... and BOOM! the end result was a 1.1TB VHDX file! 😁 Which if you are keeping tabs on the story from beginning to end... it means I went from 10TB down to 7TB up to 12TB  and finally down to 1.1TB  for a total reclaimed space of 9TB! 🤣🤣 And... the whole time I did all of this I had zero system downtime. 😉

Before Migration

After Migration

👮Disclaimers: Now, I'll say this much... While I did do this all live with zero down time, I wouldn't say that's a wise choice for everyone. I got away with doing it live mainly because the contents of the drive are stale at this point and no user data was being read from the drive much less written to the drive during this process. I would not recommend doing this live on a drive that had any significant amount user data access for fear of potential corruption or crashing of an application trying to write to disk. As was noted, the defrag program I chose was a horribly wrong choice due to it not knowing how to deal with block cloning and thus killing my storage savings... so pick a different defrag program. But, if you aren't using block cloning on your volume... O&O Defrag is a good program in all other respects! Additionally, SDelete was written to work with NTFS and nobody from the dev team (to my knowledge) has ever reported that it is actually safe or compatible for use with ReFS. The only confirmation that it's "safe" are the dozens if not hundreds of users who've said they've used it without issues other than noted above. That said... your mileage may vary! And, a final note... although the underlying VHDX file was successfully reduced in size, the ReFS volume was not... it's still 20TB... and since ReFS doesn't support drive quotas, there's nothing explicitly in place to prevent data from being written to the drive and causing it to bloat back up again... potentially all the way up to it's full 20TB capacity!

As a point of observation, here's the Disk Throughput on the hypervisor during the 1) zeroing process with SDelete, 2) migration of the 1.1TB of data between the temp drive and the production environment, 3) the completion of the migration of the zeroed out data... essentially the compacting of the VHDX file where Hyper-V was still transferring the drive, but all it was actually doing was reading zeros and not writing anything to the new VHDX file.

Disk Throughput

Anyway... that was my adventure in shrinking a 20TB virtual disk formatted with ReFS. The completion of the task at hand was only partially successful since using O&O Defrag killed the block cloning savings offered by ReFS. With the release of Veeam 12 around the corner, we should hopefully be getting the ability to migrate backups from one repository to another while maintaining ReFS/XFS block cloning benefits! 😍 Being able to stand up a new repo and move backups without losing block cloning would have rendered this whole adventure moot... I would have simply migrated and blown away the old repo. That said, this is still a good IT nugget to keep in your back pocket in case you need this to shrink a storage volume.

Note: Post migrated from old Blogger website.

The joke is that the definition of an expert is someone who's made all possible mistakes in a very narrow field of study. Well... Apparently I'm not an expert yet because I just had made a mistake this past week. Hopefully by sharing this, someone else can learn from my mistake to prevent them from going through the same pain I just did.

Thankfully, this mistake wasn't an earth shattering issue. More of an annoyance really than anything. Still... not something I would want to go through again. So, without further adieu... I share with you this "Lesson Learned In Pain: Disk Defragmentation"

As a Managed Service Provider (MSP), one of my services is to provide storage for off-site backups. And, as one would expect, I manage the server that these backups land on. One of the storage servers is winding down on its service life but is not yet completely empty. There are still a handful of backups (and most of those are for company internal use) that had either not aged off yet, or have not been moved to another storage repository. The volume that these backups are stored on uses the ReFS file system and takes advantage of Block Cloning to help save space. On this particular volume, it had 20TB of available space, of which only 3TB of it was still in use. However, if you go into Windows File Explorer to the root of the drive, select all and then right click and select properties... It would report back that there was nearly 6TB of data on the drive. And, that's not an error. The reason for the discrepancy is that the Block Cloning from ReFS gave me roughly a 2-to-1 space savings for that particular set of backups (for what it's worth, many of my ReFS backup repositories are typically closer to 5-to-1). Here's a perfect example of a 5:1 space savings with a screenshot of the two property windows side by side. On the left, it shows a 3TB volume (Drive G:) formatted with ReFS reporting that only 2.47TB of the drive is in use. On the right is the file level properties for the same volume (G:\), this time showing that 12.3TB of data is stored on the volume.

ReFS Block Clone Savings

So, how is all of this a problem? Well... In my wisdom, I decided I was going to defragment the volume. While the Windows Drive Optimizer (formerly Disk Defragmenter) tool "works" (and that's all I've been using on the volume up until now), I've found through experience on other systems that it's not actually all that good at defragmenting, and in particular compacting space. As I was planning on compacting this volume, which benefits from having as much contiguous free space as possible, I wanted to use something better than the stock Windows Defrag tool.

There are dozens if not hundreds of defragment tools on the market to choose from, but I decided to go with an old favorite of mine... O&O Defrag. I've been using that program on and off again for over two decades with great success. So, I figured why not use it in this case too? Well... before jumping into the deep end of the pool, I did what I felt was sufficient due diligence to verify that it would work. I initially did a generic Internet search that turned up nothing about block cloning and their software. I then searched the O&O website and their document repository without a single mention of ReFS much less block cloning. So, I sent out an email to O&O software asking if their defrag utility was acceptable for use with ReFS and in particular with block cloning. The email reply I got back from one of their senior technical support engineers was a wee bit confusing (ESL or Google Translate perhaps?... it's a German company.), but in general the email said to go ahead and use the "SPACE" defragmentation profile.... So... I did. (BTW, there are a bunch of profiles to pick from... defrag by name, accessed, modified, etc... each one with a different optimization algorithm).

Well... 45 hours later the drive finally finished its defragmentation. And... that's when I noticed what it did in the process. Apparently the O&O Defrag software knows how to deal with the ReFS file system in general terms... but NOT how to deal with Block Cloning within ReFS. During the defrag process and moving data around on the drive... it fully rehydrated all the backup files... essentially removing all the block cloning savings. So, now the data on that volume takes up nearly 6TB of space! 😢

As stated in the beginning... this isn't an earth shattering issue. More of an annoyance really than anything. In hindsight, I could have done a "trust but verify" trial run where I stood up a test environment and verified the results before throwing this into a production environment. At the time I didn't feel it was necessary because:

1. I've used their product hundreds of times before on other systems without a single issue.
2. O&O Defrag has been around for well over 2 decades, has won several awards, and was even certified by Microsoft for all of it's "current NTFS-based operating systems" (sooo... not for Windows 98 and earlier I guess???).
3. I had an email confirmation from one of their senior technical support engineers giving me the thumbs up to go ahead with using the product on an ReFS volume with block cloning.

In any case, I guess the ultimate moral of the story is "Trust but verify." If it's the first time you are going to attempt something, it may be worth your while to set up a test environment to try it out yourself before taking someone's word for it that it will actually work. In this case, I could have honestly lost all the data on this drive and not been any worse for the wear. It's just old backups after all. In the end, I still have all the data, just that I managed to loose all the block cloning benefits associated with them.

📌A Final Note: I am not writing this article to disparage or discredit O&O Software in any way. As I've stated, I've used their products hundreds of times in the past with much delight. I have also written to them about my experience with the ReFS block cloning issue and hopefully their development team is able to make adjustments to the software in the very near future. The main point of the article is to simply point out that you need to mind your P's and Q's when doing something new. This was a first for me using this software with ReFS. I fully intend to keep using the software, only now with a little bit more knowledge on what it can/can't do.

Note: Post migrated from old Blogger website.

The day has finally come where Prompt Critical Solutions is officially a 24/7 business. Up to this point, I've been juggling a day job at a bank alongside of servicing clients. While the dual income was nice... and needed to help buy valuable infrastructure to start the company without taking out any loans... it was also a massive workload and at times stressful. The last few years working as the IT Infrastructure & Systems Lead at the bank was also valuable to learn new skills, hone existing ones, and make new friends. But, I now step out into the unknown having never been my own boss before. It's a leap of faith for sure and it has both been a very liberating and scary experience. But, I now have the time I need to better service my existing clients as well as take on more clients to help those small businesses who can't afford to hire a dedicated IT professional to be on their payroll.

Note: Post migrated from old Blogger website.

The 3-2-1 rule for backups has been around for decades. It's a rather simple principle you can find documentation on almost anywhere backups are discussed. The rules are as follows:

• 3 copies of your data
• 2 media types
• 1 copy is off-site

3 Copies of Your Data

The principal is simple, you have 3 copies of your data (at a minimum) consisting of the following:

• 1 working copy (your production data)
• 1 local backup copy for fast backup & restoration tasks
• 1 off-site copy for disaster recovery

2 Media Types

This one is hotly debated to mean many things depending on who you talk to. On the extreme end, some will say that it means having something vastly different in storage like Hard Disk Drives (HDD) as one media type and Linear Tape-Open (LTO) as another. The thought being that if you have different "media types" that you would avoid a common mode of failure. Concerns of a bug or virus being able to exploit a weakness in a common operating system, file system, hard drive manufacturer, etc. By having completely different media types it's presumed that there would be no common mode of failure that could corrupt all 3 of your data copies.

Others will take a less extreme approach, but at the very least what everyone agrees is that your backups cannot share the same storage media with one another. I have personally witnessed people configuring backups to a second partition of the very hard drive that their production data resides on. This is a clear breach of the second rule by anyone's standards. So, at the bare minimum... your 3 copies should all be stored on 3 different "storage mediums."

1 Copy is Off-Site

This last one, being off-site, was initially intended for disaster recovery purposes. Basically protecting against things like the building burning down. Depending on how you get your data off-site, it may not be the latest and greatest copy of your data. You may experience some data loss compared to your production environment if for instance you only send backups off-site in the evening hours. But, the thought process is that it's better to have something over nothing.

The New Gold Standard

The 3-2-1 rule has been a great place to start and it's amazing how many people don't follow it. But the new sheriff in town is the 3-2-1-1-0 rule and it builds on the old standard and closes the gap on a new threat landscape and new technologies being used. The new standard is as follows:

• 3 copies of your data
• 2 media types
• 1 copy is off-site
• 1 copy is off-line
• Zero defects in your backups

The first 3 rules as you can see are the same. It's the last two that need some discussion.

1 Copy is Off-Line

The adage that "the only safe computer is one that's turned off" is more or less what this rule hints at. In the initial rule about one backup being off-site, depending on how you conducted your off-site backups, that may fulfil this rule too. In my best grandpa voice "Back in the old days, we had to backup all of our data to big tape drives and someone would drive them to the backup storage facility"... and there you would have fulfilled your "off-line backups" by using that particular off-site backup scheme.
Fast forward to today and high speed connections to almost anywhere in the world are available for not just big business, but small businesses and even individuals. Furthermore, these data connections are often times available as persistent connections via site-to-site VPNs, MPLS, SD-WAN, or some other site bridging technology. If you are trying to protect against a building burning to the ground, these persistent connections won't usually present any problem. However, with the threat landscape now including ransomware viruses... some of which are smart enough to seek out and destroy or disable your backups.. now those persistent connections become a real liability issue.

When Off-Line Isn't?

While in an ideal situation your backups would be truly "off-line"... aka "powered off"... the reality is that it just needs to be controlled in such a fashion that it is inaccessible from the other two backup systems. If for instance you send your data off to a cloud storage provider that you only have access to via API calls and only during the specified backup window, then for all intents those backups are "off-line" with respect to your production system even though the remote servers are still clearly up and running. That's not to say they are invulnerable though because there's the real possibility that the malicious attack could have some understanding of your backup system and initiate the control session with that remote station and corrupt your backups. In fact, even if you use a LTO system, it's possible that it could hijack the tape library system and corrupt your backups.
So, short of actually powering it down or pulling drives or tapes from a system and placing them on a shelf, how does one deal with this in a system that mandates as much automation as possible with as little human intervention as possible? Immutable Backups!

Immutable Backups

These are relatively new but help (can't stress "help" enough) keep up the automation of things while adding an extra layer of protection. With immutable backups, when data is written to the storage device a file lock is placed on it. The lock is similar to a time capsule that says "do not delete or modify until XYZ." Once that date is expired, the lock is lifted and the user or applications are free to do as they please (pruning or purging old backups). But, until that time passes, not even the highest privileged user on the system has rights to delete or modify the data. Ransomware will therefore be stopped cold in it's tracks from compromising your data right? WRONG! Well... mostly wrong. While it can't alter or delete the files, it remains a possibility for it to reach into the underlying subsystems of the host device and flat out destroy the drive partitions which contain the data. It's not much in the way of a ransom at that point... but still a noteworthy issue for consideration (and bear in mind that the disgruntled employee is just as much a threat as a virus).
The point being, plan appropriately to ensure the most likely of attack vectors in your environment are covered. Having an always on connection between all of your backups is the worst case scenario with the highest risk for something like a ransomware attack being able to destroy everything. If you can't achieve an ideal true "off-line" status with one of your data copies, there are a multitude of ways to achieve a pseudo-off-line status to enable automation while simultaneously mitigating some of the more prevalent threat vectors.

Zero Defects In Your Backups

There's an old saying that goes "trust but verify." This final rule is basically an embodiment of that statement. You go through great efforts to purchase necessary hardware, software, planning, etc. But, it's all for not if they day you need to use them you find out that they don't work. Depending on where you look you'll find statistics that will say all kinds of shocking numbers. The short version of them all is simply that an amazing number of people never bother to test if they can actually recover from their backups. Going hand in hand with that will be the crazy numbers of backups that simply fail to work when they finally are called to do the very service for which so much time, money, and effort was spent.
Sometimes these failures are human error such as only backing up the data partition of a server and forgetting to also backup the boot partition so that a server can be restored. Other times it can be as innocent as the underlying storage media undergoing "bit-rot" and rendering an otherwise good backup useless. Regardless of the failure mechanism, if your backups don't work in your moment of need, you've just purchased a ticket to be a part of another grueling statistic... the number of companies that close their doors in X years after a significant data loss event (again, statistics vary but are non the less dismal).
How you go about verification can take on several methods. One is to do a complete read of your data and do a CRC or hash check. Others will add on a partial or complete restore of your data. Some will go even further and not only restore the data, but conduct a series of battery tests in an isolated lab environment. As an example, Veeam Backup & Replication has a feature called SureBackup which can do this type of testing as an automated scheduled task. But, regardless of what you use, the underlying point is simply trust but verify!

Summary

While the 3-2-1 rule has served as the gold standard of backups in the IT community for many years, there are inherent shortcomings in the limited list of requirements it provides. Solving some of those problems is possible through the implantation of the 3-2-1-1-0 rule. But, bear in mind that although it is titled as a "rule" it is really more of a "guideline" and simply following it is not a get out of jail free card. In fact, no plan, no matter how elaborate or well executed, will ever be 100% perfect. Any backup plan will require a balance of the likely risks weighed against the cost and complexity. The 3-2-1-1-0 rule simply establishes a foundational set of questions you need to answer during your planning and ongoing implementation of any given solution.

Note: Post migrated from old Blogger website.