Here’s a thought experiment for you (h/t to Dan Meyer for the sports analogy).
Imagine you start learning the game of basketball by learning how to shoot free throws. At no point are you told what the point of shooting free throws is, or how being a good free throw shooter will help you play the game of basketball, or even that there is a game of basketball. Worse, you are occasionally asked to shoot as many free throws you can in a minute, and then judged against your classmates based on your performance.
You are at some point asked to start practicing all of your free throws blindfolded, possibly after unsuccessfully learning how to shoot free throws earlier. If you are lucky, your coach tells you how many free throws you sank, and how many you missed. You finally have some understanding that free throws are important in basketball after years of not having a clue why you were practicing them but no matter how many times you practice, you never seem to get better at free throws.
By way of analogy, this is almost exactly how addition and multiplication facts are taught to students. They spend the earliest years learning addition and multiplication facts with only a superficial explanation of how these facts might be useful later, and most do not learn how addition and multiplication fit into mathematics as a whole and they certainly never get to experience "the game."
In their later grades, their teacher (although lacking the time to give students feedback on their "basic skills") expects students to work on their higher level mathematics without a calculator or any aid of any kind for their foundational numeracy skills. The premise behind these calculator-less classrooms is that students will likely forget their addition and multiplication facts if they get to use a calculator, and so the use of a calculator is banned. Unfortunately, in most of these classes, very little time is spent reteaching addition and/or multiplication facts, and almost no feedback is given to students as to whether they have even done their addition and/or multiplication correctly, so if you are a student who never understood addition or multiplication in the first place, this further practice without support is unlikely to be useful.
If you are going to ban students from using calculators in your class for basic arithmetic operations, then you must at least take the blindfolds off of your students and help them improve their arithmetic skills. On the other hand, I prefer not to ban tools, but instead find ways that these tools are used productively (and unproductively) and change my teaching to compensate.