You don’t need integration to define expected value or covariance in the discrete case. TBH I’m not sure if you can get around integration in the general continuous case or not.
If not, you could use some limiting argument to handle the moments of a continuous uniform RV, at least, in terms of the discrete analog.
You don’t need calculus to derive least squares estimators. You can follow the logic in this quora answer [1] to show that (e.g.) the mean is the minimum MSE estimator among constant functions, and that the conditional mean is the minimum MSE estimator among “general” (measurable L2) functions.
This derivation is familiar to many who have studied these concepts. It’s clever, it does not need differentiation, just expectation and logic.
It could be that your studies in probability were done using a certain pedagogical path, and that’s blinding you to the fact that other paths are possible.
[1] https://www.quora.com/Why-is-minimum-mean-square-error-estim...