Backyard gardens are generally unobjectionable, it is the front yard that gets neighbors riled up when they see dirt and messy looking plants. Even then, if you make an effort to have it present as ornamental (eg. herbs, cherry tomatoes, mini peppers, etc. in attractive raised beds and containers that don't displace the entire lawn) you may be able to get away with it.
Oh, and definitely keep the compost pile out of sight.