I'm not sure what it is, but this is connected with modules/content or changes done to the template. On Your website there is no sidebar at all. Code is displaying a big <a></a> element instead of sidebar. My first guess will be some HTML validation error, so to exclude this possibility try unpublish all modules or create a page without anything and than put something to sidebar just to check if it's still broken.
If it will be still broke I would search for wrong code in <head></head> section. Probably some unclised tag during adding extra files or a broken 3rd party extension which is providing a bad code.
Third possibility is an old version of PHP. Acctually this is the fastest thing to check by updating it and will be good for site performance, so start with that.
I don't know what you did, but know I unpublish modules and they still appearing on the website. Also the HOME link is not loading on the site, never seen something like this before.
How come the sidebar works when the users are logged in, and doesn't work when the users access the site as visitors? In the K2 item pages, the CSS doesn't loads when a not logged in user reads a K2 item. What could be doing this?
It's really hard to tell. I suspect that this is caused by an error in a module or content from an article. This can be checked pretty easy by unpublishing extension after extension till sidebar starts to work properly. When it does than the last unpublished will need to checked closer for it's html code.