-15 deg affecting all devices/connections or am i chasing a ghost?

[graemlourens]

Hi everybody.

I have successfully been remote & unattended imaging for the past few months now, what a delight.

Now we're getting around -15 deg every night, and since then, every single piece of software is getting an error somewhere during the night.

It started with the Atik EFW2 not wanting to report back to SGP, then Timeouts during Framedownload with the 460EX.

Next EQMOD started getting timeouts during Read-COM-Errors, and finally last night the AAG Weatherstation software reports that the connection is lost (device still powered). This morning the ScopeDome Driver Software is hanging (and that to date was my most reliable piece of software that never died before).

For me this smelt like the USB-Connections are failing.

But after a restart of the computer, all devices & software work flawlessly for around 10 Minutes, then the problems start again - what would demolish the theory of the USB-Cables beeing affected.

The next idea i have is that the Memory of the computer is having random failures - i will be testing this today when i visit the observatory as well as totally switching to the backup-computer and see if this one has the same problems (what will get me a great step further towards finding the source of these errors)

In general i would like to ask anybody if you have experienced these kind of seemingly random errors in connections, and what your experiences are with far below freezing temperatures and your connection/equipment. Maybe you have an idea that i can tester later today when i drive to the obsy.

Kind regards, Graem

[ollypenrice]

We don't get to be quite so cold other than very occasionally. We fairly often see -9C, sometimes -12C and occasionally -19C. I've always been surprized by the fact that the kit seems unaffected by this. So while the cold might be your problem I'd keep an open mind.

There are various ways of keeping things a bit warmer. Obviously you can put PCs in insulated boxes and put dew heaters around things like the focus motor and filterwheel. You can also buy small heater pads for keeping puppies and other small animals warm! I have two of these gadgets, and since puppies have accidents, they are designed to work when wet. 

http://www.ebay.co.uk/bhp/puppy-heat-pad Mine are stainless steel and have been reliable for several years.

Olly

[graemlourens]

Hi Olly.

Thx for your thoughts.

I think i'll be hunting this problem for a while just when i thought i had everything settled... this hobby never gets booring does it!

The thing is i don't think its any of the real hardware, Mount is tracking perfectly, focuser works and filterwheel is actually moving when i tell it to move.

Its more that the signals are not reaching the computer anymore. And as its all software, i really think its the computer as single common source of all the devices...

Just ran a remote memory test without any errors, i was hoping it was this.

Only thing left it to put on my warm clothes and give the obsy a visit today, changing the computer (got a fully setup spare always at hand) and seeing what happens.

Kind regards, Graem

[opticalpath]

I think that some electronic components, such as electrolytic capacitors, can be affected by low temps., but -15 does not sound too extreme.  

Is it possible that you are getting increased problems with condensation?  That could be a problem for hard drives and maybe for connectors - just a guess.  Do you monitor humidity in the observatory, and have you got some means of drying out remotely after a session? 

Adrian

[graemlourens]

Hi Adrian.

The Humidity while imaging is currently not exceeding 80%, and in the dome does not form condensation of the equipment as far as i have been able to observed (went there twice in very cold nights to check). During the day its around 50-60% due to the dry cold air. The dehumidifier is not running now, as its not necessary until humidity reaches over 80% in my opinion.

After the imaging session the dome closes and a small heater turns on for 4 hours to raise the temperate (and therefore lower the humidity back to around 50-60%).

I will today check though if i maybe have humidity inside the computer, or on the usb connectors, this could lead to the problems i have been seeing too.

I found out that i can remotely run a memorytest with automatic restart, this is currently on its way and will see if memory problems are seen.

Heading out to the obsy in an hour.

Kind regards, Graem

[opticalpath]

Sounds like humidity is well under control and unlikely to be a factor, but perhaps still worth checking connectors for any visible signs of deterioration.

Are you using something like MemTest86 to test memory (better than the Windows built-in tool) ?  Maybe also worth removing and re-seating the memory sticks.

Adrian

[graemlourens]

Hi Adrian.

Just created a bootable MemTest86 stick and in the car now heading obsy Got the tip from a fried this morning to test memory like that. The normal windows test did not reveal any problems.

I am currently simulating an imaging session with around 0 degrees in the obsy (heater running) to see if this makes the difference.

Actually first time in my life hoping for clouds tonight so i don't feel bad running tests instead of imaging

[andrew s]

Graem, I had a similar problem some years back. In my case it was memory boards that lost connection intermittently as  the temperature dropped. If it is a desk top PC I would advice removing all the slot in boards cleaning the contacts and re-seating them. Make sure you are earthed to avoid static which can be an issue if cold and dry. I would also do the same with any external connections USB, serial, Ethernet etc.

Regards Andrew s

[graemlourens]

Hi Andrew.

Interesting. Do you mean with 'cleaning' air preassure cleaning, or some washing procedure?.

Kind regards, Graem

[Davey-T]

Just a thought Graem, what is supplying your power, one thing that is susceptible to low temp's is power supplies.

Dave

[andrew s]

Hi Andrew.

Interesting. Do you mean with 'cleaning' air preassure cleaning, or some washing procedure?.

Kind regards, Graem

Just standard spray on electrical contact cleaner. As an example http://www.maplin.co.uk/p/servisol-super-10-switch-contact-cleaner-200ml-jp17t there are many others.

I recently had a very old CCD camera that used a special 10 pin RJ45 type connector and by spraying some in the can lid and using a cotton bud to apply it I manage to clean both the plug and socket so it now works reliably again.

Regards Andrew

[alan potts]

Not as advanced as you equipment but I have never had my Meade SC goto scope (7years) or Dell laptop fail in temperatures down to minus 20 and lower, though this is not everyday you understand.

Could it be power supply related? 

Alan

[Physopto]

The tip about contacts on the motherboard seem relevant. Some years ago I had a sort of opposite problem in a Physics research Lab. The heat in the lab was giving an intermittent fault due to expansion. I eventually tracked it down to a faulty board socket. I sat for hours holding each plugin board one by one until I found the culprit. A replacement socket solved that problem as time was not on our sides and the boards were especially made for the experimental equipment.

A pc is a different kettle of fish as it will mean replacing the motherboard if the socket cannot be cleaned. Then of course it means re-validating your software with Microsoft as a replacement is seen as a new computer.

Hope you can find a simple answer.

Derek

[graemlourens]

Thx guys for your answers & thoughts.

I tried today various things for 3 hours without success. Timeouts everywhere, from filterwheels to cameras to whatever is running, just doesn't want to run.

I connected the backup computer that slightly varies in software & hardware, and i am up and imaging now since an hour....

Now i have the night to think about what could cause this.

As the backup computer works perfectly fine, it can not be the USB cables or power supply itsself, it has to be either the computer or some software on it (even if pretty much everything is the same drivers & so on)

I'll write here when i have found the problem!

[ian_bird]

There are various ways of keeping things a bit warmer. Obviously you can put PCs in insulated boxes and put dew heaters around things like the focus motor and filterwheel. You can also buy small heater pads for keeping puppies and other small animals warm! I have two of these gadgets, and since puppies have accidents, they are designed to work when wet. 

http://www.ebay.co.uk/bhp/puppy-heat-pad Mine are stainless steel and have been reliable for several years.

Olly

Hi Olly

Stainless Steel? Really? Are they intended for Stainless Steel puppies?

[Horwig]

My obsy Pc refused to boot because of the seating of the ram simms. It was never as cold as -15, but damp is my problem here.

Since reseating the memory, it's on 24/7 and seems to be working (famous last words)

Huw

[graemlourens]

Hi Huw.

Thank you for your reply.

I would have hoped it to only be the ram, then at least i know what it is but testing in various manners (without windows running) revealed no problems, also window internal logs does not show any problems with ram.

Currently the temperatures have risen again, and i will go the obsy today to do a rater stupid test, but hopefully efficient. I'll just put back the original computer without doing anything to it, and see if it really was merely the temperature.

If not, then i know that this must be software / driver related.

This is like searching for a needle in a haystack, but i'm at least lucky i got the backupcomputer that is not letting me down.

Kind regards, Graem

[opticalpath]

By coincidence, I have just had some unusual intermittent problems (affecting boot-up) with my desktop PC at home (indoors).  A Google search suggested that a RAM problem could cause it.   All tests on the memory showed no faults.  However, after pulling the four memory cards out and reseating them firmly in their sockets, the problems have gone.  If you haven't tried this already .... worth doing to see if it helps.

Adrian

[graemlourens]

Hey guys, a quick update:

My Backupcomputer ran perfectly for 9 hours straight imaging at -13 deg with 4x target switching (i was mosaicing (is this in the dictionary? )) without me changing anything to any other part of the obsy.

So this has ruled out anything to do with powersupply or anyhing besides the computer itsself (software or hardware)

My next test was to see if this whole thing is temperature related or software related.

I heated the obsy today since the morning gently to a nice 2 degrees, and have just switched out the computers.

I have been simulating imaging now for over 2 hours with the 'faulty' computer, and everything is ok. so this leads me to believe that it is not software related, but really something with the computer hardware itsself.

My next test tonight will be do shut down the heaters and let the temperature drop back down (sadly now we're at -5 tonight, not -15 so the test will not be conclusive) to see if i can trigger these faults again.

@Davey, sry somehow skipped your answer. As the backupcomputer ran perfectly at the same temperature, i can rule out power supply problems for now.

@Adrian. This is very promising and could well be worth trying. I am for now testing various things step by step or else i'll have no idea what fixed the problem, but will for sure try this if nothing else helps!

Kind regards, Graem

[Oddsocks]

Graem,

May not be relevant to your problem but prior to retirement I used to work on integrated computer systems every day and I saw lots of thermally induced intermittent problems on processor sockets of the intel processors using the sockets LGA1150 and LGA1151 (series i5 & i7).

Although we did not go below freezing our systems frequently cycled between 5 deg and 50 deg every day with the processors themselves reaching 65 degrees at times.

The windows logs never picked up anything, the systems either crashed or hung and the boot hardware diagnostic tools we ran rarely ever found intermittent faults, only hard permanent faults.

In most cases, just removing the processor from the socket and refitting it, adding a smear of fresh heat sink compound, seemed to fix a lot of faults.

As other have stated by far the biggest problem is with the memory module sockets and removing the modules, wiping the contacts with clean dry paper and refitting them cures most evils.

William.

[perfrej]

Hi folks!

Just a couple of quick notes for your pleasure that aid in understanding what is going on

Temperature

Active electronic components are screened after manufacture and sorted according to behavior. They are then sold as different operating ranges:

• Commercial: 0 - 85°C
• Industrial: -40 - 100°C
• Automotive (or Extended): -40 - 125°C
• Military: -55 - 125°C

Anything we get our hands on is usually made from commercial grade chips. Extreme requirements, like missiles and other aerospace stuff, often use devices that are military grade and tested to MIL-STD-883C. That test procedure involves subjecting parts to 1500 G shocks and actually pulling the bond wires... I have a few old chips of that type in my stash On a side note, you'd be surprised how little it takes to generate 1500 G. The part is mounted on a 10 cm thick steel plate that is then cause to fall perfectly flat on another such steel plate (they have groves to let air out at impact). The height it is dropped from? 72 cm! I witnessed this test on a visit to Israel in 1983.

Corrosion and oxidation

This is another enemy. Any connector will corrode or get smudged. It will start to show its ugly face at connectors with high currents and connectors with high frequency signals - like power supplies and your computer or CCD camera.

Mitigation?

• Never turn your outdoor computer off.
• Allow air in the enclosure from the bottom, but not too much.
• Avoid contact spray (it helps in the short run but leaves too much residue).
• For permanently connected connectors, use connector gel.
• Modify fans to blow at half speed (with a switch so that you can disable that feature in the summer).
• Use high quality (expensive) connectors.
• Measure ground loop currents and eliminate them. When a potential difference takes the path through your signal cable you have problems.
• Replace USB cables regularly and prefer mini USB over standard.

The last point requires some elaboration. A "full quality" USB standard  connector has a rating of 1,500 connect/disconnect cycles. The cheap stuff probably won't even be near that figure. Mini connectors are rated at 5,000 cycles, and micro connectors at 10,000 or 15,000. So go mini where you can (haven't seen a micro one yet in astro).

There is more, but I'll stop there. I have designed outdoor systems since the late 70's and it actually works if you pay attention to the little details. Maybe I should write a book on this...

All the best, and let's make 2016 the best astro year to date!

/per

[graemlourens]

Very interesting, i do indeed have an i5 running in that computer.

Yesterday when the obsy started cooling down, it didn't take more than 10 minutes for the first problems to show up.

Today i'll be refitting the Ram & running the same test again, and will see what happens.

But i'm playing with the idea of just keeping the backupcomputer running and selling the other one, i've started to have enough

Kind regards, Graem

[opticalpath]

............

Mitigation?

• Never turn your outdoor computer off.
• Allow air in the enclosure from the bottom, but not too much.
• Avoid contact spray (it helps in the short run but leaves too much residue).
• For permanently connected connectors, use connector gel.
• Modify fans to blow at half speed (with a switch so that you can disable that feature in the summer).
• Use high quality (expensive) connectors.
• Measure ground loop currents and eliminate them. When a potential difference takes the path through your signal cable you have problems.
• Replace USB cables regularly and prefer mini USB over standard.

Interesting stuff here, Per.  A couple of questions (with apologies to Graem for going off-topic):

Connector gel: Is that some kind of lubricant, or did you mean the stuff that's used as a sealant in outdoor connector housings?

How do you check for/ measure/ eliminate ground loop currents? For example, I have a mount, cameras, laptop, focuser, dew-heater strip .... running from an array of separate DC power supplies, which in turn are all connected to the same mains AC supply at the end of a long cable coming from the house fusebox;  USB connections from laptop to cameras, mount, focuser. No (intentional) earth connection in the observatory, only back at the house fusebox. That's probably a fairly common set up - where to start? 

Adrian

[gajjer]

Hi Graem

My concern would be with the connectors too. At -20C I wouldn't have thought you would have any problems with devices. Especially since your system seems to be running. However, condensation could hit you anywhere - device connections, board connections, cables etc.

In automotive and aerospace the boards are coated - I doubt that many of those in your system are. Similarly connectors have some degree of protection.

If you are starting up from cold it could be frost melting as equipment warms up.

You may need to provide an environment where the equipment can be either kept warm and dry or can be warmed up in advance of a session.

Just my thoughts.

I am envious that you have nights that are clear!

cheers

gaj

[graemlourens]

Thx Gaj for your answer.

Today i'm heading out again, and trying various things, also reseating the memory.

This will be the last hardwaretest before i reinstall the whole computer from scratch with identically the same software as the backupcomputer.

Maybe some of you are thinking why the heck i just don't keep the backupcomputer running and thats it - the problem is that that PC has got a problem in summer to keep its 'cool' and overheats, thats why i replaced it in the first place.

Kind regards, Graem