HelioHost Posted July 23, 2017 Posted July 23, 2017 I have an account on HelioHost and I created a cron job.It's a fairly long job (on my computer it runs for 1.5h-2h) but it doesn't eat much resources (on my 2-core 2GHz, 4GB RAM computer htop shows that the job takes 6% of CPU and 1% of RAM). I made it run every 12 hours not to exceed the limit.But the job is randomly killed, sometimes after 0.5min, sometimes after 4min, sometimes after a different period.The main command generates a SQLite DB file, which after killing is almost empty and there is the db-journal file left (which means that the transaction didn't succeed nor rolled-back, just died). I redirect stdout and stderr of this command to file and that log contains no errors, and debug log is just cut. Other commands (compressing, moving the file, etc.) should redirect their output to email (as Your cron job page states) but I didn't get any email (I checked spam as well).Is there any limit on execution time, or is that some other problem?--Qeebmisk
HelioHost Posted July 24, 2017 Author Posted July 24, 2017 My username is qeebmisk > -------- Original Message --------> Subject: Re: [Hh#95780] Killing Cron Jobs> Local Time: July 23, 2017 4:22 PM> UTC Time: July 23, 2017 2:22 PM> From: support@heliohost.org> To: qEebMisK@protonmail.com> What is your username or main domain?> You may view the status of your ticket by visiting:> https://www.helionet.org/index/index.php?showtopic=28967> Thank you,> Heliohost support> https://www.heliohost.org/> https://www.helionet.org/
Krydos Posted July 24, 2017 Posted July 24, 2017 It looks like you caused high load and the infinite loop protection kicked in. Why does your script take several hours to complete?
HelioHost Posted July 24, 2017 Author Posted July 24, 2017 It generates my city's public transport timetable, and as it's not behind some reasonable API and my city is completely OK with scraping their websites the script makes al lot of requests to their website (every stop for every line is separate) and probably parsing the HTMLs also takes some time.The thing is it finished successfully exactly one time the previous week. The resulting DB file was corrupted, though, so maybe singular commands were interrupted.I'm actually sorry for unintentionally making a protection mechanism kick in. It sounds so serious and I don't want to cause damage to Your servers. > -------- Original Message --------> Subject: Re: [Hh#95780] Killing Cron Jobs> Local Time: July 24, 2017 3:01 PM> UTC Time: July 24, 2017 1:01 PM> From: support@heliohost.org> To: qEebMisK@protonmail.com> It looks like you caused high load and the infinite loop protection kicked in. Why does your script take several hours to complete?> You may view the status of your ticket by visiting:> https://www.helionet.org/index/index.php?showtopic=28967> Thank you,> Heliohost support> https://www.heliohost.org/> https://www.helionet.org/
Krydos Posted July 24, 2017 Posted July 24, 2017 I whitelisted your account to run long scripts again. We'll see how high the load gets. Another option that you could look in to is instead of one long several hour long script (which will always have a tendency to break anyways) you could find a way to break it down into smaller pieces. For instance, say you need to scrape 100 urls per day you could keep the current page saved in a database, and each time the script is called it checks the database to see what it needs to do next, scrapes the one page, increments the database, and then quits. If you made the script executable from a web browser (like in cgi-bin directory) I can set up an external cron job to execute the script as often as every 5 minutes. External cron jobs aren't limited to 2 per day like internal cron jobs are. This way also if one portion got corrupted it would only affect the small section instead of messing the entire script up.
HelioHost Posted July 24, 2017 Author Posted July 24, 2017 Thank You a lot.I've planned to create an incremental version of the script, but firstly I created something that at least works. Now I'm finishing another--urgent--assignment and I'll turn my attention to this. > -------- Original Message --------> Subject: Re: [Hh#95780] Killing Cron Jobs> Local Time: July 24, 2017 4:10 PM> UTC Time: July 24, 2017 2:10 PM> From: support@heliohost.org> To: qEebMisK@protonmail.com> I whitelisted your account to run long scripts again. We"ll see how high the load gets. Another option that you could look in to is instead of one long several hour long script (which will always have a tendency to break anyways) you could find a way to break it down into smaller pieces. For instance you say you need to scrape 100 urls per day you could keep the current page saved in a database, and each time the script is called it checks the database to see what it needs to do next, scrapes the one page, increments the database, and then quits. If you made the script executable from a web browser (like in cgi-bin directory) I can set up an external cron job to execute the script as often as every 5 minutes. External cron jobs aren"t limited to 2 per day like internal cron jobs are. This way also if one portion got corrupted it would only affect the small section instead of messing the entire script up.> You may view the status of your ticket by visiting:> https://www.helionet.org/index/index.php?showtopic=28967> Thank you,> Heliohost support> https://www.heliohost.org/> https://www.helionet.org/
Piotr GRD Posted July 24, 2017 Posted July 24, 2017 With "so long scripts" you would have a problem even on a paid shared webhosts, maybe not every time, but quite often. Make it short and incremental, it's not hard if you're familiar with coding. For external cronjobs you can even use some free services.A good reliable example that I use is Cronjob.de - it works very well with no problems for many years for me, you just need to login once every 8 weeks to keep the free version running.
Ableson Posted August 1, 2017 Posted August 1, 2017 WebCron is a great method of automating tasks on your server if you’re not blessed with cron jobs, you can be found a list of free webcron services at: http://www.cronjobservices.com.
Recommended Posts