Advanced search

Message boards : News : New beta app for analyzing Planck data

1 · 2 · Next
Author Message
Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20758 - Posted: 31 Jan 2016, 17:34:04 UTC

Today we are releasing a new app (in beta for the moment) to help analyze Planck data. Read more about it in the comments. Start crunching this app now for a chance to receive thanks in the acknowledgements of our forthcoming paper!

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20759 - Posted: 31 Jan 2016, 17:34:17 UTC - in response to Message 20758.
Last modified: 31 Jan 2016, 17:42:08 UTC

Part of the promise of all the updates, Virtualbox, the boinc2docker jobs, etc... was to make it easier for us to deploy apps, so we could go from "cool science idea" to "functioning Cosmology@Home app" quickly and make the results that you guys are calculating as relevant as possible for science going on at this very moment. Today this starts to become reality as we are releasing a new app called "planck_param_sim". We will give more details in a post when it comes out of beta (hopefully very soon). For now though a brief summary,


  • The science: The goal of C@H is "to find the model of the universe which best agrees with cosmological data." Right now in cosmology we have a standard model, which we call LCDM, that seems to explain all of our data fairly well. The CAMB jobs you are running have been scanning through the parameters of this model. These new jobs are going to help test the consistency of the model itself. To do so, each job simulates a universe, simulates what observing this universe with the Planck satellite would look like, and simulates an analysis of this (simulated) data (can you tell we love simulations...). This lets us build up a distribution of what results look like when we analyze the data in certain different ways, which we can compare to how our actual (non-simulated) Planck results looks like. If they look pretty much the same, LCDM passes another precision test. If they don't it means we've found something new and interesting, the most exciting of which is that we've seen hints of new physics!

  • Job specs: its impossible to determine the exact length of any particular job ahead of time. We've set the progress bar so that most jobs will finish somewhere between when it is at 20% and 90%. We're also giving fixed credit per job, based on the average job length. This is 50 credits per job.

  • We will be using these results directly in a forthcoming paper from the Planck collaboration. To give you extra incentive to run these jobs we'll give thanks in the acknowledgments of the paper to the top 3 users and the top team according to credits gained on this app (we'll give a two week warning before picking the winners). Credits gained during the beta will count, so go ahead and get started now!


Thanks, and don't hesitate to give us any feedback below!

Crystal Pellet
Send message
Joined: 12 Feb 13
Posts: 21
Credit: 351,882
RAC: 139
Message 20761 - Posted: 31 Jan 2016, 18:29:33 UTC - in response to Message 20759.

Thanks, and don't hesitate to give us any feedback below!

First task - validate error http://www.cosmologyathome.org/result.php?resultid=36244200

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20762 - Posted: 31 Jan 2016, 18:57:55 UTC - in response to Message 20761.

Thanks, and don't hesitate to give us any feedback below!

First task - validate error http://www.cosmologyathome.org/result.php?resultid=36244200

Issue should be fixed now. I'll go back and revalidate your first couple of jobs that I see were marked invalid.

Crystal Pellet
Send message
Joined: 12 Feb 13
Posts: 21
Credit: 351,882
RAC: 139
Message 20763 - Posted: 31 Jan 2016, 21:08:29 UTC - in response to Message 20762.

Issue should be fixed now. I'll go back and revalidate your first couple of jobs that I see were marked invalid.

I returned 3 tasks so far. 2 were invalid and 1 was valid.
This valid one turned into invalid too now.
I'll hold back the 'In Progress' tasks waiting for a better fix.

Crystal Pellet
Send message
Joined: 12 Feb 13
Posts: 21
Credit: 351,882
RAC: 139
Message 20764 - Posted: 31 Jan 2016, 21:47:51 UTC - in response to Message 20763.

I returned 3 tasks so far. 2 were invalid and 1 was valid.
This valid one turned into invalid too now.
I'll hold back the 'In Progress' tasks waiting for a better fix.

Ooh that's fast. Now all three invalids turned to Completed and validated.
I'll start another task.

Profile robertmiles
Send message
Joined: 26 Oct 11
Posts: 47
Credit: 262,559
RAC: 359
Message 20766 - Posted: 1 Feb 2016, 1:36:00 UTC

Could you mention the approximate amount of memory this new app requires, relative to the previous app, so users who have had to set No new tasks due to the memory demands of the previous app can decide whether to enable ONLY the new app?

Henrik
Send message
Joined: 7 Dec 15
Posts: 7
Credit: 164,248
RAC: 0
Message 20770 - Posted: 1 Feb 2016, 11:26:33 UTC

How can I get the new Jobs? It only gives me boinc2docker jobs when I update.

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20773 - Posted: 1 Feb 2016, 12:05:45 UTC - in response to Message 20766.

Could you mention the approximate amount of memory this new app requires, relative to the previous app, so users who have had to set No new tasks due to the memory demands of the previous app can decide whether to enable ONLY the new app?

Thanks, yea, this is currently listed as having the same memory requirement as camb_boinc2docker, which is 2GB. I left it the same not realizing anyone was holding back due to memory requirements. But in truth it only uses between 500-1000 MB, so I will go ahead and lower it.

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20774 - Posted: 1 Feb 2016, 12:11:39 UTC - in response to Message 20770.

How can I get the new Jobs? It only gives me boinc2docker jobs when I update.

To those wishing but unable to get new jobs, the checklist is,

  • Have a 64bit OS
  • Ensure VM extensions are enabled (and possibly check that you're not affected by this bug.)
  • Go to your C@H preferences and make sure that "Run test applications" is checked and that "planck_param_sim" is checked under "Run only the selected applications"

Henrik
Send message
Joined: 7 Dec 15
Posts: 7
Credit: 164,248
RAC: 0
Message 20782 - Posted: 1 Feb 2016, 13:38:08 UTC - in response to Message 20774.

Hi and thanks for the reply. Got some jobs now and wanted to run them but after a minute or so it just ends with a computation error. You need any info for troubleshooting? http://www.cosmologyathome.org/result.php?resultid=36258867

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20783 - Posted: 1 Feb 2016, 13:47:44 UTC - in response to Message 20782.

Hi and thanks for the reply. Got some jobs now and wanted to run them but after a minute or so it just ends with a computation error. You need any info for troubleshooting? http://www.cosmologyathome.org/result.php?resultid=36258867

Yea, you're not the only one, I'm seeing an unusually high number of these "run for ~1min, computation error" jobs from everyone. Maybe like 1/3 of them. For some people it seems to just work after a while, for others it never does. Yours for example give back no log, which is extra confusing. For others it looks to be related to failures pulling the Docker images from inside the job. Looking into it...

Henrik
Send message
Joined: 7 Dec 15
Posts: 7
Credit: 164,248
RAC: 0
Message 20784 - Posted: 1 Feb 2016, 14:01:40 UTC - in response to Message 20783.

Alright. After removing the project and adding it again I managed to get past a minute without any errors but then after ten minutes I got this error:

http://i.imgur.com/n3TUtjA.png

What also makes me wonder is why my CPU usage is always at 0% or 1% but the status on the projekt is "Running" and the progressbar is also filling. Let me know if you need any further information.

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20785 - Posted: 1 Feb 2016, 14:08:15 UTC - in response to Message 20784.

Ah, for that error there's some discussion in this thread this thread. Downgrading to Virtualbox 4.X might be a solution. Otherwise we actually also just made an update aimed at giving us better logs to figuring out what this error is, so I'll be curious to look at yours from this job.

Henrik
Send message
Joined: 7 Dec 15
Posts: 7
Credit: 164,248
RAC: 0
Message 20787 - Posted: 1 Feb 2016, 14:23:00 UTC - in response to Message 20785.

I tried the latest VBox installation and now CPU usage is actually back at 100% if I start a task, thats good. I'll let it run for half an hour now and report back if I run into any errors.

Jesse Viviano
Send message
Joined: 29 Nov 14
Posts: 26
Credit: 310,634
RAC: 470
Message 20792 - Posted: 1 Feb 2016, 15:48:50 UTC

Validate errors still are happening. I have noticed that two results are being generated for each planck_param_sims (beta test) work unit. I also noticed that when the second result is computed and then reported, it always is a validate error (possibly unless the first result is declared invalid). A possible stopgap solution could be to program either the work unit generator or the result generator to generate only one result per work unit. The long term solution would be to program the validator to be able to validate all non-tardy results despite the fact that some other result was reported first and tardy results that are reported before their replacement work units are reported.

Profile Steve Hawker*
Send message
Joined: 10 Feb 13
Posts: 9
Credit: 244,929
RAC: 3
Message 20797 - Posted: 1 Feb 2016, 22:54:34 UTC

All tasks fail on my Linux box with "process exited with code 1"

This suggests some library problem I think. I'll try on OSX later.

S.

Profile Marius
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 29 Jun 15
Posts: 416
Credit: 4,276
RAC: 0
Message 20798 - Posted: 1 Feb 2016, 23:06:28 UTC - in response to Message 20797.

All tasks fail on my Linux box with "process exited with code 1"

This suggests some library problem I think. I'll try on OSX later.

S.

Thanks, yea there's a ton of these errors unfortunately and I'm digging into it. Definitely not a library thing though, these jobs are running inside Docker inside a VM so they don't interact in any way with what you actually have installed on your machine. The calculation is getting killed by SIGILL, I'm working on getting the right logs out of jobs to figure out what's going on.

Validate errors still are happening

Yea, a few of these sorry about that. I will go back and re-validate them. You pretty much debugged the problem on our end exactly.

Profile DoctorNow
Volunteer tester
Avatar
Send message
Joined: 22 May 07
Posts: 22
Credit: 193,286
RAC: 0
Message 20799 - Posted: 2 Feb 2016, 9:11:25 UTC

Well, looks like I don't get the new app working on one of my hosts - an AMD Phenom II X6 1045 with Windows Vista.
Installed is BOINC 7.6.22 and VirtualBox 5.0.14 now. I tried different VB-versions from 4.3.20 up to the one now, but it doesn't look as if it can work correctly - CPU is seemingly using only one core when I look into the task manager and the percentage won't go over 0,1%, already running over an hour now.

I cannot test it on my other machine (AMD FX 8320 with Win 7) currently, BOINC manager says there are no tasks available, yet the server status says something else... ?
____________
Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@Heidelberg
My BOINC-Stats

Henrik
Send message
Joined: 7 Dec 15
Posts: 7
Credit: 164,248
RAC: 0
Message 20800 - Posted: 2 Feb 2016, 9:50:06 UTC - in response to Message 20799.

Did you try to remove the C@H Project in your Manager and then readding it, only allowing Plank tasks? That fixed it for me, my Xeon was crunching numbers all night, maybe Marius can check some of my tasks if they are really valid. They finished without error in ~6 minutes. Cheers

1 · 2 · Next

Message boards : News : New beta app for analyzing Planck data