Reddit mentions of Probability Theory: The Logic of Science

• Probability Theory: The Logic of Science
  
• Bayesian Spectrum Analysis and Parameter Estimation
  
• Maximum Entropy in Action
  
• Data Analysis--A Bayesian Tutorial

Of course efforts like this won't fly because there will be people who sincerely want to can them because it's "computerized racial profiling," completely missing the point that, if race does correlate with criminal behavior, you will see that conclusion from an unbiased system. What an unbiased system will also do is not overweight the extent to which race is a factor in the analysis.

Of course, the legitimate concern some have is about the construction of prior probabilities for these kinds of systems, and there seems to be a great deal of skepticism about the possibility of unbiased priors. But over the last decade or two, the means of constructing unbiased priors have become rather well understood, and form the central subject matter of Part II of E.T. Jaynes' Probability Theory: The Logic of Science, which I highly recommend.

At the moment, psychology is largely ad-hoc, and not a modicum of progress would've been made without the extensive utilization of statistical methods. To consider the human condition does not require us to simply extrapolate from our severely limited experiences, if not from the biases of limited datasets, datasets for which we can't even be certain of their various e.g. parameters etc..

For example, depending on the culture, the set of phenotypical traits with which increases the sexual attraction of an organism may be different - to state this is meaningless and ad-hoc, and we can only attempt to consider the validity of what was stated with statistical methods. Still, there comes along social scientists who would proclaim arbitrary sets of phenotypical features as being universal for all humans in all conditions simply because they were convinced by limited and biased datasets (e.g. making extreme generalizations based on the United States' demographic while ignoring the entire world etc.).

In fact, the author(s) of "Probability Theory: The Logic of Science" will let you know what they think of the shaky sciences of the 20th and 21st century, social science and econometrics included, the shaky sciences for which their only justifications are statistical methods.
_

With increasing mathematical depth and the increasing quality of applied mathematicians into such fields of science, we will begin to gradually see a significant improvement in the validity of said respective fields. Otherwise, currently, psychology, for example, holds no depth, but the field itself is very entertaining to me; doesn't stop me from enjoying Michael's "Mind Field" series.

For mathematicians, physics itself lacks rigour, let alone psychology.
_

Note, the founder of "psychoanalysis", Sigmund Freud, is essentially a pseudo-scientist. Like many social scientists, he made the major error of extreme extrapolation based on his very limited and personal life experiences, and that of extremely limited, biased datasets. Sigmund Freud "proclaimed" a lot of truths about the human condition, for example, Sigmund Fraud is the genius responsible for the notion of "Penis Envy".

In the same century, Einstein would change the face of physics forever after having published the four papers in his miracle year before producing the masterpiece of General Relativity. And, in that same century, incredible progress such that of Gödel's Incompleteness Theorem, Quantum Electrodynamics, the discovery of various biological reaction pathways (e.g. citric acid cycle etc.), and so on and so on would be produced while Sigmund Fraud can be proud of his Penis Envy hypothesis.

If you like logic and the scientific method, I recommend E. T. Jaynes' Probability Theory: The Logic of Science. You can buy it here:
http://www.amazon.com/Probability-Theory-The-Logic-Science/dp/0521592712/

or read a PDF here:
http://shawnslayton.com/open/Probability%2520book/book.pdf

Wellll I'm going to speak in some obscene generalities here.

There are some philosophical reasons and some practical reasons that being a "pure" Bayesian isn't really a thing as much as it used to be. But to get there, you first have to understand what a "pure" Bayesian is: you develop reasonable prior information based on your current state of knowledge about a parameter / research question. You codify that in terms of probability, and then you proceed with your analysis based on the data. When you look at the posterior distributions (or posterior predictive distribution), it should then correctly correspond to the rational "new" state of information about a problem because you've coded your prior information and the data, right?

WELL let's touch on the theoretical problems first: prior information. First off, it can be very tricky to code actual prior information into a true probability distribution. This is one of the big turn-offs for frequentists when it comes to Bayesian analysis. "Pure" Bayesian analysis sees prior information as necessarily coming before the data is ever seen. However, suppose you define a "prior" whereby a parameter must be greater than zero, but it turns out that your state of knowledge is wrong? What if you cannot codify your state of knowledge as a prior? What if your state of knowledge is correctly codified but makes up an "improper" prior distribution so that your posterior isn't defined?

Now'a'days, Bayesians tend to think of the prior as having several purposes, but they also view it as part of your modeling assumptions - something that must be tested to determine if your conclusions are robust. So you might use a weakly regularizing prior for the purposes of getting a model to converge, or you might look at the effects of a strong prior based on other studies, or the effects of a non-informative prior to see what the data is telling you absent other information. By taking stock of the differences, you can come to a better understanding of what a good prediction might be based on the information available to you. But to a "pure" Bayesian, this is a big no-no because you are selecting the prior to fit together with the data and seeing what happens. The "prior" is called that because it's supposed to come before, not after. It's supposed to codify the current state of knowledge, but now'a'days Bayesians see it as serving a more functional purpose.

Then there are some practical considerations. As I mentioned before, Bayesian analysis can be very computationally expensive when data sets are large. So in some instances, it's just not practical to go full Bayes. It may be preferable, but it's not practical. So you wind up with some shortcuts. I think that in this sense, modern Bayesians are still Bayesian - they review answers in reference to their theoretical understanding of what is going on with the distributions - but they can be somewhat restricted by the tools available to them.

As always with Bayesian statistics, Andrew Gelman has a lot to say about this. Example here and here and he has some papers that are worth looking into on the topic.

There are probably a lot of other answers. Like, you could get into how to even define a probability distribution and whether it has to be based on sigma algebras or what. Jaynes has some stuff to say about that.

If you want a good primer on Bayesian statistics that has a lot of talking and not that much math (although what math it does have is kind of challenging, I admit, though not unreachable), read this book. I promise it will only try to brainwash you a LITTLE.

Imagine you have a dataset without labels, but you want to solve a supervised problem with it, so you're going to try to collect labels. Let's say they are pictures of dogs and cats and you want to create labels to classify them.

One thing you could do is the following process:

1. Get a picture from your dataset.
   
2. Show it to a human and ask if it's a cat or a dog.
   
3. If the person says it's a cat or dog, mark it as a cat or dog.
   
4. Repeat.
   
   (I'm ignoring problems like pictures that are difficult to classify or lazy or adversarial humans giving you noisy labels)
   
   That's one way to do it, but is it the most efficient way? Imagine all your pictures are from only 10 cats and 10 dogs. Suppose they are sorted by individual. When you label the first picture, you get some information about the problem of classifying cats and dogs. When you label another picture of the same cat, you gain less information. When you label the 1238th picture from the same cat you probably get almost no information at all. So, to optimize your time, you should probably label pictures from other individuals before you get to the 1238th picture.
   
   How do you learn to do that in a principled way?
   
   Active Learning is a task where instead of first labeling the data and then learning a model, you do both simultaneously, and at each step you have a way to ask the model which next example should you manually classify for it to learn the most. You can than stop when you're already satisfied with the results.
   
   You could think of it as a reinforcement learning task where the reward is how much you'll learn for each label you acquire.
   
   The reason why, as a Bayesian, I like active learning, is the fact that there's a very old literature in Bayesian inference about what they call Experiment Design.
   
   Experiment Design is the following problem: suppose I have a physical model about some physical system, and I want to do some measurements to obtain information about the models parameters. Those measurements typically have control variables that I must set, right? What are the settings for those controls that, if I take measurements on that settings, will give the most information about the parameters?
   
   As an example: suppose I have an electric motor, and I know that its angular speed depends only on the electric tension applied on the terminals. And I happen to have a good model for it: it grows linearly up to a given value, and then it becomes constant. This model has two parameters: the slope of the linear growth and the point where it becomes constant. The first looks easy to determine, the second is a lot more difficult. I'm going to measure the angular speed at a bunch of different voltages to determine those two parameters. The set of voltages I'm going to measure at is my control variable. So, Experiment Design is a set of techniques to tell me what voltages I should measure at to learn the most about the value of the parameters.
   
   I could do Bayesian Iterated Experiment Design. I have an initial prior distribution over the parameters, and use it to find the best voltage to measure at. I then use the measured angular velocity to update my distribution over the parameters, and use this new distribution to determine the next voltage to measure at, and so on.
   
   How do I determine the next voltage to measure at? I have to have a loss function somehow. One possible loss function is the expected value of how much the accuracy of my physical model will increase if I measure the angular velocity at a voltage V, and use it as a new point to adjust the model. Another possible loss function is how much I expect the entropy of my distribution over parameters to decrease after measuring at V (the conditional mutual information between the parameters and the measurement at V).
   
   Active Learning is just iterated experiment design for building datasets. The control variable is which example to label next and the loss function is the negative expected increase in the performance of the model.
   
   So, now your procedure could be:
   
   
5. Start with:
   
   • a model to predict if the picture is a cat or a dog. It's probably a shit model.
     
   • a dataset of unlabeled pictures
     
   • a function that takes your model and a new unlabeled example and spits an expected reward if you label this example
     
6. Do:
   
   1. For each example in your current unlabeled set, calculate the reward
      
   2. Choose the example that have the biggest reward and label it.
      
   3. Continue until you're happy with the performance.
      
7. ????
   
8. Profit
   
   Or you could be a lot more clever than that and use proper reinforcement learning algorithms. Or you could be even more clever and use "model-independent" (not really...) rewards like the mutual information, so that you don't over-optimize the resulting data set for a single choice of model.
   
   I bet you have a lot of concerns about how to do this properly, how to avoid overfitting, how to have a proper train-validation-holdout sets for cross validation, etc, etc, and those are all valid concerns for which there are answers. But this is the gist of the procedure.
   
   You could do Active Learning and iterated experiment design without ever hearing about bayesian inference. It's just that those problems are natural to frame if you use bayesian inference and information theory.
   
   About the jargon, there's no way to understand it without studying bayesian inference and machine learning in this bayesian perspective. I suggest a few books:
   
   

• Information Theory, Inference, and Learning Algorithms, David Mackay - for which you can get a pdf or epub for free at this link.
  
  Is a pretty good introduction to Information Theory and bayesian inference, and how it relates to machine learning. The Machine Learning part might be too introductory if already know and use ML.
  
  
• Bayesian Reasoning and Machine Learning by David Barber - for which you can also get a free pdf here
  
  Some people don't like this book, and I can see why, but if you want to learn how bayesians think about ML, it is the most comprehensive book I think.
  
  
• Probability Theory, the Logic of Science by E. T. Jaynes. Free pdf of the first few chapters here.
  
  More of a philosophical book. This is a good book to understand what bayesians find so awesome about bayesian inference, and how they think about problems. It's not a book to take too seriously though. Jaynes was a very idiosyncratic thinker and the tone of some of the later chapters is very argumentative and defensive. Some would even say borderline crackpot. Read the chapter about plausible reasoning, and if that doesn't make you say "Oh, that's kind of interesting...", than nevermind. You'll never be convinced of this bayesian crap.
  
  

Bayes is the way to go: Ed Jayne's text Probability Theory is fundamental and a great read. Free chapter samples are here. Slightly off topic, David Mackay's free text is also wonderfully engaging.

https://www.amazon.com/Probability-Theory-Science-T-Jaynes/dp/0521592712

Ok, it's kinda expensive (~$60), but it's on amazon:

http://www.amazon.com/Probability-Theory-Logic-Science-Vol/dp/0521592712/ref=sr_1_1?ie=UTF8&s=books&qid=1252600006&sr=8-1

*edit for link pwnage.

Jaynes: Probability Theory. Perhaps 'rigorous' is not the first word I'd choose to describe it, but it certainly gives you a thorough understanding of what Bayesian methods actually mean.

I'm only part way through it myself, but here's one I've been recomended in the past that I've been enjoying so far:

Probability Theory: The Logic of Science by E.T. Jaynes

http://www.amazon.com/Probability-Theory-The-Logic-Science/dp/0521592712

http://omega.albany.edu:8008/JaynesBook.html

The second link only appears to have the first three chapters in pdf (though it has everything as postscript files), but I would be shocked if you couldn't easilly find a free pdf off the whole thing online with a quick search.

If you can find this at your library, I suggest you pour over it in the weekend. You will not regret it.

it most certainly is! There's a whole approach to statistics based around this idea of updating priors. If you're feeling ambitious, the book Probability theory by Jaynes is pretty accessible.

Casella & Berger is the go-to reference (as Smartless has already pointed out), but you may also enjoy Jaynes. I'm not sure I'd say it's quick but if gaps are your concern, it's pretty drum-tight.

I've heard good things about (but have not read) Probability, the logic of science.

A complete table of contents + the first 3 chapters are available here. This should tell you if it covers the appropriate material and if the explanations are to your satisfaction.

Probability Theory: The Logic of Science. This is an online pdf, possibly of an older version of the book. Science covers knowledge of the natural world, and mathematics and logic covers knowledge of formal systems.

Probability Theory: The Logic of Science

You may also enjoy Probability Theory: The Logic of Science by E.T. Jaynes, and Information Theory, Inference and Learning Algorithms by David McKay.

If you want a math book with that perspective, I'd recommend E.T. Jaynes "Probability Theory: The Logic of Science" he devolves into quite a lot of discussions about that topic.

If you want a popular science book on the subject, try "The Theory That Would Not Die".

Bayesian statistics has, in my opinion, been the force that has attempted to reverse this particular historical trend. However, that viewpoint is unlikely to be shared by all in this area. So take my viewpoint with a grain of salt.

Depends what your goal is. As you have a good background, I would not suggest any stats book or deep learning. First, read trough Probability theory - The logic of science and the go for Bishop's Pattern Recognition or Barbers's Bayesian Reasoning and ML. If you understand the first and one of the second books, I think you are ready for anything.

I'm really fond of Jaynes' Probability Theory: The Logic of Science and Rudin's Principles of Mathematical Analysis. Both are excellent, clearly written books in their own way.

FYI, Jaynes actually wrote a whole probability textbook that essentially put together all his thoughts about probability theory. I haven't read it, but many people say it got some good stuff.

"Bayesian" is a very very vague term, and this article isn't talking about Bayesian networks (I prefer the more general term graphical models), or Bayesian spam filtering, but rather a mode of "logic" that people use in everyday thinking. Thus the better comparison would be not to neural nets, but to propositional logic, which I think we can agree doesn't happen very often in people unless they've had lots of training. My favorite text on Bayesian reasoning is the Jaynes book..

Still, I'm less than convinced by the representation of the data in this article. Secondly, the article isn't even published yet to allow anyone to review it. Thirdly, I'm suspicious of any researcher that talks to the press before their data is published. So in short, the Economist really shouldn't have published this, and should have waited. Yet another example of atrocious science reporting.

> For one, you need a categorical definition by which to justify your "probability" with. What, does each time you tell a god to speak deduct 1%? That's absurdly vague, stupid, and unheard of, so no wonder I never thought you'd actually be arguing this.

I don't happen to know the appropriate decibel-values to assign to E and not-E in this case. But I know the fucking SIGNS of the values.

No, I don't know how many times god needs to appear for me to believe that I wasn't drugged or dreaming or just going crazy. But god appearing is evidence for the existence of god, and him not appearing is evidence against.

Does it really matter if we are talking intervals of 5-seconds versus lifetimes?

3 pages, and you don't even have to go to a library! Check it out:

http://www.amazon.com/reader/0521592712?_encoding=UTF8&ref_=sib%5Fdp%5Fpt#reader

Click on "First Pages" to get to the front.

You can lead a horse to water...

I'd suggest MATP 4600, Probability Theory & Applications. Only prerequisite is Calc if I remember right.

Or if you're confident in your time management, maybe read this textbook on your own; it's pretty accessible: https://www.amazon.com/gp/aw/d/0521592712/

(Neither of these will teach you a bunch of statistical tests, but those are easy to abuse if you don't understand the fundamentals ... and very easy to look up if you do understand the fundamentals.)

>All I'm saying is that the origin of a claim contains zero evidence as to that claim's truth.

I had a look back though your other posts and found this, which explains a lot, for me anyway. Most people would put some more options in there - yes, no, im pretty sure, its extremely unlikely etc..

Heres what I think is the problem, and why I think you need to change the way you are thinking - Your whole concept of what is "logical" or what is "using reason" seems to be constrained to what is formally known as deductive logic. You seem to have a really thorough understanding of this type of logic and have really latched on to it. Deductive logic is just a subset of logic. There is more to it than that.

I was searching for something to show you on other forms of logic and came across this book - "Probability Theory - The Logic of Science" Which looks awesome, Im going to read it myself, it gets great reviews. Ive only skimmed the first chapter... but that seems to be a good summary of how science works- why it does not use just deductive logic. Science draws most of its conclusions from probability, deductive logic is only appropriate in specific cases.

Conclusions based on probability - "Im pretty sure", "This is likely/unlikely" are extremely valid - and rational. Your forcing yourself to use deductive logic, and only deductive logic, where its inappropriate.

>You have no way of knowing, and finding out that this person regularly hallucinates them tells you nothing about their actual existence.

Yeah I think with the info you've said we have it would be to little to draw a conclusion or even start to draw one. Agreed. It wouldnt take much more info for us to start having a conversation about probabilities though - Say we had another person from the planet and he says its actually the red striped jagerwappas that are actually taking over - and that these two creatures are fundamentally incompatible. ie. if x exists y can't and vice-versa.

This one? Damn, it's £40-ish. Any highlights or is it just a case of this book is the highlight?

It's on my wishlist anyway. Thanks.

> There are some philosophical reasons and some practical reasons that being a "pure" Bayesian isn't really a thing as much as it used to be. But to get there, you first have to understand what a "pure" Bayesian is: you develop reasonable prior information based on your current state of knowledge about a parameter / research question. You codify that in terms of probability, and then you proceed with your analysis based on the data. When you look at the posterior distributions (or posterior predictive distribution), it should then correctly correspond to the rational "new" state of information about a problem because you've coded your prior information and the data, right?

Sounds good. I'm with you here.

> However, suppose you define a "prior" whereby a parameter must be greater than zero, but it turns out that your state of knowledge is wrong?

Isn't that prior then just an error like any other, like assuming that 2 + 2 = 5 and making calculations based on that?

> What if you cannot codify your state of knowledge as a prior?

Do you mean a state of knowledge that is impossible to encode as a prior, or one that we just don't know how to encode?

> What if your state of knowledge is correctly codified but makes up an "improper" prior distribution so that your posterior isn't defined?

Good question. Is it settled how one should construct the strictly correct priors? Do we know that the correct procedure ever leads to improper distributions? Personally, I'm not sure I know how to create priors for any problem other than the one the prior is spread evenly over a finite set of indistinguishable hypotheses.

The thing about trying different priors, to see if it makes much of a difference, seems like a legitimate approximation technique that needn't shake any philosophical underpinnings. As far as I can see, it's akin to plugging in different values of an unknown parameter in a formula, to see if one needs to figure out the unknown parameter, or if the formula produces approximately the same result anyway.

> read this book. I promise it will only try to brainwash you a LITTLE.

I read it and I loved it so much for its uncompromising attitude. Jaynes made me a militant radical. ;-)

I have an uncomfortable feeling that Gelman sometimes strays from the straight and narrow. Nevertheless, I looked forward to reading the page about Prior Choice Recommendations that he links to in one of the posts you mention. In it, though, I find the puzzling "Some principles we don't like: invariance, Jeffreys, entropy". Do you know why they write that?

All gone now. (05:30 UMT 10 August) LiSP and Probability Theory: The Logic of Science are still in the top two slots, but amazon.ca appears to have sold out of new copies.

Jaynes' Probability Theory is fantastic.

> Honestly, both of our arguments have become circular. This is because, as I have stressed, there is not enough data for it to be otherwise. Science is similar to law in that the burden of proof lies with the accuser. In this case there is no proof, only conjecture.

((Just in case it is relevant: Which two arguments do you mean exactly, because the circularity isn't obvious to me?))

In my opinion you can argue convincingly about future events where you are missing important data and where no definitive proof was given (like in the AI example) and I want to try to convince you :)

I want to base my argument on subjective probabilities. Here is a nice book about it. It is the only book of advanced math that I worked through \^\^ (pdf).

My argument consists of multiple examples. I don't know where we will disagree, so I will start with a more agreeable one.

Let's say there is a coin and you know that it may be biased. You have to guess the (subjective) probability that the first toss is head . You are missing very important data: The direction the coin is biased to, how much it is biased, the material .... . But you can argue the following way: "I have some hypotheses about how the coin behaves and the resulting probabilities and how plausible these hypotheses are. But each hypothesis that claims a bias in favour of head is matched with an equally plausible hypothesis that points in the tail direction. Therefore the subjective probability that the first toss is head is 50%"

What exactly does "the subjective probability is 50%" mean? It means if I have to bet money where head wins 50 cent and tail wins 50 cent, I could not prefer any side. (I'm using small monetary values in all examples, so that human biases like risk aversion and diminishing returns can be ignored).

If someone (that doesn't know more than me) claims the probability is 70% in favour of heads, then I will bet against him: We would always agree on any odds between 50:50 and 70:30. Let's say we agree on 60:40, which means I get 60 cent from him if the coin shows tail and he gets 40 cent from me if the coin shows head. Each of us agrees to it because each one claims to have a positive expected value.

This is more or less what happened when I bet against the brexit with my roommate some days ago. I regularly bet with my friends. It is second nature for me. Why do I do it? I want to be better at quantifying how much I believe something. In the next examples I want to show you how I can use these quantifications.

What happens when I really don't know something. Let's say I have to guess my subjective probability that the Riemann hypothesis is true. So I read the Wikipedia article for the first time and didn't understand the details ^^. All I can use is my gut feeling. There seem to be some more arguments in favour of it being true, so I set it to 70%. I thought about using a higher value but some arguments might be biased by arguing in favour to what some mathematicians want to be true (instead of what is true).

So would I bet against someone who has odds that are different from mine (70:30) and doesn't know much more about that topic? Of course!

Now let's say in a hypothetic scenario an alien, a god, or anyone that I would take serious and have no power over him appears in front of me, chooses randomly a mathematical conjecture (here: it chooses the Rieman hypotheses) and speaks the following threat: "Tomorrow You will take a fair coin from your wallet and throw it. If the coin lands head you will be killed. But as an alternative scenario you may plant a tree. If you do this, your death will not be decided by a coin, but you will not be killed if and only if the Riemann hypothesis is true"

Or in other words: If the subjective probability that the Riemann hypothesis is true is >50% then I will prefer to plant a tree; otherwise, I will not.

This example shows that you can compare probabilities that are more or less objective (e.g. from a coin) with subjective probabilities and that you should even act on that result.

The comforting thing with subjective probabilities is that you can use all the known rules from "normal" probabilities. This means that sometimes you can really try to calculate them from assumptions that are much more basic than a gut feeling. When I wrote this post I asked myself what the probability is that the Riemann hypothesis will be proven/disproven within the next 10 years. (I just wanted to show you this, because the result was so simple, which made me happy, but you can skip that).

• assumption 1: Given a single arbitrary mathematical statement I know nothing about. And lets say I consider only those with a given difficulty, which means it is either easy to solve or difficult to solve from an objective point of view. Now I use the approximation that if it wasn't solved for n days, then the probability that it will be solved within the next day is like throwing a dice - it is independent of n. This behaviour is described by an exponential function "exp(-r t)", where the result is the probability that it remains unsolved after t years and a given difficulty parameter r. You could use better models of course, but given I know nothing about that statement, it is OK for me to expect a distribution which looks like an exponential function.
  
• assumption 2: Most mathematical problems and subproblems are solved rather fast/instantly, because they are simple. The outstanding problems are the difficult ones. This can be described by a difficulty parameter probability distribution where each possible parameter value has the same subjective probability. This is only one way to describe the observation of course, but I also get this probability distribution if I use the principle of indifference, according to which the problem should be invariant with respect to the timescale (= nothing changes if I change the units from months to decades).
  
• result: Ok I don't know how difficult the Riemann hypothesis is to prove, but integrating over all possible difficulties and weighting them by their subjective probability (=assumption 2) and the plausibility of not being solved for past years "p", I can calculate the odds that it will be solved within the next years "t". The solution = "t:p". So given, that it wasn't solved for 100 years the odds are very small (10:100).
  
  And this result is useful for me. Would I bet on that ratio? Of course! Would I plant a tree in a similar alien example? No I wouldn't, because the probability is <50%. Again, it is possible to use subjective probabilities to find out what to do.
  
  And here is the best part, about using subjective probabilities. You said "Science is similar to law in that the burden of proof lies with the accuser. In this case there is no proof, only conjecture." But this rule is no longer needed. You can come to the conclusion that the probability is too low to be relevant for whatever argument and move on. The classic example of Bertrand Russel's teapot can be solved that way.
  
  Another example: You can calculate which types of supernatural gods are more or less probable. One just needs to collect all pro and contra arguments and translate them to likelihood ratios . I want to give you an example with one type of Christian god hypothesis vs. pure scientific reasoning:
  
  
• Evidence "The species on planet earth can be organized by their genes in a tree shape.": evolution predicts this (therefore p=1) and Christian-god-intelligent-design-hypothesis says "maybe yes maybe something else" (p= 1/2 at most). Therefore the likelihood ratio is 1:2 in favour of pure scientific reasoning.
  
• more arguments, contra: problem of evil, lawful universe and things that follow from that, ...
  
• more arguments, pro: Fine-tuned Universe problem, existence of consciousness, ...
  
  In the end you just multiply all ratios of all arguments and then you know which hypothesis of these two to prefer. The derived mathematical formula is a bit more complicated, because it takes into account that the arguments might depend on each other and that there is an additional factor (the prior) which is used to indicate how much you privilege any of these two hypotheses over all the other hypotheses (e.g. because the hypothesis is the most simple one).
  
  I wanted to show you that you can construct useful arguments using subjective probabilities, come to a conclusion and then act on the result. It is not necessary to have a definitive proof (or to argue about which side has the burden of proof).
  
  I can imagine two ways were my argument is flawed.
  
  
• Maybe there will be too much to worry/ things to do, if one uses that method consequently. But all extreme examples I can think of either have too low probability (e.g. Pascal's Wager), or there is not much that can be done today (most asteroids are detected too late), or it is much easier to solve the problem when it arrives instead of today.
  
• Subjective probabilities are formalized and can be used consistently for environmental uncertainty. But there are problems if you try to reason under logical uncertainty. This is not yet formalized. Assuming it will never be, then my argument cannot be used.

learn Haskell, learn how to make money, or learn some probability theory

I really love Probability Theory: The Logic of Science by Jaynes. While it is not a physics book, it was written by one. It is very well written, and is filled with common sense (which is a good thing). I really enjoy how probability theory is built up within it. It is also very interesting if you have read some of Jaynes' more famous works on applying maximum entropy to Statistical Mechanics.

I enjoyed

Introduction to Probability Theory, Hoel et. al

Also,

Probability Theory, Jaynes

is essential. For probabilistic programming I would also look into

Bayesian Methods for Hackers