Comments on: The unicorn problem http://www.statisticsblog.com/2012/10/the-unicorn-problem/ In Monte Carlo We Trust Thu, 02 May 2013 18:14:27 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Matt Asher http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-16878 Matt Asher Tue, 12 Feb 2013 16:40:25 +0000 http://www.statisticsblog.com/?p=695#comment-16878 Hi Pascal, Nice! Reminds me of trading cards to McDonald's Monopoly game as a kid, trying to complete a particular set. We always suspected that they made all but the last card in a set very rare, to prevent people from winning many of the big prizes. Given that our we were just a tiny fraction of the people who collected the cards, we were effectively sampling with replacement from the overall collection. Makes me wonder about trading, though, and how much it would increase your chances of finding your "unicorn". Could make the simulation more interesting. Hi Pascal,

Nice! Reminds me of trading cards to McDonald’s Monopoly game as a kid, trying to complete a particular set. We always suspected that they made all but the last card in a set very rare, to prevent people from winning many of the big prizes. Given that our we were just a tiny fraction of the people who collected the cards, we were effectively sampling with replacement from the overall collection.

Makes me wonder about trading, though, and how much it would increase your chances of finding your “unicorn”. Could make the simulation more interesting.

]]> By: Pascal http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-16877 Pascal Tue, 12 Feb 2013 15:53:25 +0000 http://www.statisticsblog.com/?p=695#comment-16877 Hi, I stumbled upon your nice blog. The "unicorn problem" is related to the classic coupon collector's problem (http://en.wikipedia.org/wiki/Coupon_collector%27s_problem), where you have n urns and at each step you place a ball into a randomly selected urn. The question is then how many balls you have to use so that every urn contains a ball. There are many generalizations of this problem, a Google search will help you find them. The relation with your problem: instead of saying that the probability of finding a species is 1/20 at each trip, it is roughly equivalent to say that at each trip you find one (random) species, and then counting 50 trips as one (this approximation is ok, because 50 not much bigger than the square root of 1000). The expected number of trips necessary to find the 1000 species should therefore be roughly 1000 * log(1000) / 50, which is approximately 140. Just checking at the graph: Seems to fit :) You can also do the calculations directly for your model. This is actually very easy, since the random variables N_1,...,N_k,...,N_1000, where N_k is the number of trips you have to do to find species k, are independent. Each one is geometric with success probability 1/20, therefore their maximum is roughly 20*log(1000) (standard result from extreme value theory). Hi, I stumbled upon your nice blog. The “unicorn problem” is related to the classic coupon collector’s problem (http://en.wikipedia.org/wiki/Coupon_collector%27s_problem), where you have n urns and at each step you place a ball into a randomly selected urn. The question is then how many balls you have to use so that every urn contains a ball. There are many generalizations of this problem, a Google search will help you find them.

The relation with your problem: instead of saying that the probability of finding a species is 1/20 at each trip, it is roughly equivalent to say that at each trip you find one (random) species, and then counting 50 trips as one (this approximation is ok, because 50 not much bigger than the square root of 1000). The expected number of trips necessary to find the 1000 species should therefore be roughly 1000 * log(1000) / 50, which is approximately 140.

Just checking at the graph: Seems to fit :)

You can also do the calculations directly for your model. This is actually very easy, since the random variables N_1,…,N_k,…,N_1000, where N_k is the number of trips you have to do to find species k, are independent. Each one is geometric with success probability 1/20, therefore their maximum is roughly 20*log(1000) (standard result from extreme value theory).

]]> By: Landmine detection revisited; the inverse unicorn problem « Probability and statistics blog http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-16835 Landmine detection revisited; the inverse unicorn problem « Probability and statistics blog Mon, 04 Feb 2013 15:28:27 +0000 http://www.statisticsblog.com/?p=695#comment-16835 [...] the existence of mines if they were very sparse in an area? In a sense, this is the inverse of the unicorn problem; instead of trying to find every last mine, we’re concerned with finding the very first one, [...] [...] the existence of mines if they were very sparse in an area? In a sense, this is the inverse of the unicorn problem; instead of trying to find every last mine, we’re concerned with finding the very first one, [...]

]]> By: Simulation of landmine clearing with Massoud Hassani’s Mine Kafon « Probability and statistics blog http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-16710 Simulation of landmine clearing with Massoud Hassani’s Mine Kafon « Probability and statistics blog Fri, 11 Jan 2013 00:29:30 +0000 http://www.statisticsblog.com/?p=695#comment-16710 [...] you look on my blog you’ll see a post I did about something I call The Unicorn Problem, related to finding all of the new species in an environment. The problem there, as with this [...] [...] you look on my blog you’ll see a post I did about something I call The Unicorn Problem, related to finding all of the new species in an environment. The problem there, as with this [...]

]]> By: Matt Asher http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-13396 Matt Asher Mon, 15 Oct 2012 14:14:28 +0000 http://www.statisticsblog.com/?p=695#comment-13396 @Jan Galkowski Very interesting problem! Kind of like the German Tank Problem (http://www.statisticsblog.com/2010/05/how-many-tanks-gtp-gets-put-to-the-test/) for species. Would be interesting to test the theorem you linked with an MC simulation. @Jan Galkowski

Very interesting problem! Kind of like the German Tank Problem (http://www.statisticsblog.com/2010/05/how-many-tanks-gtp-gets-put-to-the-test/) for species. Would be interesting to test the theorem you linked with an MC simulation.

]]> By: jcb http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-13395 jcb Mon, 15 Oct 2012 07:32:12 +0000 http://www.statisticsblog.com/?p=695#comment-13395 DX hunting. That's a radio amateur activity and the uniform discoverability rate you assume above does not hold at all. DX hunting. That’s a radio amateur activity and the uniform discoverability rate you assume above does not hold at all.

]]> By: Jan Galkowski http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-13393 Jan Galkowski Mon, 15 Oct 2012 00:43:32 +0000 http://www.statisticsblog.com/?p=695#comment-13393 How about an even more interesting problem: Given the rate of finding new species, and "juicing" the calculation by using estimates of weights on how efficacious certain sampling places and schemes are ("importance weights"), determine how many more species remain to be found. References for students of the matter, as I am: http://oregonstate.edu/instruct/st571/urquhart/var_prob/sld011.htm Overton, Stehman, "The Horvitz-Thompson Theorem as a unifying perspective for probability sampling: With examples from natural resources sampling," THE AMERICAN STATISTICIAN, 49(3), August 1995, pp 261ff. A. R. Solow, W. K. Smith, "Estimating species number under an inconvenient abundance model," JOURNAL OF AGRICULTURAL, BIOLOGICAL, AND ENVIRONMENTAL STATISTICS, 14: 242-252, 2009. How about an even more interesting problem: Given the rate of finding new species, and “juicing” the calculation by using estimates of weights on how efficacious certain sampling places and schemes are (“importance weights”), determine how many more species remain to be found.

References for students of the matter, as I am:

http://oregonstate.edu/instruct/st571/urquhart/var_prob/sld011.htm

Overton, Stehman, “The Horvitz-Thompson Theorem as a unifying perspective for probability sampling: With examples from natural resources sampling,” THE AMERICAN STATISTICIAN, 49(3), August 1995, pp 261ff.

A. R. Solow, W. K. Smith, “Estimating species number under an inconvenient abundance model,” JOURNAL OF AGRICULTURAL, BIOLOGICAL, AND ENVIRONMENTAL STATISTICS, 14: 242-252, 2009.

]]> By: M.P. http://www.statisticsblog.com/2012/10/the-unicorn-problem/comment-page-1/#comment-13391 M.P. Sun, 14 Oct 2012 19:25:15 +0000 http://www.statisticsblog.com/?p=695#comment-13391 The biggest problem w the assumptions is that all species are equally likely to be encountered each time. if you made some more rare than others I bet it would take even longer to find your unicorn. The biggest problem w the assumptions is that all species are equally likely to be encountered each time. if you made some more rare than others I bet it would take even longer to find your unicorn.

]]>