tag:blogger.com,1999:blog-33280816920218860542024-03-21T22:07:05.356+01:00TechnicalMumboJumboAnonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.comBlogger52125tag:blogger.com,1999:blog-3328081692021886054.post-45490856831074589762017-05-11T21:38:00.000+02:002017-05-11T21:38:53.262+02:00Air Quality in Poland #16 - This is the end?<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: center;">
Initial idea </h3>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Hello and welcome to the last post published under <a href="https://temuju.blogspot.com/search/label/Get%20Noticed%202017!" target="_blank">Get Noticed 2017!</a> tag. But fear nothing, it is just last post submitted to <a href="http://devstyle.pl/daj-sie-poznac/" target="_blank">Get Noticed</a> competition, not last post written and published by me. I decided to dedicate this post for Air Quality In Poland project summary. This topic was my main project for this competition and I spent 15 + 1 blog posts for working on it. So, let me discuss it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What was my original idea for this project? When I started to think about topic to pick for this competition, there was fresh discussion in Polish media about very poor quality of air in many of Polish biggest cities. In general this discussion was narrated as "Breaking News! We have new record of pollution! Are we living in places more polluted than ... (put your favorite polluted place here)?". I am very interested in health and living quality, so I was also interested in following this topic. After some time I started to notice some flaws with TV reports about air quality. There were some misinterpretations and possible mistakes in reasoning. Naturally, my question was: Can I do it better? Can I produce reproducible report which would not have such errors? And that questions formed my general goal for this project - <b>Get raw air quality data and produce as accurate, meaningful and as reproducible report as possible</b>.</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: center;">
Goals</h3>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
After forming this wide goal I decided to play with general idea and formulate additional goals: Which place is most polluted in Poland? Which is least polluted? Can I visualize pollution? Can I calculate changes of air quality? Can I visualize those changes? Can I interpolate missing values so I could estimate air quality in every place in Poland? Can I predict future values of pollution? Can I predict future values for each place in Poland? Maybe can I build application which will advise best time for outdoor activities? Will there be possibility to scrap GIOS webpage for current pollution measurements?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So many questions and ideas, and so little time. How much did I accomplished? Well, I wasn't able to properly implement much of those ideas, but I believe that I did something useful.</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: center;">
Implementation</h3>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
First of all, I prepared pipeline which gathers values of main pollutants from various years which was scattered in various csv files. Since this data wasn't always perfectly labeled I had to do some data cleaning in order to gather it in one place. Finally I produced multi-index data frame with date time and location as index and pollutants as columns. You can check it in <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/01-preproc.ipynb" target="_blank">notebook number one</a>. It doesn't answers any of those questions, but builds fundaments for answering each of them.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
When I had developed consistent data structure I focused on finding worst and best place to breathe in Poland. To do that I had to learn how air quality is classified and apply those rules on my data structure. You can find this analysis in <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/02-exploration.ipynb" target="_blank">notebook number two</a>. When I was preparing it, I found that since there are many data points missing, I cannot tell with full certainty which places are best and worst. But I addressed this problem clearly, so shouldn't be any misunderstandings there. Since my script is capable of calculating overall air quality for all places for selected time ranges, it could be easily used to calculate day to day, week to week (and so on ...) changes of air quality. So it also answers one of my questions. I totally skipped geo-spatial visualizations at this point - I decided it will take me too much time to produce something which wouldn't be useless colorful blob.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Next steps of my projects were directed to interpolation and prediction of pollutants, but in meantime, GIOS made a surprise and released official API which allowed to get current air data. It was pleasant surprise because I didn't had to focus on java-script web scraping with Python. Such scraping seems to be quite mundane, but maybe I'm missing something. Anyway, as you can see in <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/03-api.ipynb" target="_blank">notebook number three</a>, I tested this API and it seems to be working as promised. Very cool feature.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Fourth task which I planned to do was to predict future values based on time series. I picked fresh data from API and tired to predict values for next few hours. Since I don't have experience with time series I decided to try naive regression approach. As you can see in <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/04-timeseries.ipynb" target="_blank">notebook number four</a>, it wasn't successful idea. So I'm assuming that I didn't completed this task and there is still much to do here.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Last task which I was planning to complete, was to interpolate missing data, which, if implemented successfully, would lead me to building pollution estimator for each place in Poland. Most promising idea was to use nearest neighbors regression and apply it on geo-spatial features. And this implementation also failed miserably - you can check analysis code in <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/05-estimation.ipynb" target="_blank">notebook number five</a>. I'm not sure why it failed so badly. Maybe there is no way to properly interpolate such data based just on geo-spatial coordinates. Or maybe other algorithms would be better suited for such purpose? Who knows? One thing would be very useful in this and previous (prediction) problem - data structure with weather data. I believe that there might be some correlations or even causation related to various weather parameters. I would definitely check them out if I had access to such data.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<b>Leftovers and future </b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What ideas I didn't touched at all? There are two ideas which I would like to see implemented in this project but I didn't had time to take care of them. One is this advisor web application supposed to help with picking best time for outdoors activity. I didn't bother to start with it because fourth and fifth tasks were completed with unsatisfactory results. Another idea was to rewrite everything using Dask - Python module designed to parallelize array/data frame operations. I tried start with it, but it seems that it is little bit more complicated to use than Pandas. But those ideas are rather minor, so I'm not too much sad about not completing them.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The future? For now I'm not planning to continue to work on this project. I will put it into my maybe/far future projects shelf, so maybe I will go back to it when far future occurs ;). On the other hand if someone will be interested in discussing it or/and using my code I will happily share my thoughts about it and maybe do some additional work on it. Again, who knows?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Thanks for reading my blog and code. I hope that I created something useful and maybe I showed something interesting related to Python, reproducible research and data science. See you soon! </div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-78638768876857358622017-05-10T19:36:00.000+02:002017-05-10T19:36:58.491+02:00Not enough RAM for data frame? Maybe Dask will help?<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: center;">
What environment do we like?</h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8nj1h0Df_Dspp337tFCyHXTS2wj3hroGl4-8YQPFBs66eHcefuhtVhBwfsk7xmD5bbYjiVtb72dti5-IfgR5S_bWBi6FAPn9eoE2pYNYK4fpB4EQUFsZ6pYbCFzCmgEB_phUpWIoEMmgY/s1600/dask_stacked.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8nj1h0Df_Dspp337tFCyHXTS2wj3hroGl4-8YQPFBs66eHcefuhtVhBwfsk7xmD5bbYjiVtb72dti5-IfgR5S_bWBi6FAPn9eoE2pYNYK4fpB4EQUFsZ6pYbCFzCmgEB_phUpWIoEMmgY/s200/dask_stacked.png" width="167" /></a></div>
<div style="text-align: justify;">
When we are doing data analysis we would like to do it in as much interactive manner as possible. We would like to get results as soon as possible and test ideas without unnecessary waiting. We like to work in that manner because great proportion of our work leads to dead ends. As soon as we get there we can start thinking about other approach to problems. When you were working with R, Python or Julia backed by Jupyter Notebook, you are familiar with this workflow. And everything works well when your workstation has enough RAM to handle all data, its processing, cleaning, aggregation and transformations. Even Python, which is considered as rather <i>slow</i> language performs nicely here. Especially when you are using strongly optimized modules like NumPy. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But what with case, where data which we want to process is larger than available RAM? Well, there are three simple outcomes of your attempts: 1) your operating system will start to swap and possibly will be able to complete ordered task, but it will take way more time than expected; 2) you will receive warning or error message from functions which can estimate needed resources - rare case; 3) your script will use all available RAM and swap and will hang your operating system. But are there maybe any other options?</div>
<br />
<h3 style="text-align: center;">
What can we do?</h3>
<br />
<div style="text-align: justify;">
According to <a href="https://www.youtube.com/watch?v=RTiAMB2tQjo" target="_blank">this (Rob Story: Python Data Bikeshed)</a> and <a href="https://www.youtube.com/watch?v=QgjkgNVXSjQ" target="_blank">this (Peader Coyle - The PyData map: Presenting a map of the landscape of PyData tools)</a> presentations, there are many possibilities to build tailored data processing solution. From all of those modules, <a href="http://dask.pydata.org/en/latest/" target="_blank">Dask</a> looks especially interesting to me, because it mimics <a href="http://pandas.pydata.org/" target="_blank">Pandas</a> naming convention and functions behavior.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Dask offers three data structures - array, data frame and bag. Array structure is designed to implement methods known from NumPy array. So everyone experienced with NumPy will be able to use it without much problems. The same approach was used to implement data frame which is inspired by Pandas data frame structure. Bag on the other hand is designed to be equivalent of json dictionaries or other Python data structures.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What is key difference between Dask data structures and their archetypes? Dask structures are divided into small chunks and every operation on them is evaluated when its needed. It means that when we have data frame and series of transformations, aggregations, searches and similar operations Dask will calculate what and when to do with each chunk and will take care of executions of those operations and do garbage collection immediately. If we would like to do that in original Pandas data frame it will have to fit into RAM entirely, and every of steps in pipeline will also have to store its results in RAM even if some operations could be executed with <b>inplace=True</b> parameter.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
How can we use it? As I mentioned, Dask data structures were designed to be "compatible" with NumPy and Pandas data structures. So, if we check data frame <a href="http://dask.pydata.org/en/latest/dataframe-api.html" target="_blank">API reference</a> we will see that many Pandas methods was re-implemented with the same arguments and results.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But the problem is that not all original methods from NumPy and Pandas are implemented in Dask. So it is not possible to blindly substitute Pandas with Dask and expect that code will work. On the other hand, in cases where you are unable to read your data at all, it might be worth to spend some time to rework your flow and adjust it to Dask.</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Second problem with Dask it that despite it tries to execute various operations on chunks in parallel, it may take more time to produce final results than in simple all in RAM data frame. But if you know <a href="https://docs.python.org/2/library/profile.html" target="_blank">time execution characteristic</a> of your scripts you can try to substitute some heavy time parts of it and compare with Dask execution. Maybe it will make sense.</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: center;">
The future?</h3>
<br />
<div style="text-align: justify;">
What is the future of such modules? It depends. One my ask "Why to bother when we have <a href="https://spark.apache.org/docs/0.9.0/python-programming-guide.html" target="_blank">PySpark</a>?". And this is valid question. I would say, that Dask and similar solutions fits nicely in niche where data is to big for RAM but on the other hand it fits on directly attached hard drives. If data fits nicely in RAM I wouldn't bother to work with Dask and similar modules - I would just stick to plain old and good NumPy and Pandas. Also if I had to deal with such amount of data that wouldn't fit on available attached to workstation disks, I would consider going into big data solutions which also gives me redundancy over hardware failures. But Dask is still very cool, and at least worth to test.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I hope that Dask developers will port more of NumPy and Pandas methods into. I also saw some works towards integrating it with <a href="https://github.com/dask/dask-searchcv" target="_blank">Scikit-Learn</a>, <a href="https://github.com/dask/dask-xgboost" target="_blank">XGBoost</a> and <a href="https://github.com/dask/dask-tensorflow" target="_blank">TensorFlow</a>. You should definitely check this out when you will be considering buying more RAM next time ;).</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-1962706231718003842017-05-05T21:44:00.000+02:002017-05-05T21:44:45.311+02:00Air Quality in Poland #15 - How to estimate missing pollutants? Part 2<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: center;">
Current status</h3>
<br />
<div style="text-align: justify;">
In last post we prepared data for machine learning process. We selected one date time point of measurements and found actual coordinates of measurement stations. As I mentioned, I planned to use Nearest Neighbor Regressor applied on geographical coordinates.</div>
<h3 style="text-align: center;">
Approach</h3>
<div style="text-align: justify;">
We already should have column with tuple of latitude and longitude. I decided to change it to list and split into two columns</div>
<br />
<script src="https://gist.github.com/QuantumDamage/1e71ba4db129c08991a335c4b6b1c3ef.js"></script><br />
<br />
<div style="text-align: justify;">
Since I have no idea how to tackle number of neighbors to use for prediction I decided to create loop and try to find best answer for that. I also generated different model for each pollutant since measurement stations are not identically distributed. Here is the code:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/2a5bf51e5355e550d5131b8b967046ad.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTvDTlkPBCV_SOsXyV4cc59FzkdWJEsqI1B6VP2jCeXZNwM9HiDfV8SCh24JkJNqoBycrQcQg8O7MxhALqBaDTSS8GtviQ0LJYIcdhp6MUSlqa9k0s52AudZPB7WjCxWsxLUdFW1SEQrV3/s1600/15-worst.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
What is going on here? In line 4 we are selecting only interesting data for each pollutant - measurements values and latitudes and longitudes. Then in line 5 we are selecting points which have no values - we will predict those values in future. In line 6 we are doing the same selection but without "~" which is used for negation. We are taking every point which has measurement data. In lines 8 and 9 I'm splitting data frame into features and target data frames. In line 11 I'm preparing train and test sets with 3:1 size. I'm using constant random state so it should be reproducible. And in line 12 magic is starting to happen.<br />
<br />
As I mentioned, I have no clue of how many neighbors should I use for value calculation. So I prepared loop from 1 to (all - 1) points, which is making fit, and calculates score of its accuracy. Scores are then added to list and plotted. And here is the place where magic turns out to be just cheap trick. Scoring function used for evaluating test set is coefficient of determination. And as far as I know it should give score between 0 and 1. In my cases it gives quite much negative values, so it might mean that this model is very bad. Lets see example of best and worst scenarios:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinEntI9NgKVhRLRw15LJcW8aIOqwcLdDFZs3bUnqh2RXzzXM2RzsR_nyILuY8rT4tGb3SsloFvcZsVe2QCNdLU5d_AxuOknOkvlHDZqs_D15QCUvpHfM2V8u3HIV-4VlE6y63-cP-YvKLZ/s1600/15-best.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinEntI9NgKVhRLRw15LJcW8aIOqwcLdDFZs3bUnqh2RXzzXM2RzsR_nyILuY8rT4tGb3SsloFvcZsVe2QCNdLU5d_AxuOknOkvlHDZqs_D15QCUvpHfM2V8u3HIV-4VlE6y63-cP-YvKLZ/s320/15-best.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Best results?</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTvDTlkPBCV_SOsXyV4cc59FzkdWJEsqI1B6VP2jCeXZNwM9HiDfV8SCh24JkJNqoBycrQcQg8O7MxhALqBaDTSS8GtviQ0LJYIcdhp6MUSlqa9k0s52AudZPB7WjCxWsxLUdFW1SEQrV3/s1600/15-worst.png" style="margin-left: auto; margin-right: auto;"><img border="0" height="216" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTvDTlkPBCV_SOsXyV4cc59FzkdWJEsqI1B6VP2jCeXZNwM9HiDfV8SCh24JkJNqoBycrQcQg8O7MxhALqBaDTSS8GtviQ0LJYIcdhp6MUSlqa9k0s52AudZPB7WjCxWsxLUdFW1SEQrV3/s320/15-worst.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Worst results?</td></tr>
</tbody></table>
<br />
As you can see both of those results are far from something useful. That's quite disappointing, but to be honest, I'm getting used to it ;).</div>
<h3 style="text-align: center;">
Conclusion?</h3>
<div style="text-align: justify;">
What can we learn from this failure? We can think about possible causes of it. One cause might be that I'm using wrong scoring function - it was designed to measure effectiveness of linear fit. Maybe our situation is not linear? Other cause might be that in this date time there was not enough meaningful data - nothing was happening, so it was hard to predict anything. Another reason might be related to overall nature of measured data - it might be not to much related to geo-spatial distribution but maybe to weather, industry production, national park presence or something like this. Without this data it might be not possible to predict missing data.<br />
<br />
Anyway, I believe I had build quite nice base for further experimentation. When I will have more spare time maybe I will try to scientifically find why my initial approach didn't worked. Or maybe you have any idea?</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-5820795469190313722017-04-28T19:39:00.000+02:002017-04-28T19:39:37.599+02:00Air Quality in Poland #14 - How to estimate missing pollutants? Part 1<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
<h3>
What do we have now?</h3>
</div>
<br />
<div>
<div style="text-align: justify;">
In previous blog posts about air quality in Poland, we developed pipeline which produced overall quality summaries for selected measuring station and date ranges. But when we tried to look for best and worst place to breath, we encountered typical analytical problem - not every station was measuring all pollutants. It implies, that there might be stations which are in worse or in better zones but we cannot point them because of lack of measurement data. Can we do something about it?</div>
<br />
<div style="text-align: center;">
<h3>
Can we go further?</h3>
<h3>
</h3>
<h3>
</h3>
</div>
</div>
<div>
<div style="text-align: justify;">
Actually yes. We can use machine learning methods to calculate missing values. We are particularly interested in supervised learning algorithm called <b>Nearest Neighbors Regression</b>. This algorithm is easy to understand and to apply. But before we can play with it, we must prepare data to be usable for scikit-learn.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To play with nearest neighbors regression we should have values from neighbors of data points which we would like to predict. Our data points are rather complete in term of time - when measuring station starts to measure pollutant, it continues measurements until decommissioning. So predicting data before and after would be rather guess work. But if we approach this problem from geo-spatial context it might give us better results. To do that, we must select every hour separately, pick up one of pollutants, check which stations don't have measurements and fill them. Also, we need to load coordinates for each station - we currently have only their names. Example code will look like that:</div>
<div style="text-align: justify;">
<br /></div>
<script src="https://gist.github.com/QuantumDamage/f2e9a9077ea6f22a8daec292196521d6.js"></script></div>
<div>
</div>
<div>
</div>
<div>
</div>
<div>
</div>
When we will have data from all measuring stations from one hour aggregation and coordinates for each station we can then select particular pollutant and all station which are measuring it:<br />
<br />
<script src="https://gist.github.com/QuantumDamage/f2e9a9077ea6f22a8daec292196521d6.js"></script><br />
<div>
</div>
<div>
Now, we have nice and clean data frame which contains stations, values and coordinates.</div>
<div>
<br />
<div style="text-align: center;">
<h3>
What will we do next?</h3>
</div>
</div>
In next blog post, I will split generated data frame into test and train subsets and use them to build nearest neighbors regression model. Stay tuned!<br />
<br />
As usual, code is <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/05-estimation.ipynb" target="_blank">available at GitHub</a>.</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-41142243659504419242017-04-27T17:50:00.002+02:002017-04-27T17:50:58.409+02:00Why should you participate in machine learning competitions<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
<h3>
What are machine learning competitions?</h3>
</div>
<div style="text-align: justify;">
<br />Machine learning competitions are competitions, in which goal is to predict either class or value, based on data. Data could be tabular, time series, audio, image or something similar and is often related to real life problems, which are tried to be solved with it. Rules are formulated in such way that every participant can immediately compare his results with every other participant. Data used in competitions is usually pretty much clean and processed so it is quite easy to start working with own solution.<br /><br /><h3 style="text-align: center;">
Why should I bother?</h3>
<br />Machine learning competitions are prepared by machine learning specialists for machine learning specialists. It means that problems which are used there represents current problems which are present in social, academic out business environments. Those problems also can be solved by machine learning methods, but sponsors are looking for some novel approach which may be proposed by competitors. This gives us <b>first reason</b> - this type of problems is currently present and solved with machine learning approach.<br /><br />When you register to completion and accept its rules, you will be able to download train and test data. This data always comes from real life processes and has its every flaws. There are some unexpected values, data leaks, broken files and repetitions. But on the other hand, process of its obtaining is usually well described and data itself is also described. In machine learning research such real but usable data is very valuable. This is <b>second reason</b> - access to real word data.<br /><br />As I mentioned, every participant receive train and test data. Train data, as usual is used to train prediction model. But test data is not used to test your model in usual way. You will receive this data but without target class or value. It's purpose is to be used for prediction. Those predicted classes or values can then be submitted into competition system and scored by defined function based on true and hidden real classes or values. You don't know target classes and values, so it is very hard to cheat. It's hard because every team or individual competitor is limited to result submissions per day. But since you know loss function, you can estimate effectiveness of your model before submitting results. After calculation, loss is placed on public leaderboard so every competitor can see how well is his model performing compared to others. After competition ends, leaderboard is recalculated with inclusion of another hidden data set, which wasn't used to calculate score on public leaderboard. This means that even if someone submitted just random results and luckily achieved great score, his results after recalculation will be closer to random than to high score.<br /><br />After competition, top contestants are interviewed to share their approach towards tackling problems. All of this gives us <b>third reason</b> - you are often competing with and comparing your solutions to top solutions in industry/academia so it teaches you humility in thinking about your “brilliant” solutions.<br /><br /><h3 style="text-align: center;">
Where can I compete?</h3>
<br />I participated in some competitions and it always was entertaining and educating for me. I didn’t had much of leaderboard success. But I learned patience and careful from a to z thinking. I really recommend participating in them. Currently there are at least two companies organizing such competitions: <a href="https://www.kaggle.com/competitions" target="_blank">Kaggle</a> and <a href="https://www.drivendata.org/competitions/" target="_blank">DrivenData</a>. First one is bigger and rather business driven, second one is definitely smaller but aims toward solving social/greater good problems, so it might be better suited for morally motivated competitors. Either way, both are using flow described above. Good luck!</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-38810340941288507512017-04-23T14:33:00.000+02:002017-04-23T14:33:15.752+02:00Air Quality In Poland #13 - How to predict furutre ... lazy way?<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: center;">
Can we predict air quality?</h3>
<div style="text-align: justify;">
One of fields covered by machine learning, is prediction of future values based on time series. Those values might be for example: stock market prices, number of products sold, temperature, clients visiting store and so on. The goal is to predict values for future time points based on historical data. The problem here is that available data consist only time point and value, without any additional information. In the other words, in this types of problems you have to predict shape of plot having only plot (and actual data frame) of historical data. How to do that? I have not much of ideas, but maybe naive and brute force approach will give me something.</div>
<h3 style="text-align: center;">
Yes, we can!</h3>
<div style="text-align: justify;">
What kind of brute force I have in mind? Well, even if only data we have, are pollutant measurements values and dates and time over which it was averaged, we can still treat is as regression problem ... with very small number of features. Actually it will be only one feature, which will be number of hours since first available measurement. So in addition of <b>date time</b> and <b>value</b> columns I will add calculated <b>relative time</b> column. That was "naive" part. Time for brute force.<br />
<br />
In this case, by "brute force" approach I mean to push data into TPOTRegressor and see what will be the result. It is quite lazy and not too smart, but since I don't have to much time now it have to be enough.<br />
<br />
After about 10 minutes of model mutating we can use it to predict values for next 24 hours and plot them to see if it make any sense.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUOuketJJ7LE0-T2wFoziVA5QTQ1gCquU7f5lc7clTf9sa07WQAfOJR9sKLukfqStf7ckovZTXb0w_z1yXQCpCZE4uS6wwgRxXEFe9nOf7ZwLB7Bqv903KuPdBf7DJW1RScx8ZbE1Us4Gv/s1600/13-plot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="149" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUOuketJJ7LE0-T2wFoziVA5QTQ1gCquU7f5lc7clTf9sa07WQAfOJR9sKLukfqStf7ckovZTXb0w_z1yXQCpCZE4uS6wwgRxXEFe9nOf7ZwLB7Bqv903KuPdBf7DJW1RScx8ZbE1Us4Gv/s400/13-plot.png" width="400" /></a></div>
</div>
<h3 style="text-align: center;">
Well ... it's something.</h3>
<div style="text-align: justify;">
As you can see on plot above, we were able to generate "some" predictions for further pollutant concentration. Are they valid? I don't know because I didn't bother to perform too much cross validation and comparison with actual measurements. But even without them, we can see that shape of perditions is not as it suppose to be. I know that this approach make not too much sense, but I wanted to test how quickly can I make at least small step toward time series prediction. If you would like to check my code - <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/04-timeseries.ipynb" target="_blank">here it is</a>.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-45981826649050229562017-04-20T20:57:00.000+02:002017-04-20T20:57:08.826+02:00Air Quality In Poland #12 - What is this? Is this .... API?<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: center;">
Surprise from GIOS</h2>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwKvMEoh7qoAna2bTul9VQ9oWohOFHEVt5NM8PmHCFOKChatUQuRtd7fOVXniSOtl8lnUzbsxgJIBrXXns_Ygjh4mAPG8XcmoPd4jqq19rZ5nabi3FodiqQhSOJUmy-WP_Jo-lNMEZtJhG/s1600/data.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="156" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwKvMEoh7qoAna2bTul9VQ9oWohOFHEVt5NM8PmHCFOKChatUQuRtd7fOVXniSOtl8lnUzbsxgJIBrXXns_Ygjh4mAPG8XcmoPd4jqq19rZ5nabi3FodiqQhSOJUmy-WP_Jo-lNMEZtJhG/s200/data.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Image from <a href="https://memegenerator.net/instance/56157047/chappelle-crackhead-yall-got-anymore-of-them-data" target="_blank">memegenerator.net</a></td></tr>
</tbody></table>
<div style="text-align: justify;">
When I was starting playing with air quality data from GIOS, only reasonable way to obtain it was download pre-packed archives with data for each year separately. Also when I'm publishing this post, last data archive is dated 2015, so there is no data from whole 2016 and 2017-now available. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But recently, GIOS with collaboration with <a href="https://epf.org.pl/en/" target="_blank">ePaństwo Foundation</a> released initial and experimental version of data <a href="http://powietrze.gios.gov.pl/pjp/content/api" target="_blank">RESTful API</a>. It is very simple, but you don't have to identify yourself with private key and there are <a href="http://powietrze.gios.gov.pl/pjp/content/terms_of_service" target="_blank">no significant limitations</a> of how one can use it. Lets see what can we do with it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Currently, four types of request are handled which can give us following data: measurement stations, sensors, measured data and air quality index.</div>
<h3 style="text-align: center;">
Measurement stations</h3>
<div style="text-align: justify;">
First API request we should check is station/findAll. This API request should give us JSON response with list of measuring stations and some information about them. Most important field from this response is top level <b>id</b>, which contains id of station. We will need that value as part of further requests. To receive data from this request, parse it as data frame, and (for example) select interesting place we can do those simple operations:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/1b562721e35269dd74830fad369cab33.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
We will receive following data frame:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/68f809df22e463ec62e9be8f20bc8308.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
In my example I picked station <b>AM5 Gdańsk Szadółki</b> which has id 733.</div>
<div style="text-align: center;">
<h3>
Sensors</h3>
<div style="text-align: justify;">
Since we have station id we can explore its sensors now. Overall idea of sending request and transforming its response is the same as in measurement stations example:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/19066306e3c57db0417e9067fdbfd5cc.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
As result we have neat data frame with information about sensors located in this measurement station:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/2d4ddfad2a4c302c1527547fe5ef50d5.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
We can easily see that we have 5 sensors located there. Lets then explore data from <b>PM10</b> sensor which has id 4727.</div>
<h3 style="text-align: center;">
Measured data</h3>
<div style="text-align: justify;">
Now we arrived to probably most interesting part of API. We can get actual measurement data here. As we can expect, complexity of getting this data is similar as above, with one distinction. We are receiving list of dictionaries, and each dictionary contains two pairs of key/value. So if we want nice data frame we have to add additional transformation. But fear no more - it is quite simple and gives us wanted results immediately:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/1e2611fd922c457245cb38fabb8c00db.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Example results:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/04af42a68c4cd63f0c95fd0ac1c93fd6.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
But as we can expect, there are some gotchas here. If you have "trained" eye, you probably spotted fact, that date time data is written in 12h format without AM/PM distinction. Well, this is because ... there is no such information provided in API response. I'm assuming that received data is sorted, so first 01 is one hour after midnight and second occurrence of 01 during the same date will correspond to 13 in 24h time format. For now I didn't bother to recalculate it according to above assumption - I'm hoping that this will be fixed soon so I don't have to deal with it. Second gotcha here is about range of data. Received data points are from range of three calendar days including current day, so it will contain at most 24 * 3 points. There is no way to modify that range, so if our data retrieving application crash, and we fail to notice it over three days, we will have data gap, which would not be filled until yearly data package will be released. Also, if someone is interested only in current values, he will always receive unneeded data which basically wastes bandwidth. Apart of those little flaws I didn't found other problems. Here's plot of this data:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNYKAvVmdhatweP6sRgxlX120vWEHmtn6qCNpJKeEk4HKXk2cdhg-u9Wt279rqq1FV-x4NFoQsE7gJ3N0jj2uiTZfkPi1a2AI8vQWz0KQ12OYuq6hiBhbG0kz9U4BnXJS55KkYcnGVDnnH/s1600/plot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNYKAvVmdhatweP6sRgxlX120vWEHmtn6qCNpJKeEk4HKXk2cdhg-u9Wt279rqq1FV-x4NFoQsE7gJ3N0jj2uiTZfkPi1a2AI8vQWz0KQ12OYuq6hiBhbG0kz9U4BnXJS55KkYcnGVDnnH/s400/plot.png" width="400" /></a></div>
<h3 style="text-align: center;">
Air quality index</h3>
Last data which we can get with API is current air quality. It doesn't seems to be very interesting - it just gives current air quality category for each sensor in station and overall air quality for that station. If you like to see how to access it I invite you to check my <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/03-api.ipynb" target="_blank">notebook dedicated</a> to operations with API. It also contains all mentioned API requests and data processing.</div>
<div style="text-align: center;">
<h3>
Conclusion</h3>
</div>
<div style="text-align: justify;">
It's great we can access so valuable and important data trough API. Despite its simplicity and flaws it still provides good point for analysis of current air quality situation. If I could add something to that API, I would enable modifying time frame for measurements data, so users could fill the gaps in their copies of data for different time frames analysis. If only other public government data would be so nice...</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-48190205616159610962017-04-15T12:19:00.000+02:002017-04-15T12:19:02.964+02:00TPOT - your Python Data Science Assistant<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAiudrve-y3GPR05kvIzPcSbupAkXYdbn9hfXVk7jzN-SbS49SzMPuMQFnJIaFKxeTJ5I_5pYlmIrE3FhxovIKZvw1HaLvmIICcyT1ayI0jGsYI8vzwA3XqJFsM6NkCVLXBKd1wtVWiC2F/s1600/tpot-logo.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="175" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAiudrve-y3GPR05kvIzPcSbupAkXYdbn9hfXVk7jzN-SbS49SzMPuMQFnJIaFKxeTJ5I_5pYlmIrE3FhxovIKZvw1HaLvmIICcyT1ayI0jGsYI8vzwA3XqJFsM6NkCVLXBKd1wtVWiC2F/s200/tpot-logo.jpg" width="200" /></a></div>
<div style="text-align: justify;">
While dealing with supervised learning problems in Python, such as regression and classification, we can easily pick from <a href="http://scikit-learn.org/stable/supervised_learning.html#supervised-learning" target="_blank">many algorithms</a> and use whichever we like. Each of those algorithms has parameters, so we can use them to adjust its execution to our needs. Everything is fine and we are free to tinkering with our model. But this works when we know what to do and we have experience with data we are playing with. And how to start when you have just little knowledge of your data? Here comes <a href="https://rhiever.github.io/tpot/" target="_blank">TPOT</a>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
TPOT is a Python module which can be used as stand alone application or it could be imported to Python script and used there. Its purpose is to test data processing and modeling pipelines build from various algorithms with various parameters.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So how to start using TPOT? You need to start as you usually start with building machine learning model. I decided to use Heart Disease Data Set for my example. First, loading and preparing data:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/bdc2c441d69fc808c0839a56774808c9.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Then, we need to decide how to deal with missing data. I decided to fit it with mean of column:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/99eefb894464c41c3c7bc07731cdf0c4.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Finally we can run TPOT:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/e3e3eb5d9bf93e4a7658e5de1f073096.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
As you can see, to start TPOT "evolution", you need to input numbers for <b>generations</b> and <b>population_size</b>. Generations value says how much generations will be created and population_size determines size of each generation. It means, that with generations = 5 and population_size = 50 there be 5*50 + 50 pipelines build and tested. In my case, best pipeline was:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: left;">
Best pipeline: XGBClassifier(input_matrix, XGBClassifier__learning_rate=0.01, XGBClassifier__max_depth=6, XGBClassifier__min_child_weight=2, XGBClassifier__n_estimators=100, XGBClassifier__nthread=1, XGBClassifier__subsample=0.55)</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It says that I should use XGBClassifier on input data with mentioned parameters and I should receive 0.823039215686 CV score. To see how to actually use such pipeline we can examine generated Python file:</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/09703d4980deab88a68672d34c600808.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Only action which is needed to use this generated file is to fill missing input in line 7. And voilà, we have nice (and cheap) starting point for further analysis.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com1tag:blogger.com,1999:blog-3328081692021886054.post-67876160688098487042017-04-13T19:47:00.000+02:002017-04-13T19:47:19.555+02:00Air Quality In Poland #11 - Tricity and Kashubia<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
After seeing my analysis, my friends from work said "to hell with Cracow and their smog, we would like to know whats the situation where we live". So I decided to analyze air quality in places particularly interesting for us, which are: Gdańsk, Gdynia, Sopot and Kościerzyna. In order to obtain data from those location we need to load description file and select proper measuring stations:
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/4de80f68734d842772efe48eb576ca7c.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
After picking station codes we can select interesting part of <b>bigDataFrame</b> by slicing using them: </div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/QuantumDamage/0398dbc21a62e0b10d79b8d15fed0c3f.js"></script></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Remark: As you probably saw in above line, I shifted time slice for +1 hour. I did it because I found that this is actual way notation used in data files. So midnight is counted towards previous year. It is quite important, because station names can change over year, and in this way we avoiding single measurements for some stations.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Rest of the analysis could be performed in the same way as previously.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
OK, so what is the best place to live (breath?) in da hood? The winner is <b>PmSopBitPl06</b> which is <b>Sopot</b>, ul. Bitwy pod Płowcami. Results are below:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0H6kjm_nrWmEXXv3R-SafSmdDccdnPgx0SGmN4lEX6FeMaIpYi3-CBUZYdcqRiW8FfF6JJ2Uur8Frh35LZ0besmJsYIOArWwRQQvYx2keNbFaccgh1Ray0pbIjbmBcbqXHwTaav6WCTgW/s1600/11-best.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="115" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0H6kjm_nrWmEXXv3R-SafSmdDccdnPgx0SGmN4lEX6FeMaIpYi3-CBUZYdcqRiW8FfF6JJ2Uur8Frh35LZ0besmJsYIOArWwRQQvYx2keNbFaccgh1Ray0pbIjbmBcbqXHwTaav6WCTgW/s400/11-best.png" width="400" /></a></div>
<div style="text-align: justify;">
Remark: As you can see, only 4 pollutants out of 7 are actually measured there. So we cannot be sure if this place is really best in this region.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And what is the worst place? It is <b>PmKosTargo12</b> which corresponds to <b>Kościerzyna</b>, ul. Targowa. Tabular results:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6NP8rjJ8b4ok4abBv1zif93aEh9dt4wtmvRxmkZMKlvQj-NbIhnMekOeNW5JzTIoMt9uzknMsZ76suuj4URUJ0RGjBi3SwEyr6rrxWphyphenhyphen7j_nVINAOgzh1mUREI9kyyh_tBtiSeY442Qj/s1600/11-worst.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="131" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6NP8rjJ8b4ok4abBv1zif93aEh9dt4wtmvRxmkZMKlvQj-NbIhnMekOeNW5JzTIoMt9uzknMsZ76suuj4URUJ0RGjBi3SwEyr6rrxWphyphenhyphen7j_nVINAOgzh1mUREI9kyyh_tBtiSeY442Qj/s400/11-worst.png" width="400" /></a></div>
<div style="text-align: justify;">
Results are significantly worse than in Sopot and we are measuring 6 out of 7 pollutants here.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I'm not sure how to analyze it further ... yet. But maybe later I will return here to mess with it more.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-68964138655370288262017-04-08T14:18:00.001+02:002017-04-08T14:18:23.536+02:00Air Quality In Poland #10 - And the winner is?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
So, is Rybnik really most polluted place in Poland in 2015? Maybe not. If we would like to know that answer we just need to change data frame slice from previous post
<script src="https://gist.github.com/QuantumDamage/780c770df57df1514af8fde22f00d011.js"></script>
and run the same procedures. While running rest of the previous code, I found unexpected behavior. It turns out, that some stations measured pollution levels which were below zero. Previously I assumed that data is more or less clean, so I don't have to look for such errors. But I was wrong. Luckily, it was pretty easy to fix that - I just need to put NaN everywhere where pollutant concentration is below zero. Example of fixed classification:
<script src="https://gist.github.com/QuantumDamage/4fc4006e9b1619542d5f4da912119f79.js"></script>
How to find really bad place after those changes? We need to apply proper selections and we will know it immediately:
<script src="https://gist.github.com/QuantumDamage/71e3b23be52a04371fb25553be29dd22.js"></script>
And here is the table with results:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdQP_aBMtagrQBdX-RlzJhaUNvqitGcB3FTbgjlSrweU38nZRnSEVV6aRm2YJJdwqyPg4wTSQh9o-57WuVA8wFAvDDth-lOQY-D84Hz-gzN0dR0fZABbiSegVMyqSeq1gwK7TlGfL3885i/s1600/10-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="131" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdQP_aBMtagrQBdX-RlzJhaUNvqitGcB3FTbgjlSrweU38nZRnSEVV6aRm2YJJdwqyPg4wTSQh9o-57WuVA8wFAvDDth-lOQY-D84Hz-gzN0dR0fZABbiSegVMyqSeq1gwK7TlGfL3885i/s400/10-01.png" width="400" /></a></div>
<div style="text-align: justify;">
So, from which place we have such bad results? The place is .... <b>MpKrakAlKras</b> which is 13, Aleja Zygmunta Krasińskiego, Półwsie Zwierzynieckie, Zwierzyniec, Krakow, Lesser Poland Voivodeship, 31-111, Poland. Here is the map of its neighborhood:</div>
<div style="text-align: justify;">
<iframe frameborder="0" height="350" marginheight="0" marginwidth="0" scrolling="no" src="https://www.openstreetmap.org/export/embed.html?bbox=19.92254734039307,50.055692239906854,19.929842948913578,50.05972176766064&layer=mapnik" style="border: 1px solid black;" width="425"></iframe><br /></div>
<div style="text-align: justify;">
<small><a href="https://www.openstreetmap.org/#map=17/50.05771/19.92620">View Larger Map</a></small></div>
<div style="text-align: justify;">
And what about best air quality place? I don't know. There are places which are not measuring all important pollutants. I think that I could fill missing values and then repeat those <span style="color: #0000ee;"><u>analysis</u></span>. But this is material for further blog posts ;). Stay tuned.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Repository for source code used for this analysis: <a href="https://github.com/QuantumDamage/AQIP">https://github.com/QuantumDamage/AQIP</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-6298216657795734342017-04-04T20:07:00.000+02:002017-04-04T20:07:04.082+02:00Air Quality In Poland #09 - Is it worst in Rybnik?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
As I promised in last post, I should add calculation of overall air quality for each time point. Such quality is defined by worst category of quality for each measured pollutant. In order to determine this category, we need to examine each row and put proper category based on descriptive values:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> for quality in qualities:
reducedDataFrame.loc[(reducedDataFrame[["C6H6.desc", "CO.desc", "NO2.desc", "O3.desc", "PM10.desc",
"PM25.desc", "SO2.desc"]] == quality).any(axis=1),"overall"] = quality
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It might be not optimal procedure, but it seems to be quite fast, at least on reduced data frame. Since our <b>qualities</b> are sorted, if there is worse value in following iterations, this worse value is overwriting previous value in overall column:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> qualities = sorted(descriptiveFrame.index.get_level_values(1).unique().tolist())
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
After generating additional column we need also to concatenate it with descriptive data frame</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> overall = reducedDataFrame.groupby(level="Station")["overall"].value_counts(dropna =
False).apply(lambda x: (x/float(hours))*100)
descriptiveFrame = pd.concat([descriptiveFrame, overall], axis=1)
descriptiveFrame.rename(columns={0: "overall"}, inplace=True)
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And what are the results?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> LuZarySzyman NaN NaN
1 Very good 9.601553
2 Good 57.266811
3 Moderate 26.955132
4 Sufficient 3.482133
5 Bad 0.890513
6 Very bad 0.308254
MzLegZegrzyn NaN NaN
1 Very good 1.255851
2 Good 50.941888
3 Moderate 31.693116
4 Sufficient 8.425619
5 Bad 3.950223
6 Very bad 2.580203
MzPlocKroJad NaN NaN
1 Very good 21.965978
2 Good 60.806028
3 Moderate 15.983560
4 Sufficient 0.947597
5 Bad 0.102751
6 Very bad 0.011417
OpKKozBSmial NaN NaN
1 Very good 2.922708
2 Good 54.446855
3 Moderate 30.117593
4 Sufficient 6.302089
5 Bad 4.144309
6 Very bad 2.009362
PmStaGdaLubi NaN NaN
1 Very good 43.155611
2 Good 38.075123
3 Moderate 12.204590
4 Sufficient 3.539217
5 Bad 1.758192
6 Very bad 1.164516
SlRybniBorki NaN NaN
1 Very good 1.541272
2 Good 56.444800
3 Moderate 27.662975
4 Sufficient 6.781596
5 Bad 3.242379
6 Very bad 3.014043
Name: overall, dtype: float64
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It seems that amount of <b>very bad</b> data points in Rybnik are without change. But for example OpKKozBSmial data station has 2.009362 percent of very bad data points, but individually worst pollutant there has 1.198767 percent of very bad air quality time. So it seems that other pollutants are also significant - which is true with values 0.913346 and 0.399589 there.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Next post - looking for beast air quality place in Poland. I hope that my laptop would not explode.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-61998539952225681322017-04-02T12:00:00.000+02:002017-04-02T12:00:20.698+02:00Air Quality In Poland #08 - Is it so bad near SlRybniBorki?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
In previous post we found that measurement station <b>SlRybniBorki</b> won two times in terms of extreme values across year 2015 for main pollutants. Can we tell more about actual situation there?</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
If we check <a href="http://powietrze.gios.gov.pl/pjp/home?lang=en" target="_blank">Chief Inspectorate for Environmental Protection</a> web page, we can find documentation, which says, that each of main seven pollutants can be divided into one of six categories: "Very good", "Good", "Moderate", "Sufficient", "Bad" and "Very bad". First category says that amount of pollutant is very small, last says that measurement just went above last threshold and situation is indeed very bad. To categorize each pollutant I wrote simple function (example for C6H6):</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> def C6H6qual (value):
if (value >= 0.0 and value <= 5.0):
return "1 Very good"
elif (value > 5.0 and value <= 10.0):
return "2 Good"
elif (value > 10.0 and value <= 15.0):
return "3 Moderate"
elif (value > 15.0 and value <= 20.0):
return "4 Sufficient"
elif (value > 20.0 and value <= 50.0):
return "5 Bad"
elif (value > 50.0):
return "6 Very bad"
else:
return value
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
and applied it on previously cross sectioned data frame</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> reducedDataFrame = bigDataFrame['2015-01-01 00:00:00':'2015-12-31 23:00:00'].loc[(slice(None),pollutedPlaces), :]
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In order to estimate severity of pollution I grouped data points by station name, counted occurrences of each category and then divided it by total number of measurement point times 100. After this procedure I received series of measurement stations with percentage of data points assigned to each category:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> for pollutant in bigDataFrame.columns:
reducedDataFrame[pollutant+".desc"] = reducedDataFrame[pollutant].apply(lambda x: globals()[pollutant+"qual"](x))
tmpseries = reducedDataFrame.groupby(level="Station")[pollutant+".desc"].value_counts(dropna = False).apply(lambda x: (x/float(hours))*100)
descriptiveFrame = pd.concat([descriptiveFrame, tmpseries], axis=1)
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So what are the results?</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhs8oaaMsL9lrtI30se8uX-7Gf9hiMj-OvMB5t5407Wzd5S_1uQGGy6X8I_4AtnZpiERmgWSPhR8X7VC4TA9L8mdMsrzUwqFCQ3O7dJ5ODESkqW1YQuc54HrNt6XHLTKAcK_x3yfVHKKNFa/s1600/08-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhs8oaaMsL9lrtI30se8uX-7Gf9hiMj-OvMB5t5407Wzd5S_1uQGGy6X8I_4AtnZpiERmgWSPhR8X7VC4TA9L8mdMsrzUwqFCQ3O7dJ5ODESkqW1YQuc54HrNt6XHLTKAcK_x3yfVHKKNFa/s640/08-01.png" width="377" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It looks like over <b>3%</b> of measurements of <b>PM10</b> near <b>SlRybniBorki</b> falls into worst possible category. It means that there was more or less 260 hours when no one should be in open air near that station. This is crazy. And if we check address of that station we will find <b>Rybnik, ul. Borki 37 d</b>. As you can see on the map below, there is quite big park nearby, but it doesn't help much. If I would have to put my bets on source of pollution, I would point to power plant on north of there. What is pretty sad, you can see that there is Hospital located in direct neighborhood of most polluted place in Poland. Is it sealed constantly, so patients are not loosing their health just by breathing? </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<iframe frameborder="0" height="350" marginheight="0" marginwidth="0" scrolling="no" src="https://www.openstreetmap.org/export/embed.html?bbox=18.51102948188782%2C50.10917127209365%2C18.52208018302918%2C50.113196306796105&layer=mapnik" style="border: 1px solid black;" width="425"></iframe><br />
<small><a href="https://www.openstreetmap.org/#map=17/50.11118/18.51655">View Larger Map</a></small></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And if you assume that overall quality of air equals worst quality in that station for that time, those numbers can be even worst. But that is the material for next post. Stay tuned!</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-70215932505959513032017-03-31T17:28:00.000+02:002017-03-31T17:28:45.381+02:00Bike Sharing Demand #01<div dir="ltr" style="text-align: left;" trbidi="on">
<style type="text/css">
p, li { white-space: pre-wrap; }
</style>
<br />
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
Bike sharing system is one of most cool feature of modern city. Such system allows citizens and tourists to easily rent bikes, and return them in different places that they were rented. In <a href="https://wroclawskirower.pl/en" target="_blank">Wrocław</a>, it is even free to ride for first 20 minutes, so if your route is so short, or you can spot automated bike stations in this time interval, you can use rented bikes practically for free. It is very tempting alternative to crowded public transportation or private cars on jammed roads.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
On 28-th May 2014 Kaggle started knowledge competition, in which goal was to predict bike rental number in city bike rental system. Bike system is owned by <a href="https://www.capitalbikeshare.com/" target="_blank">Capital Bikeshare</a> which describe themselves: "Capital Bikeshare puts over 3500 bicycles at your fingertips. You can choose any of the over 400 stations across Washington, D.C., Arlington, Alexandria and Fairfax, VA and Montgomery County, MD and return it to any station near your destination."</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
Problem with bike sharing system is that, it need to be filled with ready to borrow bikes. Owner of such system need to estimate demand for bikes and prepare appropriate supply for them. If there will be not enough bikes, system will generate more disappointment and won't be popular. If there will be to much unused bikes, it will generate unnecessary maintenance costs on top of initial investment cost. So it seems that finding good estimate for renting demand could improve customer satisfaction and reduce unnecessary spendings.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
How to approach such problem? Usually, first step should be dedicated to get some initial and general knowledge about available data. It is called <a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis" target="_blank">Exploratory Data Analysis</a>. I will perform EDA on data available in this competition.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzgdpQT36970En6HSU9IjRPbkrmt9G7yeX-4wm4AoEKJWl7cSzGGbHVtV0gySsJ2mjvNmYHhVsTtPbGUQZSUiVy8eADiLpPjD_YoAoBCyNfw02ID6CtNFmKYbRuxBrMQTbgxiWHiEdZ5QD/s1600/scatter-casual-registered-count.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="234" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzgdpQT36970En6HSU9IjRPbkrmt9G7yeX-4wm4AoEKJWl7cSzGGbHVtV0gySsJ2mjvNmYHhVsTtPbGUQZSUiVy8eADiLpPjD_YoAoBCyNfw02ID6CtNFmKYbRuxBrMQTbgxiWHiEdZ5QD/s320/scatter-casual-registered-count.png" width="320" /></a>As we can see, in train data we have three additional columns: {'casual', 'count', 'registered'}. Our goal its to predict 'count' value for each hour for missing days. We know that 'casual' and 'registered' should sum nicely to total 'count'. We can also observe their relations on scatter plots. 'registered' values seems to be nicely correlated with 'count', but 'casual' are also somewhat related. This plot can give idea, that instead of calculating 'count' one may calculate 'registered' and 'casual' and based on this numbers submit total 'count'.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidJ18P_y0ZYb2KiGudJB9jdjEJYP6q9PYIaHSnZ3-llCs64U-jZDn-xW4FeyKrXtsCAUC2wWrOjhZE3lJiVwzAz_3jPwvwkWLMOgzpFjGnA_mcgAVLSxQy7SmXjldpAMszl97gUbGHg6C4/s1600/histograms.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidJ18P_y0ZYb2KiGudJB9jdjEJYP6q9PYIaHSnZ3-llCs64U-jZDn-xW4FeyKrXtsCAUC2wWrOjhZE3lJiVwzAz_3jPwvwkWLMOgzpFjGnA_mcgAVLSxQy7SmXjldpAMszl97gUbGHg6C4/s320/histograms.png" width="320" /></a>Every data point is indexed by round hour in datetime format, so after splitting it to date and time components we have sixteen columns. We can easily generate histogram for each of them. By visually examining this histogram we can point to some potentially interesting features: '(a)temp', 'humidity', 'weather', 'windspeed', and 'workingday'. Are they important? We don't know that now.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEju4DBZdfYOIieZ9Lgm05VifkI_tJjcimi8YgT3JQb9Uz9Ge5nGjri63dq_nuhcf9Ln1nCwmcRIctnj91q8MzRIGHJjC8RlUTi4xGsPzWKScQJIJdYoDeCPg5YeBr1OXiioXRfdNVlg1-li/s1600/season.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEju4DBZdfYOIieZ9Lgm05VifkI_tJjcimi8YgT3JQb9Uz9Ge5nGjri63dq_nuhcf9Ln1nCwmcRIctnj91q8MzRIGHJjC8RlUTi4xGsPzWKScQJIJdYoDeCPg5YeBr1OXiioXRfdNVlg1-li/s320/season.png" width="320" /></a>We can examine more those features and pick those which have rather discrete values which unique count is less or equal 10 (my arbitrary guess). I will sum them for each hour for each unique value. Sounds complicated? Maybe plots will bring some clarity ;). First plot shows aggregation with keeping 'season' information. We have clear information that at some hours there were twice times more bike borrowing for season '3' than for season '1'. It is not so surprising if we assume that season '3' is summer. It will automatically lead to '1' being winter, and it is not true according to data description: <b>season</b> - 1 = spring, 2 = summer, 3 = fall, 4 = winter. Fascinating.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpdSeWVQ_BsWMAAYpI74ieEqvvWKwHQl6Wlfv56e7iVqpdC84zsjVShTOpr3mdazvkjkf33H7TP7kUvEanPTiZV0dagK-S8qbquE3rHo3qY4aMDVccFeT2Tw_bUhEeSsMc1S4clMguS29P/s1600/wather.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="228" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpdSeWVQ_BsWMAAYpI74ieEqvvWKwHQl6Wlfv56e7iVqpdC84zsjVShTOpr3mdazvkjkf33H7TP7kUvEanPTiZV0dagK-S8qbquE3rHo3qY4aMDVccFeT2Tw_bUhEeSsMc1S4clMguS29P/s320/wather.png" width="320" /></a><br />
Second feature which might be interesting in this analysis is 'weather'. This feature is described as: <b>weather</b> - 1: Clear, Few clouds, Partly cloudy, Partly cloudy; 2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist; 3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds; 4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog. It seems that 'weather' value corresponds with not niceness of weather conditions. Higher value - worst weather. We can see that on histogram and on aggregated hourly plot.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUTK7qWd3sWio3eJsPIcsnOzglhErLQ76IMIo5Z5BD2Lyx5dBupuDxPGx8vN-X3usizErvwshetEN1Eb0Je_6Hdql__R2H3Q7_m8QhCRobuVGjze_lspDUE20-dsXUnV-IlyzHAuIqDQHo/s1600/dayofweek.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUTK7qWd3sWio3eJsPIcsnOzglhErLQ76IMIo5Z5BD2Lyx5dBupuDxPGx8vN-X3usizErvwshetEN1Eb0Je_6Hdql__R2H3Q7_m8QhCRobuVGjze_lspDUE20-dsXUnV-IlyzHAuIqDQHo/s320/dayofweek.png" width="320" /></a> </div>
<div style="text-align: justify;">
Another interesting information might be extracted from calculated feature 'dayofweek'. For each date there was day of week calculation and its results were encoded as 0 = Monday up to 6 = Sunday. As we can see on related image, there are two different trends on bike sharing based on day of week. There are two days which build first trend. They are '5' and '6' which means 'Saturday' and 'Sunday'. In western countries those days are part of weekend and are considered as non working days for most of population. Rest of days, which are all building second trend, are considered as working days. We can easily spot peaks on working days, which are by my guess related to traveling to and from work place. In second "weekend" trend we can observe smooth curves which are probably reflecting general humans activity over weekend days.</div>
<div style="text-align: justify;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2wTr53m9t5QV3QPVhSUyDJfdNCSVX8XZag-Tz3IX2e10I045O7RM-fghJbkBzrIMOn9MlivUkV0oIheuOFt4u-3stqN267pWiDor1tl9DWJ14ACBCO7128iW4Ys2TSzCMeE7-r4Q8RLoJ/s1600/correlationagainstcount.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2wTr53m9t5QV3QPVhSUyDJfdNCSVX8XZag-Tz3IX2e10I045O7RM-fghJbkBzrIMOn9MlivUkV0oIheuOFt4u-3stqN267pWiDor1tl9DWJ14ACBCO7128iW4Ys2TSzCMeE7-r4Q8RLoJ/s320/correlationagainstcount.png" width="320" /></a></div>
<div style="text-align: justify;">
OK, it seems to be good time to examine correlations in this data set. Lets start with numerical and categorical correlations against our target feature 'count'. It is not surprising that 'registered' and 'casual' features are nicely correlated, we saw that earlier. 'atemp' and 'temp' seems to also correlate at some level. Rest of features have rather low correlation, but it seems, that there is 'humidity' among them which might also be worth of consideration for further possible investigations.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPdfXF9k8iFt75ikyshDvH8lRMkhaCwhnYgf7CZFnefJ1Y-dUlAirdOApS4Y5ImFz7XT8IEVzzL46LSAbReF9HDNmugC14TBWltZ5W6FiHxktU03OCgSiBZTX_vQ_602tkSFhR-KbWpMis/s1600/correlations.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPdfXF9k8iFt75ikyshDvH8lRMkhaCwhnYgf7CZFnefJ1Y-dUlAirdOApS4Y5ImFz7XT8IEVzzL46LSAbReF9HDNmugC14TBWltZ5W6FiHxktU03OCgSiBZTX_vQ_602tkSFhR-KbWpMis/s320/correlations.png" width="320" /></a>What are overall every to every feature correlations? We can examine it visually on correlations heatmap. On the plot, we can spot correlations between 'atemp' and 'temp'. In fact 'atemp' is defined as "feels like" temperature in Celsius. So it is subjective, but it correlates nicely with objective temperature in Celsius. Other correlation is in "month" and "season" which is not surprising at all. Similar situation we can observe with 'workingday' and 'dayofweek'.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuj_Er0PIiEHT2QGtW2Ylhbd4xErl9kk6xgLIJiOLHV1bOeoklGJf_d0LOYblvtMBJhw0pjeqq_uLpAKcsoTetNoU6k1fY8iMU9_8383KqgnsVefzYZ7LkeIj-38ygMOlOnap4eNjQOEyM/s1600/pca.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuj_Er0PIiEHT2QGtW2Ylhbd4xErl9kk6xgLIJiOLHV1bOeoklGJf_d0LOYblvtMBJhw0pjeqq_uLpAKcsoTetNoU6k1fY8iMU9_8383KqgnsVefzYZ7LkeIj-38ygMOlOnap4eNjQOEyM/s320/pca.png" width="320" /></a></div>
So how much not redundant inform<span id="goog_1251642811"></span><span id="goog_1251642812"></span>ation do we have in this data set? After removing target features, there are thirteen numerical or numerically encoded features. We can run Principal Component Analysis procedure on this dataset and calculate which resulting features cover how much of the data set. After applying cumulative sum on results we receive plot which tells us, that first 6 components after PCA explains over 99% of variance in this data set.<br />
<br />
As you can see, despite Bike Sharing Demand dataset is rather simple, it allows to do some data exploration and checking some of "common sense" guess about human behavior. Will this data work well with machine learning context? I don't know yet, but maybe I will have time to check it. If you would like to look at my step by step analysis you can check my github repository with <a href="https://github.com/QuantumDamage/bike-sharing-demand/blob/master/workspace/Exploratory%20Data%20Analysis.ipynb" target="_blank">EDA Jupyter notebook</a>. I hope you will enjoy it!</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-35394414863965952492017-03-26T13:26:00.000+02:002017-03-26T13:26:46.034+02:00Air Quality In Poland #07 - Is my data frame valid?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
OK, so we have Jupyter Notebook which handles all data preprocessing. Since it got quite large I decided to save data frame from it on disk and leave it as is. I perform further analysis on separate notebook, so whole flow will be more clean now. Both notebooks are located in the same <a href="https://github.com/QuantumDamage/AQIP/tree/master/workspace" target="_blank">workspace</a> so using them didn't change.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While working with <b>one big data frame</b> I encountered unwanted behavior of my process. Data frame which I created has 1 GB after saving to hard drive. Whole raw data I processed has 0,5 GB. So basically after my preprocessing I got additional 0,5 GB of allocated working memory. On my 4 GB ThinkPad X200s when I'm trying to read already read file, my laptop hangs. To avoid that I'm checking if maybe this variable is already created, and if it is present, I'm skipping its loading:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> if 'bigDataFrame' in globals():
print("Exist, do nothing!")
else:
print("Read data.")
bigDataFrame = pd.read_pickle("../output/bigDataFrame.pkl")
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Since I have this data frame it should be easy to recreate result of searching for most polluted stations over year 2015. And in fact it is:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> bigDataFrame['2015-01-01 00:00:00':'2015-12-31 23:00:00'].idxmax()
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
We also receiving byproduct of such search which is date and time of such extreme measurement:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijsJaTaSANh-qolRi5ebxpVAEHEHJHPmahcnavWn6SfoG5yQQ3s8H9ONm4iOv5PHw1AtMohyphenhyphenDGl92Bj_fSMYvw7bGkuTKuq-q_UZTMToO9-wJQPDmm4eEWk3ph_G1EakpTtTB-TEzHxKpn/s1600/07-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijsJaTaSANh-qolRi5ebxpVAEHEHJHPmahcnavWn6SfoG5yQQ3s8H9ONm4iOv5PHw1AtMohyphenhyphenDGl92Bj_fSMYvw7bGkuTKuq-q_UZTMToO9-wJQPDmm4eEWk3ph_G1EakpTtTB-TEzHxKpn/s1600/07-01.png" /></a></div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It looks like we have the same values as in previous approach, so I'm assuming that I merged and concatenated data frames correctly. If we would like to know actual values we can swap <b>idxmax()</b> with <b>max()</b>.<br />
<br />
It looks like this was all I prepared for in this short post. Don't get overwhelmed by your data and keep doing science! Till next post!</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-33827548807506563222017-03-24T18:00:00.000+01:002017-03-24T18:00:00.373+01:00Air Quality In Poland #06 - Big Data Frame part 2<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
Since we know how to restructure our data frames and how to concatenate them in proper way, we may start with building one big data frame. To select interesting pollutants and years of measurements we have to build two lists:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> pollutants = importantPollutants
years = sorted(list(dataFiles["year"].unique()))
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
and then we have to run two nested loops which will walk over relevant files and concatenate or merge generated data frames</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;">1: bigDataFrame = pd.DataFrame()
2: for dataYear in years:
3: print(dataYear)
4: yearDataFrame = pd.DataFrame()
5: for index, dataRow in tqdm(pollutantsYears[pollutantsYears["year"] == dataYear].iterrows(), total=len(pollutantsYears[pollutantsYears["year"] == dataYear].index)):
6: data = pd.read_excel("../input/" + dataRow["filename"] + ".xlsx", skiprows=[1,2])
7: data = data.rename(columns={"Kod stacji":"Hour"})
8:
9: year = int(dataRow["year"])
10: rng = pd.date_range(start = str(year) + '-01-01 01:00:00', end = str(year+1) + '-01-01 00:00:00', freq='H')
11:
12: # workaround for 2006_PM2.5_1g, 2012_PM10_1g, 2012_O3_1g
13: try:
14: data["Hour"] = rng
15: except ValueError:
16: print("File {} has some mess with timestamps".format(dataRow["filename"]))
17: continue
18:
19: data = data.set_index("Hour")
20: data = data.stack()
21: data = pd.DataFrame(data, columns=[dataRow["pollutant"]])
22: data.index.set_names(['Hour', 'Station'], inplace=True)
23:
24: yearDataFrame = pd.concat([yearDataFrame, data], axis=1)
25:
26: bigDataFrame = bigDataFrame.append(yearDataFrame)
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This code is more or less the same as in previous post, but you can see some differences. Line (10) is for generating time indexes for different year each time, so we don't have to care of leap year. Lines (12-17) on the other hand cover problems with <b>3</b> data files which are not starting on first hour of new year. Perhaps I will take care of them later, for now they are ignored.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Why do we need nested loops? I previous post I worked towards merging data frames containing different pollutants into one. So if we have such data frame we just need to append it to our target big data frame.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
After creating data frame with all interesting data points we should save it to disk, so later we will only have to read it in order to start analysis.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> bigDataFrame.to_pickle("../output/bigDataFrame.pkl")
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Thats all for today, thanks for reading!</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-27240604537631494412017-03-19T17:33:00.000+01:002017-03-24T17:33:06.330+01:00Air Quality In Poland #05 - Big Data Frame part 1<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
In last post I showed how to find example information - maximal values of each pollutant across year 2015. I believe that this example was good, but poorly executed. I basically iterated over relevant files and saved calculated values. I didn't saved content of files in memory, so if I would like to find minimal values I would need to execute this loop again and basically waste time.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Better approach would be to iterate over files once, and store their content in properly organized data frame. Actually, some people claim that such organizing and preprocessing of data takes up to 80% of their usual analytics process - and when you have nice and clean data frame you can start to feel like in home.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So my target now is to prepare one big data frame, which will contain all measurements for all measuring stations for all main pollutants across years 2000-2015. Since procedure to create such data frame will take some non trivial steps I decided to split it into two blog posts.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
OK, so we have to start with reading data, and renaming column which is wrongly labeled:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> data1 = pd.read_excel("../input/2015_PM10_1g.xlsx", skiprows=[1,2])
data1 = data1.rename(columns={"Kod stacji":"Hour"})
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
After reading xlsx data into pandas data frame we can observe that there is some kind of anomaly in reading datetime fields. It seems that since row 3 there is constant addition of 0.005 s to previous row. It accumulates to 43.790 s over whole file.</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjimV2ZD1O24DF31D9lY8gcNBc2Q5wKBZe8bvvc_Ceswp7dStzTH8WerIVJq0b96YnXob3z3NiQfRbCY8ZUqLb2CdMamJs3JtLZaalcsjooth-_B5oUGrXjlu04Y6NiirPeVc-CsuTWbFsW/s1600/05-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjimV2ZD1O24DF31D9lY8gcNBc2Q5wKBZe8bvvc_Ceswp7dStzTH8WerIVJq0b96YnXob3z3NiQfRbCY8ZUqLb2CdMamJs3JtLZaalcsjooth-_B5oUGrXjlu04Y6NiirPeVc-CsuTWbFsW/s1600/05-01.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It looks like Microsoft has used own timestamp method across xlsx files which is different from common Unix timestamp. There are probably methods for dealing with it, but I decided to recreate this index by hand:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> rng = pd.date_range(start = '2015-01-01 01:00:00', end = '2016-01-01 00:00:00', freq='H')
data1["Hour"] = rng
data1 = data1.set_index("Hour")
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Now we have nice and clean data frame with measurements for one pollutant. How to merge it with other pollutants data frames? Answer for that question was probably most difficult answer for that problem so far. As you can see in raw xlsx files, each measurement is located in three dimensional space. First dimension is "pollutant" and we can get it from filename. Second dimension is "date and time" and we can get it from spreadsheet index. Third dimension is "measuring station" and it is located in spreadsheet columns. Since target data frame is two dimensional, I had to decide if I want multilevel columns or multilevel index, and where to put each dimension. "Date and time" is obvious pick for index, since it is natural way to analyze instances with such index. Next I had to pick one or more "features". I plan to work on "main pollutants" only, so it seems to be good pick for features/columns. "Measuring station" was what was left. Such stations are build and decommissioned at different times, so I decided to treat them as additional "spacial" level of index, so even if station was working for some weeks/months it will generate less "NaN" cells than when it would be treaded as column level. I hope that this make sense and would not make problems in future. What is most funny, to do that I just need to use <b>stack()</b> function, recreate data frame from series and add proper multiindex names:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> data1 = data1.stack()
data1 = pd.DataFrame(data1, columns=["PM10"])
data1.index.set_names(['Hour', 'Station'], inplace=True)
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Only programmers know the pain of hours of thinking projected to few lines of code. Thanks god no one is paying me for produced lines of code. Not that anyone is paying me anything for this analysis ;). If we do the same transformations by hand for other pollutant and create second data frame, we can easily merge them (<b>dataMerged = pd.concat([data1, data2], axis=1)</b>) and receive (head of) foundations for target data frame:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOm8OEsyyHmDUAC9jcaaq2bljA_itESjI9MLj4iRMDmVxcM2444CpUldxgJewSBDkYqJpXskf2G_m437uuvo98Cl8iRANK2BssCWRE5ZSkYNBcv00Bal1uL2zzYGaB5yuA5Nvb1glhCLDp/s1600/05-02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOm8OEsyyHmDUAC9jcaaq2bljA_itESjI9MLj4iRMDmVxcM2444CpUldxgJewSBDkYqJpXskf2G_m437uuvo98Cl8iRANK2BssCWRE5ZSkYNBcv00Bal1uL2zzYGaB5yuA5Nvb1glhCLDp/s1600/05-02.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Thats all for today. In next post I will try to wrap this code into iterators for years and pollutants, so hopefully after running them I will receive my target big data frame. Thanks.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-41366756232403746962017-03-15T18:42:00.000+01:002017-03-24T17:32:32.302+01:00Air Quality In Poland #04<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
Hey! Welcome in 4-th post dedicated to messing with air quality in Poland data. In last post we prepared data files for easy manipulation. It is time now, for actually analyze something.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For fist analysis I decided to look for most extreme values for each of most important pollutants across year 2015. So basically I will look for dead zones across Poland ;). And what are those <b>important pollutants</b>? In Poland, air quality is calculated as worst class of classes among "PM10", "PM25", "O3", "NO2", "SO2", "C6H6", "CO".</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In order to find such places we need to locate proper files. To do that we need simple selection from data frame:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> importantPollutants = ["PM10", "PM25", "O3", "NO2", "SO2", "C6H6", "CO"]
pollutants2015 = dataFiles[(dataFiles["year"] == "2015") & (dataFiles["resolution"] == "1g") &
(dataFiles["pollutant"].isin(importantPollutants))]
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
which will give us much smaller data frame with list of interesting files:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVkOyCVcpU-REjGK3ucR3R9kRJhZ4dKS1lRFgwbXStSBnwlzxisuvP3EE2LV002w6sZRlaHzTCF4_R3Z_PvmTnCP4oIGy4IAZr8ewzyrqs6c0po2DVi1X75Reexssaisqz-h5IMzf05ZLA/s1600/04-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVkOyCVcpU-REjGK3ucR3R9kRJhZ4dKS1lRFgwbXStSBnwlzxisuvP3EE2LV002w6sZRlaHzTCF4_R3Z_PvmTnCP4oIGy4IAZr8ewzyrqs6c0po2DVi1X75Reexssaisqz-h5IMzf05ZLA/s1600/04-01.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Since we have relevant list of files, we can write simple (and probably not super efficient) loop over them, which will find maximum vale of each pollutant and corresponding measurement station. It will look like that:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> worstStation = {}
for index, dataRow in tqdm(pollutants2015.iterrows(), total=len(pollutants2015.index)):
dataFromFile = pd.read_excel("../input/" + dataRow["filename"] + ".xlsx", skiprows=[1,2])
dataFromFile = dataFromFile.rename(columns={"Kod stacji":"Godzina"})
dataFromFile = dataFromFile.set_index("Godzina")
worstStation[dataRow["pollutant"]] = dataFromFile.max().sort_values(ascending = False).index[0]
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This loop is taking 2 minutes on my ThinkPad X200s and produces only dictionary with pollutants as keys and codenames of stations as values. We may easily count values occurrences and see worst "dead zone":</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> Counter({u'LuZarySzyman': 1,
u'MzLegZegrzyn': 1,
u'MzPlocKroJad': 1,
u'OpKKozBSmial': 1,
u'PmStaGdaLubi': 1,
u'SlRybniBorki': 2})
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Since "SlRybniBorki" doesn't says much, we must consult "Metadane_wer20160914.xlsx" file which allows us to decode this station as "Rybnik, ul. Borki 37 d". Whoever lives there, I feel sorry for you!</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Thats all for today, thanks for reading! ;) </div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-10257982771991757162017-03-12T15:24:00.000+01:002017-03-24T17:32:10.379+01:00Air Quality In Poland #03<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
We have now nice data frame which contains list of data files and descriptions of content in them. It looks like that (first 5 rows):</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr9y2-Sz7t2a8Y96xHYXHOYb0SvfyfYdM7pnV83XVAnMZKJmanSRm89XUMgWNvfqVSl28CXmAuNZL5lBSW6X_9gD6ludUwls3GhJvQe02__9EZcNtafpS_iqmtMks_mq8Csx6sJLdQkkil/s1600/03-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr9y2-Sz7t2a8Y96xHYXHOYb0SvfyfYdM7pnV83XVAnMZKJmanSRm89XUMgWNvfqVSl28CXmAuNZL5lBSW6X_9gD6ludUwls3GhJvQe02__9EZcNtafpS_iqmtMks_mq8Csx6sJLdQkkil/s1600/03-01.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
We can now easily check how much data for each year we have,</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgErhjOhEv6sBscACJWj86LzW6kvNjhe9WgmcA1V_K3Z31OLt2fww8OxYsbar5f6uezE1KV_CmjLfU6vWnVBPdtlJzdx3zxcVu1Hti7YSS7Zm_syVJnCeKBbiYDZsd7fb6_TrLcBjkJlteG/s1600/03-02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgErhjOhEv6sBscACJWj86LzW6kvNjhe9WgmcA1V_K3Z31OLt2fww8OxYsbar5f6uezE1KV_CmjLfU6vWnVBPdtlJzdx3zxcVu1Hti7YSS7Zm_syVJnCeKBbiYDZsd7fb6_TrLcBjkJlteG/s1600/03-02.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
what pollutants were measured</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUTqReIQJVq5kCtViX_I_oUK3agqz7O-eIGI63rpf_iPaVIUJ9C_5lMOeWV2smqMpICFlPw8Hqmue-qs8E5kHAIRmcNe1QjrVx5PgFvXZrmLjGVg4vPdkzHTsGcQ_J_VSoUcDi8CLHtrX1/s1600/03-03.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUTqReIQJVq5kCtViX_I_oUK3agqz7O-eIGI63rpf_iPaVIUJ9C_5lMOeWV2smqMpICFlPw8Hqmue-qs8E5kHAIRmcNe1QjrVx5PgFvXZrmLjGVg4vPdkzHTsGcQ_J_VSoUcDi8CLHtrX1/s1600/03-03.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
and how much files is available for each resolution:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiThgDguurJTI8ROYVM1_wVOnXSWyPqf9aSFwn47jGgMb0zRMANYrV26eeCe1UrJ5yZqCS-WBB8BgZB79Rqg0bgStPwmLvcCVNiKnbrjpLk1CAIi5cDexSngl3UmNREYFCA-TnS594Na8L9/s1600/03-04.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiThgDguurJTI8ROYVM1_wVOnXSWyPqf9aSFwn47jGgMb0zRMANYrV26eeCe1UrJ5yZqCS-WBB8BgZB79Rqg0bgStPwmLvcCVNiKnbrjpLk1CAIi5cDexSngl3UmNREYFCA-TnS594Na8L9/s1600/03-04.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As we can see, this is the place where something isn't exactly as it supposed to be. 9 files have some mess within resolution column. To fix that, we need to find rows with invalid <b>resolution</b> and replace <b>pollutant</b> and <b>resolution </b>values by hand in them:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<pre style="background: rgb(240, 240, 240) none repeat scroll 0% 0%; border: 1px dashed rgb(204, 204, 204); color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 99%;"><code style="color: black; word-wrap: normal;"> dataFiles.ix[dataFiles["resolution"] == "(PM2.5)_24g", 'pollutant'] = "SO42_(PM2.5)"
dataFiles.ix[dataFiles["resolution"] == "(PM2.5)_24g", 'resolution'] = "24g"
</code></pre>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
After figuring proper name for all messed files (details in <a href="https://github.com/QuantumDamage/AQIP/blob/master/workspace/EDA.ipynb" target="_blank">notebook on github</a>) we can check overall status of files data frame by issuing <b>dataFiles.describe()</b>:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpEfDMSJzKXmwqExIitrT8GXZfyIX7tpsQw1S1JLDgfdw7XQ5abcV8rmGVzlGuVAVyFa4VuFuuBlPopO9SKhOknwIkP7ceB38mn18yLHg2rBUSWm0bM3Z3VQcF4OkF74UqG2pzu0VfNp1A/s1600/03-05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpEfDMSJzKXmwqExIitrT8GXZfyIX7tpsQw1S1JLDgfdw7XQ5abcV8rmGVzlGuVAVyFa4VuFuuBlPopO9SKhOknwIkP7ceB38mn18yLHg2rBUSWm0bM3Z3VQcF4OkF74UqG2pzu0VfNp1A/s1600/03-05.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As we can see cont value in <b>resolution</b> column doesn't sum to target 359 values. It is because there is one data file called "2015_depozycja" which has data about <a href="https://simple.wikipedia.org/wiki/Deposition" target="_blank">deposition</a> experiments which are out of my scope for now. I decided to remove this row from data frame.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So now we have clean data frame with filenames and file contents description in separate columns. Thanks to this, we will be able to easily access needed data and use it for further analysis.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-7586434736665404032017-03-07T19:50:00.000+01:002017-03-24T17:31:38.916+01:00Air Quality In Poland #02<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
In order to do analysis, we have to obtain relevelant data. In Poland, best place for air data is on official website of <a href="http://powietrze.gios.gov.pl/pjp/home?lang=en" target="_blank">Chief Inspectorate for Environmental Protection</a>. This inspectorate is main and official Polish government agency responsible for measuring and analyzing changes in natural environment. This agency is providing <a href="http://powietrze.gios.gov.pl/pjp/archives" target="_blank">packages witch archival measurement data</a>. Currently those packages covers years range 2000-2015. They are not updated in real time, but it seems that this will be enough for my analysis.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Before reproducing research, everyone should check if available data is exactly the same as originally used data. For this purpose I will calculate md5sums of downloaded files and archives. I should calculate sum for every file inside archives, but I believe that decompressing them without errors should be enough to assume that we are working on the same files. Heres my list:</div>
<div style="text-align: justify;">
<br /></div>
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> $ md5sum *
b6f86aec3bee46d87db95f0e5e93ea70 2000.zip
a7c045e40179b297c282d745d9cbc053 2001.zip
ba05c06c7a2681f1aaa54c6e9dd88a34 2002.zip
3f4215d89a64a5a6e52b205eec848a83 2003.zip
4053dcc35f228bd8233eb308d67f2995 2004.zip
9e23571c25bf8bb6ad77fc006007a047 2005.zip
b37ff6a8f0d12539a8d026b882ecbb49 2006.zip
5fe5b74264d1d301190446ed13b5ffa0 2007.zip
d63f9e4fcc9672b1136eb54188e12d2f 2008.zip
b437a9d17e774671a334796789489d9f 2009.zip
3a3cd0db3d14501d07db5f882225d584 2010.zip
d0e0e19f7517ed0b1a67260e9840bd89 2011.zip
58ebcdd2c36c5ef0f7117a42e648822a 2012.zip
36eefbd5ae62651807fa108c64ac155e 2013.zip
47836093ac1d4aa1b71edc6964a53a3c 2014.zip
4030e4d5b1e5ba6c1c5876b89b7aaa55 2015.zip
71665e79bf0a6a2f3765b0fcbb424b70 Kopia Kody_stacji_pomiarowych.xlsx
b7ff94632d6c60842980ea882ae1b091 Metadane_wer20160914.xlsx
bfa2680d5fbb08f9067f467c8a864235 Statystyki_2000-2015_wer20160914.xlsx
</code></pre>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
After obtaining the same files we can start unzipping into <b>input</b> directory, which is located on the same level as <b>workspace</b> directory. I'm not including input data files because of following reasons: 1) I'm not sure if license allows redistributing those files. It might be that only valid way to obtain them is through mentioned website. 2) Those files have some significant size - they have 490 MB unpackaged. It would be much waste of transfer if anyone interested only in source code would have to download them. 3) Those files are xlsx, which are binary. It is not good practice to put binary files into source code version control system.<br />
<br />
So what data do we have exactly? After unpacking all zip files, we should obtain 359 files with data and 3 additional files which were previously unpacked. Data files have following naming convention "xx_yy_zz.xlsx". xx means year. We have data from 2000-2015, so we expect number in this range in first filename section. yy part is responsible for pollutant, for example it might be "NO2". Last part (zz) tells us about measurement value averaging - "1g" means, that data was averaged over one hour for each hour, and "24g" means that data was averaged over 24 hours once for each day.<br />
<br />
To read all filenames I run double for loop:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> filenames = [ os.path.splitext(wholeFilename)[0] for wholeFilename in
[ basename(wholePath) for wholePath in glob.glob("../input/2*.xlsx") ] ]
</code></pre>
<br />
After creating list with data filenames I'm building data frame with filename and columns responsible for year, pollutant and resolution<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> dataFiles = pd.DataFrame({"filename": filenames})
dataFiles["year"], dataFiles["pollutant"], dataFiles["resolution"] = dataFiles["filename"].str.split('_', 2).str
</code></pre>
<br />
Since we created data frame with columns which are describing file contents, we can easily access data which will be interesting in future measurements.<br />
<br />
But as usually when dealing with data, there is additional problem with this approach. I will describe it in next post. Stay tuned.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com2tag:blogger.com,1999:blog-3328081692021886054.post-57571332804446939022017-03-04T12:00:00.000+01:002017-03-24T17:31:01.775+01:00Air Quality In Poland #01<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<style type="text/css">
p, li { white-space: pre-wra</style>Dear reader! Welcome to my first post tagged "Get Noticed 2017!". As you can see, this is not first post on this blog. But as you can also see, I'm not publishing regularly. This is my attempt to change it. By change it, I mean to publish at least two posts weekly. One post dedicated to my open source project and other post related to IT. By doing it, I will fulfill requirements of competition "<a href="http://devstyle.pl/daj-sie-poznac/" target="_blank">Get Noticed!</a>". So I will try to kill two birds with one stone. Or something like that. I was inspired to compete in "Get Noticed!" by <a href="http://www.michalgellert.pl/blog/" target="_blank">Michał</a> and encouraged by <a href="http://blog.wurbanski.me/" target="_blank">Wojtek</a> which are also competing in it. We will see how will it roll.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><img border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbDuQh67bt0sOyNj1jdKmKvgZPmUiVUpb4Yxz-mAE9RFNAYKbRiY-_WkSzVfVJjMQeySXc8L8TP8ifCnEfXYOR5BqzNaekwvqRC-piBgU8qFsM7ZjXCP2Dsy9coMncGJs0Jkml29BiKTNH/s200/Zrzut+ekranu+z+2017-02-09+19%253A44%253A55.png" width="200" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">It looks like there is difference<br />
in air quality between two<br />
central points ...</td></tr>
</tbody></table>
<div style="text-align: justify;">
What open source project I will develop here? Since there were no restrictions about technology, programming language, topics and purposes, I decided to perform exploratory data analysis on air quality data of Poland. What exactly do I meant by that? I would like to build Jupyter Notebook in which I will do step by step research analysis. My goal will be to build usable analysis which will be worthy, scientifically correct and engineeringly valid. And it will be fully reproducible of course!</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhESPrUIjCrjHSEcXIfjRkIChfjp9K1vp_U-6ywL5g3OjihI92RH_zUb2nYs3OICAfzxTicRBgpw0Utq5W0CXwze7iEX1IZpzaEUNEhkthQjoWy3tFUOjrxsW600pTdhr5eyRNcn5MoERjH/s1600/Zrzut+ekranu+z+2017-02-09+19%253A45%253A02.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhESPrUIjCrjHSEcXIfjRkIChfjp9K1vp_U-6ywL5g3OjihI92RH_zUb2nYs3OICAfzxTicRBgpw0Utq5W0CXwze7iEX1IZpzaEUNEhkthQjoWy3tFUOjrxsW600pTdhr5eyRNcn5MoERjH/s200/Zrzut+ekranu+z+2017-02-09+19%253A45%253A02.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">... but the yellow point show<br />
only data for PM10 ...</td></tr>
</tbody></table>
My first step will be toward downloading and preparing <a href="http://powietrze.gios.gov.pl/pjp/archives" target="_blank">GIOŚ</a> air pollution data. I will try to estimate missing values, so every point on map will have various pollutant estimates. Then I will work on visualizing those estimates. Next step will be dedicated to gathering and discussing various facts related to performed analysis. When I will be able to complete mentioned steps I will try to build predictive model for predicting best time for physical outdoor activities, for various places in Poland. So those are my initial ideas for project development.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijUEGZ5hHPpftvUp44nk2D1O-eWh0pn4NjmIGQw0f_ZhJ3u7sDaiex1AD8SluYCzbDD2rdxDPae3mh9IJ8BuLIOvkcg-cbo9VUsnFnYk01IOWDpqT5SGUYmiYM_C0RygIDwgO65P3B6-5r/s1600/Zrzut+ekranu+z+2017-02-09+19%253A45%253A09.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijUEGZ5hHPpftvUp44nk2D1O-eWh0pn4NjmIGQw0f_ZhJ3u7sDaiex1AD8SluYCzbDD2rdxDPae3mh9IJ8BuLIOvkcg-cbo9VUsnFnYk01IOWDpqT5SGUYmiYM_C0RygIDwgO65P3B6-5r/s200/Zrzut+ekranu+z+2017-02-09+19%253A45%253A09.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">... and orange one shows also<br />
PM 2.5. Is air really better in<br />
yellow one?</td></tr>
</tbody></table>
Which technologies will I use? I'm big fan of Python, so I will use it exclusively. I'm thinking about using <a href="http://pandas.pydata.org/" target="_blank">Pandas</a>, <a href="http://www.numpy.org/" target="_blank">NumPy</a>, <a href="http://matplotlib.org/" target="_blank">Matplotlib</a> and <a href="https://python-visualization.github.io/folium/" target="_blank">Folium</a> modules, but this list will probably change over time. My product will have form of <a href="https://jupyter.org/" target="_blank">Jupyter</a> Notebook, but I may not restrict myself to only one Notebook. If my work will be effective I might consider building standalone Python scripts to perform some parts of analysis. Time will show what ideas will pop up.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
What was my motivation to pick up such idea for project? Recently in Poland we had some discussions about poor air quality around couple of big cities known of heavy industrial profile. I found that some persons are getting wrong conclusions about data points. Worst case was when reporter was comparing two not so distant points and air quality index (AQI) in them. One point was "safe" and other "unsafe", and that was clearly highlighted by reported. But the problem with "safe" point was that measuring station weren't measuring all pollutants there, so system probably filled missing values with zeros. This case leads to situation when someone think that he is chilling in "safe" air quality zone, while actually he knows nothing about it. This problem pushed me to thinking about better way to estimate air quality in points where there are no direct measurements available. I don't have any experience with working with air quality and geospatial data but I think I will be able to perform at least some basic analysis and produce some useful pieces of code.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
I also hope that participating in "Get Noticed 2017!" competition will give me much fun and possibly some feedback related to my work. Michal is very enthusiastic about it, so I need at least try to maintain development and writing momentum and see what I will build. Last but not least: source code for my project is located <a href="https://github.com/QuantumDamage/AQIP" target="_blank">here</a>.</div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
<br /></div>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
Ideas to implement:</div>
<ul style="text-align: justify;">
<li>Fill missing values for measuring stations</li>
<li>Interpolate pollutants values over Poland</li>
</ul>
<div style="text-align: justify;">
Random ideas:</div>
<ul style="text-align: justify;">
<li>Scrap current data from GIOS server</li>
</ul>
<div style="margin: 0px; text-align: justify; text-indent: 0px;">
Ideas graveyard:</div>
<ul style="text-align: justify;">
<li>Build mobile application based on my work</li>
</ul>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-53037765370788601692016-04-22T21:00:00.000+02:002016-04-22T21:00:18.542+02:00AIAR Weekly #20<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
Annoucement:<br />
This is last, and as you can see, incomplete issue of AIAR Weekly. AIAR Weekly was very interesting experiment, and based on results from it, I decided to suspend it without due date. Thanks you very much for reading my materials and following news from world of artificial intelligence and robotics! I hope that my work was usable to someone. See you soon on my other projects and experiments!<br />
<br />
AIAR Weekly</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div style="text-align: center;">
Issue #20</div>
<div style="text-align: center;">
22.04.2016<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /><div style="text-align: justify;">
Articles and videos:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://scs.ryerson.ca/~aharley/vis/conv/" target="_blank">[AI] 3D Visualization of a Convolutional Neural Network</a> - Neat real time 3D visualization of CNN. You can adjust your handwriting and see how it is affecting results from neural network.</li>
<li style="text-align: justify;"><a href="https://www.youtube.com/playlist?list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal" target="_blank">[AI] Machine Learning Recipes with Josh Gordon</a> - Google just started short video tutorials series dedicated to ML. There are two videos for now based on scikit, but they promise harder and harder topics and usage of tensor flow in further materials. Nice place to start with machine learning.</li>
<li style="text-align: justify;"><a href="https://medium.com/the-bleeding-edge/a-dummy-s-guide-to-deep-learning-part-3-of-3-642ef2c3e26d#.8v494ia0z" target="_blank">[AI] A dummy’s guide to Deep Learning (part 3 of 3)</a> - Third and last part in simple introduction to deep learning. This time, author prepared some hands on examples for playing with MNIST data set.</li>
</ul>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Crowdfunding:</div>
<ul style="text-align: justify;">
<li><a href="https://www.kickstarter.com/projects/1818505613/codeybot-new-robot-who-teaches-coding?ref=nav_search" target="_blank">[R] Codeybot: New Robot Who Teaches Coding</a> - Codeybot is a simple robot which is designed to combine free play fun with teaching to program actual physical device. It looks very modern and based on presentation it gives positive look and feel. </li>
</ul>
<ul style="text-align: left;">
</ul>
</div>
<div style="text-align: justify;">
Book of the week:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://amzn.to/22C2cLG" target="_blank">[AI] Artificial Intelligence for Humans, Volume 3: Deep Learning and Neural Networks</a> - Third book in a series dedicated to teaching different artificial intelligence methods. This time, author focuses on explaining neural networks and deep learning possibilities.</li>
</ul>
<div style="text-align: justify;">
Courses:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://www.coursera.org/learn/ml-regression" target="_blank">[AI] Machine Learning: Regression</a> - "In this course, you will explore regularized linear regression models for the task of prediction and feature selection. You will be able to handle very large sets of features and select between models of various complexity. You will also analyze the impact of aspects of your data -- such as outliers -- on your selected models and predictions. To fit these models, you will implement optimization algorithms that scale to large datasets."</li>
</ul>
<div>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Jobs:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://jobs.lever.co/cruise/9561e23e-7b7c-4bf9-ba47-1111ec1a36c5" target="_blank">[AI] Computer Vision / Machine Learning Engineer</a> @ Cruise - Cruise looking for skilled engineers for developing autonomous car systems. If you have some experience in machine learning, this might be good option for you. <b>Location</b>: San Francisco, USA. <b>Tags</b>: python, c++, theano, caffe, computer-vision.</li>
</ul>
<div style="text-align: justify;">
Kudos:</div>
<div style="text-align: center;">
Michał Neonek, MrValgad, Tompul, Magdalena, Mucha</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Appendix: </div>
<div style="text-align: justify;">
Do you have link to cool news, article, tutorial or video and want to share with other robot/AI fans? Send it to me and if meet quality standards I will include it in next issue of AIAR Weekly.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
Don't forget to <a href="http://temuju.blogspot.com/p/subscribe-me.html" target="_blank">subscribe AIAR Weekly!</a><br />
You can <a href="http://temuju.blogspot.com/p/sponsorship-by-sponsoring-newsletter-we.html" target="_blank">sponsor</a> this magazine also through Patreon.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<a href="http://temuju.blogspot.com/search/label/AIAR%20Weekly" target="_blank">Archive</a><br />
<br />
License: <a href="https://creativecommons.org/licenses/by-nc-sa/3.0/" target="_blank">CC BY-NC-SA 3.0</a></div>
<ul style="text-align: left;">
</ul>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-83768328756131285022016-04-15T21:00:00.000+02:002016-04-15T21:00:02.361+02:00AIAR Weekly #19<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
AIAR Weekly</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div style="text-align: center;">
Issue #19</div>
<div style="text-align: center;">
15.04.2016<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-mBm4rzUyhieTFoHiNdcz3s4pr3aZ6pad4ozd5B6eFBMbOdNleJI7YeEIg4Ulszt2-hg3c4izUBB3g25Jz_JVvEFdQJ3rLTDtC3i4-859TP3KPSZhWUOVhI81JlEj52hSfdnkSYLplzIC/s1600/schaft-robot-two-legs-21-1000x400.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-mBm4rzUyhieTFoHiNdcz3s4pr3aZ6pad4ozd5B6eFBMbOdNleJI7YeEIg4Ulszt2-hg3c4izUBB3g25Jz_JVvEFdQJ3rLTDtC3i4-859TP3KPSZhWUOVhI81JlEj52hSfdnkSYLplzIC/s200/schaft-robot-two-legs-21-1000x400.jpg" title="Image credit: Schaft/mehdi_san/YouTube" width="200" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
Featured material:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://singularityhub.com/2016/04/12/watch-google-x-unleash-awesome-two-legged-robot-on-tokyo/" target="_blank">[R] Watch Google X Unleash Awesome Two-Legged Robot on Tokyo</a> - Since beginning of Robotics Weekly (now called Artificial Intelligence and Robotics Weekly), when I was presenting robots, very often those robots were human inspired. Newest robot from Google X is slightly different. Basically, it is "just" legs with some electronics and batteries as a center of mass between them. But surprisingly, it behaves and walks quite effortless. The problem for now is that, it doesn't have any hands. Bud does it need them?</li>
</ul>
<div style="text-align: justify;">
Articles and videos:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://www.huffingtonpost.com/quora/the-pitfalls-of-deep-lear_b_9662160.html" target="_blank">[AI] The Pitfalls of Deep Learning</a> - What shortcomings do you see with deep learning? This question was originally asked on Quora, and Oren Etzioni tries to explain her view point on current limitations of Deep Learning.</li>
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=HW8oQbLKAUM" target="_blank">[R] What can we learn from Robot Athletes?</a> - How to stimulate research and development in robotics? Design decathlon competitions and let people challenge them with their creations. With this approach you can develop really robust and unbreakable systems. TED talk by Jacky Baltes.</li>
<li style="text-align: justify;"><a href="https://medium.com/the-bleeding-edge/a-dummy-s-guide-to-deep-learning-part-2-of-3-e28c109fc30f#.letgpzb97" target="_blank">[AI] A dummy’s guide to Deep Learning (part 2 of 3)</a> - Continuation of simple course which has to give better view on Deep Learning. </li>
<li style="text-align: justify;"><a href="http://futurism.com/images/quest-lifelike-robots/?src=home" target="_blank">[R] The Quest For Lifelike robots</a> - Never too much of good infographics. This time authors from futurism.com developed infographics dedicated to robots which are designed to look as much alike humans as possible. Worth to check.</li>
<li style="text-align: justify;"><a href="http://playground.tensorflow.org/" target="_blank">[AI] Neural Network Playground</a> - Ever wanted to check out those neural networks but you were afraid of math? Fear no more! On this website, you can easily set-up simple neural network and solve simple problem with it immediately. You can also observe its progress with every iteration. Cool stuff.</li>
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=9K46YrbpDyQ" target="_blank">[R] How babies can inspire us to build intelligent robot</a> - In this TED talk, Alex Pitti presents his view on idea, that to really develop intelligence, you should give it body, and some senses, so it can learn and perceive surrounding world as average toddler can.</li>
<li style="text-align: justify;"><a href="http://googleresearch.blogspot.com/2016/04/announcing-tensorflow-08-now-with.html" target="_blank">[AI] Announcing TensorFlow 0.8</a> - Latest release of open source deep learning framework adds support for distributed computing. Thanks to it, some models can be easily trained in hours vs in weeks if you only have enough computational power.</li>
<li style="text-align: justify;"><a href="https://medium.com/@willjack/deep-learning-the-truth-behind-the-hype-7872d8aa49b9#.2j6v3e3cw" target="_blank">[AI] Deep learning: the truth behind the hype</a> - If you are not too eager to jump into deep learning hype train, this article might reinforce your feelings. It also shows advantages and disadvantages of deep learning, and some ideas how to combine it with other techniques in the future.</li>
</ul>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Crowdfunding:</div>
<ul style="text-align: justify;">
<li><a href="https://www.kickstarter.com/projects/988476124/purplekit?ref=nav_search" target="_blank">[R] PurpleKit</a> - PurpleKit is all-in-one box, aimed to help amateur tinkers with collecting their supplies for projects. It contains various pieces of aluminum, bolts, screws and similar stuff. I assume, based on interest, that every tinker want rather to gather such components on his/her own, than just buy box with most popular parts. I'm wrong maybe on this?</li>
</ul>
<ul style="text-align: left;">
</ul>
</div>
<div style="text-align: justify;">
Book of the week:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://amzn.to/1Nl3sfb" target="_blank">[AI] Artificial Intelligence for Humans, Volume 2: Nature-Inspired Algorithms</a> - Second book in a series. This time, author focuses on algorithms closely related to biology and nature in general, for example: genetic mutation and cellular automats.</li>
</ul>
<div style="text-align: justify;">
Courses:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://www.coursera.org/learn/ml-foundations" target="_blank">[AI] Machine Learning Foundations: A Case Study Approach</a> - "In this course, you will get hands-on experience with machine learning from a series of practical case-studies. At the end of the first course you will have studied how to predict house prices based on house-level features, analyze sentiment from user reviews, retrieve documents of interest, recommend products, and search for images. Through hands-on practice with these use cases, you will be able to apply machine learning methods in a wide range of domains."</li>
</ul>
<div>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Jobs:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://careers-appliedminds.icims.com/jobs/1245/robotics-software-engineer/job" target="_blank">[R] Robotics Software Engineer</a> @ Applied Minds - Applied Minds is a company which provides various consulting solutions for its clients. They aim to build interdisciplinary teams which work together towards finding best solution for targeted problems. They also looking for robotics specialist this time. <b>Location</b>: Burbank, USA. <b>Tags</b>: python, c++, ros, opencv, slam.</li>
</ul>
<div style="text-align: justify;">
Humor:</div>
<ul style="text-align: justify;">
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=8AKZk2ldPWs" target="_blank">[R] Simone Giertz's Popcorn Machine!</a> - Totally must have device for next season of Game of Thrones!</li>
</ul>
<div style="text-align: justify;">
Kudos:</div>
<div style="text-align: center;">
Michał Neonek, MrValgad, Tompul, Magdalena, Mucha</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Appendix: </div>
<div style="text-align: justify;">
Do you have link to cool news, article, tutorial or video and want to share with other robot/AI fans? Send it to me and if meet quality standards I will include it in next issue of AIAR Weekly.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
Don't forget to <a href="http://temuju.blogspot.com/p/subscribe-me.html" target="_blank">subscribe AIAR Weekly!</a><br />
You can <a href="http://temuju.blogspot.com/p/sponsorship-by-sponsoring-newsletter-we.html" target="_blank">sponsor</a> this magazine also through Patreon.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<a href="http://temuju.blogspot.com/search/label/AIAR%20Weekly" target="_blank">Archive</a><br />
<br />
License: <a href="https://creativecommons.org/licenses/by-nc-sa/3.0/" target="_blank">CC BY-NC-SA 3.0</a></div>
<ul style="text-align: left;">
</ul>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-32247224977106982652016-04-08T15:23:00.000+02:002016-04-08T19:25:55.080+02:00AIAR Weekly #18<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
AIAR Weekly</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div style="text-align: center;">
Issue #18</div>
<div style="text-align: center;">
08.04.2016<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZpCBczW5wqKG3q3DPwEXI9W6mFRR_jqLTo6G4m98NWVOpL8_UvMi2MyVrQKH6QZdJ7hNms7QmfUwLafYmEetVhnn67qHa0nPQ08dKF9ERF3g9XUtk0-DBMD4QQ2ikAnfdh9IfzKALpKn9/s1600/1*WDh2Pua8RqxT-OIHmyoRHQ.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="113" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZpCBczW5wqKG3q3DPwEXI9W6mFRR_jqLTo6G4m98NWVOpL8_UvMi2MyVrQKH6QZdJ7hNms7QmfUwLafYmEetVhnn67qHa0nPQ08dKF9ERF3g9XUtk0-DBMD4QQ2ikAnfdh9IfzKALpKn9/s200/1*WDh2Pua8RqxT-OIHmyoRHQ.gif" title="Image: https://www.kickstarter.com/projects/scanse/sweep-scanning-lidar" width="200" /></a> <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
Featured material:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://medium.com/@comet_labs/engineer-explains-lidar-748f9ba0c404#.uskitfihc" target="_blank">[R] Engineer Explains: Lidar</a> - Very short but complete article about concept of lidar. Must read material for "seeing" robotics newbies ;)</li>
</ul>
<div style="text-align: justify;">
Articles and videos:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=6Viwwetf0gU&feature=youtu.be" target="_blank">[R] Killerdrone</a> - Some time ago someone attached a gun to a drone. And it was pretty dangerous proof of concept. This time, two Finnish farmers attached something less lethal but way much scary. You have to see it for yourself.</li>
<li style="text-align: justify;"><a href="https://medium.com/the-bleeding-edge/a-dummy-s-guide-to-deep-learning-part-1-of-3-ea8ae8d93e2a#.l5lsgdps4" target="_blank">[AI] A dummy’s guide to Deep Learning (part 1 of 3)</a> - First part of this guide wouldn't give you hands on snippets and source code, but "just" tell you some basic information about deep learning. Author promises that in next parts there will be more technical materials. Lets see. </li>
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=2Y5eC9Vp5Do" target="_blank">[R] Could a Robot Be a Bona Fide Hero?</a> - TEDx talk by Prof. Selmer Bringsjord. In this talk, he presents his view on heroic actions done by robots, and he discusses problems with moral classifications of heroic and civic actions.</li>
<li style="text-align: justify;"><a href="http://futurism.com/delivery-drones-used-rwanda-ferry-medical-supplies/" target="_blank">[R] Delivery Drones To Be Used in Rwanda to Ferry Medical Supplies</a> - I believe that this was just matter of time, but somehow that time was too long in my opinion. Finally someone will test transportation of medical supplies by drones. I hope that this will be great success!</li>
<li style="text-align: justify;"><a href="https://cds.cern.ch/record/2142287" target="_blank">[AI] Deep Learning and the Future of AI</a> - Fresh talk from CERN. Yann LeCun discusses recent breakthroughs in AI research and tries to predict future :).</li>
<li style="text-align: justify;"><a href="http://robohub.org/robots-hadrian-bricklaying-robot/" target="_blank">[R] Robots Podcast #205: Hadrian Bricklaying Robot, with Mark Pivac</a> - Would you like to build whole house from bricks just in two days? Soon, thanks to this Australian company, you will be able. Maybe not personally but with help from their pretty sophisticated and neat machine ;).</li>
</ul>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Crowdfunding:</div>
<ul style="text-align: justify;">
<li><a href="https://www.kickstarter.com/projects/georjaiwin/jet-pack?ref=nav_search" target="_blank">[R] JetPack - Bluetooth Shield for Arduino Robots</a> - There is too much Arduino shields already... said no one ever! This time shield will have Bluetooth module for wireless communication and motor controller. Fundraising is already successful, but I still recommend it.</li>
</ul>
<ul style="text-align: left;">
</ul>
</div>
<div style="text-align: justify;">
Book of the week:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://amzn.to/1RxlDDq" target="_blank">[AI] Artificial Intelligence for Humans, Volume 1: Fundamental Algorithms</a> - Volume 1 of Artificial Intelligence for Humans series is effect of Kickstarter campaign. Jeff Heaton selects some starting level and fundamental algorithms related to machine learning and artificial intelligence and presents them in very simple way. He also provides source code for examples in most popular languages. I didn't had a chance to read this book, but for that price it looks like nice starting point for further education in AI.<br /> </li>
</ul>
<div style="text-align: justify;">
Courses:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://lagunita.stanford.edu/courses/SelfPaced/Haptics/2014/about" target="_blank">[R] Introduction to Haptics</a> - "Participants in this class will learn how to build, program, and control haptic devices, which are mechatronic devices that allow users to feel virtual or remote environments. In the process, participants will gain an appreciation for the capabilities and limitations of human touch, develop an intuitive connection between equations that describe physical interactions and how they feel, and gain practical interdisciplinary engineering skills related to robotics, mechanical engineering, electrical engineering, bioengineering, and computer science."</li>
</ul>
<div>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Jobs:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://www.bosch.us/content/language1/html/13485.htm" target="_blank">[R] Autonomous Driving Software Developer</a> @ Bosch - When I'm thinking about Bosch I'm usually thinning about car related hardware or tools used in building construction. But it seems that Bosch also invests in autonomous cars technology. If you are robotics nerd, you can add Bosch to your interesting company lists. <b>Location</b>: Palo Alto, USA. <b>Tags</b>: c++, ros, linux, python, algorithms.</li>
</ul>
<div style="text-align: justify;">
Humor:</div>
<ul style="text-align: justify;">
<li style="text-align: justify;">Sadly I didn't found nothing funny this time ;(</li>
</ul>
<div style="text-align: justify;">
Kudos:</div>
<div style="text-align: center;">
Michał Neonek, MrValgad, Tompul, Magdalena, Mucha</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Appendix: </div>
<div style="text-align: justify;">
Do you have link to cool news, article, tutorial or video and want to share with other robot/AI fans? Send it to me and if meet quality standards I will include it in next issue of AIAR Weekly.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
Don't forget to <a href="http://temuju.blogspot.com/p/subscribe-me.html" target="_blank">subscribe AIAR Weekly!</a><br />
You can <a href="http://temuju.blogspot.com/p/sponsorship-by-sponsoring-newsletter-we.html" target="_blank">sponsor</a> this magazine also through Patreon.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<a href="http://temuju.blogspot.com/search/label/AIAR%20Weekly" target="_blank">Archive</a><br />
<br />
License: <a href="https://creativecommons.org/licenses/by-nc-sa/3.0/" target="_blank">CC BY-NC-SA 3.0</a></div>
<ul style="text-align: left;">
</ul>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-10899189418342986312016-04-01T21:00:00.000+02:002016-04-01T21:00:06.134+02:00AIAR Weekly #17<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
AIAR Weekly</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div style="text-align: center;">
Issue #17</div>
<div style="text-align: center;">
01.04.2016<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHPA-rVdtOHfr1tcBoy8Azhg2zKMjtI92-cx1tlsi2dyQMMxLhHPdes4CsLlHLJhUs5qjkh7PO7VQ1-F9jOKzLkfjIcsFEoJPnjAKA1RXfLpLH8vvaEeR8PPzn8qjt0hLyOvmRNxA_LTlE/s1600/header_ESSAY-USE-v2-GS3522999.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHPA-rVdtOHfr1tcBoy8Azhg2zKMjtI92-cx1tlsi2dyQMMxLhHPdes4CsLlHLJhUs5qjkh7PO7VQ1-F9jOKzLkfjIcsFEoJPnjAKA1RXfLpLH8vvaEeR8PPzn8qjt0hLyOvmRNxA_LTlE/s200/header_ESSAY-USE-v2-GS3522999.jpg" title="Image: aeon.co" width="200" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
Featured material:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://aeon.co/essays/intelligent-machines-might-want-to-become-biological-again" target="_blank">[AI] Where do minds belong?</a> - Essay about idea that
mechanical/electronic intelligence may be not optimal in terms of energy
consumption and maybe after its developments it will go back to
biological mechanisms like human brain. Quite interesting view on
potential evolutionary path. </li>
</ul>
<div style="text-align: justify;">
Articles and videos:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=D6qMh1quGwo" target="_blank">[R] Past, present, future of surgical robotics for fracture surgery</a> - I'm not a big fan of surgeries. Actually, I'm not sure if anyone is a fan of surgeries. In general, fracture surgeries are quite complicated and not funny for patients and surgeons. In this TED video Sanja Dogramadzi demonstrates her approach to tackling this problem with robotics. Their prototype is not ready to perform surgeries yet, but it looks very promising and who knows, maybe it will operate you ;).</li>
<li style="text-align: justify;"><a href="http://www.digitaltrends.com/cool-tech/japanese-ai-writes-novel-passes-first-round-nationanl-literary-prize/" target="_blank">[AI] A Japanese AI program just wrote a short novel, and it almost won a literary prize</a> - Did you ever felt proud after writing sort or maybe even long fiction text? Did you felt that this text was so abstract that only you could write it in such form? Well, it seems that soon artificial intelligence writers might start to write their own novels and they will be at least as good as those written by humans. No job is secure now?</li>
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=ZrU2bClkYQ8" target="_blank">[AI] Artificial Intelligence & Music as a Communications Medium</a> - Not every TED talk starts with drum solo. This time, Sean Holden explains his views on idea that human level artificial intelligence is just around corner, and why he believe it is not a case yet.</li>
<li style="text-align: justify;"><a href="https://medium.com/life-learning/the-near-future-of-ai-the-road-to-super-intelligent-apps-and-machines-7fb8b71acf2f#.sgenzhwbl" target="_blank">[AI] The Near Future of AI: The Road to Super Intelligent Apps and Machines</a> - What could our smartphone do, if we harness AI into applications? Standard risks still applies.</li>
</ul>
<div style="text-align: justify;">
Crowdfunding:</div>
<ul style="text-align: justify;">
<li><a href="https://www.kickstarter.com/projects/primotoys/cubetto-hands-on-coding-for-girls-and-boys-aged-3?ref=hero" target="_blank">[R] Cubetto</a> - Cubetto creators have very high ambitions. They want to start teaching programming three years old kids, with robots. And with such cute wooden robot and simple brick programming board it seems as quite achievable goal. Definitely worth checking.</li>
</ul>
<ul style="text-align: left;">
</ul>
</div>
<div style="text-align: justify;">
Book of the week:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://amzn.to/1UrB7v7" target="_blank">[R] Robotics: Everything You Need to Know About Robotics From Beginner to Expert (Robotics Mastery, Robotics 101)</a> - Very short "book" which should get you familiar with basic concepts of robotics.</li>
</ul>
<div style="text-align: justify;">
Courses:</div>
<ul style="text-align: left;">
<li style="text-align: justify;">Nothing interesting this time ;(.</li>
</ul>
<div>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Jobs:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://jobs.lever.co/skydio/657fc242-62b2-4d5c-a8f2-da8a17049ecd" target="_blank">[AI] Software Engineer - Computer Vision</a> @ Skydio - In <a href="http://temuju.blogspot.nl/2016/03/aiar-weekly-16.html" target="_blank">previous issue</a> of AIAR Weekly I presented product from Skydio team - autonomous and very sophisticated drone. If you are interested in designing computer vision system which can be used in such devices - Skydio team might be perfect place for you. <b>Location</b>: Redwood City, USA. <b>Tags</b>: slam, c++, deep-learning, neural-networks, software-design.</li>
</ul>
<div style="text-align: justify;">
Humor:</div>
<ul style="text-align: justify;">
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=LSZPNwZex9s&feature=youtu.be" target="_blank">[R] Introducing the self-driving bicycle in the Netherlands</a> - Where can I buy it?</li>
</ul>
<div style="text-align: justify;">
Kudos:</div>
<div style="text-align: center;">
Michał Neonek, MrValgad, Tompul, Magdalena, Mucha</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Appendix: </div>
<div style="text-align: justify;">
Do you have link to cool news, article, tutorial or video and want to share with other robot/AI fans? Send it to me and if meet quality standards I will include it in next issue of AIAR Weekly.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
Don't forget to <a href="http://temuju.blogspot.com/p/subscribe-me.html" target="_blank">subscribe AIAR Weekly!</a><br />
You can <a href="http://temuju.blogspot.com/p/sponsorship-by-sponsoring-newsletter-we.html" target="_blank">sponsor</a> this magazine also through Patreon.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<a href="http://temuju.blogspot.com/search/label/AIAR%20Weekly" target="_blank">Archive</a><br />
<br />
License: <a href="https://creativecommons.org/licenses/by-nc-sa/3.0/" target="_blank">CC BY-NC-SA 3.0</a></div>
<ul style="text-align: left;">
</ul>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0tag:blogger.com,1999:blog-3328081692021886054.post-42222144245625460762016-03-25T21:00:00.000+01:002016-03-25T21:00:15.868+01:00AIAR Weekly #16<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: center;">
AIAR Weekly</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div style="text-align: center;">
Issue #16</div>
<div style="text-align: center;">
25.03.2016<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLXIB39A4zPKBdUgqofZx0-oJbezMGdhh_9LWSkE1e2IqvciLyhRBfqbQvQphB0pWVVl1cRkz-TiEA3E3R5Avex4TXtVZsQheC61AZzHaFwtq5_-JwA_mIXn2jfMw7e0KqI9lMJMoacws6/s1600/image00.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLXIB39A4zPKBdUgqofZx0-oJbezMGdhh_9LWSkE1e2IqvciLyhRBfqbQvQphB0pWVVl1cRkz-TiEA3E3R5Avex4TXtVZsQheC61AZzHaFwtq5_-JwA_mIXn2jfMw7e0KqI9lMJMoacws6/s200/image00.png" title="Image: Google" width="200" /></a> <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
Featured material:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://cloudplatform.googleblog.com/2016/03/Google-takes-Cloud-Machine-Learning-service-mainstream.html" target="_blank">[AI] Google Cloud Machine Learning</a> - Ever wanted to build business based on powerful machine learning tools and maybe even deep learning techniques, but infrastructure hold you up? Now you can use almost the same infrastructure as Google, based on offering from Google. In their service, you should be able to easily use tensor flow or other frameworks and integrate them nicely with other Google services. It seems that only sky (and wallet) is the limit now.</li>
</ul>
<div style="text-align: justify;">
Articles and videos:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://www.wired.com/2016/03/boeings-monstrous-underwater-robot-can-wander-ocean-6-months/" target="_blank">[R] Boeing’s Monstrous Underwater Robot Can Wander the Ocean for 6 Months</a> - Underwater robots is a topic that I'm trying to represent here as often as possible. This time, you can read article about newest product from Boeing company. And no, it isn't a airplane. ;)</li>
<li style="text-align: justify;"><a href="https://medium.com/singularity-university-blog/scientific-research-in-the-age-of-artificial-intelligence-ec0216be718c#.d6x7xmrdd" target="_blank">[AI] Scientific Research in the Age of Artificial Intelligence</a> - If you are doing research, you often struggle with manual matching publications and other materials. Some cool information are scattered on the edge on your scope and looks seemingly unrelated. What about building AI designed to visually connect those publications and present them to you in a way that immediately shows their relations. Maria Ritola and her team build such tool and called it Iris. I wonder how doing scientific research will look in the future. </li>
<li style="text-align: justify;"><a href="http://spectrum.ieee.org/automaton/robotics/drones/skydio-camera-drone-autonomous-flying" target="_blank">[R] Skydio's Camera Drone Finally Delivers on Autonomous Flying Promises</a> - Yet another drone which promises possibility to follow moving target and simultaneously avoid different emerging obstacles. But this time, creators of Skydio claims that their drone not only safely avoid obstacles but also maps its surroundings and dynamically calculates optimum path to travel. But still no actual product to buy on market.</li>
<li style="text-align: justify;"><a href="http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit/" target="_blank">[AI] Microsoft deletes 'teen girl' AI after it became a Hitler-loving sex robot within 24 hours</a> - Lets develop Twitter bot which will be AI and have ability to learn from its discussion partners. What can go wrong? Well... it seems that in case of "Tay" wrong things escalated pretty quick. I believe that this is another hint about developing safety circuits into every general purpose artificial intelligence.</li>
<li style="text-align: justify;"><a href="http://robohub.org/robots-satellite-assembly-in-space/" target="_blank">[R] Robots Podcast #204: Satellite Assembly in Space, with John Lymer</a> - This time John Lymer tells a story about his experience in designing and building space robots. He also explains general idea of building precise and agile robots designed to fix, operate and deploy satellites in space. Quite nice bit of space robotics.</li>
<li style="text-align: justify;"><a href="http://singularityhub.com/2016/03/23/6-reasons-why-industry-needs-to-be-agile-as-software-to-survive/" target="_blank">[R] 6 Reasons Why Industry Needs to Be Agile as Software to Survive</a> - Article about general trends in modern industry. Topics like 3d printing, robotics, sensors networks and automation are discussed there.</li>
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=-EYOSnn7-_Y" target="_blank">[AI] What if we could be inspired by AI?</a> - In this TEDx talk, Alex Berman and Valencia James presents combination of dance and artificial intelligence with glitches. My art sense is probably too limited to fully enjoy this presentation. Can you tell me meaning of it?</li>
</ul>
<div style="text-align: justify;">
Crowdfunding:</div>
<ul style="text-align: justify;">
<li><a href="https://www.kickstarter.com/projects/frobotics/zeroborg-robotics-for-the-raspberry-pi-zero?ref=nav_search" target="_blank">[R] ZeroBorg</a> - Since Raspberry Pi Zero is on the market for some time, it is time that we can expect Zero size add-on boards flooding DIY online shops. This time, PiBorg group is crowdfunding their latest motor controller board for Raspberry Pi Zero. They seems to be experienced with this type of add-ons and price also seems to be reasonable.</li>
</ul>
<ul style="text-align: left;">
</ul>
</div>
<div style="text-align: justify;">
Book of the week:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="http://amzn.to/25gj5iX" target="_blank">[AI] Python Machine Learning</a> - This time I would like to point you to something more practical - machine learning done with Python programming language and its tools. This book is quite fresh and has lot of positive reviews.</li>
</ul>
<div style="text-align: justify;">
Courses:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://www.open2study.com/courses/mobile-robotics" target="_blank">[R] Mobile Robotics</a> - "What will I learn? What is, and what is not a robot – and more specifically, a mobile robot. Why we need robots. What subsystems robots are made up of. Different ways that mobile robots can move themselves around, and which are most suitable for different environments. How a variety of sensors receive information about the environment around then. Ways to classify sensors: proprioceptive vs exteroceptive; active vs passive. How a feedback system works. That robots follow logical sequential instructions in order to function. To create basic flow diagrams and pseudo code to program what a robot will do. How to develop a list of design requirements for a robotic system. How to design, implement and troubleshoot a robotic system."</li>
</ul>
<div>
<ul style="text-align: left;">
</ul>
<div style="text-align: justify;">
Jobs:</div>
<ul style="text-align: left;">
<li style="text-align: justify;"><a href="https://www.piborg.org/careers" target="_blank">[R] Electronics Design Engineer</a> <complete id="goog_1117696181">@ PiBorg - If you liked board from this issue of AIAR Weekly Crowdfunding link and you would like to work with PiBorg team on designing their future inventions, this job might fit you perfectly. <b>Location</b>: Near St.Ives, Cambridgeshire, UK. <b>Tags</b>: circuit-design, soldering, python, linux, raspberry-pi.</complete></li>
</ul>
<div style="text-align: justify;">
Humor:</div>
<ul style="text-align: justify;">
<li style="text-align: justify;"><a href="https://www.youtube.com/watch?v=ub6mA2m3rq4" target="_blank">[R] Sock Removal Robot</a> - [gore warning] Ha, I bet everyone would like to have such invention in his/her home!</li>
</ul>
<div style="text-align: justify;">
Kudos:</div>
<div style="text-align: center;">
Michał Neonek, MrValgad, Tompul, Magdalena, Mucha</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
Appendix: </div>
<div style="text-align: justify;">
Do you have link to cool news, article, tutorial or video and want to share with other robot/AI fans? Send it to me and if meet quality standards I will include it in next issue of AIAR Weekly.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
Don't forget to <a href="http://temuju.blogspot.com/p/subscribe-me.html" target="_blank">subscribe AIAR Weekly!</a><br />
You can <a href="http://temuju.blogspot.com/p/sponsorship-by-sponsoring-newsletter-we.html" target="_blank">sponsor</a> this magazine also through Patreon.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<a href="http://temuju.blogspot.com/search/label/AIAR%20Weekly" target="_blank">Archive</a><br />
<br />
License: <a href="https://creativecommons.org/licenses/by-nc-sa/3.0/" target="_blank">CC BY-NC-SA 3.0</a></div>
<ul style="text-align: left;">
</ul>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/00240328826912976646noreply@blogger.com0