data.table

The data.table package may also make a huge difference in performance, and often simplifies the code as well https://github.com/Rdatatable/data.table

data.table is basically a highly optimized C library https://github.com/Rdatatable/data.table

R's come a long way in the last decade. The tibble and data.table packages both address this issue. data.table (https://github.com/Rdatatable/data.table) is the more strongly-typed of the two, by default it fails loudly when it encounters data that doesn't conform to the column type. It's also quite fast--binds to C code that parallelizes with OpenMP. It has very terse and expressive syntax, I find it so much more intuitive and easy to work with than pandas. If you're happy with Python, by all means keep using it. I use both languages. Just suggesting that if you gave up on R that long ago, you might be pleasantly surprised by how much better it's gotten since then.

Almost always faster, actually. https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-G...

Have you seen the R package data.table (https://github.com/rdatatable/data.table)? Not sure whether data.table is in the same domain as xsv, and certainly a lot of it is written in C. But for comparison's sake: fread("cities.csv") 1.30 s And then the rest of the computations will be faster of course: count -- 0.005 ms freq -- 0.2 s sort -- 0.1 s It's so useful that I often just use csvs between 10 and 100GB as a database as the difference in performance between fread and a 'proper' database aren't enough to justify the latter.

As with Python, the fast libraries written for R are usually implemented with something else under the hood. Take the data.table library, for example: https://github.com/Rdatatable/data.table It's wicked fast for many kinds of tasks, but its R API is just a thin layer on top of C.

I'd say native data frames in R aren't great at these (maybe #4, maybe #9). I'm excited to see how Arrow can perform, and hopefully we'll see solid bindings to R as well. The data.table package (https://github.com/Rdatatable/data.table/wiki) does make progress on some of these - I'd say #1, #3, maybe #7, #8. Dplyr has a query planner too, fwiw.

"Did you mean..." or "this is a bad idea because..." type notes in error/warning messages are underrated teaching tools. The R package "data.table" does a good job at this. Just scroll through https://github.com/Rdatatable/data.table/blob/master/R/data.... and search for "warning(".

I went from barely using data.table to only using data.table for basically everything in less than a few years. I think this is the trend given it's faster than basically everything: https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A...

And when the data gets bigger, there's data.table[1], which performs amazingly well at certain tasks (vectorized ops ftw!), though the syntax can get a little clunky (if you squint at it hard, it's SQL-ish). On my 2012 macbook pro, I'm able to do (some) transformations of tables containing 10s of millions of rows in only a few seconds (and sometimes faster). It's possible to use dplyr and data.table together, as well, to good effect[2]. [1] https://github.com/Rdatatable/data.table/wiki [&] https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A... [2] http://stackoverflow.com/questions/21435339/data-table-vs-dp... [&] https://twitter.com/hadleywickham/status/553169339751215104

data.table and dplyr take data frames up a notch! https://github.com/Rdatatable/data.table/wiki/Getting-starte... https://cran.rstudio.com/web/packages/dplyr/vignettes/introd... xts is quite useful as well, when you need to do modeling/analysis in an especially time-oriented manner: http://joshuaulrich.github.io/xts/

data.table's `DT[i, j, by]` is quite consistent actually and is comparable to SQL's - i = where, j = select | update and by = group by. This form is always intact. For example: require(data.table) DT = data.table(x=c(3:7), y=1:5, z=c(1,2,1,1,2)) DT[x >= 5, mean(y), by=z] ## calculates mean of y while grouped by z on ## rows where x >= 5 DT[x >= 5, y := cumsum(y), by=z] ## updates y in-place with it's cumulative sum ## while grouped by z on rows where x >= 5 "Harder to read after you've written it" and "harder to learn" are all very subjective and pointless. One could make very similar observations about `dplyr`, but I'll refrain from it here. I implore the readers to take a look at over 100+ reviews on crantastic: http://crantastic.org/packages/dat

Here's a benchmark Matt recently did comparing data.table, dplyr and pandas on 50GB and 100GB: https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A...

We're in the process of adding more detailed vignettes illustrating more clearly the philosophy behind data.table's `i, j, by`. Should make things lot easier for beginners - https://github.com/Rdatatable/data.table/issues/944 data.table is not just fast, but is also more memory efficient - we want to highlight in the vignettes as well. And timings are quite relevant even on 10 million rows: https://gist.github.com/arunsrinivasan/db6e1ce05227f120a2c9 For much larger data, check the project page: https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A...