Tech tip of the day for those who have to deal with CSV files

[Klaus]

I discovered these life-changing utilities today:
https://csvkit.readthedocs.io/en/latest/

Instead of going through the intense pain of importing tables into something like Excel, they allow you to just SQL-query your data on the command line instead.

For instance, if we have data about exam results in a CSV file and I want a histogram of the result distribution, I just do

> csvsql -I --table grades examresults.csv --query 'SELECT Grade,COUNT(*) AS Count FROM grades GROUP BY Grade'
Grade,Count
"1,0",3
"1,7",13
"2,0",24
"2,3",25
"2,7",42
"3,0",53
"3,3",53
"3,7",81
"4,0",32
"5,0",132

Ever tried something like that in Excel? It's a PITA.

I can't believe I've managed to live my life until this point without this tool.

[Horace]

That would come in handy if I could craft SQL queries off the top of my head like that.

[mark]

[Klaus]

@Horace said in Tech tip of the day for those who have to deal with CSV files:

That would come in handy if I could craft SQL queries off the top of my head like that.

Believe it or not, but mastering SQL is a tiny task compared to mastering Excel. Also, using SQL is already the advanced use case. Many of the tools work out-of-the-box without the need for SQL.

Their true power comes from their composability, thanks to the Unix philosophy. Ken Thompson and Dennis Ritchie should have been award the Nobel peace prize or something.

[Axtremus]

“csvsql” … nice, and it doesn’t look that hard to program too, amazing it’s not already more popular.

[taiwan_girl]

ISnt that a US pharmacy? LOL

[George K]

@taiwan_girl said in Tech tip of the day for those who have to deal with CSV files:

ISnt that a US pharmacy? LOL

"CVS"

[taiwan_girl]

@George-K