Schedule stream recordings from the command line

Schedule stream recordings from the command line

Micz Flor | Sourcefabric Blog | Feb 10 2014

Note: the original article was published on our Sourcefabric blog, celebrating the UNESCO World Radio Day 2014. It was split into two parts.

“Radio continues to evolve in the digital age.” says UNESCO on their website celebrating World Radio Day. It’s also the most inviting medium when it comes to tinkering with technology. With World Radio Day 2014 coming up on February 13, I would like to share a hack I developed to help radio-makers discover and utilize rare content.

I’ve loved radio ever since I built my first crystal radio receiver as a kid with my dad. Radio has changed a lot since then, so has the way I listen. I used to do a lot of taping at home. Now I’m a podcast addict, and for me that involves hunting for and collecting spoken-word radio jewels that are not included in any podcast.

To do this, I wanted a script to schedule recordings for web streams, one that could run on my server and not my desktop machine (because I don’t want my desktop switched on all the time). The result is streamplan.sh, a bash script that can be started from the command line.

I wrote this post in two parts, stay tuned for part two in the coming days:

Which application to use for recording live streams, and why
How to configure and schedule stream recording with streamplan.sh

Which application to use for recording live streams, and why

The first step in building a scheduler for stream recording is researching what recorder to use. This post is dedicated to the comparison of various options. The next post will go into more detail regarding the configuration and scheduling part. The recording app would need to…

… Be controlled from the command line. I want this app to run on my server and that I access through the terminal.
… Transcode files. Streams come in different formats like mp3, ogg, windows media and more. I want to transcode all into mp3 or ogg.
… Be told to stop after a certain time. The player should kill itself and stop recording. Otherwise the hard disk would fill up in case an external kill signal didn’t come.
… Handle constant and variable bitrate, because both systems are used in popular streams.
… Read playlists as well as mount points, because many links to live streams are actually links to playlists. More about this in relation to avconv / ffmpeg below.

Many media players can record streams, and there are a number of applications dedicated to stream ripping. Based on the criteria above, I shortlisted a couple:

Streamripper: Recording tracks like sliced bread

Streamripper converts music streams into files

Streamripper is great when it comes to converting music streams into single files. It records the stream, chops it up, changes the ID3 tags according to the stream information, and saves individual files in a separate folder. Since I mainly record larger shows in one piece, I added options on the command line to not add a directory for each stream (-s) and rip to a single file (-a followed by the filename) into a specified directory (-d followed by the directory) and it would terminate automatically after a given time (-l followed by the duration in seconds):

streamripper http://link.to/stream -d /path/to/ -a localfile.mp3 -s -l duration

But Streamripper did not work for some mp3 streams. I assume it cannot handle variable bitrate and/or not extract mount points from playlist; the Streamripper tutorial mentions this. When I tried to record mp3 streams, it worked with an NPR stream, but not with Dradio.de which is encoded in a variable bitrate.

MPlayer: The player that records

MPlayer can capture live streams of media

I had used MPlayer before but I didn’t know it is also very good for grabbing live streams. Simply add the option -dumpstream, tell it what file to record to (-dumpfile followed by the filename) and the stream to listen to (-playlist followed by the stream url):

mplayer -dumpstream -dumpfile "/path/to/localfile.mp3" -playlist "http://link.to/stream"

To the best of my knowledge, transcoding happens automatically. The file ending of the dumpfile will specify what format to transcode to (e.g. .mp3).

But that the way in which Mplayer can be told to stop after a certain time (using the option -endpos) works only for static files, not for streams — which is what this blog post is about.

Avconv (aka ffmpeg): The Swiss army knife with an Achilles heel

Avconv transcodes multimedia files from the command line

Avconv is a command line tool for transcoding multimedia files. It is part of a fork of the widely known ffmpeg. When I began doing this research, I was certain that I would end up working with avconv. And if you take a look at their documentation page, you get the impression that avconv can do pretty much everything.

But I immediately ran into problems recording my sample streams. It took me some time to understand what was actually happening (or rather not happening). Many links to web streams are actually links to playlist files, not to the mount point of the stream itself. To give you an example, try running these two commands in your terminal window:

wget http://www.dradio.de/streaming/dlf_hq_ogg.m3u
cat dlf_hq_ogg.m3u

The first line retrieves the content of the file, which is the link to the live stream of dradio.de. You might expect this to be the audio stream, but it is not. It is actually a text file that contains a single line - which points to another URL. You can see this URL using the second command which spits out the content of the linked file, the link to the actual stream.

Try again with another example, NPR:

wget http://www.npr.org/streams/mp3/nprlive24.pls
cat nprlive24.pls

This file contains metadata and a link to the actual stream in row three:

[playlist]
NumberOfEntries=1
File1=http://nprdmp.ic.llnwd.net/stream/nprdmp_live01_mp3
Title1=NPR 24 Hour Program Stream

There are many reasons for using a playlist file instead of pointing directly to the stream. It allows you to change the streaming server in the background without having to change the links to the stream. You can also play one or more static files (e.g. advertisement or jingles) before the live stream starts. And, as is the case in the NPR example, you can add metadata for the player. You can also include a fallback stream as the second URL in the playlist.

Most players will take the file, read it and then start playing the content. Not avconv. It requires the actual stream and cannot interpret playlists.

VLC : My choice for recording live streams

VLC can be used to rip and transcode files from various formats

Everybody seems to know the VLC player. Many people also use it to rip media and some to transcode files from one format to another. VLC does everything on my checklist, but there is one little problem.

VLC runs from the command line with the command “cvlc”. The player transcodes files, it also handles constant and variable bitrates. VLC also correctly reads and interprets playlists, the one feature that avconv fell short on.

But you will remember that I needed process that could kill itself before filling up the hard disk. And while VLC can do this, it behaved strangely when I tested it.

The following command was suggested in the VLC forum to stop the recording automatically after $LENGTH seconds:

cvlc "http://link.to/stream" --sout file/mp3:/path/to/localfile.mp3 --run-time $LENGTH --stop-time=$LENGTH vlc://quit

I tried it in different combinations, using only run-time or only stop-time or both, but it didn’t work as I expected. Instead, the command runs twice and then stops. For example, I want to record for one hour ($LENGTH in seconds = 3600) at 8pm. Acting on the command above, VLC records from 8 to 9pm, then stops. Then it starts recording again, from 9 to 10pm, overwriting the first recording.

Is it just me? If you know what I am doing wrong here, please comment. I am happy to learn.

Now the good news: VLC does everything else that I want. It even kills itself after a certain time, but in an unexpected way, which means I need to come up with a work-around.

How to configure and schedule stream recording

I will now go into more detail on how best to configure VLC to do the job of recording and transcoding streams. I will also explain a simple but efficient way to schedule a recording on Linux using the “at” command. I finish this post by putting it all together in a bash script that I frequently use to schedule and record shows from web radio streams: streamplan.sh

Install VLC

sudo apt-get install vlc
sudo apt-get install libavcodec-extra-53

The second package is suggested by VLC for transcoding, which I will discuss later.

Using VLC to record mp3 streams

Now let’s record some streams. The first example is the German talk radio using a variable bitrate, the second is the stream from NPR using a constant bitrate. Both streams will record 20 seconds worth of audio.

cvlc "http://www.deutschlandradio.de/streaming/dkultur.m3u" --sout file/mp3:DLF-test.mp3
cvlc "http://www.npr.org/streams/mp3/nprlive24.pls" --sout file/mp3:NPR-test.mp3

What happens? To run VLC on the command line on Linux you use “cvlc”. This is followed by the stream URL. And following “–sout” we specify that we are recording a file in mp3 with the filename given after “:”.

You can use the absolute path along with the filename and the file will be written to that location on your machine, given VLC has the rights to write the file.

To stop the recording process, press “Ctrl+C” in the terminal window where VLC is running.

Make VLC stop automatically after recording

As I mentioned in my previous post, the ways in which VLC can be told to stop after a certain number of seconds look good in writing, but behave strangely in action. The command line help lists two options to terminate VLC with parameters from the command line:

--run-time=/ The stream will run this duration (in seconds).
--stop-time= / The stream will stop at this position (in seconds).

While “run-time” is meant to kill VLC after running for a given number of seconds, “stop-time” looks into the file VLC is playing and stops at a position of the file given in seconds.

Dealing with a continuous live stream (and not a file with a beginning and an end), it seems that “run-time” is the command to use. But it turns out that all players I looked at had issues with stopping the process automatically; this goes for VLC as well.

After digging through support requests, tutorials and discussions, I concluded that the only bullet-proof way to terminate VLC meant using both commands.

The following examples will record 20 seconds worth of audio and then automatically stop.

cvlc "http://www.deutschlandradio.de/streaming/dkultur.m3u" --sout file/mp3:DLF-test.mp3 --run-time=20 --stop-time=20 vlc://quit
cvlc "http://www.npr.org/streams/mp3/nprlive24.pls" --sout file/mp3:NPR-test.mp3 --run-time=20 --stop-time=20 vlc://quit

Look in the folder and you will see two files with very different sizes. NPR is about 80 kB in size, DLF about 320 kB. This is due to the compression of the stream. DLF streams in stereo with 128 kbps and a sample rate of 44100 Hz while NPR streams in mono with 32 kbps and a sample rate of 22050 Hz.

In the previous post I mentioned that stopping VLC after a given duration gets unexpected results. The two lines listed above are examples of this strange behaviour. While the second command line records the NPR stream as expected and quits after 20 seconds, the first line command line records 20 seconds of the DLF stream and then records 20 seconds again, overwriting the original stream. I haven’t tested this properly, but I blame the variable bitrate (DLF). We will deal with this issue later.

Schedule your stream recording with the “at” command

Now that we have a command line that records a stream and stops after a given duration, we can schedule our first recording. To do so, use the “at” command, which schedules commands to be executed once in the future at a specified time. (If you plan to record events periodically at fixed times, you should familiarise yourself with cron.)

“At” is a neat way to tell your machine to do something “at” a certain time and date. There are a number of ways to specify time, but we will use the format:

at hh:mm YYYY-MM-DD

e.g. the April 1, 2015 at 10:15 pm would be:

at 22:15 2015-04-01

If you type this line in the terminal and hit return, the cursor line starts with “at>” and you can schedule one or more commands. Once you are done, press Control-D at the beginning of the line and you will return to the prompt.

Recording 20 seconds of NPR on the first of April 2015 at 10:15pm would look like this in your terminal:

$ at 22:15 2015-04-01
warning: commands will be executed using /bin/sh
at> cvlc "http://www.npr.org/streams/mp3/nprlive24.pls" --sout file/mp3:/path/to/NPR-test.mp3 --run-time=20 --stop-time=20 vlc://quit
at> 
job 117 at Wed Apr 1 22:15:00 2015

Congratulations! You just scheduled your first stream recording! One thing is important: replace the relative path of the local file with an absolute path to which you want to record the file, like your home directory.

You can see the job ID right here in the last line. If you make a mistake, you can remove this scheduled process by typing:

at -r 117

Displaying a list of all scheduled tasks, type:

atq

Kill the recording with sleep and pkill

As mentioned before, the way VLC terminates itself does not always work as expected. Here’s a more reliable way to kill the recording:

Start the recording with a run-time that is longer than what you actually need
Kill the recording in a second scheduled “at” process using pkill

Firstly, install pkill as part of the procps package

sudo apt-get install procps

Now you can schedule the above command and replace the 20 seconds with 200 seconds or more if you want. As the second command, type:

echo "sleep 20; pkill vlc" | at 22:15 2015-04-01

What this does: At the same time you start the recording with “at”, a second process is started. It sleeps for 20 seconds and then kills the vlc process.

Transcoding the recorded stream to mp3

Let’s take a closer look at the simple recording command we use:

cvlc "http://www.deutschlandradio.de/streaming/dkultur.m3u" --sout file/mp3:DLF-test.mp3

It says “mp3” twice. It is tempting to replace “mp3” with “ogg” and hope it will transcode the stream from one format into another. If you try this, you can even open the resulting .ogg file in VLC and play it. But it has not been transcoded. If you check the audio information on the file, it will still give you the mp3 information you originally had. VLC is just smart enough to play the file even though it has the wrong file ending, technically speaking.

Most audio players play MP3, but not all play OGG or Windows Media files. So let’s alter our command to automatically transcode any stream into MP3. The following example will grab the Windows Media stream of BBC Radio 5 Live and transcode it into an MP3-file in stereo with 128 kbps and a sample rate of 44100 Hz; “dst” is followed by the local filename, but you can also use an absolute path.

cvlc --sout "#transcode{acodec=mp3,ab=128,channels=2,samplerate=44100}:std{access=file,mux=mp3,dst=BBC5.mp3}" http://bbc.co.uk/radio/listen/live/r5l.asx

As you can see, this command does not terminate after a certain time. It just keeps running until the drive is full. I tried to use run-time and stop-time in relation to the transcoding, but that did not work at all.

To use the transcoding option of VLC when recording live streams, we have to use the alternative way of killing the recording process described above.

Based on this little research, I put together streamplan.sh which is a command line tool to schedule and record streams. You can download or fork the script streamplan.sh from https://github.com/MiczFlor/streamplan.

As a prerequisite, you will need to install the id3v2 command line tool to alter ID3 tags in the recorded file, plus your choice of recorder, such as cvlc and pkill to stop the recording. On Debian or Ubuntu, you can do this with the command:

sudo apt-get install id3v2 vlc-nox procps