E.Smith [Tue, 2 Oct 2018 22:28:32 +0000 (23:28 +0100)]
ui: Use background-size to scale fanart to fit dvr dialog box.
Previously the fanart could be too big for the dialog. Now we
scale to fit the dialog.
We use 'cover' which can make the image slightly exceed the
dialog dimensions, compared to 'contain' which would leave
space at the top/bottom if the fanart is an odd dimension.
The 'cover' seems better with the fanart I've seen so far.
E.Smith [Tue, 2 Oct 2018 13:50:03 +0000 (14:50 +0100)]
xmltv: Add option to save epgdb after xmltv import.
The "periodic save database" means that for xmltv you can
import your daily listings, crash, restart, and not have xmltv
data since the periodic epgdb timer has not elapsed.
So, add an option so the user can save the database after the
import has completed, assuming changes occurred.
This save is delayed by a couple of minutes in case the user
is importing from several different xmltv guides, in which case
the save occurs after the last import.
We keep track of uuids for upcoming recordings. Once an hour we
get one entry from the list and pre-fetch artwork for it so it
can be displayed in the UI "upcoming recordings" dialog.
This is only done if the user has explicitly enabled fetch
artwork on the entry's profile.
In Configuration->Recording->Digital Video Recorder Profiles there
is now a "fetch artwork for new recordings" button, and a place
to enter additional arguments (such as --tmdb-key).
python: Support multiple grabber modules for movies and tv artwork.
The "--modules-movie=a,b,c" command line option will load each of
these modules in turn and ask them to provide artwork. TV modules are
selected via "--modules-tv" option.
The default (if no --modules-movies is provided) is to search the
Python path for python files that are named "tv_meta_*py" and then
call the module's "get_capabilities" to determine if the grabber
supports tv, movie, or both.
Each grabber module is called in turn until all artwork is
retrieved. So, if the first module only provides a fanart, then the
next module can supply the poster image.
Modules can be passed command line options from tvhmeta by prefixing
them with the shortened module name.
So, for a grabber module such as "tv_grab_tmdb", we pass through
command line arguments from tvhmeta to it. So if the tvhmeta is called
with an option of "--tmdb-key" then we pass through "key" as an option
to the tv_grab_tmdb (stripping the "--tmdb").
This allows third parties to produce grabber modules that
automatically integrate.
Also switched from deprecated OptionParser to argparse.
install: Add tvhmeta and tv_meta_tmdb.py to install files.
The tvhmeta program allows lookups of metadata for recordings.
The tv_meta_tmdb.py is the underlying file that does lookups
and is both an executable and a library.
python: Add basic tmdb lookup scripts to retrieve artwork.
The scripts attempt a tmdb lookup by using the title+year from
the dvr record associated with a particular uuid.
It then sets artwork and fanart.
The script can either be invoked manually or run automatically
from Tvheadend when a recording occurs (pre-recording rule).
In that case it needs to be passed the arguments:
"/usr/local/bin/tvhmeta --uuid %U --tmdb-key abcdef"
...where the key is from the tmdb website sign up.
The tv_meta_tmdb library is stand-alone and can be used
to test specific lookups to determine why they do not work.
A per-user cache is kept inside the tmdb3 library and this
is stored in /tmp.
The tmdb3 library has to be installed. This can be installed via "pip
install tmdb3" or "synth install www/py-tmdb3" depending on OS.
The scripts support a "--debug" option. I'd expect we are likely to
get several wrong/no results, especially with non-English movies until
we have a larger set of failure reasons to work with.
Once we get the tmdb working, we can try and "modularize" it so
different providers can be installed, and add a grabber for tv
episodes.
We now display fanart (if available) on the background of the dvr
dialog. This fanart image is also displayed every ten seconds where
the existing image is displayed.
We have to put the image inside a fixed width container, otherwise if
you alternate between a long thin image and a wide image then the text
reflows.
E.Smith [Mon, 1 Oct 2018 17:05:26 +0000 (18:05 +0100)]
FreeBSD: Add libunwind trap support for FreeBSD only.
Although the existing backtrace works correctly on Linux, on
FreeBSD it frequently generates a backtrace with completely
wrong function names. (FreeBSD 11.2, current latest version).
For example, making htsp_build_dvrentry crash with SEGV, it
would either not generate a stacktrace or would generate a
backtrace of:
-pthread_sigmask
-pthread_getspecific
-service_remove_unseen
-htsp_get_subscription_status
-htsp_init
-tcp_server_done
-tvhthread_create.
...instead of the correct backtrace of:
-<signal>
-htsp_build_dvrentry
-htsp_method_async
-htsp_read_loop
-htsp_serve...
So on FreeBSD only, we use libunwind to generate the
backtrace and function names. We explicitly make
libunwind and libexecinfo mutually exclusive since
FreeBSD has both.
Line are logged similar to:
CRASH: htsp_build_dvrentry+5d (ip=11f659d sp=7fffd8bc3930)
Note that it does not have line numbers since the addr2line
does not appear to work on FreeBSD (even with the original
backtrace code).
An example of the problem with the old backtrace code using
the frame from htsp_method_async from within the tvheadend
traphandler after the retrieval of the stack frames:
(gdb) print dladdr(0x11f1638, &dli) <--- addr of htsp_method_async from frame 4.
$39 = 1 <--- success
(gdb) print dli
$40 = {dli_fname = 0x7fffffffef97 ".../build.freebsd/tvheadend", dli_fbase = 0x1021000, dli_sname = 0x1044f91 "service_remove_unseen", <--- but wrong name
dli_saddr = 0x11eff80 <service_remove_unseen>} <--- and this is nearest symbol address
(gdb) print htsp_method_async+1640
$41 = (htsmsg_t *(*)(htsp_connection_t *, htsmsg_t *)) 0x11f1638 <htsp_method_async+1640> <---but gdb knows the original address is htsp_method_async
(gdb) print service_remove_unseen
$42 = {void (const char *, int)} 0x11eff80 <service_remove_unseen> <--- and gdb knows sevice_remove_unseen is at the dli_saddr.
By contrast, with libunwind, we get:
(gdb) print buf
$50 = "htsp_method_async", '\000' <repeats 110 times> <--- libunwind detected correct function name
(gdb) where 10 <--- even though our signal has been delivered on its own stack
#0 traphandler_libunwind () at src/trap.c:162
#1 0x000000000120cf06 in traphandler (sig=11, si=0x7fffdbbdb860, UC=0x7fffdbbdb4f0) at src/trap.c:221
#2 0x0000000806673954 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) disass 18814904 <--- and gdb knows that ip address is for the same method as libunwind detected
Dump of assembler code for function htsp_method_async:
0x00000000011f1150 <+0>: push %rbp
E.Smith [Mon, 1 Oct 2018 15:57:36 +0000 (16:57 +0100)]
trap: Allow chdir /tmp even if prctl not supported.
Even though prctl is Linux specific, other platforms allow core
dumps to occur in the cwd, so it's useful to allow the "cd /tmp"
for those platforms if the existing --dump option is specified.
E.Smith [Mon, 1 Oct 2018 17:32:26 +0000 (18:32 +0100)]
build: Add hardening options.
Add some hardening options from:
https://wiki.debian.org/Hardening
These protect against basic buffer overruns.
Although debian/rules can have an "export DEB_BUILD_HARDENING=1",
it's useful to have these available across all builds that support
the compiler options.
Previously if multiple services were mapped to same channel then
we would get duplicate entries in the csv, such as a string of
'DVB-T,DVB-T,DVB-S,DVB-S'.
Previously we added the filename to the dvr_entry at the start of the
recording, but did not persist it. This meant that if tvheadend
crashed before the programme completed then we would leave a file on
disk which is not referenced by any recording, hence will never be
deleted.
So we persist after the file is created/stream opened. This entry then
has filename, stream info, and (actual) start time, but no (actual)
stop time.
Currently this fanartImage is set by the user in recording
post-processing. So, a "%U" format specifier gives the user the
uuid of the recording and they can use api/idnode/{load,save}
to add fanart/image artwork from appropriate sources.
This fanart is then displayed in Kodi via pvr.hts.
python: Add tvhmeta program for setting artwork on a dvr recording.
Very basic program for setting artwork/fanart in a dvr entry,
primarily as a proof of concept. Retrieving of artwork from an
external source is not done.
The program demonstrates retrieving existing data from the server for
the recording, updating the artwork, saving it, then re-fetching to
show the change has been applied.
The uuid can be found via the %U format specifier in the recording
post-processing commands.
Fix building with gcc 8
- Patch for nasm taken from Fedora 28
- Added CFLAGS -Wno-stringop-truncation -Wno-stringop-overflow
- fdk-aac requires -fPIC to link properly
Some broadcasts can have different charsets (such as iso-8859-1) but
we assume utf-8 unless user has set it correctly. So when decode fails
we get an exception. So we now attempt to decode with error
replacement so user sees incorrect character.
This gives "u'Denise Th\ufffd\ufffd':" as the string returned instead
when the received name contains an é that is in iso-8859-1 instead of
utf-8.
Ensure files are compatible with python2 and python3.
Main differences:
- print requires brackets
- string is bytes in python2 and unicode in python3
- need to use struct to pack/unpack binary data
- need to convert socket data to bytearray to allow data extraction
Old dvr_entry records (tvh4.2) do not have a create timestamp, so we
fake one based on the start time of the recording. This will allow
us to send the timestamp to the UI in the future knowing that it
will give consistent results for user sorting since it may be
useful for user to sort upcoming recordings by when they were
created (i.e., scheduled).
We have to ensure we don't async reschedule if we have only just been
created then destroyed, otherwise we can enter a loop where every few
minutes the autorec is checked, realizes it can schedule against a
current broadcast, create the dvr_entry, then find it's a duplicate
of an existing recording/dvr_entry, so destroy the new entry, which
then causes the loop to start again later.
So, if a dvr_entry is created and destroy quickly then we avoid
the async reschedule. This breaks the loop since we no longer
trigger a second async reschedule.
dvr: Mark dvr entries as duplicates early to avoid logging.
When autorecs are modified, we can do an async reschedule to ensure
recordings are correctly updated in case the autorecs interact.
However this can cause excessve logging in some circumstances.
Effectively, if there is a recording on disk and the autorec matches
the same recording in the future then the dvr_entry_create_by_autorec
can exit the "bcast" loop with no match at all (since entry on disk
frequently, but not always, has a null de_bcast).
This then means we create the dvr_entry, and, if the programme is
broadcasting now, then we immediately schedule a timer to start the
recording and log an info to say scheduled. We then immediately cancel
the recording, delete the dvr_entry, which then causes us do an async
schedule. But this async schedule realizes the autorec matches the
broadcast, so we go through the process again.
So, first stage is to delete the duplicate before we do any logging.
Unfortunately, due to the way dup matching works (only dvr entry vs
dvr entry, not vs bcast), we have to create a dvr_entry to do the
matching.
We also need to check duplicate event after it is inserted in to
de_global_link (to avoid assertion fault).
dvr: Ensure non-scheduled dvr_entry is always "best".
Previously in dvr_is_better_recording_timeslot we would sanity check
that the old channel and broadcast exists before further checks.
However, it probably makes sense to ensure that a non-scheduled
dvr_entry (such as already recorded) is always the better match even
though it might not be linked to any broadcasts.
webui: Add Previous button to epg and dvr, fix minor issues (If you select the first/last row the previous/next are active when the popup window is opened first time)
FreeBSD: Support different stat format in Makefile.webui.
The stat program on FreeBSD requires different arguments to GNU
stat. In the past, this is done by the ports patching the Makefile
post-extract.
Instead, we'll configure the program's arguments based on platform.
We'll also use %z (filesize) instead of the port's %b (file blocks) so
we generate equivalent output to the Linux version..
dvr: Async reschedule autorecs whenever a log entry is destroyed.
Destroying a log entry can cause another autorec rule to be valid and
need scheduling.
For example, if you have two autorec rules that match the same
programme then only one autorec rule will schedule the programme
and the other autorec will see the programme is scheduled and do
nothing.
However, if you then disable the first autorec rule, then we need the
second autorec rule to re-arm the timer for that programme since it
still matches it.
This resheduling is done async after a delay. This avoids large
changes causing constant rescheduling. So, if user deletes a thousand
log files, we do not want a thousand reschedules to occur. Instead,
we dispatch a single timer after the last update has occurred.
My listings provider thinks every old film is worth 5/5, so we add
minyear to allow an autorec of "good films" to filter out old
films. Also add corresponding maxyear for people who think the
opposite and only want old programmes recorded.
ui: Fix createToolbar2 issues on some recent browsers.
Recent versions of Firefox had an exception saying the createToolbar2
function does not exist when it was called to create the secondary
toolbar that is used for category searching in the EPG.
I believe this was due to scoping issues, so rescoped the variables.
This worked, but although the toolbar was added it was not displayed
on Firefox. So changed the way the toolbar is added and it now works
on Firefox and Safari.
The official build runs with gcc which checks for some incompatible
pointers for assignment of const to non-const, whereas clang
build disabled such warnings.
So, update to re-enable the check on clang and fixup couple of
locations that failed.
dvr: Avoid recording partial programme if autorec created mid-programme.
When creating an autorec, it can match a programme that has already
started. If so, it's better to prefer the later recording so you get a
full recording.
So, if it's 09:10 and there is an hour long programme that started at
09:00 but is repeated at 10:00, then let's record at 10:00 instead and
get the full hour, instead of at 09:10 and only get 50 minutes.
dvr: Only do time slot scheduling in the scheduled state.
We do not want a currently recording programme to be cancelled
due to a better channel being found. For example recording a SD
channel and then the OTA updates and finds an HD version of the
show at the same time and so tries to schedule that HD version
instead.
The configuration to enable finding a 'best' time slot for
recording is now configurable as an advanced option in
Configuration->Recording->DVR Profiles, with the default being
disabled.
When toggled, the autorec rules themselves need to be manually
toggled to cause them to reschedule.
projects / thirdparty / tvheadend.git / log
