Restoring an older VHS anime - "Rainbowing" and other various issues

Hey,

I'm attempting to restore some footage for an older anime on VHS. I recently had some help solving a field blending issue, and now I'm trying to find the right denoisers, sharpeners, and smoothers to make the footage look a little better. I've been doing research and have found the following threads with some settings that I've tried:

https://forum.videohelp.com/threads/397357-90-s-Japanese-anime-VHS-clean-up-attempt#post2584187

https://forum.videohelp.com/threads/355366-anime-filtering-via-avisynth#post2235757
(minus deinterlacing stuff)

https://forum.videohelp.com/threads/362895-VHS-Anime-to-DVD-in-Virtualdub#post2306889
(minus deinterlacing stuff, looked pretty good but had a bit of a red hue on all white scenes)

https://forum.videohelp.com/threads/401326-Avisynth-filters-on-anime#post2615463
(this worked quite well for my video as well and made the colors pop a lot more)

I've also been following this guide:
https://www.animemusicvideos.org/guides/avtechbeta/avspostqual.html#rainbows

After trying various approaches, I tried coming up with a script with settings that I thought were the most helpful. Unfortunately, I'm still running into a few issues that I'm having trouble solving:

Code:

AVISource("source.avi")
ConvertToYV12()
ColorYUV(levels="TV->PC")
MCTemporalDenoise(settings="very high")
vmToon()

Pretty simple, but only because I have MCTemporalDenoise doing a lot of the heavy lifting. MCTD at "very high" is needed to reduce some "rainbowing" that I've been seeing in some scenes. I really don't like having this setting so high, I end up losing so much detail, but it's the only thing I've found that gets rid of that rainbowing. None of the other scripts I've tried, or any setting lower than "very high", seems to work. I've even tried using DFMDeRainbow with "mask=true" to help me determine which threshold value is needed (I ended up having to set it to "maskthresh=55"). I've also tried applying a sharpener (vmToon was recommended from that guide) to try and mitigate some of the blurriness from MCTD.

I'm feel like I'm going in circles and am making no progress, and thought about reaching out for help. I'll upload a couple clips, one daytime and one nighttime. I've found the rainbowing is most noticeable during the nighttime one, fwiw.

Thank you for any help!

Originally Posted by Selur

looking at Day.avi
a. don't see rainbows
b. try adding the line darkening before applying MCTemporalDenoise
c. to me, the most annoying thing are the halos and the chroma shift
since it's an anime, and you are using Avisynth, I would suggest trying some of the avs-mlrt filters, since there are tons of filters that are trained to clean up animes.

Cu Selur

Ps: Quick&Dirty2 using 2x_LD-Anime-Compact for cleanup

Appreciate it! I didn't even realize AI ML was a thing for this. This is my first attempt at restoring something, and now that I feel like I have a somewhat decent understanding of the basics, I feel like this is a whole new thing to learn now.

I went ahead and tried using one of the models from this thread: https://forum.doom9.org/showthread.php?t=184768

The specific one I'm trying to use is 2x_LD-Anime_Skr_v1.0.onnx (it's in the same folder as the mlrt.dll). Here's my script right now:

Code:

AVISource("source.avi")
ConvertToPlanarRGB()
mlrt_ncnn(network_path="2x_LD-Anime_Skr_v1.0.onnx")

But it's giving an error when I try to open it in Vdub:

Code:

Avisynth open failure:
mlrt_ncnn: expects clip with type fp32

I've tried looking up info on the error, and I can't seem to find anything. From what I can see, fp32 is the data type of the model? Would you happen to know how I could go about solving this error?

Originally Posted by jagabo

Regarding using MCTemporalDenoise to reduce chroma noise, you can apply it differently to the luma and chroma using MergeChroma():

Code:

light = MCTemporalDenoise(settings="low", chroma=false)
heavy =  MCTemporalDenoise(settings="very high")
MergeChroma(light, heavy) # luma from light, chroma from heavy

Or all on one line:

Code:

MergeChroma(MCTemporalDenoise(settings="low", chroma=false), MCTemporalDenoise(settings="very high"))

Oh! I didn't realize I could do that, I'll go ahead and give it a try, thank you!

Regarding fp32, see ConvertToFloat()
http://avisynth.nl/index.php/ConvertToFloat

Code:

ConvertToPlanarRGBA()
ConvertToFloat()

Or use z_ConvertFormat()
http://avisynth.nl/index.php/Avsresize#z_ConvertFormat

Originally Posted by jagabo

Regarding fp32, see ConvertToFloat()
http://avisynth.nl/index.php/ConvertToFloat

Code:

ConvertToPlanarRGBA()
ConvertToFloat()

Or use z_ConvertFormat()
http://avisynth.nl/index.php/Avsresize#z_ConvertFormat

Awesome! Thank you so much, that helped solve the issue. I also spent all day yesterday messing around with different models, and "2x_LD-Anime_Skr_v1.0" produced incredible results. I was baffled at how good it looked.

One issue that I noticed though, is that in my source, there's something that looks similar to haloing, but it isn't haloing. I'm not exactly sure what the term to describe it is (is this the chroma shift that Selur was referring to?). But basically, it looks like some of the color bleeds outside the lines, and then LD-Anime ends up enhancing it so that it becomes more obvious and prominent. I'll uploaded a couple of pictures and a short 2 second clip.

For what it's worth, here's something a little different for removing that ghost to the right of the dark areas:

Code:

LWLibavVideoSource("birds.avi", cache=false, prefer_hw=2) 
ConvertToYV12()

Spline36Resize(400, height) # closer to the true resolution of VHS
Overlay(last, last, x=3, mode="subtract", opacity=0.2).ColorYUV(gain_y=60) # subtract ghost
dehalo_alpha(rx=2.5, ry=2.0, darkstr=1.2, brightstr=1.2) # then halo reduction

# sharpen and upscale back to the original frame size
aWarpSharp2(depth=5)
CAS(0.7)
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=720, fheight=480)

before and after:
[Attachment 78740 - Click to enlarge]

Originally Posted by jagabo

For what it's worth, here's something a little different for removing that ghost to the right of the dark areas:

Code:

LWLibavVideoSource("birds.avi", cache=false, prefer_hw=2) 
ConvertToYV12()

Spline36Resize(400, height) # closer to the true resolution of VHS
Overlay(last, last, x=3, mode="subtract", opacity=0.2).ColorYUV(gain_y=60) # subtract ghost
dehalo_alpha(rx=2.5, ry=2.0, darkstr=1.2, brightstr=1.2) # then halo reduction

# sharpen and upscale back to the original frame size
aWarpSharp2(depth=5)
CAS(0.7)
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=720, fheight=480)

before and after:
[Attachment 78740 - Click to enlarge]

You. Are. A. Life. Saver. This worked wonderfully, thank you so much! Between this and your help with the QTGMC Srestore, you've seriously helped me a ton. Thank you again!

Glad to help out.